Information forgery detection method, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of information detection, and discloses an information forgery detection method, computer equipment, a storage medium and a program product, and the method comprises the steps: obtaining to-be-detected information which comprises at least two types of modal information; determining interaction characteristics among different modal information in the to-be-detected information; on the basis of the interaction features between the different modal information, a feature connection diagram is constructed, and initial state information of each modal information in the feature connection diagram is determined by the interaction features between the different modal information; performing iterative updating on the state information of the modal information based on the feature connection diagram to obtain the state information after the iterative updating of the modal information; and determining a counterfeit detection result of the to-be-detected information based on the state information after the modal information is iteratively updated. According to the invention, the accuracy of information forgery detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information detection, and in particular to an information forgery detection method, a computer device, a storage medium and a program product. Background Art

[0002] In the information age, the fusion of information in multiple modalities, such as text, images, and audio, has become the norm. However, the emergence of this multimodal information has also brought new challenges, especially in the detection of information forgery. Forgers can hide the traces of forgery by cleverly exploiting the complex relationship between different modalities, making detection extremely difficult.

[0003] The multimodal information forgery detection technology in the related art mines the complex relationship between multimodal information based on a fusion method such as simple splicing or addition of the features of multimodal information, thereby realizing the forgery detection of multimodal information.

[0004] However, this approach cannot effectively mine the complex relationships between multimodal information, resulting in low accuracy in detecting information forgery. Summary of the invention

[0005] In view of this, the present invention provides an information forgery detection method, computer device, storage medium and program product to solve the problem that the multimodal information forgery detection technology in the related art cannot effectively mine the complex relationship between multimodal information, resulting in low accuracy of information forgery detection.

[0006] In a first aspect, the present invention provides a method for detecting information forgery, the method comprising: Acquire information to be detected, where the information to be detected includes at least two types of modal information; Determining interaction characteristics between different modal information in the information to be detected; Based on the interaction features between the different modal information, construct a feature connection graph, wherein the initial state information of each modal information in the feature connection graph is determined by the interaction features between the different modal information; Iteratively updating the state information of the modal information based on the feature connection graph to obtain the iteratively updated state information of the modal information; Based on the state information after iterative update of the modal information, a forgery detection result of the information to be detected is determined.

[0007] The information forgery detection method provided in this embodiment obtains the information to be detected, determines the interaction features between different modal information in the information to be detected, constructs a feature connection graph based on the interaction features between different modal information, the initial state information of each modal information in the feature connection graph is determined by the interaction features between different modal information, iteratively updates the state information of the modal information based on the feature connection graph, obtains the updated state information of the modal information, and determines the forgery detection result of the information to be detected based on the iteratively updated state information of the modal information. The information forgery detection method can effectively capture the interaction features and complex information between different modal information in the information to be detected, better adapt to the complexity of multimodal information, and improve the accuracy of information forgery detection.

[0008] In an optional implementation, the determining the interaction feature between different modal information in the information to be detected includes: Obtaining a characteristic vector of modal information in the information to be detected; Based on the feature vector of the modal information, interaction features between the different modal information are determined.

[0009] The information forgery detection method provided in this embodiment obtains feature vectors of information in different modalities and determines the interaction features between them to capture the relationship between these cross-modal information, performs forgery detection based on the relationship between these cross-modal information, and improves the accuracy of forgery detection.

[0010] In an optional implementation, the acquiring a feature vector of the modal information in the information to be detected includes: For any modal information in the information to be detected, preprocess the modal information to obtain the preprocessed modal information; Based on the preprocessed modal information, a feature vector of the modal information is determined.

[0011] The information forgery detection method provided in this embodiment can remove noise and irrelevant parts in the modal information through a preprocessing step, so that the characteristic vector of the determined modal information is more accurate, thereby improving the accuracy of the forgery detection result of the information to be detected.

[0012] In an optional implementation, the determining the interaction feature between the different modal information based on the feature vector of the modal information includes: Performing a linear transformation on the feature vector of the modal information to obtain a query vector, a key vector, and a value vector of the modal information; Based on the query vector, key vector and value vector of the modal information, interaction features between the different modal information are determined.

[0013] The information forgery detection method provided in this embodiment can capture the complex relationship between different modal information by introducing query vectors, key vectors and value vectors, enhance the understanding of the information to be detected, and thus improve the accuracy of forgery detection of the information to be detected.

[0014] In an optional implementation, performing a linear transformation on the feature vector of the modal information to obtain a query vector, a key vector, and a value vector of the modal information includes: The query vector, key vector and value vector of the modal information are obtained by the following formula:

[0015]

[0016]

[0017] in, For the A query vector of modality information, For the The key vector of modal information, For the A value vector of modal information, For the The feature vector of modal information, , , For the The weight matrix of each modal information.

[0018] The information forgery detection method provided in this embodiment performs a linear transformation on the feature vector of the modal information to obtain the query vector, key vector and value vector of the modal information, and then determines the interaction characteristics between different modal information based on the query vector, key vector and value vector of the modal information, thereby enhancing the understanding of the information to be detected and improving the accuracy of forgery detection of the information to be detected.

[0019] In an optional implementation, the determining the interaction features between the different modal information based on the query vector, the key vector, and the value vector of the modal information includes: The interaction characteristics between any modal information and other modal information are obtained by the following formula:

[0020] in, is the interaction feature between the i-th modal information and the j-th modal information, S represents the activation function, is the query vector of the i-th modal information, is the key vector of the jth modal information, is the value vector of the jth modal information, represents the range of feature dimensions, is the dimension of the key vector.

[0021] The information forgery detection method provided in this embodiment determines the interaction characteristics between different modal information by introducing query vectors, key vectors and value vectors, thereby being able to capture the complex relationship between different modal information, enhance the understanding of the information to be detected, and thereby improve the accuracy of forgery detection of the information to be detected.

[0022] In an optional implementation, constructing a feature connection graph based on the interaction features between the different modal information includes: Taking each type of modal information as a node, and taking the interaction features between different modal information as the feature vectors of the nodes, to generate the feature connection graph; Different nodes interact with each other through edges.

[0023] The information forgery detection method provided in this embodiment generates a feature connection graph by taking each modal information as a node and the interaction characteristics between different modal information as the feature vector of the node. This can intuitively represent the relationship and interaction between different modal information, and can better capture the complex relationship between global information and modal information, thereby improving the overall understanding of the information to be detected.

[0024] In an optional implementation, the iteratively updating the state information of the modal information based on the feature connection graph to obtain the iteratively updated state information of the modal information includes: For any node in the feature connection graph, determine a neighbor node of the node based on the feature connection graph; Determining aggregation information of the node based on the state information of the neighboring node; Based on the aggregated information, updating the state information of the node; Return to the step of determining the aggregation information of the node based on the status information of the neighboring nodes, until the number of updates of the status information of the node reaches a preset update threshold, and obtain the status information of the modal information corresponding to the node after iterative updates, wherein one node corresponds to one modal information.

[0025] The information forgery detection method provided in this embodiment can dynamically capture the complex interactive relationship between different modal information by determining the node's aggregation information based on the status information of neighbor nodes and updating the node's status information based on the node's aggregation information. Each iterative update will provide a deeper understanding of the mutual influence between each modal information, thereby improving the ability to understand the information to be detected and improving the accuracy of forgery detection of the information to be detected.

[0026] In an optional implementation, determining the aggregation information of the node based on the state information of the neighboring node includes: The aggregation information of the node is determined by the following formula:

[0027] in, For the tth update, the node Aggregate information; is the set of edges in the feature connection graph. If edge (j, i) exists in the set of edges, then the node is determined Is a node Neighbor nodes of For Node Point to Node The weight of the edge; For Node The input matrix of For Node The output matrix of For the (t-1)th update, the node status information.

[0028] The information forgery detection method provided in this embodiment determines the aggregation information of a node through the status information of neighbor nodes and the weights of the edges between the node and its neighbor nodes, and can more accurately capture the impact of neighbor nodes on the current node, thereby improving the ability to understand the information to be detected, and further improving the accuracy of forgery detection of the information to be detected.

[0029] In an optional embodiment, the method further includes: The node is determined by the following formula Point to Node The edge weights are:

[0030] in, is the activation function, is the weight matrix, For Node The corresponding eigenvector of modal information, For Node The corresponding feature vector of the modal information, m is the number of nodes in the feature connection graph, For Node The eigenvector of the corresponding modal information.

[0031] The information forgery detection method provided in this embodiment can adaptively adjust the weight of each edge by using an activation function and a weight matrix, dynamically learn and optimize the importance of the edge according to the node characteristics, thereby more accurately capturing the relationship between nodes and improving the accuracy of forgery detection of the information to be detected.

[0032] In an optional implementation, updating the state information of the node based on the aggregate information includes: The state information of the node is updated by the following formula:

[0033] in, For Node Status information after the tth update; For Node Status information after the (t-1)th update; For the tth update, the node Aggregate information; For Node Initial status information.

[0034] The information forgery detection method provided in this embodiment ensures the consistency and progressiveness of information fusion by introducing the initial state information of the node as the residual and updating the state information of the node based on the previous state information and the current aggregated information. This iterative mechanism can gradually adjust and optimize the state information of each node, more deeply explore the potential relationship between different modal information, and thus improve the accuracy of forgery detection of the information to be detected.

[0035] In an optional implementation, the determining the forgery detection result of the information to be detected based on the state information after iterative update of the modal information includes: For any modal information, inputting the state information after iterative update of the modal information into two different prediction heads to obtain the prediction score of the modal information and the weight of the modal information; The prediction score of the information to be detected is determined by the following formula:

[0036] in, is the prediction score of the information to be detected, is the weight of the i-th modal information, is the prediction score of the i-th modal information, and m is the number of nodes in the feature connection graph; Based on the predicted score of the information to be detected, a falsification detection result of the information to be detected is determined.

[0037] In the information forgery detection method provided in this embodiment, after the state information of each modal information is iteratively updated, two different prediction heads are input to obtain the prediction score of the modal information and the weight of the modal information, and the prediction score of the information to be detected is determined by the prediction score of the modal information and the weight of the modal information, and then the forgery detection result of the information to be detected is determined, thereby ensuring the accuracy of the forgery detection result of the information to be detected.

[0038] In a second aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the information forgery detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a third aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute the information forgery detection method of the first aspect or any corresponding embodiment thereof.

[0040] In a fourth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the information forgery detection method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a schematic diagram of a flow chart of an information forgery detection method according to an embodiment of the present invention; Figure 2 is a flow chart of another information forgery detection method according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a prediction head according to an embodiment of the present invention; Figure 4 is a flow chart of another information forgery detection method according to an embodiment of the present invention; Figure 5It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0044] In the information age, the fusion of information in multiple modes such as text, images, and audio has become the norm. Multimodal information is complementary and complex. Complementarity refers to the mutual complementation of information in multiple modes. For example, image information can provide visual content, text information can describe semantic information, and audio information can convey sound-related content. When describing an event, image information may show the scene, text information explains the background and details, and audio information supplements the environmental sound.

[0045] Complexity refers to the complexity of the structure and relationship of multimodal information. There may be multiple semantic, temporal and spatial associations between multiple modal information, such as the synchronization relationship between image frames and audio information in video information, and the semantic matching relationship between image information and related text information description.

[0046] In this context, forgery detection of multimodal information also faces new challenges. Among them, forgery detection of multimodal information refers to the technology of judging the authenticity of content containing multiple modal information (such as text, image, audio and other modalities) to determine whether it has been forged.

[0047] The forgery of multimodal information can be carried out within a single modality or by exploiting the complex relationships between different modalities for cross-modal forgery. Forgery within a single modality can be maliciously modified using image editing software to modify the objects, scenes or characters in the image, or by using audio editing tools to edit, remix or add false sound effects to audio information, or by using text generation algorithms to generate reasonable but actually false text information.

[0048] Cross-modal forgery using the complex relationship between different modalities can be achieved by generating seemingly real image information (generated through an image generation model) that matches the false text information, generating related text information based on the forged image information, or generating matching image information based on the false audio information, etc.

[0049] In the related technology, the forgery detection of single modal information is already very mature, but it is very difficult to detect cross-modal forged information by utilizing the complex relationship between different modalities, because the forgery detection method of single modal information cannot fully consider the relationship between different modalities.

[0050] Text, images, audio and other modal information all have their own unique feature representations, different data structures and different semantic understandings. For example, the feature representation of image information is a pixel matrix, the feature representation of text information is a single sequence, and the feature representation of audio information is a waveform or spectrum feature. The semantic understanding of image information requires the identification of objects, scenes, etc., and the semantic understanding of text information is based on vocabulary, grammar and semantic relationships. Therefore, after fusing these information with different data structures and different semantic understandings, it is difficult to detect whether the fused information is forged.

[0051] And with the development of technology, the means of forgery are constantly updated, including technologies such as Generative Adversarial Network (GAN), which can generate highly realistic forged image information or audio information. New text generation algorithms can also produce forged text information that is difficult to distinguish between true and false, making the detection of forgery of multimodal information more difficult.

[0052] In the related art, forgery detection of multimodal information is achieved by mining the complex relationship between multimodal information through a fusion method based on simple splicing or addition of features of multimodal information, thereby achieving forgery detection of multimodal information.

[0053] However, this method cannot effectively mine the complex relationship between multimodal information because it does not design a reasonable fusion strategy. Effective fusion of multimodal information requires the design of a reasonable fusion strategy, such as early fusion, late fusion, or hybrid fusion. Moreover, this method does not consider the weight distribution of different modal information in the fusion process. Different modal information may have different importance for forgery detection in different scenarios. For example, when detecting forgery of news reports, text information may be more decisive than image information, while when detecting forgery of movie clips, image information and audio information may be more important. This results in low accuracy in information forgery detection.

[0054] An embodiment of the present invention provides an information forgery detection method, which obtains information to be detected, determines the interaction characteristics between different modal information in the information to be detected, constructs a feature connection graph based on the interaction characteristics between different modal information, and the initial state information of each modal information in the feature connection graph is determined by the interaction characteristics between different modal information. The state information of the modal information is iteratively updated based on the feature connection graph to obtain the updated state information of the modal information. Based on the iteratively updated state information of the modal information, the forgery detection result of the information to be detected is determined to effectively capture the interaction characteristics and complex relationships between different modal information in the information to be detected, better adapt to the complexity of multimodal information, and improve the accuracy of information forgery detection.

[0055] According to an embodiment of the present invention, an embodiment of a method for detecting information forgery is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In this embodiment, a method for detecting information forgery is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 1 is a flow chart of an information forgery detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, obtaining information to be detected, where the information to be detected includes at least two types of modal information.

[0057] The information to be detected is multimodal information. For example, the information to be detected may be information including image information and text information. The information to be detected may also include information including image information, text information and audio information. In other words, the modal information may be image information, text information, audio information, etc.

[0058] Step S102, determining the interaction characteristics between different modal information in the information to be detected.

[0059] Step S103, constructing a feature connection graph based on the interaction features between different modal information, wherein the initial state information of each modal information in the feature connection graph is determined by the interaction features between different modal information.

[0060] Step S104, iteratively update the state information of the modal information based on the feature connection graph to obtain the state information of the modal information after iterative update.

[0061] Step S105, determining a forgery detection result of the information to be detected based on the state information after iterative update of the modal information.

[0062] It should be noted that steps S102 to S105 may be implemented by a target information forgery detection model, and the target information forgery detection model is determined according to the type of modal information included in the information to be detected.

[0063] Exemplarily, when the information to be detected is information including image information and text information, it is determined that the types of modal information included in the information to be detected are image information and text information.

[0064] When the types of modal information included in the information to be detected are image information and text information, the target information forgery detection model is a model trained using training samples containing both image information and text information.

[0065] When the types of modal information included in the information to be detected are image information, text information and audio information, the target information forgery detection model is a model trained using training samples containing three types of information: image information, text information and audio information.

[0066] It can be understood that the target information forgery detection model is a trained model.

[0067] Specifically, the target information forgery detection model is generated through the following steps: 1. Obtain a data set, in which the types of modal information included in the training samples in the data set are the same as the types of modal information included in the information to be detected.

[0068] Exemplarily, when the types of modal information included in the information to be detected are image information, text information and audio information, each training sample in the data set contains three types of information: image information, text information and audio information.

[0069] It should be noted that the dataset includes real samples and forged samples.

[0070] 2. For any training sample in the data set, input the training sample into the modal coding layer to obtain the feature vector of the modal information included in the training sample.

[0071] 3. Input the feature vector of the modal information into the multimodal connection layer to obtain the interaction characteristics between the modal information.

[0072] 4. Input the interaction features between modal information into the feature connection graph generation module to obtain the feature connection graph corresponding to the training sample.

[0073] 5. Input the feature connection graph corresponding to the training sample into the multimodal information association loop module to obtain the state information after iterative update of any modal information in the training sample.

[0074] 6. Input the iteratively updated state information of any modal information into the dual-head prediction module to obtain the prediction result of the training sample. The prediction result is used to characterize whether the training sample is forged information.

[0075] 7. Use the data set to train the information forgery detection model to obtain the target information forgery detection model.

[0076] It should be noted that during the training process, an optimization algorithm can be used to minimize the loss function. For example, the Adaptive Moment Estimation (Adam) algorithm is used, the learning rate is set to 0.001, the exponential decay rate of the first-order moment estimation is set to 0.9, and the exponential decay rate of the second-order moment estimation is set to 0.999.

[0077] To balance the ratio of positive and negative samples, an equal number of real samples and fake samples are randomly selected for each training batch to ensure that the performance of the trained model is optimal.

[0078] The information forgery detection model is trained using the data set to obtain the target information forgery detection model, including: The predicted results of the training samples and the actual results represented by the labels of the training samples are input into the preset loss function to obtain the loss value.

[0079] Based on the loss value, the model parameters of the information forgery detection model are updated, and the model training is continued based on the data set until the training termination condition is reached to obtain the target information forgery detection model.

[0080] Among them, the preset loss function is:

[0081] in, is the loss value, N is the total number of training samples in the data set, is the label of the i-th training sample (real information or fake information), is the prediction score of the i-th training sample.

[0082] Based on the loss value, the model parameters of the information forgery detection model are updated, and the model training is continued based on the data set until the training termination condition is reached to obtain the target information forgery detection model, including: Based on the loss value, the model parameters of the information forgery detection model are updated, and the model training is continued based on the data set until the number of training iterations reaches the preset iteration threshold, and the target information forgery detection model is obtained.

[0083] Or, based on the loss value, update the model parameters of the information forgery detection model, continue to train the model based on the data set until the loss value reaches a preset loss value threshold, and obtain the target information forgery detection model.

[0084] The training termination condition may be that the number of training iterations reaches a preset iteration threshold, or that the loss value reaches a preset loss threshold. The specific training termination condition may be set according to the specific situation. The preset iteration threshold and the preset loss threshold are set by the technicians and are not specifically limited here.

[0085] Among them, the information forgery detection model includes a modal encoding layer, a multimodal connection layer, a feature connection graph generation module, a multimodal information association loop module and a dual-head prediction module.

[0086] After the target information forgery detection model is determined, the information to be detected is input into the target information forgery detection model to obtain a forgery detection result corresponding to the information to be detected output by the target information forgery detection model.

[0087] The forgery detection result is used to indicate whether the information to be detected is forged information.

[0088] The information forgery detection method provided in this embodiment obtains the information to be detected, determines the interaction features between different modal information in the information to be detected, constructs a feature connection graph based on the interaction features between different modal information, the initial state information of each modal information in the feature connection graph is determined by the interaction features between different modal information, iteratively updates the state information of the modal information based on the feature connection graph, obtains the updated state information of the modal information, and determines the forgery detection result of the information to be detected based on the iteratively updated state information of the modal information. The information forgery detection method can effectively capture the interaction features and complex information between different modal information in the information to be detected, better adapt to the complexity of multimodal information, and improve the accuracy of information forgery detection.

[0089] The information forgery detection method provided in this embodiment obtains a loss value by inputting the predicted results of training samples and the actual results represented by the labels of the training samples into a preset loss function. The model can quantify the gap between its predicted results and the actual results. Based on the loss value, the model can continuously adjust parameters to gradually reduce the error until the training termination condition is reached, thereby obtaining a target information forgery detection model and improving the accuracy of forgery detection of the target information forgery detection model.

[0090] The information forgery detection method provided in this embodiment constructs an information forgery detection model including a modal encoding layer, a multimodal connection layer, a feature connection graph generation module, a multimodal information association loop module, and a dual-head prediction module, and uses a data set to train the information forgery detection model to obtain a target information forgery detection model, so as to use the target information forgery detection model to perform forgery detection on the information to be detected. The target information forgery detection model can effectively capture the paired interaction features and complex relationships between different modal information, thereby improving the accuracy of forgery detection.

[0091] The information forgery detection method provided in this embodiment can effectively optimize model performance, improve training efficiency, ensure training quality, and enhance the stability and generalization ability of the model by updating model parameters based on loss values ​​and setting reasonable training termination conditions.

[0092] In this embodiment, a method for detecting information forgery is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 2 is a flow chart of an information forgery detection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, obtain the information to be detected, the information to be detected includes at least two modal information. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0093] Step S202, determining the interaction characteristics between different modal information in the information to be detected.

[0094] Specifically, the above step S202 includes: Step S2021, obtaining the feature vector of the modal information in the information to be detected.

[0095] After the information to be detected is obtained, a feature vector of the modal information in the information to be detected is obtained based on the information to be detected.

[0096] It should be noted that this step is implemented by the modal encoding layer in the target information forgery detection model.

[0097] Step S2022: Determine the interaction characteristics between different modal information based on the feature vector of the modal information.

[0098] It should be noted that this step is implemented by the multimodal connection layer in the target information forgery detection model.

[0099] Step S203, constructing a feature connection graph based on the interaction features between different modal information, wherein the initial state information of each modal information in the feature connection graph is determined by the interaction features between different modal information.

[0100] Specifically, the above step S203 includes: Step S2031, taking each type of modal information as a node, taking the interaction features between different modal information as the feature vectors of the nodes, and generating a feature connection graph.

[0101] Different nodes interact with each other through edges.

[0102] It should be noted that this step is implemented by the feature connection graph generation module in the target information forgery detection model.

[0103] Different from the related art that concatenates or combines features of different modal information, this embodiment uses a feature connection graph structure to represent multimodal information features. Presenting multimodal information in the form of a feature connection graph structure can intuitively display the relationship between different modal information.

[0104] Each modal information is regarded as a node, that is, each node in the feature connection graph corresponds to a modal information, and the edge represents the interaction between different modalities, which makes the complex multimodal interaction relationship visualized and helps researchers better understand the connection and dependency between different modal information.

[0105] Each node has a feature vector that represents the attribute information of the node. In multimodal information forgery detection, nodes can correspond to different modal information, and the feature vector of the node can be a multimodal feature vector after preprocessing and modal encoding layer processing. The dimension and specific content of the node's feature vector depend on the nature of the input data and the task requirements. For example, for image modality, the node feature vector can contain image color, texture, shape and other features; for text modality, the node feature vector can contain word vectors, semantic information, etc.

[0106] In this embodiment, the feature vector of a node is an interaction feature between different modal information obtained by the multimodal connection layer.

[0107] The structure of the feature connection graph makes the target information forgery detection model interpretable. The edge weights and node status information can reflect the importance of the interaction between different modal information and the flow of information, providing a certain explanation basis for the model's decision-making. This is very important for understanding the working principle of the model and improving the credibility of the model, especially in some application scenarios with high interpretability requirements, such as medical and financial fields.

[0108] Therefore, the information to be detected is represented as a feature connection graph , where each node Corresponding to a modal information , information of different modalities can interact through edges, , is the number of modal information contained in the information to be detected. The elements in B are the nodes of the feature connection graph, and the elements in D are the weights of the edges connecting the nodes.

[0109] Through the above steps, the multimodal information forgery detection task (binary classification task) is converted into a task of node interaction on the modeling graph, and the information to be detected corresponds to a feature connection graph, which improves the interpretability of the model.

[0110] Step S204, iteratively update the state information of the modal information based on the feature connection graph to obtain the state information of the modal information after iterative update. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0111] Step S205: Determine the forgery detection result of the information to be detected based on the state information after iterative update of the modal information. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0112] The information forgery detection method provided in this embodiment obtains feature vectors of information in different modalities and determines the interaction features between them to capture the relationship between these cross-modal information, performs forgery detection based on the relationship between these cross-modal information, and improves the accuracy of forgery detection.

[0113] The information forgery detection method provided in this embodiment generates a feature connection graph by taking each modal information as a node and the interaction characteristics between different modal information as the feature vector of the node. This can intuitively represent the relationship and interaction between different modal information, and can better capture the complex relationship between global information and modal information, thereby improving the overall understanding of the information to be detected.

[0114] In some optional implementations, the above step S2021 includes: Step a1, for any modal information in the information to be detected, preprocess the modal information to obtain the modal information after preprocessing.

[0115] In the forgery detection of multimodal information, each modal information can be regarded as an independent field. First, for each modal information, appropriate preprocessing is performed.

[0116] Specifically, it is necessary to clarify the types of modal information included in the information to be detected that needs to be processed. Exemplarily, the types of modal information included in the information to be detected are image information, text information, and audio information.

[0117] Different modal information has different characteristics and representation forms, which will affect the subsequent processing. Therefore, different preprocessing is required for different modal information.

[0118] Exemplarily, for image information, preprocessing includes operations such as adjusting the image size and normalizing the image information to ensure the consistency and stability of the input data.

[0119] For text information, preprocessing may include word segmentation, removal of stop words, etc.

[0120] For audio information, preprocessing may include noise reduction, frame division, and the like.

[0121] Step a2, determining the eigenvector of the modal information based on the preprocessed modal information.

[0122] Specifically, after preprocessing the modal information in the information to be detected, the preprocessed modal information is input into the multimodal pre-training model to obtain a feature vector of the modal information.

[0123] It should be noted that the multimodal pre-trained model can be a model that is trained through comparative learning using a large amount of image information, text information and audio information and has the ability to process image information, text information and audio information.

[0124] The multimodal pre-training model is used to extract features of the preprocessed modal information to obtain the feature vector of the modal information.

[0125] For example, for image information, the image encoder part of the multimodal pre-trained model is used to extract features from the pre-processed image information, and the image encoder outputs a 512-dimensional feature vector, i.e., the image feature vector , whose length is Each element in this feature vector represents some abstract feature of the image information, which may include encoding of color distribution, texture information, object shape, etc.

[0126] For text information, the text encoder part of the multimodal pre-training model is used to extract features from the preprocessed text information. The text encoder outputs a 256-dimensional feature vector, namely the text feature vector , the length is This feature vector reflects the semantic information, grammatical structure and possible association between the text information and other modal information. The positions and relationships between different words in this feature vector space reflect their semantic similarity and contextual relationship.

[0127] For audio information, the audio encoder part of the multimodal pre-trained model is used to extract features from the pre-processed audio information. The audio encoder outputs a 384-dimensional feature vector, namely the audio feature vector , the length is This feature vector contains the characteristic encoding of the rhythm, pitch, timbre and other aspects of the audio information. The positions of different audio clips in this feature vector space reflect the similarities and differences of their audio features.

[0128] It should be noted that after obtaining the feature vector of the modal information included in the information to be detected, the feature vectors of different modal information can be spliced ​​along the feature dimension.

[0129] Exemplarily, the information to be detected includes m modal information, and the feature vector of each modal information is , where i represents the modal information number, Represents the dimension of the feature vector, then the concatenated vector is:

[0130] in, The sum of the dimensions of the feature vectors representing all modal information. Indicates the range of feature dimensions, "That is, it should not exceed the sum of the dimensions of the eigenvectors of all modal information." The "operation" represents a concatenation operation, which converts the feature vector of sparse modal information into a more expressive and dense feature vector to prepare for subsequent forgery detection. In other words, information from different modalities can be fused together to provide a unified input for subsequent processing.

[0131] It should be further noted that during model training, the concatenated vectors can be further processed and optimized. For example, regularization techniques can be used to reduce overfitting, or batch normalization can be applied to accelerate model training and improve model stability.

[0132] The modal encoding layer can also be adjusted and improved according to specific task requirements and dataset characteristics to improve the performance of multimodal information forgery detection.

[0133] The information forgery detection method provided in this embodiment can remove noise and irrelevant parts in the modal information through a preprocessing step, so that the characteristic vector of the determined modal information is more accurate, thereby improving the accuracy of the forgery detection result of the information to be detected.

[0134] Feature extraction of modal information is performed through a multimodal pre-training model. This type of multimodal pre-training model can capture the commonalities and differences between different modal information, extract more expressive and discriminative features, and thus improve the accuracy of forgery detection of the information to be detected.

[0135] In some optional implementations, the above step S2022 includes: Step b1, performing a linear transformation on the feature vector of the modal information to obtain the query vector, key vector and value vector of the modal information.

[0136] Among them, the multimodal connection layer mainly uses the attention mechanism to capture the interaction characteristics of different modal information in different semantic subspaces after passing through the modal encoding layer.

[0137] According to the feature vectors of multiple modal information included in the information to be detected, the query vectors, key vectors and value vectors of the multiple modal information can be obtained, and then according to the query vectors, key vectors and value vectors of the multiple modal information, the feature representation of the pairwise interaction between any modal information in the information to be detected and other modal information in the information to be detected, that is, the interaction feature between any modal information in the information to be detected and other modal information, can be determined.

[0138] Step b2, based on the query vector, key vector and value vector of the modal information, determine the interaction characteristics between different modal information.

[0139] The information forgery detection method provided in this embodiment can capture the complex relationship between different modal information by introducing query vectors, key vectors and value vectors, enhance the understanding of the information to be detected, and thus improve the accuracy of forgery detection of the information to be detected.

[0140] In some optional implementations, the above step b1 includes: Step b11, obtain the query vector, key vector and value vector of the modal information through the following formula:

[0141]

[0142]

[0143] in, For the A query vector of modality information, For the The key vector of modal information, For the A value vector of modal information, For the The feature vector of modal information, , , For the The weight matrix of each modal information.

[0144] It should be noted that , , is a learnable weight matrix.

[0145] The information forgery detection method provided in this embodiment performs a linear transformation on the feature vector of the modal information to obtain the query vector, key vector and value vector of the modal information, and then determines the interaction characteristics between different modal information based on the query vector, key vector and value vector of the modal information, thereby enhancing the understanding of the information to be detected and improving the accuracy of forgery detection of the information to be detected.

[0146] In some optional implementations, the above step b2 includes: Step b21, obtaining the interaction characteristics between any modal information and other modal information by the following formula:

[0147] in, is the interaction feature between the i-th modal information and the j-th modal information, S represents the activation function, is the query vector of the i-th modal information, is the key vector of the jth modal information, is the value vector of the jth modal information, represents the range of feature dimensions, is the dimension of the key vector.

[0148] It should be noted that S is the softmax layer, that is, the softmax activation function.

[0149] The information forgery detection method provided in this embodiment determines the interaction characteristics between different modal information by introducing query vectors, key vectors and value vectors, thereby being able to capture the complex relationship between different modal information, enhance the understanding of the information to be detected, and thereby improve the accuracy of forgery detection of the information to be detected.

[0150] In order to make the above steps b11 and b21 clearer, the information to be detected includes image information, text information, and audio information. The multi-head attention mechanism is introduced to dynamically determine the importance weights between different model information, that is, to determine the interaction features between different modal information.

[0151] First, the feature vectors of the three modal information are linearly transformed to obtain the query vector (Query), key vector (Key) and value vector (Value). For image information: , , .in, is the query vector of image information, is the feature vector of image information, is the key vector of image information, is the value vector of image information, , and is the learnable weight matrix of image information.

[0152] Similarly, similar operations are performed on text information and audio information to obtain the query vector of text information , key vector Sum value vector , the query vector of audio information , key vector Sum value vector .

[0153] Then, the interaction features between different modal information are calculated. It can be understood that the interaction features between image information and text information are the same as the interaction features between text information and image information, the interaction features between image information and audio information are the same as the interaction features between audio information and image information, and the interaction features between text information and audio information are the same as the interaction features between audio information and text information. Therefore, it is only necessary to calculate the interaction features between image-text, text-audio, and audio-image.

[0154] Among them, the interactive features between image and text are: ,in is the dimension of the key vector, is the interaction feature between image and text, marked as . The interactive features between text and audio are: , is the interaction feature between text and audio, marked as .

[0155] The interactive features between audio and image are: , marked as . It should be noted that after obtaining the interaction features between different modal information, the interaction features between different modal information can also be spliced ​​together.

[0156] The features after splicing are: ,in is the concatenation operation (along the feature vector embedding dimension), m is the number of modal information, is the feature dimension after concatenation, is an activation function. That’s all the multimodal interaction information.

[0157] In some optional implementations, the above step S204 includes: Step c1, for any node in the feature connection graph, determine the neighbor nodes of the node based on the feature connection graph.

[0158] The multimodal information association loop module aims to model the node interactions on the feature connection graph in a flexible and explicit way. Each is associated with a hidden state , which is the state information of the node. The state information of the feature connection graph consists of the state information of these nodes: ,in, is the state information of the feature connection graph at the tth update, and t is the number of times the state information is updated. Obtained by the multimodal connection layer as the initial state information of the feature connection graph.

[0159] It should be noted that the initial state information of each node in the feature connection diagram is determined by the interaction characteristics between different modal information. In other words, the initial state information of each modal information in the feature connection diagram is determined by the interaction characteristics between different modal information.

[0160] Exemplarily, the initial state information of the image information may be an interaction feature between image and text, the initial state information of the text information may be an interaction feature between text and audio, and the initial state information of the audio information may be an interaction feature between audio and image.

[0161] It can be understood that the initial state information of multiple modal information is different, and the initial state information of each modal information is determined from the interaction characteristics between the modal information and other modal information.

[0162] The multimodal information association loop module includes two parts: state aggregation and state update. Each node starts from the initial state information and aggregates and updates its state information in a cyclic manner. In each state information update of each node, the node aggregates the state information of neighboring nodes (after being updated) to obtain aggregated information. Based on the aggregated information and the node's historical state information, the node's state information is updated through the Long Short-Term Memory (LSTM) network and residual.

[0163] First, the neighbor nodes of the node are determined according to the feature connection graph.

[0164] Step c2: determining the aggregation information of the node based on the state information of the neighboring nodes.

[0165] After the neighbor nodes of the node are determined, the aggregation information of the node is determined according to the state information of the neighbor nodes.

[0166] Step c3: Update the node status information based on the aggregated information.

[0167] After the aggregation information of the node is obtained, the state information of the node is updated according to the aggregation information of the node.

[0168] Step c4, returns to the step of determining the node's aggregation information based on the status information of neighboring nodes, until the number of updates of the node's status information reaches a preset update threshold, and obtains the iteratively updated status information of the modal information corresponding to the node, wherein one node corresponds to one modal information.

[0169] It should be noted that the preset update number threshold is set by technical personnel and is not specifically limited here.

[0170] It should be further explained that the above steps c1 to c4 are implemented by the multimodal information association loop module in the target information forgery detection model.

[0171] The information forgery detection method provided in this embodiment can dynamically capture the complex interactive relationship between different modal information by determining the node's aggregation information based on the status information of neighbor nodes and updating the node's status information based on the node's aggregation information. Each iterative update will provide a deeper understanding of the mutual influence between each modal information, thereby improving the ability to understand the information to be detected and improving the accuracy of forgery detection of the information to be detected.

[0172] In some optional implementations, the above step c2 includes: Step c21, determine the node aggregation information by the following formula:

[0173] in, For the tth update, the node This aggregate information is obtained by collecting state information from neighboring nodes, which will be used to update the node in the subsequent state update step Status information; is the set of edges in the feature connection graph. If edge (j, i) exists in the set of edges, then the node is determined Is a node , then the node will participate in the summation operation; For Node Point to Node The weight of the edge, also an element in the adjacency matrix, reflects the importance of the interaction between two nodes; For Node The input matrix of For Node The output matrix of For the (t-1)th update, the node Status information; Represents the node The sum is calculated for all neighboring nodes.

[0174] It should be noted that the edge weights learned through the attention mechanism can adaptively adjust the interaction weights between different modal information according to different tasks and data characteristics.

[0175] In the state aggregation process, the previous state information of the neighboring nodes is converted and weighted summed to obtain the node The aggregated information realizes the fusion of multimodal information. By collecting the status information of neighbor nodes and performing weighted summation according to the weight of the edge, information from different modalities can influence and fuse each other, which helps to capture the complex interactive relationship between multimodal information and improve the accuracy of forgery detection.

[0176] The adjacency matrix describes the connection relationship between nodes in the feature connection graph. and nodes If there is an edge connection between them, the corresponding element in the adjacency matrix is ​​not zero, indicating the connection strength or weight between them.

[0177] In multimodal information forgery detection, the adjacency matrix can be determined based on the correlation or prior knowledge between different modal information. For example, if there is a certain semantic association between image information and text information, a larger connection weight can be set between their corresponding nodes; if there is no obvious relationship between the two modal information, the connection weight can be set to a smaller value or zero.

[0178] The adjacency matrix may be fixed or obtained through dynamic learning. In the embodiment of the present invention, the adjacency matrix is ​​constructed according to the edge weights by obtaining the edge weights.

[0179] The multimodal information association loop module includes a message passing layer for transmitting information between nodes. It is usually composed of multiple message passing steps, each of which includes the following operations: 1. Message collection: For each node, collect the state information of its neighbor nodes, which can be obtained by multiplying the feature vector of the neighbor node with the connection weight, that is, the weight of the edge, and then summing them. In this embodiment, the initial state information of the node is determined by the interaction characteristics between the different modal information output by the multimodal connection layer, and the subsequent state information of the node is determined by the aggregation information of the node. 2. Message aggregation: Aggregate the collected state information of the neighbor nodes to obtain a comprehensive message vector, that is, aggregation information. Common aggregation functions include summation, mean, maximum value, etc. 3. Message update: Fuse the aggregated message with the node's own feature vector to update the node's state information. This can be obtained by splicing the message vector with the node feature vector, and then passing it through a fully connected layer or other nonlinear transformation.

[0180] It can be understood that the message transmission layer is used to implement the above steps c2 to c4.

[0181] It should be further explained that the original formula for determining the aggregation information of a node is:

[0182] in, Representation Node To Node These matrices are used to transform the state information of the nodes so that they can better adapt to the characteristics and interaction requirements of different nodes during information transmission and aggregation.

[0183] However, since the fixed projection matrix on all edges cannot model and flexibly interact, it is necessary to perform adaptive transformation on each edge. However, since the feature connection graph is a complete graph containing a large number of edges, simply assigning an independent projection matrix to each edge will consume a lot of parameter space and running time.

[0184] To reduce the complexity of time and space, an input matrix and an output matrix are assigned to each node in the feature connection graph. To Node When sending its status information, the status information will first be sent by the node The output matrix Convert, then Received by the node before The input matrix Convert from node arrive Edge The projection matrix on can be written as: . Then the formula for determining the aggregation information of a node can be rewritten as:

[0185] This makes the number of parameters proportional to the number of nodes rather than the number of edges, greatly reducing the spatial complexity and practical complexity while enabling information interaction at the edges.

[0186] The information forgery detection method provided in this embodiment determines the aggregation information of a node through the status information of neighbor nodes and the weights of the edges between the node and its neighbor nodes, and can more accurately capture the impact of neighbor nodes on the current node, thereby improving the ability to understand the information to be detected, and further improving the accuracy of forgery detection of the information to be detected.

[0187] In some optional implementations, the above information forgery detection method further includes: Step d1, determine the node by the following formula Point to Node The edge weights are:

[0188] in, is the activation function, is the weight matrix, For Node The corresponding eigenvector of modal information, For Node The corresponding feature vector of modal information, m is the number of nodes in the feature connection graph, For Node The eigenvector of the corresponding modal information.

[0189] It should be noted that , is the learnable weight matrix.

[0190] The importance of interactions between different nodes can be inferred through dynamic edge weights. First, the nodes and The corresponding feature vector of the modal information Splice and then pass the weight matrix Perform a linear transformation and then h activation function processing.

[0191] Among them, the function expression of Swish is:

[0192] in is a learnable parameter or a fixed value (usually set to 1). In multimodal information, the relationships between different modal information are complex and diverse, and not all interactions between modal information are of equal importance. Through attention-based edge weights, different weights can be assigned to the interactions between different modal information, highlighting important interaction relationships and ignoring unimportant interaction relationships, thereby more accurately modeling the characteristics and relationships of multimodal information.

[0193] The weight of the edge reflects the importance of the interaction between different modal information, which enables the information forgery detection model to well explain the relationship between different modal information of the input information to be detected in a multimodal scenario. For example, in multimodal information forgery detection, if the edge weight between an image modality and a text modality is large, it means that the interaction between the two modalities is important for judging whether the information is forged. This provides a basis for understanding the decision-making process of the model and increases the interpretability of the model.

[0194] It should be noted that the above formula for calculating edge weights is applicable to When , the weight of the edge is 0.

[0195] That is, the adjacency matrix can be expressed as: , among which When , the elements in the adjacency matrix are nodes and The weight of the edge between It can be a node determined by the above formula for calculating the weight of the edge point to The weight of the edge. , the element is 0, indicating that the node itself has no edge connection.

[0196] The weight of the edge guides the information transfer and aggregation between nodes. By assigning different weights to the interactions between different modal information, the model can more specifically aggregate the state information of neighboring nodes, thereby better capturing the complex interactive relationship between multimodal information, helping to improve the accuracy and performance of multimodal information forgery detection, and well explaining the relationship between different modal information in the information to be detected.

[0197] The information forgery detection method provided in this embodiment can adaptively adjust the weight of each edge by using an activation function and a weight matrix, dynamically learn and optimize the importance of the edge according to the node characteristics, thereby more accurately capturing the relationship between nodes and improving the accuracy of forgery detection of the information to be detected.

[0198] In some optional implementations, the above step c3 includes: Step c31, update the node status information using the following formula:

[0199] in, For Node Status information after the tth update; For Node Status information after the (t-1)th update; For the tth update, the node Aggregate information; For Node Initial status information.

[0200] It should be noted that after aggregating the state information, the node updates the state information through LSTM and residual connection. The state information of the corresponding modal information is updated through LSTM based on the aggregated information of the node and the state information obtained by the node in the last update, and an additional residual connection (from the initial state information) is introduced to update the state information of the node, promoting low-order feature reuse and gradient back propagation.

[0201] The information forgery detection method provided in this embodiment ensures the consistency and progressiveness of information fusion by introducing the initial state information of the node as the residual and updating the state information of the node based on the previous state information and the current aggregated information. This iterative mechanism can gradually adjust and optimize the state information of each node, more deeply explore the potential relationship between different modal information, and thus improve the accuracy of forgery detection of the information to be detected.

[0202] In some optional implementations, the above step S205 includes: Step e1, for any modal information, input the state information after iterative update of the modal information into two different prediction heads to obtain the prediction score of the modal information and the weight of the modal information.

[0203] When the number of updates of the node's status information reaches the preset update number threshold, the final node status information is obtained. .

[0204] For any modal information, the iteratively updated state information of the modal information is input into two different prediction heads, one prediction head is used to predict the score, and the other prediction head is used to obtain the weight of the modal information.

[0205] It should be noted that the two prediction heads have the same structure but do not share parameters. Figure 3 is a schematic diagram of the structure of a prediction head according to an embodiment of the present invention. Figure 3As shown, one prediction head includes an input layer, a hidden layer 1, a hidden layer 2, a hidden layer 3, a hidden layer 4, a hidden layer 5, and an output layer. Hidden layer 1 includes 128 neurons, hidden layer 2 includes 64 neurons, hidden layer 3 includes 64 neurons, hidden layer 4 includes 32 neurons, and hidden layer 5 includes 32 neurons.

[0206] Step e2, determine the prediction score of the information to be detected by the following formula:

[0207] in, is the prediction score of the information to be detected, is the weight of the i-th modal information, is the prediction score of the i-th modal information, and m is the number of nodes in the feature connection graph.

[0208] Step e3: determining a forgery detection result of the information to be detected based on the predicted score of the information to be detected.

[0209] For example, if the prediction score of the information to be detected exceeds the score threshold, the prediction result of the information to be detected is determined to be forged information, and if the prediction score of the information to be detected does not exceed the score threshold, the prediction result of the information to be detected is determined to be true information. The set score threshold may be 0.5.

[0210] It should be noted that the above is only an exemplary description, and adaptive adjustments can be made according to actual conditions to determine the forgery detection result of the information to be detected.

[0211] It should be further explained that the above steps e1 to e3 are implemented by a dual-head prediction module in the target information forgery detection model. The output layer of the dual-head prediction module can be set to different forms according to task requirements. In this embodiment, the output layer is a classifier for outputting a forgery detection result of the information to be detected, and the forgery detection result is used to characterize whether the information to be detected is forged information.

[0212] The output layer is usually composed of one or more fully connected layers and activation functions. The fully connected layer maps the feature vector of the node to the output space, and the activation function can increase the nonlinear expression ability of the model.

[0213] In the information forgery detection method provided in this embodiment, after the state information of each modal information is iteratively updated, two different prediction heads are input to obtain the prediction score of the modal information and the weight of the modal information, and the prediction score of the information to be detected is determined by the prediction score of the modal information and the weight of the modal information, and then the forgery detection result of the information to be detected is determined, thereby ensuring the accuracy of the forgery detection result of the information to be detected.

[0214] In this embodiment, a method for detecting information forgery is provided, which can be used in mobile terminals, such as servers, central processing units, etc. Figure 4 is a flow chart of an information forgery detection method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps: Step 1: Input multimodal information.

[0215] That is, the information to be detected is obtained, and the information to be detected is input into the target information forgery detection model.

[0216] Step 2: Preprocess different modal data separately.

[0217] Corresponding to the aforementioned step a1, it will not be repeated here.

[0218] Step three, represent the preprocessed multimodal information as a connection graph.

[0219] It is understandable that the preprocessed multimodal information can be represented as a connection graph first, and then the step of obtaining the pairwise interaction features of different modal information is performed. In other words, the steps are not necessarily performed in order, and the execution order can be adjusted according to actual conditions.

[0220] Each modal information is regarded as a node, and different nodes interact through edges to generate a connection graph, which corresponds to the aforementioned feature connection graph.

[0221] Step 4: Associate a hidden state with each node in the connection graph.

[0222] It can be understood that the hidden state of a node is the state information of the node.

[0223] Step 5: Use the self-attention mechanism to capture the pairwise interaction features of different modal information.

[0224] Please refer to the corresponding description of the above step b1 and step b2 for details, which will not be repeated here.

[0225] Step 6: State aggregation: Calculate edge weights based on the initial states of nodes and construct an adjacency matrix.

[0226] Please refer to the above step d1 and the corresponding description for details, which will not be repeated here.

[0227] Step 7: Assign input and output matrices to each node for adaptive edge transformation.

[0228] Step 8: Perform state aggregation based on the adjacency matrix and adaptive edge transformation results.

[0229] The description of step 7 and step 8 can be found in detail in the corresponding description of the aforementioned step c21, which will not be repeated here.

[0230] Step 9, state update: state update through LSTM and residual connection.

[0231] Step 10: After a certain number of propagation steps, the final node state is obtained through the attention scoring layer.

[0232] The detailed description of step nine and step ten can refer to the corresponding description of the aforementioned step c31 and step c4, which will not be repeated here.

[0233] Step 11: Use the dual-head prediction head to predict the score and attention node weight respectively, and comprehensively predict the score and attention node weight to obtain the overall prediction result.

[0234] Step 12: output the forgery detection result.

[0235] The description of step 11 and step 12 can be found in detail in the description of the aforementioned steps e1 to e3, which will not be repeated here.

[0236] The information forgery detection method provided in this embodiment first converts the input multimodal information into a multimodal feature vector through a modal encoding layer, and then passes through a multimodal connection layer to align the information between the various modalities, and then constructs these modal information into a feature connection graph, which is sent to a multimodal information association loop module, and the interactive information between the modal information is extracted by aggregating and updating the state information in the feature connection graph, and finally the output information of the multimodal information association loop module is passed through a double-head prediction module to obtain the final prediction result, which can effectively capture the paired interactive features and complex relationships between different modal information. This complex model architecture can better adapt to the complexity of multimodal information and improve the accuracy of forgery detection.

[0237] The embodiment of the present invention also provides a computer device, see Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 501, memory 502, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Each component utilizes different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 501 is taken as an example.

[0238] The processor 501 may be a central processing unit, a network processor or a combination thereof. The processor 501 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0239] The memory 502 stores instructions executable by at least one processor 501 , so that the at least one processor 501 executes the method shown in the above embodiment.

[0240] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 502 may optionally include a memory remotely arranged relative to the processor 501, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0241] The memory 502 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 502 may also include a combination of the above types of memory.

[0242] The computer device also includes a communication interface 503, which is used for the computer device to communicate with other devices or a communication network.

[0243] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0244] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0245] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting information forgery, characterized in that: The method comprises: Acquire information to be detected, where the information to be detected includes at least two types of modal information; Determining interaction characteristics between different modal information in the information to be detected; Based on the interaction features between the different modal information, construct a feature connection graph, wherein the initial state information of each modal information in the feature connection graph is determined by the interaction features between the different modal information; Iteratively updating the state information of the modal information based on the feature connection graph to obtain the iteratively updated state information of the modal information; Based on the state information after iterative update of the modal information, a forgery detection result of the information to be detected is determined.

2. The method according to claim 1, characterized in that The determining of the interaction characteristics between different modal information in the information to be detected includes: Obtaining a characteristic vector of modal information in the information to be detected; Based on the feature vector of the modal information, interaction features between the different modal information are determined.

3. The method according to claim 2, characterized in that The step of obtaining a feature vector of the modal information in the information to be detected includes: For any modal information in the information to be detected, preprocess the modal information to obtain the preprocessed modal information; Based on the preprocessed modal information, a feature vector of the modal information is determined.

4. The method according to claim 2, characterized in that: The determining, based on the feature vector of the modal information, the interaction feature between the different modal information includes: Performing a linear transformation on the feature vector of the modal information to obtain a query vector, a key vector, and a value vector of the modal information; Based on the query vector, key vector and value vector of the modal information, interaction features between the different modal information are determined.

5. The method according to claim 4, characterized in that The linear transformation of the feature vector of the modal information to obtain the query vector, key vector and value vector of the modal information includes: The query vector, key vector and value vector of the modal information are obtained by the following formula: in, For the A query vector of modality information, For the The key vector of modal information, For the A value vector of modal information, For the The feature vector of modal information, , , For the The weight matrix of each modal information.

6. The method according to claim 4, characterized in that The determining, based on the query vector, the key vector, and the value vector of the modal information, the interaction features between the different modal information includes: The interaction characteristics between any modal information and other modal information are obtained by the following formula: in, is the interaction feature between the i-th modal information and the j-th modal information, S represents the activation function, is the query vector of the i-th modal information, is the key vector of the jth modal information, is the value vector of the jth modal information, represents the range of feature dimensions, is the dimension of the key vector.

7. The method according to claim 1, characterized in that The step of constructing a feature connection graph based on the interaction features between the different modal information includes: Taking each type of modal information as a node, and taking the interaction features between different modal information as the feature vectors of the nodes, to generate the feature connection graph; Different nodes interact with each other through edges.

8. The method according to claim 1, characterized in that The iteratively updating the state information of the modal information based on the feature connection graph to obtain the iteratively updated state information of the modal information includes: For any node in the feature connection graph, determine a neighbor node of the node based on the feature connection graph; Determining aggregation information of the node based on the state information of the neighboring node; Based on the aggregated information, updating the state information of the node; Return to the step of determining the aggregation information of the node based on the status information of the neighboring nodes, until the number of updates of the status information of the node reaches a preset update threshold, and obtain the status information of the modal information corresponding to the node after iterative updates, wherein one node corresponds to one modal information.

9. The method according to claim 8, characterized in that The determining, based on the state information of the neighboring nodes, the aggregation information of the node includes: The aggregation information of the node is determined by the following formula: in, For the tth update, the node Aggregate information; is the set of edges in the feature connection graph. If edge (j, i) exists in the set of edges, then the node is determined Is a node Neighbor nodes of For Node Point to Node The weight of the edge; For Node The input matrix of For Node The output matrix of For the (t-1)th update, the node status information.

10. The method according to claim 9, characterized in that The method further comprises: The node is determined by the following formula Point to Node The edge weights are: in, is the activation function, is the weight matrix, For Node The corresponding eigenvector of modal information, For Node The corresponding feature vector of the modal information, m is the number of nodes in the feature connection graph, For Node The eigenvector of the corresponding modal information.

11. The method according to claim 8, characterized in that The updating the state information of the node based on the aggregate information includes: The state information of the node is updated by the following formula: in, For Node Status information after the tth update; For Node Status information after the (t-1)th update; For the tth update, the node Aggregate information; For Node Initial status information.

12. The method according to claim 1, characterized in that The determining, based on the state information after iterative update of the modal information, a forgery detection result of the information to be detected includes: For any modal information, inputting the state information after iterative update of the modal information into two different prediction heads to obtain the prediction score of the modal information and the weight of the modal information; The prediction score of the information to be detected is determined by the following formula: in, is the prediction score of the information to be detected, is the weight of the i-th modal information, is the prediction score of the i-th modal information, and m is the number of nodes in the feature connection graph; Based on the predicted score of the information to be detected, a falsification detection result of the information to be detected is determined.

13. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the information forgery detection method according to any one of claims 1 to 12 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the information falsification detection method according to any one of claims 1 to 12.

15. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the information falsification detection method according to any one of claims 1 to 12.

Images