Ffact checking model training method and fact checking method

Abstract

The invention provides a fact checking model training method and a fact checking method. The checking method comprises the steps of obtaining a declaration to be checked; a pre-trained fact checking model is adopted to check the authenticity of the declaration, the fact checking model outputs an authenticity prediction result of the declaration, and the fact checking model comprises a multi-modal evidence retrieval module, a feature extraction and coding module, a hypergraph construction module, a hypergraph information spreading module, a line graph module and a classification module. According to the method, the authenticity of the complex declaration can be predicted and checked by accurately utilizing the multi-modal evidence.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and fact-checking technology, and in particular to a method for training a fact-checking model and a fact-checking method. Background Art

[0002] In the digital age, the Internet has become the main channel for information dissemination. However, the accompanying false information and misleading content have gradually become more prevalent. Such content not only damages the public's trust in the authenticity of information but may also have a negative impact on social stability and individual decision-making. To address this challenge, fact-checking technology has emerged, aiming to verify the authenticity of information through automated means, thereby effectively curbing the spread of false information. The core of fact-checking lies in extracting evidence related to the claim to be verified from a vast amount of data and verifying the authenticity of the claim through comprehensive evaluation and comparison.

[0003] Traditional fact-checking methods usually rely on text-based evidence. However, with the development of technology, multimodal evidence (including text, images, videos, etc.) provides a richer information source for fact-checking, which can not only provide more perspectives but also help improve the accuracy of verification results. Existing fact-checking models based on multimodal information generally suffer from the problem of only focusing on the superficial association between the claim and the evidence and ignoring the deep semantic interaction. In addition, with the popularity of social media, reports on public events increasingly appear in multimodal forms, especially content including images and text often being part of the evidence candidate set. Therefore, how to effectively utilize multimodal evidence retrieval and verification of text claims has become an important topic in current fact-checking research based on multimodal information.

[0004] Although there has been some progress in supplementing missing semantic information through visual evidence. For example, the MOCHEG framework and the CCN framework initially used neural networks for fact-checking prediction based on unimodal information and then applied multimodal fusion technology for fact-checking based on multimodal information, which involves a key issue, that is, how to extract vectors from each modality and integrate them into a unified space to complete the task of verifying the authenticity of claims with multimodal evidence. The Triple-Check framework enhances the adapter of a large basic model and uses a multimodal multi-type fusion module to clarify the relationship between modalities and different types of evidence (such as statements and documents). However, existing models still mainly focus on the superficial association between the claim and the evidence and fail to fully consider the fine-grained semantic interaction in multimodal information, thus greatly affecting the accuracy of fact-checking. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method for training a fact-checking model and a fact-checking method to eliminate or improve one or more defects existing in the prior art.

[0006] The first aspect of the present invention provides a fact-checking method, which includes the following steps: Obtain the claim to be verified; Use a pre-trained fact-checking model to verify the authenticity of the claim, and the fact-checking model outputs the authenticity prediction result of the claim. Among them, the fact-checking model includes: a multi-modal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module, and a classification module. The multi-modal evidence retrieval module is used to retrieve multiple multi-modal evidences related to the claim from the network; the feature extraction and encoding module is used to extract and encode the features of the claim and the multiple multi-modal evidences at multiple scales to obtain multiple feature embedding vector sequences corresponding to multiple feature sequences; the hypergraph construction module is used to use each feature corresponding to the feature embedding vector in the multiple feature embedding vector sequences as a node, use the claim and each evidence as a hyperedge for connecting all nodes in the corresponding feature sequence respectively, and construct a hypergraph based on all nodes and all hyperedges, where the nodes connecting multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to use the self-attention mechanism to obtain the corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected by each hyperedge in the hypergraph, and use the cross-attention mechanism to obtain the hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected by each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes respectively, construct a line graph based on the converted nodes and edges, and obtain the line graph embedding vector based on all edges in the line graph and the hyperedge embedding vectors of all nodes; the classification module is used to output the authenticity prediction result of the claim based on the hypergraph embedding vector and the line graph embedding vector.

[0007] In some embodiments of the present invention, the hypergraph information propagation module includes an intra-hyperedge information propagation module and an inter-hyperedge information propagation module. The intra-hyperedge information propagation module is used to aggregate the feature embedding vectors of multiple consecutive nodes among all the nodes connected by each hyperedge in the hypergraph by using a grouped attention mechanism, to form the aggregated feature embedding vectors of multiple aggregated nodes, and to obtain the hyperedge embedding vector of the corresponding hyperedge based on the aggregated feature embedding vectors of the multiple aggregated nodes. The inter-hyperedge information propagation module is used to obtain the attention weights between each shared node and the hyperedge embedding vectors of the respective hyperedges connected to the shared node based on the feature embedding vector of each shared node in the hypergraph and the hyperedge embedding vectors of the respective hyperedges connected to the shared node by using a cross-attention mechanism, to update the feature embedding vector of the shared node based on the weighted aggregation of the hyperedge embedding vectors of the respective hyperedges connected to the shared node and the corresponding attention weights, and to obtain the hypergraph embedding vector based on the updated feature embedding vectors of all the shared nodes.

[0008] In some embodiments of the present invention, the line graph module includes a line graph construction module and a line graph information propagation module. The line graph construction module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes corresponding to the hyperedges connected to the shared nodes respectively, and to construct a line graph based on the converted nodes and edges. The line graph information propagation module is used to construct the adjacency matrix of the line graph based on all the edges and all the nodes in the line graph by using a graph convolutional network, and to obtain the line graph embedding vector based on the matrix obtained by adding the adjacency matrix and the identity matrix and the matrix formed by the hyperedge embedding vectors of all the nodes.

[0009] In some embodiments of the present invention, the multi-modal evidence includes text evidence and image evidence, and the claims include text claims and / or image claims.

[0010] In some embodiments of the present invention, the multiple feature sequences include multiple token sequences composed of multiple tokens and multiple image patch sequences composed of multiple image patches, and the multiple feature embedding vector sequences include multiple token embedding vector sequences composed of multiple token embedding vectors and multiple image patch embedding vector sequences composed of multiple image patch embedding vectors.

[0011] The second aspect of the present invention provides a method for training a fact-checking model, the method comprising the following steps: Obtain a training set, the training set including multiple claims each provided with a respective authenticity class label and a multi-modal evidence library including multiple multi-modal evidences; Training a preset fact-checking model based on the training set to train the preset fact-checking model into a fact-checking model that outputs a truth prediction result of the claim based on the claim. Wherein, the fact-checking model includes: a multi-modal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module, and a classification module. The multi-modal evidence retrieval module is used to retrieve multiple multi-modal evidences related to the claim from the multi-modal evidence library; the feature extraction and encoding module is used to extract and encode the features of the claim and the multiple multi-modal evidences at multiple scales respectively to obtain multiple feature embedding vector sequences corresponding to multiple feature sequences; the hypergraph construction module is used to use each feature corresponding to the feature embedding vector in the multiple feature embedding vector sequences as a node, and use the claim and each evidence as a hyperedge for connecting all nodes in the corresponding feature sequence respectively, and construct a hypergraph based on all nodes and all hyperedges, wherein the nodes connecting multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to use the self-attention mechanism to obtain the corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected by each hyperedge in the hypergraph, and use the cross-attention mechanism to obtain the hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected by each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes respectively, construct a line graph based on the converted nodes and edges, and obtain a line graph embedding vector based on all edges in the line graph and the hyperedge embedding vectors of all nodes; the classification module is used to output a truth prediction result of the corresponding claim based on the hypergraph embedding vector and the line graph embedding vector.

[0012] In some embodiments of the present invention, training a preset fact-checking model based on the training set to train the preset fact-checking model into a fact-checking model that outputs a truth prediction result of the claim based on the claim includes: Training the preset fact-checking model based on the training set by minimizing the loss function until the difference between the truth prediction result of the claim and the truth category label is less than a preset threshold, and obtaining the finally trained fact-checking model; the loss function adopts the cross-entropy loss function and is calculated according to the following formula: Wherein, Loss s represents the cross-entropy loss; y k represents the truth category label of the kth claim, and the truth category label includes that the fact contained in the claim is supported by evidence, refuted, and the information is insufficient to determine the truth of the claim, that is, the claim is true, false, and unable to determine the truth; Represents the authenticity prediction result of the k-th claim, R HG Represents the hypergraph embedding vector, R LG Represents the line graph embedding vector, [:] represents the concatenation operation, W R Represents a trainable parameter matrix.

[0013] Another aspect of the present invention provides an electronic device, which includes: a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the fact-checking method described in the foregoing first aspect, or implements the steps of the fact-checking model training method described in the foregoing second aspect.

[0014] Another aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the fact-checking method described in the foregoing first aspect, or implements the steps of the fact-checking model training method described in the foregoing second aspect.

[0015] Another aspect of the present invention provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, it implements the steps of the fact-checking method described in the foregoing first aspect, or implements the steps of the fact-checking model training method described in the foregoing second aspect.

[0016] The fact-checking model training method and fact-checking method of the present invention first retrieve multi-modal evidence related to the claim through the multi-modal evidence retrieval module and the feature extraction and encoding module respectively and perform feature extraction. These features are subsequently used to construct a hypergraph. Then, the hypergraph construction module integrates multi-modal data such as evidence and claims into a hypergraph structure. Next, the hypergraph information propagation module uses the Transformer mechanism to perform effective information flow within the hypergraph structure, enhancing the high-order information integration between multi-modal claims and evidence. In addition, a line graph module is introduced to optimize the information propagation process, further improving the inference ability of the model. Finally, using a supervised loss function to evaluate and optimize the fact-checking model enables the trained model to effectively utilize multi-modal evidence to accurately predict the authenticity of the claim.

[0017] The additional advantages, objectives, and features of the present invention will be partially described below and will become partially apparent to those of ordinary skill in the art after studying the following text, or can be learned through the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the specification and the drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and the above and other objectives achievable with the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0020] Figure 1 It is a schematic flowchart of the fact-checking method in an embodiment of the present invention; Figure 2 It is a schematic structural diagram of the fact-checking model in an embodiment of the present invention; Figure 3 It is a schematic diagram of the principle of information propagation within and between hyperedges in the fact-checking model in an embodiment of the present invention; Figure 4 It is a schematic flowchart of the fact-checking model training method in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0022] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0023] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0024] Herein, it should also be noted that if not otherwise specified, the term "connection" in this document can not only refer to a direct connection, but also represent an indirect connection with an intermediate.

[0025] In the following, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0026] Regarding the problem of fact-checking based on multi-modal evidence information, the claim to be fact-checked may also exist in a multi-modal form. Since information in different modalities exists in different semantic spaces, there are also deep semantic interactions and correlations among the information within the same modality and between different modalities. For example, there are correlations such as token-token, token-image patch, and image patch-image patch in multi-modal information, as well as semantic interactions between text entities and visual objects. Therefore, only when the above complex fine-grained multi-modal information, as well as the deep and fine-grained semantic interaction relationships within the same modality and across modalities, are fully captured and integrated can the authenticity of the claim to be fact-checked be effectively verified and evaluated, and the accuracy of fact-checking be improved. That is to say, the fine-grained multi-modal information and the deep semantic interactions and correlations among the information within the same modality and between different modalities are all key information indispensable for effectively verifying the authenticity of the claim. Ignoring these fine-grained information and interaction relationships may lead to the following two problems: (1) Insufficient integration of multi-modal evidence, which is likely to result in over-reliance on information in a certain modality while ignoring key information in other modalities; (2) Unable to effectively model the high-order relationships involved in multi-modal evidence, where high-order relationships include information transmission within and between modalities, and this is crucial for comprehensive fact-checking.

[0027] Due to the complexity of multi-modal information and the existence of fine-grained semantic interactions, existing technologies often fail to fully capture and integrate the deep relationships of cross-modal information, resulting in limited accuracy of fact-checking. It can be seen from this that how to capture and fuse fine-grained multi-modal information within a unified framework remains a major challenge in the current field of fact-checking based on multi-modal information. For this reason, the embodiments of the present invention propose a method for training a fact-checking model and a fact-checking method, aiming to solve the key technical problems that have not been fully solved by existing technologies in the field of fact-checking based on multi-modal information, and ultimately be able to accurately use multi-modal evidence to verify, evaluate, and check the authenticity of complex claims.

[0028] In this process, the primary technical problem to be solved by the present invention is how to construct a framework that can effectively process and compare multimodal evidence to accurately evaluate the authenticity of statements. In response to this problem, the solution proposed by the present invention is to develop a deep learning-based multimodal fact-checking framework that can simultaneously process multimodal evidence such as text and images, and improve the accuracy of fact-checking by learning cross-modal latent features and complex semantic interactions. This solution also needs to address the following two technical problems: one is how to establish an effective information fusion mechanism between different modalities to achieve cross-modal information integration; the other is how to effectively train and optimize the model to identify and verify fine-grained multimodal information. In response to the first problem, the present invention proposes to use hypergraphs and transformers to model the high-order relationships between different modality evidence and statements, and optimize the information propagation process through line graphs to further enhance the reasoning ability of the model. In response to the second problem, the present invention adopts an end-to-end training method combined with self-supervised learning techniques to improve the performance of the model and ensure that the model can effectively learn and reason from multimodal evidence to achieve accurate fact-checking.

[0029] Figure 1 and Figure 2 are respectively the flowchart of the fact-checking method and the structural diagram of the fact-checking model in an embodiment of the present invention. As Figure 1 and Figure 2 shown, the method includes the following steps: Step S110, obtain the statement to be verified.

[0030] The statement to be verified contains the fact to be verified, which can be unimodal information or multimodal information, that is, the fact to be verified can be a statement existing solely in text form or image form, that is, a text statement or an image statement; it can also be a statement existing in both text form and image form, that is, a text statement and an image statement.

[0031] Step S120: Use a pre-trained fact-checking model to verify the authenticity of the statement, and the fact-checking model outputs the authenticity prediction result of the statement. Among them, the fact-checking model includes: a multi-modal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module, and a classification module. The multi-modal evidence retrieval module is used to retrieve multiple multi-modal evidences related to the statement from the network; the feature extraction and encoding module is used to extract and encode the features of the statement and the multiple multi-modal evidences at multiple scales to obtain multiple feature embedding vector sequences corresponding to the multiple feature sequences; the hypergraph construction module is used to use each feature corresponding to the feature embedding vector in the multiple feature embedding vector sequences as a node, and use the statement and each evidence as a hyperedge for connecting all nodes in the corresponding feature sequence, and construct a hypergraph based on all nodes and all hyperedges, where the nodes connecting multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to use the self-attention mechanism to obtain the corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected by each hyperedge in the hypergraph, and use the cross-attention mechanism to obtain the hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected by each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes respectively, construct a line graph based on the converted nodes and edges, and obtain the line graph embedding vector based on all edges in the line graph and the hyperedge embedding vectors of all nodes; the classification module is used to output the authenticity prediction result of the statement based on the hypergraph embedding vector and the line graph embedding vector.

[0032] Specifically, multimodal evidence includes text evidence and visual evidence such as image evidence and video evidence. The multimodal evidence retrieval module is a key component in the model framework designed in this method, aiming to provide rich and relevant context information for subsequent fact-checking tasks to support accurate and reliable assessment of the truthfulness of claims. The core task of this module is to retrieve multiple text evidences and multiple visual evidences related to a given claim from a large amount of evidence. Further, multiple text evidences and multiple visual evidences with a relevance greater than a preset relevance threshold are retrieved from a large amount of evidence. For text claims and text evidences, a Sentence-BERT (SBERT) model based on the BERT model (which is a model for calculating semantic similarity between sentences) can be used to screen out multiple text evidences related to the text claim from a large amount of text evidences. Specifically, the text claim and each text evidence are jointly input into the SBERT model. The SBERT model converts each sentence in the text claim and each text evidence into corresponding vector representations in a high-dimensional space, and calculates the similarity (such as cosine similarity, etc.) between the vector representations of the text claim and each sentence in each text evidence as the relevance. Finally, they are sorted from largest to smallest according to the similarity, and multiple text evidences corresponding to multiple sentences with the highest rankings (i.e., greater than the preset relevance threshold) are selected as the text evidences related to the text claim. For image evidences, a Contrastive Language-Image Pre-training (CLIP) model can be used to screen out multiple image evidences related to the text claim or image claim from a large amount of image evidences, or to screen out multiple text evidences related to the image claim from a large amount of text evidences. Specifically, the claim and each image evidence (or the image claim and each text evidence) are jointly input into the CLIP model. The CLIP model generates feature representations of the claim and each image evidence (or the image claim and each text evidence), and calculates the similarity between the feature representations of the claim and each image evidence (or the image claim and each text evidence). Finally, multiple image evidences with a similarity greater than the preset relevance threshold are also selected as the image evidences related to the claim (or multiple text evidences with a similarity greater than the preset relevance threshold are selected as the text evidences related to the image claim). Through contrastive learning, the CLIP model enables the model to understand and match the semantic relationships between images and texts, images and images, thus effectively retrieving image evidences related to the text claim or image claim and text evidences related to the image claim. The design of the multimodal evidence retrieval module takes into account the diversity and coverage of evidence to ensure that the retrieved evidence can comprehensively cover all aspects of the claim.

[0033] The main task of the feature extraction and encoding module is to convert the claims, text evidence, and image evidence in the multimodal data into high-quality embedding representations that can be further processed by the model. These high-quality embedding representations can accurately capture the semantic information features of the multimodal data and retain sufficient details to provide a basis for subsequent hypergraph construction and information dissemination. To improve the accuracy of encoding, this module also considers multi-scale feature extraction including fine-grained and coarse-grained information extraction. Among them, fine-grained features focus on the details of local and individual elements, and coarse-grained features focus on the overall structure and global context. Specifically, for each text, this module can use tools such as Spacy to extract the entity features of the text, that is, lemmas, and then form a lemma sequence corresponding to the text from the extracted multiple lemmas. It can also use the text encoder of the CLIP model (this encoder is based on the transformer architecture) or text encoders such as BERT and RoBERTa that can provide strong semantic representation capabilities to convert each lemma in the lemma sequence into a corresponding lemma embedding representation, that is, a lemma embedding vector. The multiple lemma embedding vectors corresponding to the multiple lemmas in the lemma sequence form a lemma embedding vector sequence corresponding to the lemma sequence. For each image, this module first performs multi-scale feature extraction on the image at different resolutions and perspectives to generate multiple image patches, and forms an image patch sequence corresponding to the image from the multiple image patches. It can also use the image encoder of the CLIP model or image encoders such as ViT and Resnet that can provide strong semantic representation capabilities to convert each image patch in the image patch sequence into a corresponding image patch embedding representation, that is, an image patch embedding vector. The multiple image patch embedding vectors corresponding to the multiple image patches in the image patch sequence form an image patch embedding vector sequence corresponding to the image patch sequence. That is to say, the multiple feature sequences include multiple lemma sequences composed of multiple lemmas and multiple image patch sequences composed of multiple image patches, and the multiple feature embedding vector sequences include multiple lemma embedding vector sequences composed of multiple lemma embedding vectors and multiple image patch embedding vector sequences composed of multiple image patch embedding vectors.

[0034] The purpose of the hypergraph construction module is to integrate multimodal data including statements, text evidence, and image evidence into a unified hypergraph structure so as to effectively capture and model the high-order relationships between different modalities. A hypergraph is a concept that generalizes the traditional graph, allowing a hyperedge to connect multiple nodes. In the hypergraph representing the context of multimodal fact-checking, each node represents a basic data unit. For text evidence or text statements, each token is regarded as a node, and for image evidence or image statements, each image patch is regarded as a node, and these nodes are represented by corresponding embedding vectors. A hyperedge is an edge that connects multiple nodes and is used to represent the associations between different modalities. For example, a text evidence can be represented by a hyperedge that connects all the token nodes constituting the evidence; an image evidence can also be represented by a hyperedge that connects all the image patch nodes constituting the evidence; similarly, a statement can be represented by a hyperedge that connects all the nodes constituting the statement; among them, the same token node or image patch node in different text evidences or image evidences connects multiple hyperedges (i.e., multiple text evidences or image evidences), which is the shared node. This module constructs a hypergraph based on all the nodes and all the hyperedges of the statement and all its corresponding text evidences and image evidences. The definition of the incidence matrix of this hypergraph is as follows: where v i represents a node, i represents the node serial number, e j represents a hyperedge, and j represents the hyperedge serial number.

[0035] In some embodiments, the hypergraph information propagation module includes an intra-hyperedge information propagation module and an inter-hyperedge information propagation module. The intra-hyperedge information propagation module is used to aggregate the feature embedding vectors of multiple consecutive nodes among all the nodes connected by each hyperedge in the hypergraph by using a grouped attention mechanism to form the aggregated feature embedding vectors of multiple aggregated nodes, and obtain the hyperedge embedding vector corresponding to the hyperedge based on the aggregated feature embedding vectors of the multiple aggregated nodes; the inter-hyperedge information propagation module is used to obtain the attention weights between each shared node and each hyperedge connected to the shared node respectively based on the feature embedding vector of each shared node in the hypergraph and the hyperedge embedding vectors of each hyperedge connected to the shared node by using a cross-attention mechanism, update the feature embedding vector of the shared node based on the weighted aggregation of the hyperedge embedding vectors of each hyperedge connected to the shared node and the corresponding attention weights, and obtain the hypergraph embedding vector based on the updated feature embedding vectors of all the shared nodes.

[0036] Traditional hypergraph neural networks (HGNNs) enhance node representations through information propagation between nodes and hyperedges. Specifically, based on the embedding vectors of nodes, node information related to hyperedges is aggregated onto hyperedges through the hypergraph structure to capture high-order features. The aggregated information is then propagated back to the nodes connected to the hyperedges. However, this traditional HGNN information propagation method has some limitations: (1) The CLIP model is mainly used for text-image matching, and its pre-learned fixed node semantic vectors may not well adapt to the dynamic requirements of the claim classification task when applied, thus reducing the model performance. (2) During the fact verification process, there may be differences in the importance of node information in hyperedges of the same modality and hyperedges of different modalities. To address these problems, this method proposes a novel multi-modal hypergraph population transformer, namely the hypergraph information propagation module, which utilizes the characteristics of hyperedges in the hypergraph to optimize the model, aiming to perform effective information propagation within the hypergraph structure, enhance the high-order information fusion between multi-modal claims and evidence, and combine semantic encoding and hypergraph propagation (graph topology modeling) through end-to-end joint training to improve the model performance. Specifically, this method achieves the following goals: (1) Treats the multi-modal semantic understanding parameters as trainable and establishes a joint training framework; (2) Effectively captures the importance of nodes in different modalities by implementing information propagation from nodes to hyperedges and then back to nodes. This module includes two key parts: intra-hyperedge information propagation and inter-hyperedge information propagation.

[0037] Figure 3 This is a schematic diagram of the principles of intra-hyperedge information propagation and inter-hyperedge information propagation in the fact verification model in an embodiment of the present invention. As Figure 3 shown, the purpose of intra-hyperedge information propagation is to learn the embedding representations of claims and multi-modal evidence hyperedges and capture semantic associations at different granularities. For each hyperedge e j and its connected node set V(e j ) = {v1, v2,..., v i ,..., v m}, the set of feature embedding vectors of this node set is S = {S1, S2,..., S i ,..., S m}, where S i represents the feature embedding vector of node v i . At the beginning, this method draws on the basic idea of Transformer and aggregates the elements in V into a fixed-size representation by introducing a learnable reference set Z = {Z1, Z2,..., Z i ,..., Z p}. Each Z i can be regarded as an aggregation unit to capture semantic information from V. The calculation of the self-attention mechanism between nodes within the hyperedge can be expressed as: Among them, are all trainable parameter matrices, and all belong to R p×m ; Q v , K v , V v represent the query, key, and value matrices respectively: d represents the length of the feature embedding vector, and d can take the value of 512; self_att(·) represents the self-attention function, and softmax(·) represents the activation function. Based on this, the embedding representation of the hyperedge e j is the hyperedge embedding vector E j as follows: E j = W e [Z1:Z2:...:Z i :...:Z p T Among them, W e represents the trainable parameter matrix, W e ∈R 1×p ; [:] represents the concatenation operation. However, the basic Transformer attention mechanism has limitations in the information propagation within the hyperedge, that is, it only considers the relationship between individual tokens or image patches at a single granularity level and does not consider the relationship at multiple granularity levels. To solve this problem, this method makes improvements and proposes an algorithm to enhance the traditional self-attention mechanism through a grouped attention mechanism, which is achieved by capturing the interaction between the neighborhoods of tokens or image patches.

[0038] Specifically, this method associates the neighborhoods of tokens or image patches, that is, associates multiple consecutive nodes among all the nodes connected by the hyperedge in the hypergraph to form multiple groups (i.e., aggregated nodes), and replaces the corresponding elements in the query, key, and value matrices with the aggregated embeddings of each group, thereby generating the proxy representation of each group. Aggregation can be achieved by methods such as max pooling or convolution. This method divides Q v , K v , V v into n parts, and the features of each part are represented by X i , where 1 ≤ i ≤ n, and X i represents Q v , K v or V v . The aggregated embedding representation of each part is X' i = Agg i (X i ), and then X' is generated by concatenating the aggregated embedding representations of all parts, thereby obtaining the group proxy representation Q' v ​, K' v and V' v . These grouped agents are then used for attention calculation to generate the final Z', as shown in the following formula: Next, the embedding representation of the hyperedge e j , that is, the hyperedge embedding vector E j : Z″ = W p ·Z′ = (Z″1, Z″2,..., Z" i ,..., Z" p} E j = W e [Z"1:Z″2:…:Z″ i :…:Z″ p T where W p represents a trainable parameter matrix, W p ∈R p×n . Since the grouped agent representations are involved in the attention calculation, the present invention simulates the relationships between K×K tokens, where K represents the kernel size of the aggregator, which may vary among different parts. This strategy goes beyond individual tokens or image patches, enabling the method to capture the relationships between groups of different sizes, thereby enhancing the overall performance of the model. In addition to using the above Group Transformer for information propagation within hyperedges, other variants of Transformer, such as Transformer-XL, etc., can also be used, and these variants can all capture long-range dependencies and fine-grained semantic interaction information.

[0039] The purpose of information propagation between hyperedges is to learn the attention weights between shared nodes and the respective hyperedges associated with them. It can be recognized that different shared nodes have different degrees of importance for the hyperedges they connect. This is mainly achieved by aggregating the topological information of multiple hyperedges connected to the shared node to enhance the embedding representation of the shared node. Therefore, the present method adopts the scaled dot-product attention mechanism and the heterogeneous hypergraph attention algorithm (i.e., the cross-attention mechanism) to propagate information from hyperedges to nodes.

[0040] Specifically, for each shared node v i and the set of hyperedges E(v i ) = {e1, e2, …, e j ,..., e t} connected to it, each hyperedge e j ∈E(v i ) has a corresponding hyperedge embedding representation E j . The shared node v​i The attention weights between a shared node and each of its connected hyperedges can be obtained through the following formula: where S i represents the embedding vector of the shared node v i (i.e., the feature embedding vector), E j represents the hyperedge embedding vector of the hyperedge e i , W q , W k and W v are all trainable parameter matrices and all belong to R d×d ; att(·) represents the attention weight function. Then, the feature embedding vector of the shared node is updated through the weighted aggregation of the hyperedge embedding vectors of each hyperedge connected to the shared node and the corresponding attention weights by the following formula: where represents the updated feature embedding vector of the shared node v,, β represents a hyperparameter used to control the fusion weight of the heterogeneous topological information from hyperedges in the embedding representation of the shared node, that is, the proportion 1 - β of the original information of the shared node itself retained in the embedding representation of the shared node and the proportion β of the heterogeneous topological information of the connected hyperedges, so as to balance the semantic features of the shared node itself and the global structural features carried by the hyperedges when integrating multimodal information, in order to optimize the model's ability to capture complex semantic relationships. The embedding representation of the hypergraph, that is, the hypergraph embedding vector R HG : where |V| represents the number of shared nodes in the hypergraph.

[0041] In some embodiments, the line graph module includes a line graph construction module and a line graph information propagation module. The line graph construction module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes corresponding to the hyperedges connected to the shared nodes respectively, and construct a line graph based on the converted nodes and edges; the line graph information propagation module is used to construct the adjacency matrix of the line graph by using a graph convolutional network based on all the edges and all the nodes in the line graph, and obtain the line graph embedding vector based on the matrix obtained by adding the adjacency matrix and the identity matrix and the matrix formed by the hyperedge embedding vectors of all the nodes.

[0042] The line graph module is the part of the model framework of this method for enhancing the multi-modal fact-checking reasoning ability, mainly including two parts: line graph construction and line graph information propagation. The line graph can also be called the edge graph of the hypergraph, which is a graph that can convert each hyperedge of the hypergraph into a corresponding node and create corresponding edges according to each shared node between hyperedges. The edges of the line graph are used to connect the nodes corresponding to the hyperedges that share the same node in the hypergraph, and the hyperedges that are not connected by any shared nodes in the hypergraph are converted into isolated nodes without any edges connected in the line graph. This conversion can capture the coarse-grained interactions between hyperedges and explore the semantic structure of the graph more deeply.

[0043] After the line graph construction module constructs the line graph, the line graph information propagation module can use the graph convolutional network (GCN) to propagate information on the line graph to learn the embedded representation of the line graph. This process can be expressed by the following formula: where, G represents the matrix of node embedding vectors of the line graph, σ represents the activation function; D L represents the degree matrix of (A L +I); A L represents the adjacency matrix of the line graph, A L ∈E |E|×|E| , E represents the hyperedge embedding vector in the hypergraph, that is, the node embedding vector in the line graph, E∈E |E|×d , |E| represents the number of hyperedges in the hypergraph; I represents the identity matrix; W represents the trainable parameter matrix; |V L | represents the number of nodes in the line graph, |V L | = |E|; R LG represents the embedded representation of the line graph, that is, the line graph embedding vector. By obtaining the line graph embedding representation through line graph construction and line graph information propagation, the model's ability to capture high-order semantic features is enhanced. In other embodiments, different graph upgrade network structures can also be used for line graph information propagation, such as the graph attention network (GAT), etc., to enhance the model's ability to capture complex relationships between hyperedges.

[0044] Figure 4 is a schematic flowchart of the fact-checking model training method in an embodiment of the present invention. As Figure 4 shown, this method includes the following steps: Step S410, obtain a training set, where the training set includes multiple statements with their respective authenticity category labels and a multi-modal evidence base containing multiple multi-modal evidences.

[0045] Specifically, the training set is a large-scale dataset, which can specifically contain 15,601 claims and a multi-modal evidence base. Among them, the multi-module evidence base includes a text evidence base or a text corpus, and an image evidence base or an image corpus. The text evidence base can accommodate 33,880 text paragraphs as multiple text evidences, and the image evidence base can accommodate 12,112 images as multiple image evidences. The claims in this dataset may all be unimodal, may all be multi-modal, or may be a mixture of unimodal and multi-modal, and are all labeled with corresponding truthfulness category labels. The truthfulness category labels include that the facts contained in the claim are supported by evidence, refuted, and the information is insufficient to determine the truthfulness of the claim, that is, the claim is true, false, and the truthfulness cannot be determined, etc. These data not only provide rich multi-modal information, but also ensure the quality and consistency of the data through manual annotation of a large number of claims by fact-checking journalists. The claims in the dataset cover a variety of topics, mainly including political, technological, health and other fields, which can make the trained fact-checking model applicable and generalizable in a variety of different fields.

[0046] Step S420, training a preset fact-checking model based on the training set to train the preset fact-checking model into a fact-checking model that outputs the truthfulness prediction result of the claim based on the claim. Among them, the fact-checking model includes: a multi-modal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module, and a classification module. The multi-modal evidence retrieval module is used to retrieve multiple multi-modal evidences related to the claim from the multi-modal evidence base; the feature extraction and encoding module is used to extract and encode the features of the claim and the multiple multi-modal evidences at multiple scales to obtain multiple feature embedding vector sequences corresponding to multiple feature sequences; the hypergraph construction module is used to use each feature corresponding to the feature embedding vector in the multiple feature embedding vector sequences as a node, and use the claim and each evidence as a hyperedge for connecting all nodes in the corresponding feature sequence respectively, and construct a hypergraph based on all nodes and all hyperedges, where the nodes connecting multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to use the self-attention mechanism to obtain the corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected by each hyperedge in the hypergraph, and use the cross-attention mechanism to obtain the hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected by each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes respectively, construct a line graph based on the converted nodes and edges, and obtain the line graph embedding vector based on all edges in the line graph and the hyperedge embedding vectors of all nodes; the classification module is used to output the truthfulness prediction result of the corresponding claim based on the hypergraph embedding vector and the line graph embedding vector.

[0047] The fact-checking model training process in this method is end-to-end. That is to say, the process from evidence retrieval to the prediction of claim authenticity is a continuous one. First, through the multimodal evidence retrieval module and the feature extraction and encoding module, multimodal evidence containing text and images related to the claim is retrieved respectively and feature extraction is carried out. These features are subsequently used to construct a hypergraph. Then, through the hypergraph construction module, multimodal data such as text evidence, image evidence, and claims are integrated into a hypergraph structure. Next, through the hypergraph information propagation module, the Transformer mechanism is used to perform effective information flow within the hypergraph structure, enhancing the high-order information integration between multimodal claims and evidence. In addition, a line graph module is introduced to optimize the information propagation process and further improve the inference ability of the model. Finally, a supervised loss function is used to evaluate and optimize the model. The loss function includes cross-entropy loss to ensure that the model can accurately predict the authenticity of the claim.

[0048] In some embodiments, in step S420, training a preset fact-checking model based on the training set to train the preset fact-checking model into a fact-checking model that outputs a prediction result of the authenticity of the claim based on the claim includes the following steps: Training a preset fact-checking model based on the training set by minimizing the loss function until the difference between the prediction result of the authenticity of the claim and the authenticity category label is less than a preset threshold, and obtaining the finally trained fact-checking model; the loss function uses a cross-entropy loss function and is calculated according to the following formula: where Loss s represents the cross-entropy loss, y k represents the authenticity category label of the kth claim, represents the prediction result of the authenticity of the kth claim, R HG represents the hypergraph embedding vector, R LG represents the line graph embedding vector, [:] represents the concatenation operation, W R represents the trainable parameter matrix, l c represents the number of authenticity categories; in the present invention, l c = 3. The cross-entropy loss function measures the difference between the prediction result of the authenticity of the claim predicted by the model output and the corresponding set authenticity category label. By optimizing this loss function, the prediction result of the model can be made closer to the true authenticity category label. In addition, according to the above formula, in this implementation process, first, the hypergraph embedding vector R HG output by the hypergraph information propagation module and the line graph embedding vector R LGThe fused embedding vector is concatenated to form a fused embedding vector that contains all the information of the hypergraph and the line graph. The authenticity of the corresponding input statement is then predicted based on the fused embedding vector, and the authenticity prediction result of the statement is output. The output of the classification module is a probability distribution that represents the probability that the statement belongs to each authenticity category label. The classification module can use a multi-layer perceptron (MLP) to map the fused embedding vector to the authenticity category of the statement.

[0049] In summary, the fact-checking model training method and fact-checking method of the embodiment of the present invention significantly improve the accuracy and efficiency of multimodal fact-checking. The main reasons are as follows: First, by constructing a hypergraph structure, it is possible to effectively integrate and compare evidence from different modalities, allowing the model to capture richer semantic information and high-order relationships between different modal evidence, especially fine-grained multimodal information. Secondly, the hypergraph information propagation module introduced by this method, especially the hyperedge intra- and hyperedge inter-information propagation mechanism, enables the model to more deeply understand the complex and deep semantic interactions between multimodal evidence. This fine-grained information propagation strategy not only improves the semantic understanding of evidence, but also enhances the model's ability to capture cross-modal relationships. In addition, the introduction of the line graph module further optimizes the information propagation process, and by capturing the coarse-grained interactions between hyperedges, the model's ability to capture high-order semantic features is enhanced. The line graph module provides the model with additional structured information, which helps to reveal the complex interactions between multimodal evidence, thereby improving the depth and breadth of fact-checking. This method also uses an end-to-end training method to train the model and jointly optimizes all modules so that the model can learn more effective feature representations.

[0050] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the above method.

[0051] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0052] An embodiment of the present invention further provides a computer program product, including computer instructions which, when executed by a processor, implement the steps of the foregoing method.

[0053] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present invention are programs or code segments used to execute the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link.

[0054] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0055] In the present invention, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0056] The foregoing are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the embodiments of the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A fact-checking method, characterized in that: The method comprises: A pre-trained fact-checking model is used to verify the authenticity of a statement to be verified, and the fact-checking model outputs a prediction result of the authenticity of the statement, wherein the fact-checking model includes: a multimodal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module and a classification module, wherein the multimodal evidence retrieval module is used to retrieve multiple multimodal evidences related to the statement from the network; the feature extraction and encoding module is used to extract and encode the respective features of the statement and the multiple multimodal evidences at multiple scales to obtain multiple feature embedding vector sequences corresponding to the multiple feature sequences; the hypergraph construction module is used to take the feature corresponding to each feature embedding vector in the multiple feature embedding vector sequences as a node, and take the statement and each evidence as a node for connecting the corresponding feature sequences. A hyperedge having nodes, and a hypergraph is constructed based on all nodes and all hyperedges, wherein nodes connecting multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to adopt a self-attention mechanism to obtain a corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected by each hyperedge in the hypergraph, and adopt a cross-attention mechanism to obtain a hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected to each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes, respectively, construct a line graph based on the converted nodes and edges, and obtain a line graph embedding vector based on the hyperedge embedding vectors of all edges and all nodes in the line graph; the classification module is used to output the authenticity prediction result of the statement based on the hypergraph embedding vector and the line graph embedding vector.

2. The method according to claim 1, characterized in that: The hypergraph information propagation module includes an intra-hyperedge information propagation module and an inter-hyperedge information propagation module. The intra-hyperedge information propagation module is used to adopt a grouped attention mechanism to aggregate the feature embedding vectors of multiple consecutive nodes in all nodes connected to each hyperedge in the hypergraph to form an aggregated feature embedding vector of multiple aggregated nodes, and obtain the hyperedge embedding vector of the corresponding hyperedge based on the aggregated feature embedding vector of the multiple aggregated nodes; the inter-hyperedge information propagation module is used to adopt a cross-attention mechanism to obtain the attention weights between each shared node and each hyperedge connected to the shared node based on the feature embedding vector of each shared node in the hypergraph and the hyperedge embedding vectors of each hyperedge connected to the shared node, update the feature embedding vector of the shared node based on the weighted aggregation of the hyperedge embedding vectors of each hyperedge connected to each shared node and the corresponding attention weights, and obtain the hypergraph embedding vector based on the updated feature embedding vectors of all shared nodes.

3. The method according to claim 1, characterized in that The line graph module includes a line graph construction module and a line graph information propagation module. The line graph construction module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes corresponding to the hyperedges connected by the shared nodes, respectively, and construct a line graph based on the converted nodes and edges; the line graph information propagation module is used to use a graph convolutional network to construct an adjacency matrix of the line graph based on all edges and all nodes in the line graph, and obtain a line graph embedding vector based on a matrix obtained by adding the adjacency matrix to the unit matrix and a matrix formed by the hyperedge embedding vectors of all nodes.

4. The method according to any one of claims 1 to 3, characterized in that The multimodal evidence includes text evidence and image evidence, and the statement includes a text statement and / or an image statement.

5. The method according to claim 4, characterized in that The multiple feature sequences include multiple word sequences composed of multiple word elements and multiple image block sequences composed of multiple image blocks, and the multiple feature embedding vector sequences include multiple word embedding vector sequences composed of multiple word embedding vectors and multiple image block embedding vector sequences composed of multiple image block embedding vectors.

6. A fact-checking model training method, characterized in that: The method comprises: Obtaining a training set, wherein the training set includes a plurality of statements having respective authenticity category labels and a multimodal evidence library including a plurality of multimodal evidences; A preset fact-checking model is trained based on the training set to train the preset fact-checking model into a fact-checking model that outputs a prediction result of the authenticity of the statement based on the statement, wherein the fact-checking model includes: a multimodal evidence retrieval module, a feature extraction and encoding module, a hypergraph construction module, a hypergraph information propagation module, a line graph module and a classification module, the multimodal evidence retrieval module is used to retrieve multiple multimodal evidences related to the statement from the multimodal evidence library; the feature extraction and encoding module is used to extract and encode the respective features of the statement and the multiple multimodal evidence at multiple scales to obtain multiple feature embedding vector sequences corresponding to the multiple feature sequences; the hypergraph construction module is used to take the feature corresponding to each feature embedding vector in the multiple feature embedding vector sequences as a node, and take the statement and each evidence as a node for connection. The method comprises the following steps: connecting the hyperedges of all nodes in the corresponding feature sequence, and constructing a hypergraph based on all nodes and all hyperedges, wherein the nodes connected to multiple hyperedges are used as shared nodes; the hypergraph information propagation module is used to adopt a self-attention mechanism to obtain a corresponding hyperedge embedding vector based on the feature embedding vectors of all nodes connected to each hyperedge in the hypergraph, and adopt a cross-attention mechanism to obtain a hypergraph embedding vector based on the feature embedding vectors of all shared nodes in the hypergraph and the hyperedge embedding vectors of multiple hyperedges connected to each shared node; the line graph module is used to convert the hyperedges and shared nodes in the hypergraph into nodes and edges connecting the nodes, respectively, construct a line graph based on the converted nodes and edges, and obtain a line graph embedding vector based on the hyperedge embedding vectors of all edges and all nodes in the line graph; the classification module is used to output the authenticity prediction result of the corresponding statement based on the hypergraph embedding vector and the line graph embedding vector.

7. The method according to claim 6, characterized in that The preset fact-checking model is trained based on the training set to train the preset fact-checking model into a fact-checking model that outputs a prediction result of the authenticity of the statement based on the statement, including: Based on the training set, the preset fact-checking model is trained by minimizing the loss function until the difference between the authenticity prediction result of the statement and the authenticity category label is less than a preset threshold, and the final trained fact-checking model is obtained; the loss function adopts a cross entropy loss function and is calculated according to the following formula: Among them, Loss s represents the cross entropy loss; y k represents the authenticity category label of the k-th statement, the authenticity category label includes whether the facts contained in the statement are supported by evidence, refuted by evidence, and insufficient information to determine the authenticity of the statement, that is, the statement is true, false, and the authenticity cannot be determined; represents the authenticity prediction result of the k-th statement, R HG represents the hypergraph embedding vector, R LG represents the line graph embedding vector, [:] represents the concatenation operation, W R Represents a trainable parameter matrix.

8. An electronic device comprising a processor, a memory and computer instructions stored in the memory, characterized in that: The processor is used to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images