Multimodal fake news detection method, apparatus and device

Abstract

The disclosure provides a multi-modal false information detection method, device and equipment, which can be applied to the technical field of information detection. The method comprises: acquiring a plurality of event information about a same topic keyword; generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information; extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph; processing the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event false judgment results; generating a time feature sequence based on a plurality of multi-modal features and a plurality of event false judgment results; and obtaining a false judgment result about the topic keyword according to the time feature sequence.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of information detection, and more particularly, to a multi-modal false information detection method, device and equipment. BACKGROUND

[0002] In recent years, with the rapid development of the Internet, people's way of obtaining news event information has become more and more convenient, and false news event information and the like is also rapidly spread through social platforms, which has a huge negative impact on public opinion. With the popularity of mobile devices, news event information is often rapidly spread in the form of multi-modal information. While it is convenient for readers to obtain news event information, it also causes the technology of relying solely on traditional text analysis of false news event information to be no longer applicable.

[0003] Moreover, in the traditional false news event information detection method, only a single news event information can be detected. Multiple news event information under the same theme keyword are related and influenced each other, and in the process of detecting the false of the theme keyword by using multiple news event information related to the theme keyword, the contribution degrees of the multiple news event information to the false identification of the theme keyword are different, and joint modeling is difficult. SUMMARY

[0004] Therefore, the present disclosure provides a multi-modal false information detection method, device and equipment.

[0005] One aspect of the present disclosure provides a multi-modal false information detection method, comprising:

[0006] obtaining multiple event information about the same theme keyword;

[0007] generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information;

[0008] extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph;

[0009] processing the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event false identification results;

[0010] generating a time feature sequence based on multiple multi-modal features and multiple event false identification results, wherein the time feature sequence comprises multiple local perception vectors, the multiple local perception vectors are sorted according to the publication times of the multiple event information, and each local perception vector comprises multi-modal features and event false identification results corresponding to the event information after splicing;

[0011] obtaining a false identification result about the theme keyword according to the time feature sequence.

[0012] According to an embodiment of the present disclosure, the generating, according to the external knowledge information and the event information, a heterogeneous graph corresponding to the event information comprises:

[0013] Obtaining multi-modal information of the event information, the multi-modal information comprising at least two of the following: text information in the event information, visual information in the event information, and social context information related to the event information;

[0014] Extracting, by using a preset extraction rule, entities and relationships of the text information from the text information of the multi-modal information, entities and relationships of the visual information from the visual information, and entities and relationships of the social context information from the social context information;

[0015] According to the external knowledge information, obtaining entities and relationships of the external knowledge information related to the topic keyword;

[0016] According to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information, generating the heterogeneous graph corresponding to the event information.

[0017] According to an embodiment of the present disclosure, the extracting, by using a preset extraction rule, entities and relationships of the text information from the text information of the multi-modal information, entities and relationships of the visual information from the visual information, and entities and relationships of the social context information from the social context information comprises:

[0018] Extracting, by using a pre-trained language model, the entities of the text information from the text information and the entities of the social context information from the social context information;

[0019] Extracting, by using a residual neural network model, the entities of the visual information from the visual information;

[0020] According to the entities of the text information and the text information, extracting, by using a segment convolutional neural network model, the relationships of the text information between the entities of the text information from the text information;

[0021] According to the entities of the social context information and the social context information, extracting, by using a segment convolutional neural network model, the relationships of the social context information between the entities of the social context information from the social context information;

[0022] According to the entities of the visual information and the visual information, extracting, by using a segment convolutional neural network model, the relationships of the visual information between the entities of the visual information from the visual information.

[0023] According to an embodiment of the present disclosure, the obtaining, according to the external knowledge information, the entity and the relationship related to the topic keyword comprises:

[0024] According to the external knowledge information, a plurality of knowledge graphs of different types are constructed, wherein the external knowledge information is obtained from open source information, the external knowledge information comprises information related to the topic keyword, and the plurality of knowledge graphs comprise at least two of the following: a domain knowledge graph, a general knowledge graph, a matter knowledge graph, and a space-time knowledge graph.

[0025] The entity and the relationship of the external knowledge information are obtained from the plurality of knowledge graphs.

[0026] According to an embodiment of the present disclosure, the generating, according to the entity and the relationship of the external knowledge information, the entity and the relationship of the text information, the entity and the relationship of the visual information, and the entity and the relationship of the social context information, the heterogeneous graph corresponding to the event information comprises:

[0027] The entity alignment is performed on the entity of the external knowledge information, the entity of the text information, the entity of the visual information, and the entity of the social context information by using an entity linking technology, to obtain the entity of the external knowledge information, the entity of the text information, the entity of the visual information, and the entity of the social context information with unified entity names.

[0028] The heterogeneous graph corresponding to the event information is generated by using the entity and the relationship of the external knowledge information with unified entity names, the entity and the relationship of the text information with unified entity names, the entity and the relationship of the visual information with unified entity names, and the entity and the relationship of the social context information with unified entity names.

[0029] According to an embodiment of the present disclosure, the extracting, from the entity of the heterogeneous graph, a feature to obtain the entity feature of the entity of the heterogeneous graph comprises:

[0030] The entity of the heterogeneous graph is obtained.

[0031] The entity of the heterogeneous graph is processed by using a pre-trained language model to obtain the entity feature of the entity of the heterogeneous graph.

[0032] According to an embodiment of the present disclosure, the processing, on the heterogeneous graph and the entity feature, to obtain the multi-modal feature and the event fake judgment result of the event information comprises:

[0033] The heterogeneous graph is processed to obtain a heterogeneous graph adjacency matrix.

[0034] input the above-mentioned heterogeneous graph adjacency matrix and the above-mentioned entity feature into a feature extraction module in the hierarchical attention model to obtain the above-mentioned multi-modal feature;

[0035] input the above-mentioned multi-modal feature into a full connection layer and a classification layer of the hierarchical attention model in sequence to obtain the above-mentioned event fake identification result.

[0036] According to an embodiment of the present disclosure, the generating a time feature sequence based on the plurality of multi-modal features and the plurality of event fake identification results comprises:

[0037] performing feature splicing on the multi-modal feature and the event fake identification result corresponding to the multi-modal feature to generate a local perception vector of the event information;

[0038] sorting and combining a plurality of local perception vectors according to the publishing time of each of the plurality of event information to obtain the time feature sequence.

[0039] Another aspect of the present disclosure provides a multi-modal fake information detection device, comprising:

[0040] a plurality of event information acquisition modules configured to acquire a plurality of event information about a same topic keyword;

[0041] a heterogeneous graph generation module configured to generate a heterogeneous graph corresponding to the event information according to external knowledge information and the event information;

[0042] an entity feature obtaining module configured to extract features from entities of the heterogeneous graph to obtain entity features of entities of the heterogeneous graph;

[0043] an event fake identification result obtaining module configured to process the heterogeneous graph and the entity features to obtain multi-modal features and event fake identification results of the event information;

[0044] a time feature sequence generation module configured to generate a time feature sequence based on the plurality of multi-modal features and the plurality of event fake identification results, wherein the time feature sequence comprises a plurality of local perception vectors, the plurality of local perception vectors are sorted according to the publishing time of each of the plurality of event information, and each local perception vector comprises spliced multi-modal features and event fake identification results corresponding to the event information;

[0045] a topic keyword fake identification result obtaining module configured to obtain a fake identification result about the topic keyword according to the time feature sequence.

[0046] Another aspect of the present disclosure provides an electronic device, comprising:

[0047] one or more processors;

[0048] a memory for storing one or more instructions,

[0049] wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method as described above.

[0050] According to an embodiment of the present disclosure, by acquiring a plurality of event information about a same topic keyword, and generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information, extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph, processing the heterogeneous graph and the entity features to obtain a plurality of multi-modal features of the event information and event fake identification results, local perception is realized on the multi-modal features and the event fake identification results of the plurality of event information included in the topic keyword, a time feature sequence is generated based on the plurality of multi-modal features and the plurality of event fake identification results, wherein the time feature sequence includes a plurality of local perception vectors, the plurality of local perception vectors are sorted according to the publishing times of the plurality of event information respectively, and each local perception vector includes the multi-modal features and the event fake identification results corresponding to the event information after splicing, and according to the time feature sequence, the technical means for obtaining the fake identification result about the topic keyword is realized, the time feature sequence generated by using the multi-modal features and the event fake identification results of the plurality of event information included in the topic keyword is used to globally represent the features of the topic keyword, the time feature sequence is obtained, and according to the time feature sequence, the fake identification result about the topic keyword is obtained, the fake of the topic keyword is judged by using the time feature sequence generated by using the multi-modal features and the event fake identification results of the plurality of event information included in the topic keyword, and the accuracy of the fake detection of the topic keyword is improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0052] Figure 1 schematically illustrates an exemplary system architecture to which a multi-modal fake information detection method according to an embodiment of the present disclosure can be applied;

[0053] Figure 2 schematically illustrates a flowchart of a multi-modal fake information detection method according to an embodiment of the present disclosure;

[0054] Figure 3 schematically illustrates a block diagram of a multi-modal fake information detection apparatus according to an embodiment of the present disclosure; and

[0055] Figure 4 schematically illustrates a block diagram of an electronic device suitable for the above-described method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and methods are not described in detail in order to avoid obscuring the concepts of the present disclosure.

[0057] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0058] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or excessively formal manner.

[0059] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of the items enumerated, but not limited to the items enumerated (e.g., "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.). In the case of using expressions similar to "at least one of A, B, or C, etc.", it should generally be interpreted to include at least one of the items enumerated, but not limited to the items enumerated (e.g., "a system having at least one of A, B, or C" should include a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0060] In a conventional false news event information detection method, only a single news event information can be detected. Multiple news event information under the same theme keyword are related and affect each other, and in the process of detecting the false of the theme keyword by using multiple news event information related to the theme keyword, the multiple news event information has different contribution degrees to the false identification of the theme keyword, and it is difficult to jointly model. In view of the problems existing in the process of detecting the false of the theme keyword by using multiple news event information related to the theme keyword, embodiments of the present disclosure provide a multi-modal false information detection method, device and equipment.

[0061] The embodiment of the disclosure provides a multi-modal false information detection method, which comprises: acquiring a plurality of event information about a same topic keyword; generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information; extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph; processing the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event false judgment results; generating a time feature sequence based on the plurality of multi-modal features and the plurality of event false judgment results, wherein the time feature sequence comprises a plurality of local perception vectors, the plurality of local perception vectors are sorted according to respective publishing times of the plurality of event information, and each local perception vector comprises spliced multi-modal features and event false judgment results corresponding to the event information; and obtaining a false judgment result about the topic keyword according to the time feature sequence.

[0062] According to the embodiment of the disclosure, by acquiring a plurality of event information about a same topic keyword, and generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information, extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph, processing the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event false judgment results, the multi-modal features and the event false judgment results of the plurality of event information included in the topic keyword are locally perceived, a time feature sequence is generated based on the plurality of multi-modal features and the plurality of event false judgment results, wherein the time feature sequence comprises a plurality of local perception vectors, the plurality of local perception vectors are sorted according to respective publishing times of the plurality of event information, and each local perception vector comprises spliced multi-modal features and event false judgment results corresponding to the event information, the technical means of obtaining a false judgment result about the topic keyword according to the time feature sequence, realizing that the time feature sequence generated by using the multi-modal features and the event false judgment results of the plurality of event information included in the topic keyword globally represents the features of the topic keyword, obtains the time feature sequence, and obtains the false judgment result about the topic keyword according to the time feature sequence, realizes that the time feature sequence generated by using the multi-modal features and the event false judgment results of the plurality of event information included in the topic keyword judges the false of the topic keyword, and improves the accuracy of false detection of the topic keyword.

[0063] In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and public order and good customs are not violated.

[0064] In the technical scheme of the disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0065] Figure 1An exemplary system architecture 100 to which a multi-modal fake information detection method according to embodiments of the present disclosure can be applied is schematically shown. It should be noted that Figure 1 The system architecture shown is merely an example of a system architecture to which embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0066] As Figure 1 The system architecture 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105, as shown. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0067] A user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0068] The first terminal device 101, the second terminal device 102, the third terminal device 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0069] The server 105 can be a server providing various services, such as a background management server providing support for websites browsed by users using the first terminal device 101, the second terminal device 102, the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information or data generated or obtained according to user requests, etc.) to the terminal device.

[0070] It should be noted that the multi-modal false information detection method provided by the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the multi-modal false information detection apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The multi-modal false information detection method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the multi-modal false information detection apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0071] For example, the subject keyword and the multiple event information of the same subject keyword can be stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (for example, the terminal device 101, but not limited thereto), or stored on an external storage device and can be imported into the terminal device 101. Then, the terminal device 101 can send the subject keyword and the multiple event information of the same subject keyword to the server 105, and the multi-modal false information detection method provided by the embodiments of the present disclosure is executed by the server 105 receiving the subject keyword and the multiple event information of the same subject keyword.

[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0073] Figure 2 A flowchart of a multi-modal false information detection method according to an embodiment of the present disclosure is schematically shown.

[0074] As Figure 2 The method includes operations S201-S207, as shown in the figure.

[0075] In operation 201, multiple event information about the same subject keyword is obtained.

[0076] According to the embodiments of the present disclosure, the multiple event information can represent multiple multi-modal information related to the same subject keyword. The multi-modal information can include text information, image information, video information, audio information, etc.

[0077] According to an embodiment of the present disclosure, since the multiple event information can include information related to the same topic keyword and similar in part, the multiple event information under the same topic keyword are associated with each other and influence each other, and can be used for false identification of the topic keyword. According to an embodiment of the present disclosure, the multiple event information can be, for example, multiple news information under the same topic keyword.

[0078] According to an embodiment of the present disclosure, the topic keyword can be, for example, “Star A got married”, and 3 pieces of news information related to “Star A got married” can be searched according to the topic keyword “Star A got married”, and each piece of news information can include a photo of Star A and a text description of the information about Star A getting married.

[0079] In operation 202, a heterogeneous graph corresponding to the event information is generated according to the external knowledge information and the event information.

[0080] According to an embodiment of the present disclosure, the external knowledge information is external data related to the topic keyword that can be obtained from different media platforms.

[0081] According to an embodiment of the present disclosure, entities and relationships can be extracted from the external knowledge information and the event information respectively, and then a heterogeneous graph corresponding to the event information is generated according to the entities and relationships extracted from the external knowledge information and the event information.

[0082] In operation 203, features are extracted from the entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph.

[0083] According to an embodiment of the present disclosure, the same feature extraction method can be used to extract features of multiple entities in the heterogeneous graph to obtain the entity features of the entities of the heterogeneous graph.

[0084] According to an embodiment of the present disclosure, the feature extraction method for extracting features from the entities of the heterogeneous graph can be, for example, a pre-trained language (Bidirectional Encoder Representation from Transformers, BERT) model, a residual neural network (Residual Network, RESNET50) model, a Light Gradient Boosting Machine (LGB) model, or a Very Deep Convolutional Networks (VGG-16) model, and the like, and the embodiments of the present disclosure do not limit the feature extraction method, which can be selected according to actual conditions.

[0085] In operation 204, the heterogeneous graph and the entity features are processed to obtain multiple modal features of the event information and an event false identification result.

[0086] According to an embodiment of the present disclosure, the multi-modal feature representation fuses features of multi-modal information of event information and external knowledge information.

[0087] According to an embodiment of the present disclosure, the false identification result represents the falsity of the event information. In a case where the event information is identified as true, the false identification result can be represented by 1, and in a case where the event information is identified as false, the false identification result can be represented by 0.

[0088] According to an embodiment of the present disclosure, the heterogeneous graph can be converted into an adjacency matrix form, and then the heterogeneous graph in the form of the adjacency matrix and the entity feature are input into a related text classification algorithm, and the multi-modal feature of the event information and the false identification result of the event are output according to the text classification algorithm.

[0089] According to an embodiment of the present disclosure, the related text classification algorithm can be, for example, a neural network-based algorithm, such as a Deep Neural Networks (DNN) model, a Deep Belief Network (DBN) model, a Hierarchical Attention Networks (HAN) model, and a combination technology thereof. Embodiments of the present disclosure do not limit the specific text classification algorithm, and the specific text classification algorithm can be selected according to actual conditions.

[0090] In operation 205, a time feature sequence is generated based on the plurality of multi-modal features and the plurality of false identification results of events, wherein the time feature sequence includes a plurality of local perception vectors, the plurality of local perception vectors are sorted according to the respective publishing times of the plurality of event information, and each local perception vector includes the multi-modal feature and the false identification result of the event information after splicing.

[0091] In operation 206, a false identification result of a topic keyword is obtained according to the time feature sequence.

[0092] According to an embodiment of the present disclosure, the time feature sequence can be input into a Bi-directional Long Short-Trem Memory-ATTENTION (BI-LSTM-ATTENTION) model, and the false identification result of the topic keyword is output by the BI-LSTM-ATTENTION model.

[0093] According to an embodiment of the present disclosure, the time feature sequence generated by using the multi-modal features of the multiple event information included in the topic keyword and the event fake identification result globally represents the features of the topic keyword, obtains the time feature sequence, and obtains the fake identification result of the topic keyword according to the time feature sequence, so as to realize the judgment of the fake of the topic keyword by using the time feature sequence generated by using the multi-modal features of the multiple event information included in the topic keyword and the event fake identification result, and improve the accuracy of the fake detection of the topic keyword.

[0094] According to an embodiment of the present disclosure, for the operation S202 shown in the figure Figure 2 According to the operation S202, the heterogeneous graph corresponding to the event information is generated according to the external knowledge information and the event information, which can include the following operations:

[0095] The multi-modal information of the event information is obtained, and the multi-modal information includes at least two of the following: text information in the event information, visual information in the event information, and social context information related to the event information;

[0096] The entities and relationships of the text information are extracted from the text information of the multi-modal information by using a preset extraction rule, the entities and relationships of the visual information are extracted from the visual information, and the entities and relationships of the social context information are extracted from the social context information;

[0097] According to the external knowledge information, the entities and relationships of the external knowledge information related to the topic keyword are obtained;

[0098] According to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information, the heterogeneous graph corresponding to the event information is generated.

[0099] According to an embodiment of the present disclosure, the text information represents the literal description information included in the event information. The visual information represents the information in the image and video included in the event information. The social context information related to the event information includes basic attribute information and behavior attribute information. The basic attribute information includes gender information and geographical information of the publisher of the event information. The behavior attribute information includes attention behavior information and forwarding behavior information of the publisher of the event information to the event information.

[0100] According to an embodiment of the present disclosure, a heterogeneous graph can be constructed for each of the multiple event information of the topic keyword, and each heterogeneous graph includes the multi-modal information of the event information and the external knowledge information.

[0101] According to an embodiment of the present disclosure, the heterogeneous graph can be represented as G=(V, E), where V is a set of nodes in the heterogeneous graph, and E is a set of edges in the heterogeneous graph. The nodes correspond to entities, and the edges correspond to relationships. The entities include entities of text information, entities of visual information, entities of social context information, and entities of external knowledge information. The relationships include relationships of text information, relationships of visual information, relationships of social context information, and relationships of external knowledge information.

[0102] According to an embodiment of the present disclosure, in the process of extracting entities and relationships from the multi-modal information by using the preset extraction rules, different extraction rules can be selected according to different information types in the multi-modal information.

[0103] According to an embodiment of the present disclosure, for the social context information, three types of relationships can be included, i.e., user-post-news relationship, user-propagate-news relationship, and user-follow-user relationship.

[0104] At present, in the false event information detection technology at home and abroad, a single modality information related to the detected event information is often relied on to detect the false event information, and there is a problem of missing modality information.

[0105] According to an embodiment of the present disclosure, by obtaining multi-modal information of the event information, the multi-modal information includes at least two of the following: text information in the event information, visual information in the event information, and social context information related to the event information, different types of information of the event information can be obtained, and then the different types of information of the event information can be used for subsequent false detection of the theme keyword. Since the multi-modal information includes rich information, the false detection accuracy of the theme keyword can be improved.

[0106] According to an embodiment of the present disclosure, in the process of generating the heterogeneous graph corresponding to the event information according to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information, the entities and relationships of the external knowledge information can provide additional entities and relationships for the event information, and enrich the entities and relationships of the event information.

[0107] According to an embodiment of the present disclosure, the heterogeneous graph corresponding to the event information is generated according to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information. The various types of entities and relationships related to the event information can be represented by the graph, the representation form of the various types of entities and relationships related to the event information is simplified, and the subsequent further processing of the event information is prepared.

[0108] According to an embodiment of the present disclosure, extracting, by using a preset extraction rule, an entity and a relation of the text information from the text information of the multi-modal information, extracting an entity and a relation of the visual information from the visual information, and extracting an entity and a relation of the social context information from the social context information include:

[0109] extracting, by using a BERT model, an entity of the text information from the text information and an entity of the social context information from the social context information;

[0110] extracting, by using a RESNET50 model, an entity of the visual information from the visual information;

[0111] extracting, by using a PCNN model, a relation of the text information between the entities of the text information from the text information according to the entities of the text information and the text information;

[0112] extracting, by using the PCNN model, a relation of the social context information between the entities of the social context information from the social context information according to the entities of the social context information and the social context information;

[0113] extracting, by using the PCNN model, a relation of the visual information between the entities of the visual information from the visual information according to the entities of the visual information and the visual information.

[0114] According to an embodiment of the present disclosure, for example, the entities of the text information and the sentences in the text information corresponding to the entities of the text information can be input into the PCNN model, and the PCNN model is used to extract the relations of the text information between the entities of the text information from the sentences in the text information corresponding to the entities of the text information. Similarly, the relations of the social context information and the relations of the visual information can be obtained.

[0115] According to an embodiment of the present disclosure, obtaining, according to external knowledge information, an entity and a relation of the external knowledge information related to the topic keyword includes:

[0116] constructing a plurality of knowledge graphs of different types according to the external knowledge information, wherein the external knowledge information is obtained from open source information, the external knowledge information includes information related to the topic keyword, and the plurality of knowledge graphs includes at least two of the following: a domain knowledge graph, a general knowledge graph, a matter knowledge graph, and a space-time knowledge graph;

[0117] obtaining the entity and the relation of the external knowledge information from the plurality of knowledge graphs.

[0118] According to embodiments of this disclosure, external knowledge information can be external data related to topic terms that can be obtained from different media platforms. This external data may include domain data, general data, event evolution data, and spatiotemporal data. Different types of knowledge graphs can be obtained based on the different types of data in the external data.

[0119] Currently, methods for detecting false information about events that combine external knowledge rely solely on a single knowledge graph related to that external information. However, this single knowledge graph provides limited knowledge, weak knowledge correlation, and insufficient contextual information.

[0120] According to embodiments of this disclosure, by constructing multiple knowledge graphs of different types based on external knowledge information, the various types of knowledge included in the multiple knowledge graphs can be used for subsequent false detection of topic entries. Since the multiple knowledge graphs contain rich knowledge information, the accuracy of false detection of topic entries can be improved.

[0121] According to embodiments of this disclosure, generating a heterogeneous graph corresponding to event information based on entities and relationships from external knowledge information, text information, visual information, and social context information includes:

[0122] Entity linking technology is used to align entities of external knowledge information, text information, visual information and social context information, resulting in entities of external knowledge information, text information, visual information and social context information with unified entity names.

[0123] By utilizing entities and relationships with unified external knowledge information, unified text information, unified visual information, and unified social context information, a heterogeneous graph corresponding to event information is generated.

[0124] According to embodiments of this disclosure, for example, an entity linking method based on knowledge base retrieval can be used to align entities of external knowledge information, text information, visual information, and social context information, thereby obtaining entities of external knowledge information, text information, visual information, and social context information with unified entity names.

[0125] According to embodiments of this disclosure, for example, Figure 2 Operation S203, as shown, extracts features from entities in a heterogeneous graph. Obtaining the entity features of entities in the heterogeneous graph can include the following operations:

[0126] Obtain entities from a heterogeneous graph;

[0127] The entity is processed by using the BERT model to obtain the entity features of the heterogeneous graph.

[0128] According to an embodiment of the present disclosure, for the operation S204 as shown in the figure, Figure 2 The operation of processing the heterogeneous graph and the entity features to obtain the multi-modal features of the event information and the event fake identification result can include the following operations:

[0129] The heterogeneous graph is processed to obtain a heterogeneous graph adjacency matrix.

[0130] The heterogeneous graph adjacency matrix and the entity features are input into a feature extraction module in a hierarchical attention network (HAN) to obtain multi-modal features; and the multi-modal features are sequentially input into a full connection layer and a classification layer of the HAN model to obtain the event fake identification result.

[0131] According to an embodiment of the present disclosure, the HAN model is a hierarchical attention network containing node-level attention and semantic-level attention. The HAN model learns the weight of the neighbor based on the meta path through the node-level attention, aggregates the node features of a specific semantic, and then weights all the meta paths through the semantic-level attention to obtain the optimal weighted combination of the node. The specific learning process of the HAN model is as follows.

[0132] For each type of node (type φ i ), a specific type of conversion matrix M φi is designed to map the features of nodes of different types to the same feature space, and the mapping relationship can be represented by formula (2). The type of the node is the type of the entity corresponding to the node. The type of the entity is determined by the extraction algorithm in the process of extracting the entity related to the event information and the external knowledge information.

[0133]

[0134] wherein h i represents the original feature of the node i, and h i ' represents the feature of the node i after the mapping of the original feature. The original feature of the node i can be a knowledge representation vector of the node i.

[0135] The weights of different types of nodes are learned by using a self-attention mechanism. In the case of a node pair (i, j) connected by a meta path φ, the node-level attention can be represented by formula (3).

[0136]

[0137] wherein att node The deep learning network for representing node-level attention is att node is shared under the given meta-path φ. The size of h j represents the mapped feature of the original feature of node j.

[0138] The node-level attention deep learning network can calculate the attention of all neighbors j of node i based on the meta-path φ, wherein The normalized weight coefficient is obtained by using a softmax function on The normalized weight coefficient can be represented by formula (4).

[0139]

[0140] wherein represents all neighbor nodes on the meta-path φ of node i, σ represents an activation function, and || represents a connection operation. represents the node-level attention vector of the meta-path φ.

[0141] The feature of node i based on the meta-path φ is equal to the weighted sum of neighbor features multiplied by the normalized weight coefficient corresponding to the feature of node i based on the meta-path φ The feature of node i based on the meta-path φ can be represented by formula (5).

[0142]

[0143] The semantic-level attention is repeated K times by using a multi-head attention mechanism, which can be represented as:

[0144]

[0145] Given a set of meta-paths {φ1, φ2,..., φ P}, P groups of node features can be obtained after node-level attention

[0146] The importance of different meta-paths is learned by using semantic-level attention, and the learned weight can be represented as:

[0147]

[0148] wherein att sem represents a deep learning network for semantic-level attention, P represents the number of node features, represents the weight of each meta-path.

[0149] The importance of each meta-path is learned, and the feature of each node in the meta-path is passed through a nonlinear transformation layer, and the weight importance of each meta-path φ i i The weight importance of each meta-path φ i

[0150]

[0151] wherein W represents a weight matrix, b represents a bias vector, q represents a semantic-level attention vector, and |V| represents the number of nodes of the meta-path φ i

[0152] After obtaining the weight coefficient, normalization is performed through a softmax function, and the normalized weight of the meta-path φ i The normalized weight of the meta-path φ i

[0153]

[0154] The obtained normalized weight coefficient of the meta-path is weighted and summed with the node-level feature to obtain the final feature Z, and the feature Z can be represented by formula (10).

[0155]

[0156] According to an embodiment of the present disclosure, the feature Z can be used as a multi-modal feature.

[0157] Based on the final feature Z, false detection is performed on the event information, that is, the final feature Z is input into a softmax layer of the HAN model, and a class R of the event information is output by the HAN model, that is, a false detection result of the event information. In the case of false event information, the class R of the event information is represented by 0, and in the case of true event information, the class R of the event information is represented by 1.

[0158] According to an embodiment of the present disclosure, for operation S206 as shown in Figure 2 Based on the plurality of multi-modal features and the plurality of event false detection results, generating a time feature sequence can include the following operations:

[0159] The multi-modal features and the event false detection results corresponding to the multi-modal features are feature spliced to generate a local perception vector of the event information.

[0160] According to the publishing time of each of the plurality of event information, the plurality of local perception vectors are sorted and combined to obtain a time feature sequence.

[0161] According to an embodiment of the present disclosure, for example, the theme keyword includes n event information, which can be represented by γ ia local perception vector representing an i-th event information in the plurality of event information, the local perception vectors corresponding to the plurality of event information are sorted and combined according to respective publishing time of the plurality of event information to obtain a time feature sequence γ n = {γ1, γ2,..., γ n}.

[0162] According to the embodiments of the present disclosure, the time feature sequence γ n = {γ1, γ2,..., γ n} can be input into a BI-LSTM-ATTENTION model to construct a joint model for bidirectional enhancement of global representation of topic keywords and local perception information, and output a false detection result of the topic keywords based on the model.

[0163] It should be noted that, unless it is explicitly stated that there is an execution sequence between different operations or there is an execution sequence in the technical implementation between different operations, the execution sequence between multiple operations can not be distinguished, and multiple operations can be executed simultaneously.

[0164] Figure 3 A block diagram of a multi-modal false information detection device according to an embodiment of the present disclosure is schematically shown.

[0165] As shown in Figure 3 , the multi-modal false information detection device 300 includes a plurality of event information acquisition module 310, a heterogeneous graph generation module 320, an entity feature obtaining module 330, an event false identification result obtaining module 340, a time feature sequence generation module 350, and a false identification result of topic keywords obtaining module 360.

[0166] The plurality of event information acquisition module 310 is configured to acquire a plurality of event information about a same topic keyword.

[0167] The heterogeneous graph generation module 320 is configured to generate a heterogeneous graph corresponding to the event information according to external knowledge information and the event information.

[0168] The entity feature obtaining module 330 is configured to extract features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph.

[0169] The event false identification result obtaining module 340 is configured to process the heterogeneous graph and the entity features to obtain a multi-modal feature of the event information and an event false identification result.

[0170] The time feature sequence generation module 350 is configured to generate a time feature sequence based on the plurality of multi-modal features and the plurality of event false identification results, where the time feature sequence includes a plurality of local perception vectors, the plurality of local perception vectors are sorted according to respective publishing times of the plurality of event information, and each local perception vector includes the multi-modal features and the event false identification result corresponding to the event information after splicing.

[0171] The event false identification result obtaining module 360 is configured to obtain the false identification result of the topic keyword according to the time feature sequence.

[0172] According to an embodiment of the present disclosure, the heterogeneous graph generation module includes a multi-modal information acquisition sub-module, an entity and relationship extraction sub-module related to multi-modal information, an entity and relationship obtaining sub-module of external knowledge information, and a heterogeneous graph generation sub-module.

[0173] The multi-modal information acquisition sub-module is configured to acquire multi-modal information of the event information, where the multi-modal information includes at least two of the following: text information in the event information, visual information in the event information, and social context information related to the event information.

[0174] The entity and relationship extraction sub-module related to multi-modal information is configured to extract, by using a preset extraction rule, entities and relationships of the text information from the text information of the multi-modal information, entities and relationships of the visual information from the visual information, and entities and relationships of the social context information from the social context information.

[0175] The entity and relationship obtaining sub-module of external knowledge information is configured to obtain, according to the external knowledge information, entities and relationships of the external knowledge information related to the topic keyword.

[0176] The heterogeneous graph generation sub-module is configured to generate, according to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information, a heterogeneous graph corresponding to the event information.

[0177] According to an embodiment of the present disclosure, the entity and relationship extraction sub-module related to multi-modal information includes a text and social context information entity extraction unit, a visual information entity extraction unit, a text information relationship extraction unit, a social context information relationship extraction unit, and a visual information relationship extraction unit.

[0178] The text and social context information entity extraction unit is configured to extract, by using a pre-trained language model, entities of the text information from the text information and entities of the social context information from the social context information.

[0179] The visual information entity extraction unit is configured to extract entities of visual information from the visual information by using a residual neural network model.

[0180] The text information relationship extraction unit is configured to extract relationships of the text information between the entities of the text information from the text information by using a segmental convolutional neural network model according to the entities of the text information and the text information.

[0181] The social context information relationship extraction unit is configured to extract relationships of the social context information between the entities of the social context information from the social context information by using a segmental convolutional neural network model according to the entities of the social context information and the social context information.

[0182] The visual information relationship extraction unit is configured to extract relationships of the visual information between the entities of the visual information from the visual information by using a segmental convolutional neural network model according to the entities of the visual information and the visual information.

[0183] According to an embodiment of the present disclosure, the entity and relationship obtaining submodule of the external knowledge information comprises a knowledge graph construction unit and an entity and relationship obtaining unit of external knowledge information.

[0184] The knowledge graph construction unit is configured to construct multiple knowledge graphs of different types according to external knowledge information, wherein the external knowledge information is obtained from open source information, the external knowledge information comprises information related to a theme keyword, and the multiple knowledge graphs comprise at least two of the following: a domain knowledge graph, a general knowledge graph, a matter knowledge graph, and a time-space knowledge graph.

[0185] The entity and relationship obtaining unit of external knowledge information is configured to obtain entities and relationships of external knowledge information from the multiple knowledge graphs.

[0186] According to an embodiment of the present disclosure, the heterogeneous graph generation submodule comprises an entity name unified entity obtaining unit and a heterogeneous graph generation unit.

[0187] The entity name unified entity obtaining unit is configured to perform entity alignment on the entities of external knowledge information, the entities of text information, the entities of visual information, and the entities of social context information by using entity linking technology, to obtain entities of external knowledge information, entities of text information, entities of visual information, and entities of social context information with unified entity names.

[0188] The heterogeneous graph generation unit is configured to generate a heterogeneous graph corresponding to event information by using the entities and relationships of external knowledge information with unified entity names, the entities and relationships of text information with unified entity names, the entities and relationships of visual information with unified entity names, and the entities and relationships of social context information with unified entity names.

[0189] According to an embodiment of the present disclosure, the entity feature obtaining module comprises an entity obtaining sub-module and an entity feature obtaining sub-module.

[0190] The entity obtaining sub-module is configured to obtain the entities of the heterogeneous graph.

[0191] The entity feature obtaining sub-module is configured to process the entities by using a pre-trained language model to obtain the entity features of the entities of the heterogeneous graph.

[0192] According to an embodiment of the present disclosure, the event fake judgment result obtaining module comprises a heterogeneous graph adjacency matrix obtaining sub-module, a multi-modal feature obtaining sub-module, and an event fake judgment result obtaining sub-module.

[0193] The heterogeneous graph adjacency matrix obtaining sub-module is configured to process the heterogeneous graph to obtain the heterogeneous graph adjacency matrix.

[0194] The multi-modal feature obtaining sub-module is configured to input the heterogeneous graph adjacency matrix and the entity features into a feature extraction module in a hierarchical attention model to obtain multi-modal features.

[0195] The event fake judgment result obtaining sub-module is configured to input the multi-modal features into a full connection layer and a classification layer of the hierarchical attention model in sequence to obtain the event fake judgment result.

[0196] According to an embodiment of the present disclosure, the time feature sequence generating module comprises a local perception vector generating sub-module and a time feature sequence obtaining sub-module.

[0197] The local perception vector generating sub-module is configured to concatenate the multi-modal features and the event fake judgment results corresponding to the multi-modal features to generate local perception vectors of the event information.

[0198] The time feature sequence obtaining sub-module is configured to sort and combine the local perception vectors according to the publishing time of each of the event information to obtain a time feature sequence.

[0199] Figure 4 A block diagram of an electronic device suitable for the methods described above according to an embodiment of the present disclosure is schematically shown. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0200] As Figure 4As shown, the electronic device 400 according to embodiments of the present disclosure includes a processor 401 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage section 408. The processor 401 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0201] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. The processor 401 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 402 and / or the RAM 403. Note that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0202] According to embodiments of the present disclosure, the electronic device 400 can further include an input / output (I / O) interface 405, which is also connected to the bus 404. The system 400 can further include one or more of the following components connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable medium 411 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read out therefrom is installed into the storage section 408 as necessary.

[0203] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product comprising a computer program carrying out the program codes for executing the method shown in the flow chart. In such embodiments, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the detachable medium 411. When the computer program is executed by the processor 401, the above-mentioned functions defined in the system / apparatus according to the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0204] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0205] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include but not limited to portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0206] For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 402 and / or the RAM 403 described above and / or one or more memories other than the ROM 402 and the RAM 403.

[0207] The embodiments of the present disclosure also include a computer program product comprising a computer program containing program codes for executing the method provided by the embodiments of the present disclosure, which program codes are used to make the electronic device implement the multi-modal false information detection method provided by the embodiments of the present disclosure when the computer program product is running on the electronic device.

[0208] When the computer program is executed by the processor 401, the above-mentioned functions defined in the system / apparatus according to the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0209] In one embodiment, the computer program can be embodied on a tangible memory device, such as a magnetic storage device, an optical storage device, etc. In another embodiment, the computer program can be transmitted in a signal over a network, distributed across networks, downloaded and installed, and / or installed from a removable memory media 411. The computer program comprising the program code can be transmitted using any suitable network medium, including, but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0210] According to an embodiment of the disclosure, program code for execution by a computer program can be written in any combination of one or more programming languages, and specifically, can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C", or similar programming languages. Program code can execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0212] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A multi-modal false information detection method, comprising: obtaining a plurality of event information about a same topic keyword; generating a heterogeneous graph corresponding to the event information according to external knowledge information and the event information; extracting features from entities of the heterogeneous graph to obtain entity features of the entities of the heterogeneous graph; processing the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event false judgment results; generating a time feature sequence based on a plurality of the multi-modal features and a plurality of the event false judgment results, wherein the time feature sequence comprises a plurality of local perception vectors, the plurality of local perception vectors are sorted according to publishing times of the plurality of event information, and each local perception vector comprises multi-modal features and event false judgment results corresponding to the event information after splicing; obtaining a false judgment result about the topic keyword according to the time feature sequence.

2. The method of claim 1, wherein, The generating of the heterogeneous graph corresponding to the event information according to the external knowledge information and the event information comprises: obtaining multi-modal information of the event information, the multi-modal information comprising at least two of the following: text information in the event information, visual information in the event information, and social context information related to the event information; extracting entities and relationships of the text information from the text information of the multi-modal information, extracting entities and relationships of the visual information from the visual information, and extracting entities and relationships of the social context information from the social context information by using a preset extraction rule; obtaining entities and relationships of the external knowledge information related to the topic keyword according to external knowledge information; generating the heterogeneous graph corresponding to the event information according to the entities and relationships of the external knowledge information, the entities and relationships of the text information, the entities and relationships of the visual information, and the entities and relationships of the social context information.

3. The method of claim 2, wherein, The extracting of the entities and relationships of the text information from the text information of the multi-modal information, the extracting of the entities and relationships of the visual information from the visual information, and the extracting of the entities and relationships of the social context information from the social context information by using the preset extraction rule comprises: extracting the entities of the text information from the text information and extracting the entities of the social context information from the social context information by using a pre-trained language model; extracting the entities of the visual information from the visual information by using a residual neural network model; extracting relationships of the text information between the entities of the text information from the text information by using a segment convolutional neural network model according to the entities of the text information and the text information; extracting relationships of the social context information between the entities of the social context information from the social context information by using a segment convolutional neural network model according to the entities of the social context information and the social context information; extracting relationships of the visual information between the entities of the visual information from the visual information by using a segment convolutional neural network model according to the entities of the visual information and the visual information.

4. The method of claim 2, wherein, The obtaining of the entity and the relationship of the external knowledge information related to the theme keyword according to the external knowledge information comprises: constructing multiple knowledge graphs of different types according to the external knowledge information, wherein the external knowledge information is obtained from open source information, the external knowledge information comprises information related to the theme keyword, and the multiple knowledge graphs comprise at least two of the following: a domain knowledge graph, a general knowledge graph, a matter knowledge graph, and a space-time knowledge graph; the entity and the relationship of the external knowledge information are obtained from the multiple knowledge graphs.

5. The method of claim 2, wherein, The generation of the heterogeneous graph corresponding to the event information according to the entity and the relationship of the external knowledge information, the entity and the relationship of the text information, the entity and the relationship of the visual information, and the entity and the relationship of the social context information comprises: performing entity alignment on the entity of the external knowledge information, the entity of the text information, the entity of the visual information, and the entity of the social context information by using an entity linking technology to obtain the entity of the external knowledge information, the entity of the text information, the entity of the visual information, and the entity of the social context information with unified entity names; the entity and the relationship of the external knowledge information with unified entity names, the entity and the relationship of the text information with unified entity names, the entity and the relationship of the visual information with unified entity names, and the entity and the relationship of the social context information with unified entity names are used to generate the heterogeneous graph corresponding to the event information.

6. The method of claim 1, wherein, The obtaining of the entity feature of the entity of the heterogeneous graph comprises: obtaining the entity of the heterogeneous graph; processing the entity by using a pre-trained language model to obtain the entity feature of the entity of the heterogeneous graph.

7. The method of claim 1, wherein, The processing of the heterogeneous graph and the entity feature to obtain the multi-modal feature and the event fake judgment result of the event information comprises: processing the heterogeneous graph to obtain a heterogeneous graph adjacency matrix; inputting the heterogeneous graph adjacency matrix and the entity feature into a feature extraction module in a hierarchical attention model to obtain the multi-modal feature; sequentially inputting the multi-modal feature into a full connection layer and a classification layer of the hierarchical attention model to obtain the event fake judgment result.

8. The method of claim 1, wherein, The generation of a time feature sequence based on multiple multi-modal features and multiple event fake judgment results comprises: performing feature splicing on the multi-modal feature and the event fake judgment result corresponding to the multi-modal feature to generate a local perception vector of the event information; sorting and combining multiple local perception vectors according to the publishing time of each of the multiple event information to obtain the time feature sequence.

9. A multi-modal fake information detection device, comprising: a plurality of event information obtaining modules configured to obtain a plurality of event information about a same theme keyword; a heterogeneous graph generating module configured to generate a heterogeneous graph corresponding to the event information according to external knowledge information and the event information; an entity feature obtaining module configured to extract a feature from an entity of the heterogeneous graph to obtain an entity feature of the entity of the heterogeneous graph. The event fake identification result obtaining module is configured to process the heterogeneous graph and the entity features to obtain multi-modal features of the event information and event fake identification results. The time feature sequence generation module is configured to generate time feature sequences based on the multi-modal features and the event fake identification results, wherein the time feature sequences include a plurality of local perception vectors, the local perception vectors are sorted according to the publishing times of the event information, and each local perception vector includes multi-modal features and event fake identification results corresponding to the event information after splicing. The fake identification result obtaining module for the topic keyword is configured to obtain fake identification results for the topic keyword based on the time feature sequences.

10. An electronic device, comprising: one or more processors; a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

Images