Rumor detection method and system based on multi-modal evidence fusion

Abstract

The application provides a rumor detection method and system based on multi-modal evidence fusion, and relates to the technical field of data processing. The method comprises the following steps: obtaining N initial evidences and M associated statements based on a to-be-detected statement; calculating the quality score of the initial evidence according to the information density of the text type evidence and the image type evidence contained in each initial evidence; calculating the similarity score according to the similarity of each initial evidence with the to-be-detected statement and the associated statement respectively, and calculating the weight corresponding to each initial evidence according to the similarity score and the quality score; calculating the target evidence according to the initial evidence and the corresponding weight; and obtaining the detection result according to the target evidence and the to-be-detected statement. In the application, the information density of the evidence is used as an index for evaluating the quality of the evidence, the weight of the evidence is adjusted in combination with the quality of the evidence, the similarity of the evidence with the to-be-detected statement and the associated statement, and the like, so that high-quality evidence is obtained and rumor detection is performed according to the high-quality evidence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a rumor detection method and system based on multi-modal evidence fusion. BACKGROUND

[0002] Verifying the authenticity of a statement using multi-modal evidence to enable timely detection of rumors has become a key direction to combat false information on social media. Although advanced technologies such as cross-modal fusion and attention mechanisms have improved the performance of statement detection, the effectiveness of existing methods is always limited by a core challenge: the inability to effectively handle the high heterogeneity of evidence in quality and relevance in the real world. This is because the evidence retrieved in the real world is often mixed with authoritative reports and irrelevant material pictures, and only a small part of the evidence is both relevant and credible, while current models treat all evidence equally, which severely restricts the accuracy of statement detection methods.

[0003] The limitations of existing statement detection methods are specifically reflected in three aspects. First, the cross-attention mechanism assigns similar initial weights to all evidence, resulting in low-quality evidence polluting the representation of the statement. Second, when establishing the directional relationship between the evidence, there is a lack of mechanism to suppress low-quality evidence, causing noise to spread in the network. Finally, when fusing multi-modal evidence, a fixed fusion rule is usually used, which cannot flexibly adapt to the dynamic changes in the quality distribution of evidence in different statement detection cases.

[0004] Therefore, it is urgent to develop a method that can dynamically evaluate the quality of evidence and detect the authenticity of a statement by fusing reliable evidence. SUMMARY

[0005] To solve the above technical problems, the present application provides a rumor detection method and system based on multi-modal evidence fusion, which obtains high-quality evidence by fusing cross-modal evidence, and detects rumors based on high-quality evidence for the statement to be tested, thereby improving the accuracy of the detection result. The technical solution is as follows:

[0006] In a first aspect, a rumor detection method based on multi-modal evidence fusion is provided, comprising:

[0007] Based on the statement to be tested, evidence retrieval is performed to obtain N initial evidences and M associated statements, N≥1, M≥1; wherein each initial evidence includes text type evidence and image type evidence;

[0008] For any one of the N initial evidences, the quality score of the initial evidence is calculated according to the information density of the text type evidence and the image type evidence contained in the initial evidence;

[0009] According to the similarity of the initial piece of evidence and the to-be-detected statement and M pieces of the related statements, a similarity score of the initial piece of evidence is calculated;

[0010] According to the quality score and the similarity score of the initial piece of evidence, a weight corresponding to the initial piece of evidence is calculated;

[0011] According to the initial piece of evidence and the weight corresponding to the initial piece of evidence, a target piece of evidence corresponding to the initial piece of evidence is calculated;

[0012] According to N pieces of target pieces of evidence corresponding to the initial pieces of evidence respectively and the to-be-detected statement, a detection result is obtained, the detection result being any one of a rumor and a non-rumor.

[0013] In a possible implementation, the quality score of the initial piece of evidence is calculated according to the information density of the text type evidence and the image type evidence contained in the initial piece of evidence, and the quality score of the initial piece of evidence includes:

[0014] Feature information in the text type evidence is extracted to obtain a first feature vector;

[0015] Feature information in the image type evidence is extracted to obtain a second feature vector;

[0016] The first feature vector and the second feature vector are interacted and then fused to obtain a fused feature vector;

[0017] Information entropy values of the text type evidence and the image type evidence are calculated, and the information entropy values are taken as quality scores of the fused feature vector;

[0018] The information entropy values are related to information densities of the text type evidence and the image type evidence.

[0019] In a possible implementation, the information entropy values are calculated by the following calculation formula:

[0020] ,

[0021] wherein, is an activation function, MLP gate is a function for calculating the fused feature vector z i The information entropy values are output, and the quality scores are represented.

[0022] In a possible implementation, when the initial piece of evidence contains only the text type evidence or the image type evidence, the method further includes:

[0023] The missing image type evidence in the initial piece of evidence is replaced by a specified mask, and the second feature vector obtained by extraction is a zero vector; or

[0024] The missing text type evidence in the initial piece of evidence is replaced by a specified mask, and the first feature vector obtained by extraction is a zero vector.

[0025] In a possible implementation, the to-be-detected statement at least includes text information and image information, and the similarity score of the initial piece of evidence is calculated according to the similarity between the initial piece of evidence and the to-be-detected statement and M pieces of the associated statement, and includes the following steps.

[0026] The initial weight of the initial piece of evidence is determined according to the quality score.

[0027] Features of the text information and the image information in the to-be-detected statement are extracted respectively, and a first spliced vector is obtained after splicing the extracted features.

[0028] Features of M pieces of the associated statement are extracted to obtain an associated feature vector corresponding to each piece of the associated statement.

[0029] Cosine similarities between the fusion feature vector and the first spliced vector and M pieces of the associated feature vector are calculated to obtain the similarity score of the initial piece of evidence.

[0030] In a possible implementation, the similarity score is calculated by the following calculation formula.

[0031] ,

[0032] wherein, denotes the cosine similarity between the fusion feature vector and the first spliced vector, denotes the cosine similarity between the fusion feature vector and M pieces of the associated feature vector, denotes the associated feature vector corresponding to the jth piece of the associated statement.

[0033] In a possible implementation, the detection result is obtained according to the target evidence corresponding to each of N pieces of the initial evidence and the to-be-detected statement, and includes the following steps.

[0034] A fusion evidence vector is obtained by fusing the target evidence corresponding to each of N pieces of the initial evidence.

[0035] Features of the text information and the image information are extracted respectively, and the extracted features are interacted with the target evidence corresponding to each of N pieces of the initial evidence respectively to obtain a first interaction vector corresponding to the text information and a second interaction vector corresponding to the image information.

[0036] The first interaction vector and the second interaction vector are interacted and spliced to obtain a second spliced vector;

[0037] A preset classification model is used to output a corresponding detection result according to the fusion feature vector and the second spliced vector.

[0038] In a possible implementation, the second spliced vector is calculated by the following calculation formula:

[0039]

[0040] wherein, the second spliced vector is represented by, the first interaction vector is represented by, the second interaction vector is represented by, The function is used to fuse the first interaction vector and the second interaction vector.

[0041] In a possible implementation, the method further includes:

[0042] establishing a target optimization function, the target optimization function including a first optimization function, a second optimization function and a third optimization function;

[0043] wherein, the first optimization function is used to optimize the detection result, the second optimization function is used to optimize the weight corresponding to each of the N initial evidences, and the third optimization function is used to optimize the similarity between the target evidence corresponding to each of the N initial evidences and the to-be-detected statement.

[0044] In a second aspect, a rumor detection system based on multi-modal evidence fusion is provided, including:

[0045] A data collection module is configured to perform evidence retrieval based on a to-be-detected statement to obtain N initial evidences and M related statements, N≥1, M≥1; wherein each initial evidence includes a text type evidence and an image type evidence;

[0046] A first calculation module is configured to calculate a quality score of any initial evidence in the N initial evidences according to the information density of the text type evidence and the image type evidence included in the initial evidence.

[0047] A second calculation module is configured to calculate a similarity score of the initial evidence according to the similarity between the initial evidence and the to-be-detected statement and M related statements.

[0048] ​​​a third calculation module configured to calculate a weight corresponding to the initial evidence according to the quality score and the similarity score of the initial evidence;

[0049] a fourth calculation module configured to calculate a target evidence corresponding to the initial evidence according to the initial evidence and the weight corresponding to the initial evidence;

[0050] a data generation module configured to obtain a detection result according to the target evidence corresponding to each of the N initial evidences and the to-be-detected statement, the detection result being any one of a rumor and a non-rumor.

[0051] The technical scheme provided by the embodiments of the present application can obtain the following technical effects:

[0052] (1) On the one hand, the present application fuses text type evidence and image type evidence in a cross-modal manner, and then performs quality evaluation on the fused initial evidence according to the information density of the text type evidence and the image type evidence, so as to realize automatic differentiation of high-quality initial evidence and noise and guarantee the quality of the obtained target evidence.

[0053] (2) On the other hand, the present application adjusts the weight of the initial evidence in combination with the quality of the initial evidence, the similarity of the initial evidence with the to-be-detected statement and M associated statements, and highlights the contribution of high-quality initial evidence to rumor detection, thereby providing data support for obtaining accurate detection results subsequently.

[0054] (3) In addition, the present application sets a target optimization function, the target optimization function includes multiple optimization functions for improving the accuracy of the search result from different dimensions, and through the mutual cooperation of the multiple optimization functions, the accuracy of the obtained detection result is further guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the drawings required to be used in the description of the embodiments of the present application will be briefly introduced. In the drawings:

[0056] Figure 1 is a flowchart of a rumor detection method based on multi-modal evidence fusion according to an embodiment of the present application;

[0057] Figure 2 is a whole flowchart of the method embodiment of the present application;

[0058] Figure 3 is an example diagram of obtaining a target evidence based on an initial evidence in the method embodiment of the present application;

[0059] Figure 4 is a structural block diagram of a rumor detection system based on multi-modal evidence fusion according to an embodiment of the present application;

[0060] Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is to be understood that the present application can be embodied in various forms without being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0062] It should be noted that the terms "first", "second", and the like in the description and claims of the present application and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that such terms are used interchangeably, where appropriate, to refer to the embodiments of the present application described herein, which can be implemented in sequences other than those illustrated or described herein. In addition, the term "comprising" and variations thereof are to be construed as meaning "including but not limited to", an open-ended term.

[0063] The present application provides a rumor detection method based on multi-modal evidence fusion, which uses the DERM framework to detect rumors of statements. The DERM framework is divided into four parts: origin, motivation, route and main content. The value of the DERM framework lies in that it provides a clear and easy-to-operate analysis structure, so that users can maintain critical thinking to make more rational judgments when faced with a large amount of information. The present embodiment is based on the DERM framework to realize the acquisition of high-quality evidence and the rumor detection of statements based on high-quality evidence.

[0064] As shown in Figure 1 and Figure 2 The rumor detection method based on multi-modal evidence fusion can include the following steps S101 to S107.

[0065] Step S101, obtaining a statement to be tested, the statement to be tested including at least text information and image information.

[0066] First, the statement to be tested on the web is obtained, which refers to an event or announcement to be verified whether it is a rumor. The statement to be tested includes text information and image information. The text information refers to the textual content contained in the statement to be tested, and the image information refers to the images contained in the statement to be tested. The statement to be tested can also include video information and voice information, etc. For example, a statement published through a video mainly includes video information, and a statement published through voice mainly includes voice information.

[0067] Step S102, based on the to-be-tested statement, evidence retrieval is performed to obtain N initial evidences and M associated statements, N≥1, M≥1; wherein each initial evidence contains text type evidence and image type evidence.

[0068] Based on the text information in the to-be-tested statement, text retrieval is performed in the Internet or a special database, and based on the image information in the to-be-tested statement, reverse image search is performed in the Internet or the special database to obtain N initial evidences. It should be noted that the special database refers to a database specially used for managing various types of evidence, which is obtained in advance.

[0069] Each initial evidence contains text type evidence and image type evidence. In the embodiment, when the retrieved initial evidence contains only text type evidence, a specified mask is used to replace the missing image type evidence in the initial evidence; similarly, when the retrieved initial evidence contains only image type evidence, a specified mask is used to replace the missing text type evidence in the initial evidence. It should be noted that the specified mask is set in advance, such as using "0" as the specified mask.

[0070] In addition, at least one associated statement related to the to-be-tested statement is retrieved through text retrieval and reverse image search in the Internet or the special database, or a subject word contained in the to-be-tested statement is extracted, other statements containing the same subject word are matched through subject word search, and the matched other statements are taken as associated statements. That is, the associated statement is similar to the to-be-tested statement, for example, the to-be-tested statement is "X virus does not have infectivity", and the associated statement can be "a special drug for X virus is on the market", "patients with X virus increase exponentially, and the coverage is spreading", and the like.

[0071] Step S103, for any initial evidence in the N initial evidences, the quality score of the initial evidence is calculated according to the information density of the text type evidence and the image type evidence contained in the initial evidence.

[0072] The quality of each initial evidence needs to be evaluated. In the embodiment, the quality evaluation process of each initial evidence is the same, so Figure 3 The quality of one of the N initial evidences is evaluated as an example.

[0073] First, the feature information in the text type evidence is extracted to obtain a first feature vector, and the feature information in the text type evidence refers to the information contained in the text type evidence, and in Figure 3 In the embodiment, the feature information in the image type evidence is extracted to obtain a second feature vector, and the feature information in the image type evidence refers to the information contained in the image type evidence, and in represents the first feature vector. At the same time, the feature information in the image type evidence is extracted to obtain a second feature vector, and the feature information in the image type evidence refers to the information contained in the image type evidence, Figure 3 in the middle represents the second feature vector.

[0074] Since the initial evidence described above may be missing the text type evidence or the image type evidence, when it is identified that the text type evidence has been replaced by the designated mask, the first feature vector is set to a zero vector; similarly, when it is identified that the image type evidence has been replaced by the designated mask, the second feature vector is set to a zero vector. In this way, the structure of the first feature vector and the second feature vector in each initial evidence can be guaranteed to be consistent, ensuring the unity of the feature space, so that subsequent initial evidences of only text type evidence, only image type evidence, and combination of text type evidence and image type evidence can be seamlessly processed.

[0075] Based on the obtained first feature vector and the second feature vector , the two vectors are interacted to maximize the use of cross-modal information, that is, to utilize the information of the two modalities of text and image, specifically:

[0076] Based on the first feature vector , the first feature vector and the second feature vector are interacted to obtain a third feature vector ;

[0077] Based on the second feature vector , the first feature vector and the second feature vector are interacted to obtain a fourth feature vector .

[0078] Then, the third feature vector and the fourth feature vector are fused to obtain a fused feature vector z i :

[0079] .

[0080] In this embodiment, the global feature representation CLS of the third feature vector and the fourth feature vector is fused, and CLS is a special mark used to aggregate the semantic information of the entire vector, such as the CLS of the third feature vector as a representative of the third feature vector , and the fourth feature vector CLS as the fourth feature vector The third feature vector and the fourth feature vector are compressed, so that the information amount of the third feature vector i and the fourth feature vector is reduced, and the complexity of the obtained fusion feature vector z

[0081] is 1-dimensional, while the information content of each feature is also retained.

[0081] Finally, the quality score of the fusion feature vector is calculated according to the information density of the text type evidence and the image type evidence contained in the initial evidence, and the quality score of the fusion feature vector is also taken as the quality score of the initial evidence. Specifically, the quality of the initial evidence is reflected in the information density of the text type evidence and the image type evidence contained therein. The text type evidence and the image type evidence of the high-quality initial evidence often have clear semantics and higher semantic consistency, and the information entropy values of the two are smaller at this time; on the contrary, the information entropy values of the two are larger. Therefore, the information entropy values of the text type evidence and the image type evidence are calculated first, and then the information entropy values are normalized, and the normalized values are taken as the quality score , and the calculation formula is:

[0082] ,

[0083] wherein, represents an activation function, and the MLP gate (·) function is used to output the information entropy value z i according to the fusion feature vector , wherein represents the normalized information entropy value, and represents the quality score.

[0084] Therefore, it can be seen that through the above calculation formula, not only the information entropy values of the text type evidence and the image type evidence can be obtained, but also the quality scores of the initial evidences can be obtained after the obtained information entropy values are mapped to a unified space, so that high-quality initial evidences can be screened out, and higher weight values can be allocated to the high-quality initial evidences.

[0085] In step S104, the similarity score of the initial evidence is calculated according to the similarity between the initial evidence and the to-be-measured statement and the M associated statements.

[0086] Similarly, the following still takes calculating the similarity score of one of the N initial evidences as an example.

[0087] First, the initial weight corresponding to each initial evidence is determined according to the quality score of the initial evidence, and the specific formula is: wherein, This represents the initial weight of the i-th piece of initial evidence, where w is a scaling parameter that is set in advance. This represents the quality score of the i-th initial piece of evidence.

[0088] Then, based on the similarity of the initial evidence with the statement to be tested and M related statements, a similarity score is calculated. Specifically, the initial evidence, the statement to be tested, and the related statements are first represented by vectors. The vector of the initial evidence can be represented by a fused feature vector. For the statement to be tested: features of the textual and image information in the statement to be tested need to be extracted separately, and then the two features are concatenated to obtain the first concatenated vector. The first concatenated vector is used as... Representation. For association statements: extract features from each of the M association statements to obtain the association feature vector corresponding to each association statement. Features in the association statements include information such as text and images contained in the association statement. The association feature vector is represented by... This is represented as follows. Next, the cosine similarity between the fused feature vector and the first concatenated vector and the M associated feature vectors is calculated to obtain the similarity score S. The calculation formula is as follows:

[0089] ,

[0090] in, This represents the cosine similarity between the fused feature vector and the first concatenated vector. This represents the cosine similarity between the fused feature vector and the M associated feature vectors. This represents the association feature vector corresponding to the j-th association declaration.

[0091] Step S105: Calculate the weight corresponding to the initial piece of evidence based on its quality score and similarity score.

[0092] The quality score of each piece of initial evidence is obtained through the above steps S103 and S104. After calculating the similarity score S, the weight corresponding to each piece of initial evidence is calculated. The calculation formula is:

[0093] ,in, , The function is used to calculate the similarity score S and the quality score. Output residual adjustment term Residual adjustment term Used to dynamically adjust the weights corresponding to the initial evidence. .

[0094] It should be noted that after calculating the weight corresponding to each piece of initial evidence, the weight needs to be normalized, such as by dividing the weight by the sum of all weights, to obtain the normalized weight.

[0095] Step S106, the target evidence corresponding to the initial evidence is calculated according to the initial evidence and the weight corresponding to the initial evidence.

[0096] For each initial evidence: the fusion feature vector of the initial evidence is multiplied by the weight corresponding to the initial evidence to obtain the target evidence , that is , wherein the target evidence represents the target evidence corresponding to the i-th initial evidence.

[0097] Step S107, the detection result is obtained according to the target evidence corresponding to each of the N initial evidences and the to-be-detected statement, and the detection result is any one of rumor and non-rumor.

[0098] Firstly, based on the target evidence corresponding to each of the N initial evidences, the N target evidences are fused to obtain a fusion evidence vector , and the fusion formula is: .

[0099] Then, the to-be-detected statement is represented by a vector, in order to improve the reliability of the rumor detection of the target evidence on the to-be-detected statement, the vector representation of the to-be-detected statement here needs to have context awareness, so for the text information and image information in the to-be-detected statement, not only the text information and image information need to be interacted across modalities, but also all the target evidences need to be previewed through the cross-modal attention mechanism before interaction, specifically:

[0100] The features of the text information and the image information are extracted respectively, and the extracted features are interacted with the N target evidences respectively to obtain a first interaction vector corresponding to the text information, a second interaction vector corresponding to the image information, and the calculation formulas are respectively:

[0101] , represents the feature extracted from the text information;

[0102] , represents the feature extracted from the text information.

[0103] Based on the first interaction vector and the second interaction vector , the two are interacted across modalities, and then spliced to obtain a second splicing vector , and the calculation formula is:

[0104] , wherein , The fusion function is used to fuse the first interaction vector The function is used to fuse the first interaction vector And the second interaction vector .

[0105] Finally, a preset classification model is called, and the classification model is obtained by pre-training and is composed of multiple classification perceptrons. The classification model is used to output a corresponding detection result According to the fusion feature vector and the second spliced vector The detection result Is one of rumors and non-rumors, specifically:

[0106] Among them, .

[0107] It should be noted that since the rumor or non-rumor detection result output by the classification model is output in a probability distribution manner, for example, the to-be-detected statement has a 30% probability of being a rumor and a 70% probability of being a non-rumor, which means that the to-be-detected statement is a non-rumor statement. However, when the probabilities of the to-be-detected statement belonging to rumors and non-rumors are each 50%, it is impossible to determine the classification to which the to-be-detected statement belongs. At this time, the detection result can be unverifiable, that is, the detection result can be one of rumors, non-rumors, and unverifiable.

[0108] In one possible implementation, the detection result of the to-be-detected statement in the past period of time can also be obtained, such as obtaining the detection result of the to-be-detected statement in the past month, and inputting the obtained detection result of the to-be-detected statement in the past period of time into the above-mentioned classification model. The classification model outputs a final detection result according to the fusion feature vector, the second spliced vector, and the detection result of the to-be-detected statement in the past period of time, so as to improve the accuracy of the obtained detection result.

[0109] In summary, the implementation principle of the rumor detection method based on multi-modal evidence fusion in the embodiment of the present application is as follows: first, after obtaining the to-be-detected statement, the initial evidence and the associated statement are obtained by performing evidence retrieval on the to-be-detected statement. The associated statement is a statement similar to the to-be-detected statement. Then, the quality score of the initial evidence is calculated according to the information density of the text type evidence and the image type evidence contained in the evidence. The similarity score is also calculated by the similarity of the initial evidence to the to-be-detected statement and the associated statement, and the weight corresponding to each initial evidence is calculated according to the similarity score and the quality score. The initial evidence adjusted by the weight is used as the target evidence to ensure the quality of the obtained target evidence. Finally, the rumor detection is performed on the to-be-detected statement based on the high-quality target evidence, and the classification to which the to-be-detected statement belongs is obtained, so as to achieve the purpose of rumor detection on the to-be-detected statement.

[0110] In order to guarantee the accuracy of the rumor detection result of the to-be-tested statement, the embodiment further establishes a target optimization function, the target optimization function includes a first optimization function, a second optimization function and a third optimization function,

[0111] The first optimization function adopts standard cross-entropy loss, mainly for the optimization of the detection result, and the first optimization function is:

[0112] , wherein, represents the actual detection result of the to-be-tested statement, represents the detection result of the to-be-tested statement this time, that is, the detection result obtained by the above method.

[0113] The second optimization function focuses on the initial evidence with high quality, the second optimization function introduces an entropy regularization term to promote the sparse distribution of the weight of each initial evidence, thereby optimizing the weight corresponding to each initial evidence, and the second optimization function is:

[0114] , wherein, is a value set in advance in order to make the weight of each initial evidence sparsely distributed.

[0115] The third optimization function focuses on whether the weight can reflect the semantic correlation (similarity) between the to-be-tested statement and the target evidence, the third optimization function adopts mutual information alignment loss, and the distance between the weight of the initial evidence and the target weight is minimized to make the weight reflect the semantic correlation between the to-be-tested statement and the target evidence, and the third optimization function is:

[0116] , wherein, .

[0117] Based on the first optimization function, the second optimization function and the third optimization function, the target optimization function L obtained is: , wherein, , are hyperparameters.

[0118] The embodiment is configured: when the output value of each optimization function included in the target optimization function is minimum and the output value of the target optimization function is minimum, the corresponding detection result is the optimal detection result. Therefore, based on the obtained detection result, the target optimization function is used to continuously optimize the accuracy of the obtained detection result.

[0119] It should be noted that the size of the serial number of each step in the above embodiments does not mean the order of execution, the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In actual application, all possible implementation manners described above can be combined in any combination to form possible embodiments of the present application, which will not be described one by one here.

[0120] Based on the rumor detection method based on multi-modal evidence fusion provided in each of the above embodiments, based on the same inventive concept, the embodiments of the present application also provide a rumor detection system based on multi-modal evidence fusion.

[0121] Figure 4 is a structural diagram of a rumor detection system based on multi-modal evidence fusion provided by the embodiments of the present application. As Figure 4 shown, the system can specifically include a data collection module, a first calculation module, a second calculation module, a third calculation module, a fourth calculation module and a data generation module.

[0122] The data collection module is configured to perform evidence retrieval based on the to-be-detected statement to obtain N initial evidences and M associated statements, N > 1, M > 1; wherein each initial evidence includes a text type evidence and an image type evidence.

[0123] The first calculation module is configured to calculate a quality score of any one of the N initial evidences according to the information density of the text type evidence and the image type evidence included in the initial evidence.

[0124] The second calculation module is configured to calculate a similarity score of the initial evidence according to the similarity of the initial evidence with the to-be-detected statement and the M associated statements, respectively.

[0125] The third calculation module is configured to calculate a weight corresponding to the initial evidence according to the quality score and the similarity score of the initial evidence.

[0126] The fourth calculation module is configured to calculate a target evidence corresponding to the initial evidence according to the initial evidence and the weight corresponding to the initial evidence.

[0127] The data generation module is configured to obtain a detection result according to the target evidence corresponding to each of the N initial evidences and the to-be-detected statement, wherein the detection result is any one of a rumor and a non-rumor.

[0128] In a possible implementation manner, the system described above further includes a data acquisition module, which is mainly configured to acquire a to-be-detected statement, wherein the to-be-detected statement at least includes text information and image information.

[0129] In a possible implementation, the data generation module is further configured to invoke a preset classification model, and output a final detection result according to the fusion feature vector, the second splicing vector and a detection result of the to-be-detected statement in a past period of time by using the classification model.

[0130] The rumor detection system based on multi-modal evidence fusion provided in this embodiment is used to execute the rumor detection method based on multi-modal evidence fusion provided in the above embodiment, and the implementation manners and principles are the same. For details of the implementation manners of the modules, refer to the related descriptions of the method embodiments, which will not be described here.

[0131] Based on the same inventive concept, the embodiments of the present application further provide an electronic device including a processor and a memory, the memory storing a computer program, and the processor being configured to run the computer program to execute the rumor detection method based on multi-modal evidence fusion of any one of the above embodiments.

[0132] In an exemplary embodiment, an electronic device is provided, such as Figure 5 As shown in the figure, Figure 5 The electronic device 400 shown in the figure includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the electronic device 400 can also include a transceiver 404. It should be noted that in actual application, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0133] The processor 401 can be a CPU (Central Processing Unit, central processor), a DSP (Digital Signal Processor, digital signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor 401 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0134] The bus 402 can include a path that transmits information between the above-mentioned components. The bus 402 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 402 can be divided into an address bus, a data bus, a control bus, and the like. For convenience of representation, Figure 5 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0135] The memory 403 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this.

[0136] The memory 403 is used to store computer program code for implementing the scheme of the present application, and is controlled by the processor 401 to execute. The processor 401 is used to execute the computer program code stored in the memory 403 to realize the content shown in the foregoing method embodiments.

[0137] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (such as a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 5 The electronic device shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0138] Based on the same inventive concept, the embodiments of the present application also provide a storage medium in which a computer program is stored, wherein the computer program is set to execute the rumor detection method based on multi-modal evidence fusion of any one of the above-mentioned embodiments when running.

[0139] Those skilled in the art can clearly understand the specific working process of the system, device and module described above, and can refer to the corresponding process in the foregoing method embodiments for brevity.

[0140] Those skilled in the art can understand that the technical solutions of the present application can be embodied in the form of a software product in essence or in whole or part of the technical solutions, and the computer software product is stored in a storage medium and includes program instructions to make an electronic device (such as a personal computer, a server, or a network device) execute all or part of the steps of the method described in the embodiments of the present application when the program instructions are run. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0141] Alternatively, all or part of the steps of the foregoing method embodiments can be completed by program instruction related hardware (such as an electronic device of a personal computer, a server, or a network device), and the program instructions can be stored in a computer readable storage medium, and when the program instructions are executed by a processor of the electronic device, the electronic device executes all or part of the steps of the method described in the embodiments of the present application.

[0142] The foregoing embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principle of the present application, the technical solutions recorded in the foregoing embodiments can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the protection scope of the present application.

Claims

1. A rumor detection method based on multimodal evidence fusion, characterized in that, include: Evidence retrieval is performed based on the statement to be tested, resulting in N initial pieces of evidence and M related statements, where N≥1 and M≥1; each initial piece of evidence contains both text-type and image-type evidence. For any one of the N initial pieces of evidence, the quality score of that initial piece of evidence is calculated based on the information density of the text-type evidence and the image-type evidence contained in that initial piece of evidence. Based on the similarity of this initial piece of evidence with the statement to be tested and the M related statements, the similarity score of this initial piece of evidence is calculated. Based on the quality score and similarity score of the initial piece of evidence, the weight corresponding to the initial piece of evidence is calculated; wherein the formula for calculating the weight is: , Denotes the initial weight of the i-th initial piece of evidence and w is the scaling parameter, which is set in advance. This represents the quality score of the i-th initial piece of evidence. , The function is used to calculate the similarity score S and the quality score. Output residual adjustment term Residual adjustment term Used to dynamically adjust the weights corresponding to the initial evidence. ; The target evidence corresponding to the initial evidence is calculated based on the initial evidence and the weight corresponding to the initial evidence. The detection result is obtained based on the target evidence corresponding to each of the N initial pieces of evidence and the statement to be tested, and the detection result is either a rumor or a non-rumor.

2. The method according to claim 1, characterized in that, The process of calculating the quality score of the initial evidence based on the information density of the text-type evidence and the image-type evidence contained therein includes: The first feature vector is obtained by extracting the feature information from the text-type evidence. Extract the feature information from the image type evidence to obtain the second feature vector; The first feature vector and the second feature vector are interacted and then fused to obtain a fused feature vector; Calculate the information entropy values ​​of the text-type evidence and the image-type evidence, and use the information entropy values ​​as the quality score of the fused feature vector; The information entropy value is related to the information density of the text-type evidence and the image-type evidence.

3. The method according to claim 2, characterized in that, The information entropy value is calculated using the following formula: ， in, Indicates the activation function, MLP gate The (·) function is used to determine the fused feature vector z i Output the information entropy value. This indicates the quality score.

4. The method according to claim 2, characterized in that, When the initial evidence contains only the text-type evidence or the image-type evidence, the method further includes: The missing image type evidence in the initial evidence is replaced with a specified mask, and the extracted second feature vector is a zero vector; or The missing text-type evidence in the initial evidence is replaced with a specified mask, and the extracted first feature vector is a zero vector.

5. The method according to claim 2, characterized in that, The statement to be tested includes at least text information and image information. The similarity score of the initial evidence is calculated based on its similarity to the statement to be tested and the M related statements, including: The initial weight of this initial piece of evidence is determined based on the quality score. Features of the text information and the image information in the statement to be tested are extracted respectively, and the extracted features are concatenated to obtain the first concatenated vector; Extract the features of the M association statements to obtain the association feature vector corresponding to each association statement; Calculate the cosine similarity between the fused feature vector and the first concatenated vector and the M associated feature vectors to obtain the similarity score of the initial evidence.

6. The method according to claim 5, characterized in that, The similarity score is calculated using the following formula: ， in, The cosine similarity between the fused feature vector and the first concatenated vector is represented. The cosine similarity between the fused feature vector and the M associated feature vectors is represented. This represents the association feature vector corresponding to the j-th association declaration.

7. The method according to claim 5, characterized in that, The process of obtaining the detection result based on the target evidence corresponding to each of the N initial pieces of evidence and the statement to be tested includes: The target evidence corresponding to each of the N initial pieces of evidence is fused to obtain a fused evidence vector; Features are extracted from the text information and the image information respectively. The extracted features are then interacted with the target evidence corresponding to each of the N initial pieces of evidence to obtain a first interaction vector corresponding to the text information and a second interaction vector corresponding to the image information. The first interaction vector and the second interaction vector are interacted and then concatenated to obtain the second concatenated vector; The preset classification model is used to output the corresponding detection results based on the fused feature vector and the second concatenated vector.

8. The method according to claim 7, characterized in that, The second concatenated vector is calculated using the following formula: ， in, This represents the second concatenated vector. , This represents the first interaction vector. , This represents the second interaction vector. The function is used to merge the first interaction vector and the second interaction vector.

9. The method according to claim 1, characterized in that, The method further includes: Establish an objective optimization function, which includes a first optimization function, a second optimization function, and a third optimization function; Wherein, the first optimization function is used to optimize the detection result, the second optimization function is used to optimize the weights corresponding to each of the N initial pieces of evidence, and the third optimization function is used to optimize the similarity between the target evidence and the statement to be tested corresponding to each of the N initial pieces of evidence.

10. A rumor detection system based on multimodal evidence fusion, characterized in that, include: The data collection module is used to retrieve evidence based on the statement to be tested, and obtain N initial pieces of evidence and M related statements, where N≥1 and M≥1; each initial piece of evidence contains text-type evidence and image-type evidence. The first calculation module is used to calculate the quality score of any one of the N initial pieces of evidence based on the information density of the text-type evidence and the image-type evidence contained in that initial piece of evidence. The second calculation module is used to calculate the similarity score of the initial evidence based on the similarity between the initial evidence and the statement to be tested and the M related statements respectively. The third calculation module is used to calculate the weight corresponding to the initial piece of evidence based on the quality score and the similarity score; wherein the weight is calculated using the following formula: , Denotes the initial weight of the i-th initial piece of evidence and w is the scaling parameter, which is set in advance. This represents the quality score of the i-th initial piece of evidence. , The function is used to calculate the similarity score S and the quality score. Output residual adjustment term Residual adjustment term Used to dynamically adjust the weights corresponding to the initial evidence. ; The fourth calculation module is used to calculate the target evidence corresponding to the initial evidence based on the initial evidence and the weight corresponding to the initial evidence. The data generation module is used to obtain detection results based on the target evidence corresponding to each of the N initial pieces of evidence and the statement to be tested, wherein the detection results are either rumors or non-rumors.

Images