Short video rumor detection method and system for complex incomplete data scene

Abstract

The invention discloses a short video rumor detection method and system oriented to a complex incomplete data scene, and relates to the technical field of data mining, a shared feature extraction module based on multi-modal consistency is constructed, a model adaptively skips a missing data part, information of video content is summarized from an available data part, and a short video rumor detection result is obtained. The model is helped to understand the video theme; a conflict detection module based on cross-modal inconsistency is constructed, semantic representation of all available modalities is analyzed, and the semantic representation is used for checking possible contradictions and irregularities between the modalities and serves as an auxiliary basis for identifying rumors; by constructing a transferable self-adaptive balance training mechanism, the training weight of each video mode can be automatically adjusted according to training loss, over-fitting of available modes is prevented, and under-fitting of missing modes is relieved, so that the robustness of the model for input of different mode combinations is enhanced, and the purpose of balance training is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data mining technology, and specifically to a short video rumor detection method and system for complex and incomplete data scenarios. Background Art

[0002] The rapid development of short video platforms has completely changed the way people obtain and share information. However, short videos have also accelerated the spread of fake news and misinformation, which mislead viewers through inaccurate and forged multimedia content. Compared with fake news in text form, videos have stronger visual impact and narrative ability, spread faster, and cause greater harm to society. Despite the huge challenges posed by forged multimedia content, the detection of fake news videos is also susceptible to the problem of missing modality. In practical applications, many factors may lead to missing modality on social media platforms, such as user privacy, technical limitations, and editorial preferences. Requiring modal completeness of videos in the training and inference stages is not only costly but even unrealistic. Therefore, the problem of incomplete modality significantly increases the difficulty of detecting fake news short videos.

[0003] Currently, video-based fake news detection is showing a growing trend. Existing methods usually focus on designing novel multimodal fusion structures to integrate more modal information and improve detection performance. For example, a study used a joint attention mechanism to guide the fusion of text and voice information in short videos. Another study further extracted the voice emotion and facial emotion of the speaker in the video, and then used a fully connected network to fuse the emotion representation and content representation to form a multimodal representation. Similarly, some studies use a cross-modal Transformer architecture to integrate information from different modalities.

[0004] Although existing fake news video detection methods have made some progress, these methods are mainly designed for short videos with complete picture, audio, and text modalities. These methods are usually unable to handle the problem of missing modalities in complex incomplete data scenarios. The main reason is that they cannot skip missing data and extract key information from available modalities, resulting in ineffective cross-modal feature fusion. At the same time, these methods usually rely too much on specific modalities during training, resulting in unbalanced training between modalities. These problems hinder the deployment of short video rumor detection models in practical application scenarios. Summary of the invention

[0005] In order to address the deficiencies mentioned in the above-mentioned background technology, the purpose of the present invention is to provide a short video rumor detection method and system for complex and incomplete data scenarios, which can skip the missing parts in the data set and decouple the consistency features and inconsistency semantics from the available modal data as the basis for judging rumors; at the same time, the model includes an adaptive modality balance training mechanism, which effectively balances the rumor identification capabilities of each modality during the training process.

[0006] In the first aspect, the purpose of the present invention can be achieved by the following technical solution: a short video rumor detection method for complex incomplete data scenarios, the method comprising the following steps:

[0007] Obtaining original short video data, inputting the original short video data into a pre-established multimodal encoder to obtain intra-video shared features; wherein the multimodality of the multimodal encoder includes audio modality, text modality and picture modality; performing a comparison on the original short video data using a consistency learning constraint loss to obtain inter-video shared features, inputting the intra-video shared features and inter-video shared features into a feature encoder for encoding, and obtaining consistent shared features;

[0008] Based on the consistent shared features, an orthogonal constraint loss is used to obtain a unique mapping matrix between modalities. Matrix multiplication is performed based on the unique mapping matrix between modalities and the shared features in the video to obtain a unique representation of each modality. The unique representation of each modality is used to learn cross-modal semantic conflicts through a pre-established Transformer architecture to obtain a conflict semantic representation. The conflict semantic representation and the shared features in the video are input into the feature encoder for fusion to obtain a fused representation.

[0009] Based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss, the total loss is calculated based on the adaptive modal balance training mechanism. The pre-established short video rumor detection model is trained based on the total loss to obtain the trained short video rumor detection model. The target short video is received and input into the trained short video rumor detection model to output the short video rumor detection result.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the original short video data into a pre-established multimodal encoder to obtain the shared features in the video:

[0011] The modal features are obtained by encoding the original features through a modality-specific encoder, including: encoding the audio part of the video using the VGGish encoder, encoding the text part of the video using the Bert model, and encoding the picture part of the video using VGG19. Then, based on the encoded modal features, the modal shared information is extracted through a video-level shared information encoder. The encoder consists of a projection layer and a self-attention mechanism, and the expression is as follows:

[0012]

[0013] where {v, t, a} represent the image mode, text mode and audio mode respectively, E v ,E a ,E t is the modal representation of each mode, is the extracted shared representation of each modality, SelfAttn is a single-layer self-attention network, and W s is the projection matrix of the shared subspace, H s It is the learned shared representation of the intra-modal video.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of performing a comparison on the original short video data to obtain shared features between videos by using a consistency learning constraint loss:

[0015] The short videos marked with the same event are regarded as positive samples, and the short videos marked with different events are regarded as negative samples. The contrast loss function is:

[0016]

[0017] Where i represents the target short video, P(i) represents the positive sample set of the target short video i, A(i) represents the batch set of the target short video i, τ is the contrastive learning temperature parameter, and through the consistency contrastive learning constraint, context-aware shared features between videos are obtained.

[0018] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of obtaining unique features between modalities using orthogonal constraint loss based on consistent shared features:

[0019] Based on the obtained shared subspace projection matrix W s , using orthogonal constraint loss to obtain the unique features between modalities, the expression is:

[0020]

[0021] Among them, W s represents the shared feature map matrix, Represents the screen modality-specific feature mapping matrix, represents the text modality-specific feature mapping matrix, represents the audio modality-specific feature mapping matrix, and F represents the normalization function.

[0022] In combination with the first aspect, in some implementations of the first aspect, the method further includes: based on the obtained video modality-specific feature mapping matrix Text modality-specific feature mapping matrix Audio modality-specific feature map matrix And the obtained modal characteristics E s ,E a ,E t , the unique representation of each mode is obtained through matrix multiplication, and the formula is as follows:

[0023]

[0024] Among them, *∈{v,a,t} represents each mode, Represents the unique characteristics of each mode obtained.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: performing feature denoising on the unique features of each modality, and performing a filtering operation using a fully connected gating mechanism, the expression is:

[0026]

[0027] Among them, *∈{v,a,t} represents each mode, represents the modal-specific features after filtering operations, b g Represents a trainable offset constant, ⊙ represents a Hadamard multiplication operation, and based on the obtained modality-specific features, the cross-modal semantic conflict I is learned through the Transformer architecture, which is expressed as:

[0028]

[0029] in, and They represent unique representations of the modalities respectively.

[0030] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of inputting the semantic feature representation and the shared feature in the video into a feature encoder for fusion to obtain a fused representation:

[0031]

[0032] Among them, MLP stands for fully connected layer, Represents the model prediction results.

[0033] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the total loss is calculated based on an adaptive balanced training mechanism, wherein the adaptive balanced training mechanism includes: a data completeness prompt template, a global prompt virtual memory, and a loss function regulator;

[0034] The data completeness prompt template expression is as follows:

[0035]

[0036] Among them, Q, K, V represent the query, key and value matrices in the attention calculation;

[0037] The loss function regulator expression is as follows:

[0038]

[0039] in represents the original training batch loss, [R] represents the regularization matrix, [C] represents the missing prompt matrix, ⊙ represents the Hadamard product, and u represents the predefined weight;

[0040] Generation-based contrastive learning loss Orthogonality Constraint Loss and the regularization loss term Construct the training target of the model and get the total loss The expression is as follows:

[0041]

[0042] Among them, α and β are the weights of contrastive learning loss and orthogonality constraint loss, respectively.

[0043] In the second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a short video rumor detection system for complex and incomplete data scenarios, which is characterized by comprising:

[0044] The feature extraction module is used to obtain the original short video data, input the original short video data into a pre-established multimodal encoder, and obtain the intra-video shared features; wherein the multimodal encoder includes audio modality, text modality and picture modality; compare the original short video data with the consistency learning constraint loss to obtain the inter-video shared features, input the intra-video shared features and the inter-video shared features into the feature encoder for encoding, and obtain the consistent shared features;

[0045] A feature fusion module is used to obtain unique features between modalities using orthogonal constraint loss based on consistent shared features, perform matrix multiplication calculations based on the unique features between modalities and the shared features within the video to obtain unique representations of each modality, learn cross-modal semantic conflicts for the unique representations of each modality through a pre-established Transformer architecture to obtain semantic feature representations, and input the semantic feature representations and the shared features within the video into a feature encoder for fusion to obtain fused representations;

[0046] The rumor detection module is used to calculate the total loss based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss based on the adaptive modal balance training mechanism, train the pre-established short video rumor detection model based on the total loss, obtain the trained short video rumor detection model, receive the target short video, input the target short video into the trained short video rumor detection model, and output the short video rumor detection result.

[0047] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the short video rumor detection method for complex and incomplete data scenarios as described above is adopted.

[0048] Beneficial effects of the present invention:

[0049] Aiming at complex and incomplete short video data scenarios, the present invention proposes a method for jointly learning consistent features and inconsistent semantics, which effectively improves the detection accuracy of short video rumors in scenarios where some data is missing; the present invention proposes a modal balance training mechanism, which balances the rumor recognition capabilities of each modality during the training process, greatly improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a schematic flow chart of the method of the present invention;

[0052] Figure 2 It is a schematic diagram of the structure of a consistency feature learning module according to an embodiment of the present invention;

[0053] Figure 3 2 is a schematic diagram of the structure of an inconsistent semantic comparison learning module according to an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of an adaptive training balancing mechanism according to an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of the system structure of the present invention.

[0056] Figure 6 It is a flow chart of the data preprocessing method of the present invention;

[0057] Figure 7 This is a case diagram of the rumor short video identification of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Embodiment 1:

[0060] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0061] Multimodality: Multimodality refers to the collaborative reasoning of multiple heterogeneous modal data. The external demand for multimodal data analysis and the internal demand for advanced cognitive intelligence promote each other.

[0062] like Figure 1 As shown, a short video rumor detection method for complex incomplete data scenarios includes the following steps:

[0063] S101: Obtain original short video data, input the original short video data into a pre-established multimodal encoder, and obtain intra-video shared features; wherein the multimodal encoder includes audio modality, text modality, and picture modality; compare the original short video data with a consistency learning constraint loss to obtain inter-video shared features, input the intra-video shared features and inter-video shared features into a feature encoder for encoding, and obtain consistent shared features;

[0064] Specifically, the present invention is further described below by way of embodiments:

[0065] Specifically, in the example of the present invention, the original short video data uses the public FVI dataset, which includes 1,827 short videos containing real content and 1,827 short videos containing rumors, which are classified into 738 related social events. In order to simulate the missing situation in real situations, the dataset also provides different modal data missing rates, and the expression of the missing rate is as follows:

[0066]

[0067] Among them, m i Represents the number of modalities of the i-th sample, L represents the total number of samples, and M represents the total number of modalities. Therefore, in the example of the present invention, that is, when M=3, the missing rate η is selected from [0, 0.1, 0.3, 0.5, 0.7], where [0] corresponds to the complete scene, [0.3] is approximately the scene where each short video is missing one modality, and [0.7] is approximately the scene where each short video is missing two modalities. In the training, validation, and test data sets, the example of the present invention maintains the same data missing rate. Table 1 shows the proportion of different modality missing combinations in the data set.

[0068] Table 1: The proportion of missing modal combinations in the dataset

[0069]

[0070] Step S11: Perform video content consistent semantic learning from the video level. Based on the modal features of each short video, first extract the modal shared information through a video-level shared information encoder. The encoder consists of a projection layer and a self-attention mechanism, and the expression is as follows:

[0071]

[0072] where {v, t, a} represent the image mode, text mode and audio mode respectively, E v ,E a ,E t is the modal representation of each mode, is the extracted shared representation of each modality, SelfAttn is a single-layer self-attention network, and W s is the projection matrix of the shared subspace, H s is the learned shared representation of the intra-modal video.

[0073] Step S12: A comparative consistency learning method at the event level is constructed. In this method, short videos marked as the same event are regarded as positive samples, and short videos marked as different events are regarded as negative samples. This method performs feature enhancement based on an event level higher than a single video level, performs feature coupling on short videos belonging to the same social event, and performs feature orthogonality on short videos belonging to different events. Its comparative loss function is:

[0074]

[0075] Where i represents the sample anchor point, P(i) represents the positive sample set of anchor point i, A(i) represents the batch set where anchor point i is located, and τ is the contrastive learning temperature parameter. Through the consistent contrastive learning constraint, context-aware shared features between videos are obtained.

[0076] In some embodiments, Figure 6 As shown, the encoder described in step S11 can process 7 short video input modality combinations, including "picture + text + audio", "picture + text", "picture + audio", "audio + text", "picture", "text", and "audio". Among them, the picture modality mainly includes the key frames and motion frames of the video, the audio modality mainly includes the video background music and narration, and the text modality mainly includes the title and comments of the video. The encoder described in this step needs to ensure that at least one modality is available. In addition, the encoder shares the subspace projection matrix W s The dimension is less than 1024, and the number of attention heads of the self-attention mechanism is greater than 4 to ensure sufficient feature space.

[0077] In some embodiments, the event-level contrastive learning module described in step S12 is effective in a single training batch, that is, the sampling range of positive and negative samples in each contrastive learning only involves the training batch of the current training cycle, and does not involve other training cycles, thereby avoiding data leakage problems. The contrastive learning distillation temperature in this step is greater than 0 and less than 0.2, and the weight of the contrastive learning loss function is greater than 0.2 and less than 0.5.

[0078] S102: using orthogonal constraint loss based on consistent shared features to obtain inter-modal unique features, performing matrix multiplication calculation based on inter-modal unique features and intra-video shared features to obtain unique representations of each modality, learning cross-modal semantic conflicts for the unique representations of each modality through a pre-established Transformer architecture to obtain semantic feature representations, and inputting the semantic feature representations and intra-video shared features into a feature encoder for fusion to obtain fused representations;

[0079] Based on the consistent shared features obtained in S1, the orthogonal constraint loss is used to obtain the unique features between modalities, which is used to obtain the inconsistent semantic conflicts across modalities. The expression is:

[0080]

[0081] Among them, W s represents the shared feature map matrix, represents the video-specific feature mapping matrix, represents the text-specific feature mapping matrix, Represents the audio-specific feature mapping matrix, and F represents the normalization function. This orthogonal constraint loss ensures the orthogonality between modal sharing and unique features, and enhances the feature representation ability of the model.

[0082] Step S22: Based on the video modality-specific feature mapping matrix obtained in step S21 Text modality-specific feature mapping matrix Audio modality-specific feature map matrix and the modal feature E obtained in step S11 s ,E a ,E t , the unique representation of each mode is obtained through matrix multiplication, and the formula is as follows:

[0083]

[0084] Among them, *∈{v,a,t} represents each mode, represents the unique characteristics of each mode obtained,

[0085] Step S23: The modal-specific features obtained in step S22 To perform feature denoising, a fully connected gating mechanism is used for filtering, and its expression is:

[0086]

[0087] Among them, *∈{v,a,t} represents each mode, represents the modal-specific features after filtering operations, b g Represents a trainable offset constant, and ⊙ represents a Hadamard multiplication operation. Based on the above modality-specific features, the cross-modal semantic conflict I is learned through the Transformer architecture, and its expression is: I ij =TransEncoder(E u,i ||E u,j )

[0088] Among them, E u,i With E u,jThey represent the unique representations of the modalities. The semantic conflicts learned based on the unique representations of the modalities reflect the degree of feature dispersion across modalities, which is helpful for judging rumor short videos.

[0089] Step S24: Based on the shared features and unique features generated in step S21 and step S22, a feature encoder is used to obtain a fusion representation, which is expressed as follows:

[0090] H=I va ||I ta ||I vt ||H s

[0091]

[0092] Among them, MLP stands for fully connected layer, Represents the model prediction results.

[0093] In some embodiments, step S21 includes:

[0094] The adaptive weight matrix W is introduced in the orthogonal constraint loss module. s , an adaptive weight matrix W is proposed s The design and application method of this method is to introduce an adaptive weight matrix W in the training process. s , which aims to dynamically adjust the effectiveness of multimodal features. The adaptive weight matrix W s The weight matrix W is multiplied with the features of each modality to adaptively adjust the contribution of each modality feature. This design solves the problem of insufficient adaptability of the traditional fixed weight matrix in different training stages. s In the process of back propagation, its gradient is updateable, so that it can be adjusted in real time according to the changes in data distribution during model training. In multimodal fusion tasks, the contribution of each modal feature to the final output of the model varies in different situations. s Through adaptive optimization, the model can gradually enhance the weight allocation of effective features, weaken the impact of invalid or noise features, and ultimately improve the accuracy and robustness of feature fusion.

[0095] In some embodiments, step S21 includes:

[0096] Based on the unique feature mapping matrix of each modality, an orthogonal constraint loss is calculated for each modality to ensure that the weight vectors in the model are orthogonal. When learning inconsistent semantics, the orthogonality between weights can help reduce redundant information and improve the generalization ability of the model.

[0097] In some embodiments, step S22 includes:

[0098] In the Transformer encoder architecture used, a fully connected gating mechanism is used to filter the features of each modality, and its activation function is the Sigmoid function, which is expressed as follows:

[0099]

[0100] Among them, σ(x) is the output of the Sigmoid function, x is the input value, and e is the base of the natural logarithm. The output range of this function is (0,1), which can map large positive input values ​​to outputs close to 1 and large negative input values ​​to outputs close to 0.

[0101] In some embodiments, step S23 includes:

[0102] The fully connected layer uses a linear combination of the input features and the input classifier to integrate and classify the features to identify rumors.

[0103] In some embodiments, Figure 7 As shown in the figure, the model uses a combination of shared features and unique features to classify and judge rumors. Its main basis for judgment is the degree of consistency between the unique representation of each modality and the theme content of the video. If there is obvious conflicting semantics, the video can be directly judged as a rumor short video.

[0104] S103: Based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss, the total loss is calculated based on the adaptive modal balance training mechanism, and the pre-established short video rumor detection model is trained based on the total loss to obtain the trained short video rumor detection model, receive the target short video, input the target short video into the trained short video rumor detection model, and output the short video rumor detection result.

[0105] The adaptive balanced training mechanism for different modes mainly includes: a data completeness prompt template, a global prompt virtual memory, and a loss function regulator.

[0106] The data completeness hint template is used to mark incomplete data conditions. The hint template can guide the model's attention to focus on the available modalities, and further enable the model to remember the unbalanced training modalities in a simple and efficient way. The hint template consists of 4 digits (0 or 1) and is used to mark missing data. If part of the input modality is missing, the template sets the first bit to 1, otherwise it is 0. In addition, the remaining 3 bits are used to mark the integrity of the video, text, and audio modalities in a similar manner. For example, for a short video sample with a missing audio modality, its missing hint is set to 1001; the hint template is attached to the modality feature and a joint attention calculation is performed, and its expression is:

[0107]

[0108] Among them, Q, K, V represent the query, key and value matrices in the attention calculation.

[0109] The global hint virtual memory uses software to virtualize a "key-value pair" memory for distinguishing weak modalities. In each training cycle, the memory selects candidate samples (which are more likely to contain hints of weak modalities) and merges them into the globally accessible memory. As training progresses, the weak modal distribution stored in the global memory will gradually approach the overall weak modal distribution in the dataset;

[0110] The loss function regulator takes the loss of the training batch as input and is used to balance the training process between weak and strong modalities. This module regulates the loss through the regularization loss term, which is expressed as follows:

[0111]

[0112] in represents the original training batch loss, [R] represents the regularization matrix, [C] represents the missing prompt matrix, ⊙ represents the Hadamard product, and u represents the predefined weight.

[0113] According to the contrastive learning loss generated in step S12, step S21, step S3 Orthogonality Constraint Loss and the regularization loss term The training target of the model is constructed as follows:

[0114]

[0115] Among them, α and β are the weights of the contrast module and the orthogonal constraint module, respectively, which are synchronized with the model parameters during training.

[0116] Embodiment 2: Figure 5 As shown, in the second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a short video rumor detection system for complex and incomplete data scenarios, which is characterized by comprising:

[0117] The feature extraction module 11 is used to obtain the original short video data, input the original short video data into a pre-established multimodal encoder, and obtain the intra-video shared features; wherein the multimodal encoder includes audio modality, text modality and picture modality; compare the original short video data with the consistency learning constraint loss to obtain the inter-video shared features, input the intra-video shared features and the inter-video shared features into the feature encoder for encoding, and obtain the consistent shared features;

[0118] A feature fusion module 12 is used to obtain unique features between modalities using orthogonal constraint loss based on consistent shared features, perform matrix multiplication calculation based on the unique features between modalities and shared features within the video to obtain unique representations of each modality, learn cross-modal semantic conflicts for the unique representations of each modality through a pre-established Transformer architecture to obtain semantic feature representations, and input the semantic feature representations and shared features within the video into a feature encoder for fusion to obtain fused representations;

[0119] The rumor detection module 13 is used to calculate the total loss based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss based on the adaptive modal balance training mechanism, train the pre-established short video rumor detection model based on the total loss, obtain the trained short video rumor detection model, receive the target short video, input the target short video into the trained short video rumor detection model, and output the short video rumor detection result.

[0120] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0121] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0122] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A short video rumor detection method for complex and incomplete data scenarios, characterized by: The method comprises the following steps: Obtaining original short video data, inputting the original short video data into a pre-established multimodal encoder to obtain intra-video shared features; wherein the multimodality of the multimodal encoder includes audio modality, text modality and picture modality; performing a comparison on the original short video data using a consistency learning constraint loss to obtain inter-video shared features, inputting the intra-video shared features and inter-video shared features into a feature encoder for encoding, and obtaining consistent shared features; Based on the consistent shared features, an orthogonal constraint loss is used to obtain a unique mapping matrix between modalities. Matrix multiplication is performed based on the unique mapping matrix between modalities and the shared features in the video to obtain a unique representation of each modality. The unique representation of each modality is used to learn cross-modal semantic conflicts through a pre-established Transformer architecture to obtain a conflict semantic representation. The conflict semantic representation and the shared features in the video are input into the feature encoder for fusion to obtain a fused representation. Based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss, the total loss is calculated based on the adaptive modal balance training mechanism. The pre-established short video rumor detection model is trained based on the total loss to obtain the trained short video rumor detection model. The target short video is received and input into the trained short video rumor detection model to output the short video rumor detection result.

2. The short video rumor detection method for complex and incomplete data scenarios according to claim 1 is characterized in that: The process of inputting the original short video data into a pre-established multimodal encoder to obtain the shared features in the video: The modal features are obtained by encoding the original features through a modality-specific encoder, including: encoding the audio part of the video using the VGGish encoder, encoding the text part of the video using the Bert model, and encoding the picture part of the video using VGG19. Then, based on the encoded modal features, the modal shared information is extracted through a video-level shared information encoder. The encoder consists of a projection layer and a self-attention mechanism, and the expression is as follows: where {v, t, a} represent the image mode, text mode and audio mode respectively, E v ,E a ,E t is the modal representation of each mode, is the extracted shared representation of each modality, SelfAttn is a single-layer self-attention network, and W s is the projection matrix of the shared subspace, H s is the learned shared representation of the intra-modal video.

3. The short video rumor detection method for complex and incomplete data scenarios according to claim 1 is characterized in that: The process of comparing the original short video data with the consistency learning constraint loss to obtain the shared features between videos: The short videos marked with the same event are regarded as positive samples, and the short videos marked with different events are regarded as negative samples. The contrast loss function is: Where i represents the target short video, P(i) represents the positive sample set of the target short video i, A(i) represents the batch set of the target short video i, τ is the contrastive learning temperature parameter, and the context-aware shared features between videos are obtained through consistent contrastive learning constraints.

4. The short video rumor detection method for complex and incomplete data scenarios according to claim 1 is characterized in that: The calculation process of obtaining the unique features between modalities using orthogonal constraint loss based on consistent shared features is as follows: Based on the obtained shared subspace projection matrix W s , use the orthogonal constraint loss to obtain the unique mapping matrix between modalities, expressed as: Among them, W s represents the shared mapping matrix, Represents the screen mode-specific mapping matrix, represents the text modality-specific mapping matrix, represents the audio modality-specific mapping matrix, and F represents the normalization function.

5. The short video rumor detection method for complex and incomplete data scenarios according to claim 4 is characterized in that: Based on the obtained video modality-specific mapping matrix Text modality specific mapping matrix Audio modality specific mapping matrix And the obtained modal characteristics E s ,E a ,E t , the unique representation of each mode is obtained through matrix multiplication, and the formula is as follows: Among them, *∈{v,a,t} represents each mode, Represents the unique characteristics of each mode obtained.

6. The short video rumor detection method for complex and incomplete data scenarios according to claim 5 is characterized in that: The unique features of each modality are denoised and filtered using a fully connected gating mechanism. The expression is: Among them, *∈{v,a,t} represents each mode, represents the modal-specific features after filtering operations, b g Represents a trainable offset constant, ⊙ represents a Hadamard multiplication operation, and based on the obtained modality-specific features, the cross-modal semantic conflict I is learned through the Transformer architecture, which is expressed as: in, and They represent unique representations of the modalities respectively.

7. The short video rumor detection method for complex and incomplete data scenarios according to claim 1 is characterized in that: The semantic feature representation and the shared features in the video are input into the feature encoder for fusion to obtain the calculation process of the fused representation: Among them, MLP stands for fully connected layer, Represents the model prediction results.

8. The short video rumor detection method for complex and incomplete data scenarios according to claim 1 is characterized in that: The adaptive balance training mechanism includes: data completeness prompt template, global prompt virtual memory, and loss function regulator; The data completeness prompt template expression is as follows: Among them, Q, K, V represent the query, key and value matrices in the attention calculation; The loss function regulator expression is as follows: in represents the original training batch loss, [R] represents the regularization matrix, [C] represents the missing prompt matrix, ⊙ represents the Hadamard product, and u represents the predefined weight; Generation-based contrastive learning loss Orthogonality Constraint Loss and the regularization loss term Construct the training target of the model and get the total loss The expression is as follows: Among them, α and β are the weights of contrastive learning loss and orthogonality constraint loss, respectively.

9. A short video rumor detection system for complex and incomplete data scenarios, characterized by: include: The feature extraction module is used to obtain the original short video data, input the original short video data into a pre-established multimodal encoder, and obtain the intra-video shared features; wherein the multimodal encoder includes audio modality, text modality and picture modality; compare the original short video data with the consistency learning constraint loss to obtain the inter-video shared features, input the intra-video shared features and the inter-video shared features into the feature encoder for encoding, and obtain the consistent shared features; A feature fusion module is used to obtain a unique mapping matrix between modalities based on consistent shared features using an orthogonal constraint loss, perform matrix multiplication calculations based on the unique mapping matrix between modalities and the shared features within the video to obtain a unique representation of each modality, learn cross-modal semantic conflicts for the unique representations of each modality through a pre-established Transformer architecture to obtain a conflict semantic representation, and input the conflict semantic representation and the shared features within the video into a feature encoder for fusion to obtain a fused representation; The rumor detection module is used to calculate the total loss based on the fusion representation and combined with the consistency learning constraint loss and the orthogonal constraint loss based on the adaptive modal balance training mechanism, and train the pre-established short video rumor detection model based on the total loss to obtain the trained short video rumor detection model, receive the target short video, input the target short video into the trained short video rumor detection model, and output the short video rumor detection result.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the short video rumor detection method for complex and incomplete data scenarios described in any one of claims 1 to 8 is adopted.

Images