A Video Data Behavior Compliance Detection Method Based on Multimodal Analysis

Abstract

This invention relates to the field of video data analysis technology, specifically to a video data behavior compliance detection method based on multimodal analysis. This invention can reduce the false detection rate by optimizing the classification model through a dynamic weight allocation mechanism, improving detection accuracy in complex environments, and significantly reducing the probability of normal behavior being misjudged as abnormal. Simultaneously, it can improve the fusion efficiency of multimodal information by improving multimodal feature extraction and deep fusion algorithms, fully exploring the potential relationships between image, audio, and text modalities, and improving detection performance. Furthermore, it can enhance the detection capability of long-tailed abnormal behavior by designing a generative adversarial network to generate rare behavior samples, compensating for insufficient data and improving the model's ability to identify low-frequency, high-risk behaviors. In addition, this invention can strengthen behavior interpretability by introducing an interpretable analysis framework, enabling the model to provide detailed explanations of the detection results, thereby enhancing transparency and trustworthiness in applications.

Description

Technical Field

[0001] This invention relates to the field of video data analysis technology, and more specifically, to a video data behavior compliance detection method based on multimodal analysis. Background Technology

[0002] With the widespread application of video data, particularly in areas such as content moderation, public safety, and corporate oversight, potential non-compliant behaviors within video data have brought about serious social and legal problems. These issues place higher demands on automated detection technologies. Furthermore, the widespread application of deep learning models in video data behavior compliance detection in recent years has significantly improved the processing efficiency of massive amounts of video data. These models can automatically extract multi-level feature representations from video frame sequences, providing more accurate analytical tools for behavior detection.

[0003] Current advancements in video behavior compliance detection technology focus on the fusion and analysis of multimodal information. Video data not only contains visual information but also combines audio and textual information; the interaction between these modalities can provide crucial clues for behavior recognition. For example, image data can be used to analyze specific behaviors within a scene, audio can capture verbal or nonverbal cues, and textual information can supplement the semantics of the scene. By effectively integrating this multimodal information, technological advancements have laid the foundation for behavior detection in complex scenarios.

[0004] However, existing technologies have some limitations and shortcomings in the detection of video data behavior compliance:

[0005] 1. High false detection rate: Current mainstream behavior detection models are prone to misjudging normal behavior as abnormal behavior in complex environments (such as insufficient lighting, background interference, or multi-person interaction), which limits the reliability of detection results and practical application value.

[0006] 2. Low efficiency of multimodal information fusion: Video data contains multiple modalities such as images, audio and text. Existing methods perform poorly in the extraction and fusion of cross-modal features, failing to fully explore the potential correlations between various modalities, thus limiting detection performance.

[0007] 3. Insufficient detection capability for long-tailed anomalies: Many anomalies occur very infrequently but are often highly risky. Existing models have weak detection capabilities for these long-tailed behaviors, which may lead to key risks being overlooked.

[0008] 4. Insufficient behavioral interpretability: Existing methods typically only provide detection results without offering interpretability analysis. This deficiency makes it difficult to use the detection results as a direct basis for actual decision-making, affecting transparency and trust in applications.

[0009] Based on the fusion and analysis of multimodal information, this invention proposes an innovative solution that fully considers the advantages and disadvantages of existing technologies, providing important technical support for future compliance detection of video data behavior. Summary of the Invention

[0010] The purpose of this invention is to provide a video data behavior compliance detection method based on multimodal analysis, so as to solve the problems of high false detection rate, low multimodal information fusion efficiency, insufficient long-tail abnormal behavior detection capability and insufficient behavior interpretation capability of the existing technology mentioned in the background art.

[0011] To achieve the above objectives, the present invention aims to provide a video data behavior compliance detection method based on multimodal analysis, comprising the following steps:

[0012] S1. Perform multimodal feature extraction on the video data, including:

[0013] Image modal feature extraction: generating image representation vectors based on multi-scale feature fusion using SwinTransformer;

[0014] Audio modal feature extraction: After processing by a short-time Fourier transform and a denoising network, the feature is input into an improved DeepSpeech network;

[0015] Text modality feature extraction: A domain-re-pre-trained BERT model is used to extract text features;

[0016] S2. Image, audio, and text features are concatenated into a cross-modal sequence, which is then fused using a cross-modal self-attention mechanism to dynamically adjust the weights of each modality.

[0017] S3. Generate long-tailed anomalous behavior samples using a generative adversarial network with target domain constraints, and combine them with spatiotemporal perturbation to enhance the data;

[0018] S4. Perform behavioral compliance detection based on the fused multimodal features, and output the anomaly probability or category;

[0019] S5. The interpretability of the detection results is analyzed through multimodal adversarial scoring and spatiotemporal causal reasoning.

[0020] As a further improvement to this technical solution, the specific implementation of image modal feature extraction in step S1 includes:

[0021] Extract multi-scale features from the output of each layer of the SwingTransformer;

[0022] Through the trainable mapping matrix W fusion The images are concatenated and their dimensions reduced to generate the image representation vector I.

[0023] As a further improvement to this technical solution, the audio modal feature extraction in step S1 includes:

[0024] Multi-channel spectrum S is generated using short-time Fourier transform. c ;

[0025] The spectral noise is corrected by a denoising network, and the enhanced spectrum S' is output. c ;

[0026] Audio feature vector A is extracted using an improved DeepSpeech network.

[0027] As a further improvement to this technical solution, the text modal feature extraction in step S1 includes:

[0028] The BERT model is re-pre-trained based on domain data, and the loss of the masked language model is optimized.

[0029] Text feature vector T is extracted by re-pre-training BERT.

[0030] As a further improvement to this technical solution, the specific implementation of the cross-modal self-attention mechanism in step S2 includes: using the image representation vector I as the Query, the text feature vector T as the Key, and the audio feature vector A as the Value; generating the fused feature H through multi-head attention calculation, and dynamically allocating the modal weights α. m .

[0031] As a further improvement to this technical solution, the constraints for generating the Generative Adversarial Network (GAN) in step S3 include: adversarial loss. Used to distinguish between real and generated samples;

[0032] Target domain loss Constrain the Euclidean distance between the generated samples and the real outlier data in the high-level feature space;

[0033] The joint optimization objective is Where λ is the weighting coefficient.

[0034] As a further improvement to this technical solution, step S3 also includes:

[0035] Spatiotemporal perturbations are applied to the generated samples, including random frame interpolation, rotation, lighting adjustment, and audio band occlusion;

[0036] The model is trained jointly with enhanced synthetic data and real data.

[0037] As a further improvement to this technical solution, the implementation of multimodal adversarial scoring in step S5 includes:

[0038] Map the features of each modality to the CLIP common vector space;

[0039] Add a small perturbation δ and calculate the cross-modal similarity change Δsim i→j ;

[0040] Weighted combination of similarity changes generates adversarial scores, which are used to assess the model’s sensitivity to specific modalities.

[0041] As a further improvement to this technical solution, the implementation of spatiotemporal causal reasoning in step S5 includes:

[0042] The video is segmented into temporal segments τ, and temporal edges and semantic edges are constructed.

[0043] Aggregate fragment features h using a spatiotemporal graph network τ ;

[0044] Matching fragment features and compliance rule vector r k When the similarity exceeds a threshold, a regular edge is generated.

[0045] As a further improvement to this technical solution, the compliance rule vector r k via text encoder E text (·) mapping generation, and associated with fragment feature h τ The cosine similarity is calculated using the following formula:

[0046] cos_sim(h τ ,r k )≥ε

[0047] Where ε is a preset threshold that triggers a rule violation determination.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] 1. This invention can reduce the false detection rate by optimizing the classification model through a dynamic weight allocation mechanism, improving detection accuracy in complex environments, and significantly reducing the probability of normal behavior being misjudged as abnormal. It also improves the efficiency of multimodal information fusion by improving multimodal feature extraction and deep fusion algorithms, fully exploring the potential relationships between image, audio, and text modalities, and improving detection performance. Furthermore, it enhances the detection capability of long-tailed abnormal behavior by designing a generative adversarial network to generate rare behavior samples, compensating for insufficient data and improving the model's ability to identify low-frequency, high-risk behaviors. In addition, this invention strengthens the ability to interpret behavior by introducing an interpretable analysis framework, enabling the model to provide detailed explanations of the detection results, thereby enhancing transparency and trustworthiness in applications.

[0050] 2. In this invention, image, audio and text features are fused to fully mine video scene information. The model is still robust even when modality is missing or noise is severe. At the same time, by using target domain to constrain GAN to generate scarce abnormal data and perform diversified enhancements, the detection capability of low-frequency and high-risk behaviors is greatly improved.

[0051] 3. In this invention, the CLIP-based multimodal adversarial scoring enables the model to have a more granular vulnerability discovery capability after compliance detection by perturbing and measuring the similarity of audio, image and text. At the same time, it generates audio and video evidence and text descriptions through spatiotemporal causal reasoning and compliance rule mapping, thereby greatly improving the interpretability of the detection results. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating the steps of the video data behavior compliance detection method based on multimodal analysis according to the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] In one specific embodiment, such as Figure 1 As shown, this invention provides a video data behavior compliance detection method based on multimodal analysis, specifically including the following steps:

[0055] 1. Data preprocessing and feature extraction:

[0056] Firstly, addressing the challenge of video action recognition in complex scenes, this paper fully leverages multi-source information to extract video features. For the video's image modalities, an improved SwingTransformer with multi-scale feature fusion is used to extract image vectors from video frames. This is achieved by extracting the feature F output from each layer. (l) Information at different scales is extracted from the data, and then concatenated to obtain a merged representation F. merged =Concat(F (1) ,...,F (l) W fusion Where Concat(·) represents a concatenation operation along the dimension W fusion As a trainable mapping matrix, it maps the concatenated multi-scale features back to the target dimension, generating the final representation vector I of the image modality. This preserves both local details and global context.

[0057] For the audio mode of the video, a short-time Fourier transform (STFT) is first performed on the original signal to obtain the multi-channel spectral representation S. c Then, a denoising network is used to correct the noise, resulting in the enhanced S'. c (f,τ)=M c (f,τ)S c (f,τ), where M c (f,τ) represents the denoising network. Finally, these spectra are fed into the DeepSpeech backbone network to obtain a stable audio feature vector A = E. audio ({S' c (f,τ)}), where E audio (·) represents the improved feature extraction function of DeepSpeech.

[0058] For the text modality of the video, a BERT pre-trained on domain data is used, with the loss function of Masked Language Model (MLM). To make the model more familiar with industry terminology and context, where w i It is a word that is masked, w \i Representing the context, θ represents the BERT model parameters, ultimately yielding text features T = E that are more sensitive to industry semantics. text ({w i}), E text (·) is the feature extraction function for retraining BERT.

[0059] 2. Multimodal fusion:

[0060] After completing the modal feature extraction, the image representation vector I, audio feature vector A, and text feature vector T are concatenated into a sequence. Cross-modal self-attention captures the interactions between modalities. During multi-head attention computation, Q = XW is first projected. Q K = XW K V = XW V Then through We integrate them together. Because this is a cross-modal scenario, we treat image feature I as the query, text feature T as the key, and audio feature A as the value. Finally, to handle situations where modality is missing or noise is out of control, we assign a learnable weight α to each attention output. m Then through We obtain the final fused representation, where M is the number of multi-head attention points. Thus, if the quality of a certain modality is poor, α... m It will automatically become smaller, and the weights of other modalities will relatively increase.

[0061] 3. Generation of long-tail anomalous behavior samples:

[0062] When dealing with long-tailed anomalous behavior, relying solely on multimodal fusion features H cannot fully address the data scarcity challenge. Therefore, a generative adversarial network (GAN) with target domain constraints is used to supplement the sample data. This approach is based on StyleGAN2, in addition to the conventional adversarial discriminator D... adv In addition, a domain discriminator D was added. domain In this way, the generator G can overcome the challenges posed by the conventional adversarial discriminator D. adv Domain Discriminator D domain The joint constraints generate synthetic samples that closely resemble the distribution of real anomalous data. The adversarial loss is defined as:

[0063]

[0064] Where, p data p represents the true data distribution. z The prior distribution of the noise input to the generator is given. The target domain loss, on the other hand, specifies the relationship between the generated sample G(z) and the real outlier data X in the high-level feature space φ(·). target To maintain proximity, the domain constraint loss is defined as:

[0065]

[0066] Where φ(·) is a feature mapping function compatible with multimodal fusion networks, used to measure the distance between generated samples and real anomalous data in the high-level semantic space, making GANs more closely resemble the multimodal distribution of real anomalous samples. dist(,) takes the Euclidean distance, X target This represents the feature set of the target anomaly domain data. The overall form of the joint loss is:

[0067]

[0068] Here, λ controls the weight of the target domain constraint in the overall loss. With this domain constraint, for a specific anomalous behavior l, a rare type of anomalous behavior data is synthesized using a conditional generative network (G(z,l),D(x,l)). Subsequently, spatiotemporal perturbations (such as random frame interpolation, rotation, illumination adjustment, and audio frequency band occlusion) are applied to the synthesized data, and it is used for training along with real data to enhance the model's adaptability to long-tailed scenes.

[0069] After training, a compliance detection model with multimodal fusion and long-tail anomaly adaptation will be obtained. The fused features... The sample is fed into the final detection head, using the formula:

[0070]

[0071] Output the behavior category or anomaly probability. d represents the trainable weights, d is the dimension of the fused features, and c is the number of categories. It is the bias vector, and σ(·) is the softmax function.

[0072] 4. Interpretation of Behavioral Compliance Test Results:

[0073] In practical applications, it is necessary to explain the detection results to users to improve the credibility of the model. Here, we use two methods—CLIP-based multimodal adversarial scoring and spatiotemporal causal reasoning—to enhance interpretability.

[0074] 5. Perform interpretability analysis on the test results:

[0075] CLIP-based multimodal adversarial scoring: Mapping image features I, audio features A, and text features T from a cross-modal network to the CLIP common vector space:

[0076] v I =E img (I)

[0077] v A =E audio (A)

[0078] v T =E text (T)

[0079] Among them, E img (·), E audio (·), E text (·) represents the CLIP image, audio, and text encoder. A small perturbation δ is then added to the image, resulting in the subtly affected image vector v in the CLIP common vector space. I' =E img (I+δ), this perturbation δ, when added to audio or text, also applies. The change in similarity can then be measured:

[0080] △sim I→T =|cos_sim(v I ,v T )-cos_sim(v I' ,v T )|

[0081] Similarly, we can obtain △sim I→A , △sim A→T Where cos_sim(,) is the cosine similarity function of the vectors. Then, the similarity changes of different modalities are weighted and combined to obtain the multimodal adversarial score:

[0082]

[0083] Where β ij This represents the degree of concern of each mode to the disturbance, satisfying ∑ (i,j)∈((I,T)(I,A)(T,A)) β ij =1. This way, the score allows us to understand the model's sensitivity to perturbations across image, audio, and text modalities, and identify potential over-reliance on certain modal features (such as explicit images, alarm sounds, or inappropriate text). Furthermore, this enables the model to not only answer whether features match at the modal level, but also to further supplement training data and optimize the model's network structure based on the score results.

[0084] Spatiotemporal causal reasoning: The video sequence after anomaly detection is divided into several segments τ, each segment containing an image frame, corresponding audio, and text information. To capture temporal causality and semantic relationships, two types of edges are defined between segments: if segment τ ’ If two video segments are temporally similar to each other (e.g., image, audio, text), a temporal edge is connected; if the features (image + audio + text) of the two video segments are highly similar in the multimodal common space, a semantic edge is connected. Then, a spatiotemporal graph network is used to define the aggregated features of each segment τ:

[0085]

[0086] Where N(τ') represents the adjacent segments of segment τ, W g b g Let α be a trainable parameter, σ(·) be the activation function, and α be a training parameter. τ,τ’ Attention weights are determined by segment similarity.

[0087] The compliance rules are then passed through the same encoder E as the text modality. text (·) maps to the embedding vector r k When the aggregation feature h of a certain segment is discovered τ With compliance rule vector r k When the similarity reaches a certain threshold:

[0088] cos_sim(h τ ,r k )≥ε

[0089] Where ε is the similarity threshold, a rule edge containing this rule is added to the fragment τ, making it easier for users to interpret the detection results. Tracing back to a certain fragment τ, it was detected that it violated rule r. k This improved the interpretability of the model and increased user trust.

[0090] In summary, this invention first utilizes a multi-scale fusion-improved Swin Transformer, DeepSpeech (with multi-channel spectral enhancement), and BERT re-pre-trained for compliance domains to extract deep representations of images, audio, and text from video data, respectively. Then, supported by cross-modal attention and dynamic weighting mechanisms, the features of the three modalities are mapped to a unified representation space, achieving information complementarity and global dependency modeling through cross-attention between different modalities. Next, StyleGAN2 with target domain constraints is introduced in the long-tail anomaly detection part, combining synthetic scarce behavioral scenes with spatiotemporal perturbation enhancement to alleviate the problem of insufficient real data. Based on this, an anomaly detection head is trained to perform compliance / anomaly determination on the fused multimodal features. Finally, through an interpretable analysis framework of multimodal adversarial scoring and spatiotemporal causal reasoning, a multimodal analysis and explanation chain from cause to effect is presented to users, thereby achieving a combination of high accuracy and traceability in practical scenarios.

[0091] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A video data behavior compliance detection method based on multimodal analysis, characterized in that, Includes the following steps: S1. Perform multimodal feature extraction on the video data, including: Image modal feature extraction: generating image representation vectors based on multi-scale feature fusion using SwinTransformer; Audio modal feature extraction: After processing by a short-time Fourier transform and a denoising network, the feature is input into an improved DeepSpeech network; Text modality feature extraction: A domain-re-pre-trained BERT model is used to extract text features; S2. Image, audio, and text features are concatenated into a cross-modal sequence, which is then fused using a cross-modal self-attention mechanism to dynamically adjust the weights of each modality. S3. Generate long-tailed anomalous behavior samples using a generative adversarial network with target domain constraints, and combine them with spatiotemporal perturbation to enhance the data; The constraints of the Generative Adversarial Network (GAN) include: Combating losses This is used to distinguish between real and generated samples; Target domain loss Constrain the Euclidean distance between the generated samples and the real abnormal data in the high-level feature space; The joint optimization objective is ,in These are the weighting coefficients. For generator, For conventional adversarial discriminators, For domain discriminator; S3 further includes: Spatiotemporal perturbations are applied to the generated samples, including random frame interpolation, rotation, lighting adjustment, and audio band occlusion; After training, a compliance detection model with multimodal fusion and long-tail anomaly adaptation will be obtained, which will integrate the fused features. The sample is fed into the final detection head, using the formula: ； Output behavior category or anomaly probability, where These are trainable weights. It is the dimension of the fused features. It is the number of categories. It is a bias vector. It is the softmax function; S4. Perform behavioral compliance detection based on the fused multimodal features, and output the anomaly probability or category; S5. Conduct interpretability analysis of the detection results through multimodal adversarial scoring and spatiotemporal causal reasoning; The implementation of multimodal adversarial scoring in S5 includes: Map the features of each modality to the CLIP common vector space; Add small perturbations And calculate cross-modal similarity changes ; Weighted combination of similarity changes generates adversarial scores, which are used to assess the model's sensitivity to specific modalities; The implementation of spatiotemporal causal reasoning in S5 includes: The video is divided into time-series segments. Construct temporal edges and semantic edges; Aggregating fragment features using a spatiotemporal graph network ; Matching fragment features and compliance rule vectors When the similarity exceeds a threshold, a regular edge is generated.

2. The video data behavior compliance detection method based on multimodal analysis according to claim 1, characterized in that, The specific implementation of image modal feature extraction in S1 includes: Extract multi-scale features from the output of each layer of the SwingTransformer; Through trainable mapping matrix Concatenate and reduce dimensions to generate image representation vectors .

3. The video data behavior compliance detection method based on multimodal analysis according to claim 1, characterized in that, The audio modal feature extraction in S1 includes: Multi-channel spectrum generated using short-time Fourier transform ; The spectral noise is corrected by a denoising network, and the enhanced spectrum is output. ; Extracting audio feature vectors using an improved DeepSpeech network .

4. The video data behavior compliance detection method based on multimodal analysis according to claim 1, characterized in that, The text modal feature extraction in S1 includes: The BERT model is re-pre-trained based on domain data, and the loss of the masked language model is optimized. Text feature vectors are extracted by re-training BERT. .

5. The video data behavior compliance detection method based on multimodal analysis according to claim 1, characterized in that, The specific implementation of the cross-modal self-attention mechanism in S2 includes: Image representation vector As a query, text feature vector As the key, the audio feature vector As Value; Fusion features are generated through multi-head attention computation. And dynamically allocate the modal weights. .

6. The video data behavior compliance detection method based on multimodal analysis according to claim 1, characterized in that, The compliance rule vector via text encoder Mapping generation, and with fragment features The cosine similarity is calculated using the following formula: ； in, A rule violation determination is triggered based on a preset threshold.

Images