A method and system for detecting unfriendly events in audio and video.

Abstract

This invention discloses a method and system for detecting unfriendly events in audio and video. First, the video sequence to be detected is input into an audio pre-training network and a visual pre-training network, respectively, to generate pre-trained features. Then, the pre-trained features are input into two self-attention networks to learn inter-modal temporal domain features, generating intra-modal features. Next, the intra-modal features are input into a cross-attention network to learn inter-modal correlation information, generating inter-modal features. Finally, these inter-modal features are passed through a fully connected layer to calculate prediction scores for each modality. The prediction scores are then directly summed in the temporal dimension to obtain the prediction score for the input video. The nature of the event in the audio and video is determined based on the prediction score. This invention achieves the highest average accuracy on the largest current dataset for detecting unfriendly events in audio and video, improves the generalization performance of generated audio and video features, and achieves accurate frame-level audio and video event detection.

Description

Technical Field

[0001] This invention belongs to the field of video recognition technology and relates to a method and system for detecting unfriendly events in audio and video, specifically a method and system for detecting unfriendly events in audio and video based on frame-level clustering comparison and modal step loss compensation. Background Technology

[0002] The audio and video unfriendly event detection task aims to identify unfriendly events occurring in a video by analyzing audio and visual signals, and to provide precise start and end times of the events. Research on this task has broad social application value in areas such as real-time monitoring systems and social network content moderation systems.

[0003] For this task, the general strategy of existing methods in the two key steps of frame-level feature modeling and multimodal fusion is as follows: First, the top-K MIL is used to select a fixed number of frame features with the highest prediction scores as non-friendly event frames, and comparative learning is performed to achieve frame-level feature modeling; Second, audio and video fusion features are directly used to optimize the model's objective to fuse the information of the two modalities.

[0004] However, in event frame detection, especially for unfriendly event frames, these two general strategies actually weaken the network model's ability to detect frame-level unfriendly event features. First, the number of unfriendly event frames in a video labeled as unfriendly is variable. Selecting a fixed number of frames as unfriendly event feature representations can lead to the inclusion of friendly frames or the omission of the remaining unfriendly frames, thus significantly weakening the model's ability to distinguish frame-level unfriendly event representations. Second, regarding feature fusion, although a single modality itself contains rich information, only a small portion is needed for model training. Furthermore, optimizing a single modality to approach a local optimum will cause the network to over-memorize redundant single-modal information. Therefore, this fusion strategy weakens the network's ability to aggregate unfriendly event information from two modalities, thus limiting the network model's detection performance. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an audio and video event detection method and system based on frame-level clustering comparison and modal step loss compensation, which is particularly suitable for detecting unfriendly event frames.

[0006] The technical solution adopted by the method of the present invention is: a method for detecting unfriendly events in audio and video, comprising the following steps:

[0007] Step 1: Input the video sequence to be detected into the audio pre-training network and the visual pre-training network respectively, and generate pre-trained features p respectively. a p v ;

[0008] Step 2: Transfer the pre-trained features p a p v Two self-attention networks are input separately to learn the time-frequency domain features between modalities, and intra-modal features f are generated respectively. a f v ;

[0009] Step 3: Transfer the intra-modal features f a f v Input a cross-attention network to learn relevant information between modalities and generate inter-modal features respectively.

[0010] Step 4: Extract these intermodal features The prediction scores for the two modalities are calculated separately by passing each fully connected layer. The prediction scores are then directly added together in the temporal dimension to obtain the prediction score for the input video. The nature of the events in the audio and video is determined based on the prediction scores.

[0011] Preferably, in step 1, the visual pre-trained network includes sequentially connected 3D convolutional layers, downsampling layers, Inception module layers, temporal pooling layers, and dual-stream architecture layers.

[0012] The Inception module layer consists of three Inception modules connected in sequence; each Inception module consists of several parallel convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation layer.

[0013] The dual-stream architecture layer consists of a parallel RGB stream processing module and an optical stream processing module.

[0014] Preferably, in step 1, the audio pre-trained network consists of a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a third convolutional layer, a fourth convolutional layer, a third max pooling layer, a global average pooling layer, and a fully connected layer connected in sequence.

[0015] Preferably, in step 2, the self-attention network consists of a multi-head attention mechanism layer and a feedforward neural network layer;

[0016] The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads;

[0017] The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0018] Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation. The residual connection layer is used to add the sub-layer to the input.

[0019] Preferably, in step 3, the cross-attention network consists of a multi-head attention mechanism layer and a feedforward neural network layer;

[0020] The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads;

[0021] The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0022] Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation. The residual connection layer is used to add the sub-layer to the input.

[0023] Preferably, the visual pre-trained network, audio pre-trained network, self-attention network, and cross-attention network constitute the audio-visual event detection network; the audio-visual event detection network is a pre-trained network, and during training, the intra-modal features f a f v The contrast loss L is calculated using a frame-level clustering contrast method. FCC Intermodal features Compared with the original modal features f av The target loss L is calculated using the modal step loss compensation method. MF .

[0024] Preferably, the frame-level clustering comparison method calculates the comparison loss L. FCC The specific implementation includes the following sub-steps:

[0025] (1) Unsupervised clustering;

[0026] In a batch, the intra-modal features of the audio and video modalities of non-friendly labeled samples are all unsupervised clustered using K-Means, forming two cluster centers; then, average pooling is performed in the temporal domain, generating two different pooling features e for each modal feature of each non-friendly sample. m cluster1 and e m cluster2 For the friendly-labeled samples, average pooling was performed separately to generate friendly pooling features for each modality, denoted as e. a nor and ev nor ;

[0027] (2) Similarity measurement;

[0028] Calculate the Euclidean distance between the pooled features of each non-friendly sample and friendly sample, and consider instances with larger distances as pseudo-friendly and instances with smaller distances as pseudo-normal. Finally, generate pseudo-friendly and pseudo-normal instances for each modality, denoted as follows:

[0029] (3) Comparative learning based on triple loss;

[0030] Choose two anchor points, which are pseudo-friendly features. and pseudo-normal features Construct two sets of comparison objects, denoted as {anchor point, positive sample, negative sample}, which are represented as follows for audio modalities: and Finally, triplet loss is used to calculate the contrast loss, and the calculation method is as follows:

[0031]

[0032]

[0033]

[0034] Where N represents the number of non-friendly labeled samples in a batch, B represents the batch size, ||·|| represents the Euclidean distance, [...]+ indicates that the operation within the parentheses is equal to its own value when it is greater than 0; otherwise, it is equal to 0; α represents the margin hyperparameter of the ternary loss.

[0035] Preferably, the target loss L is calculated using the modal step loss compensation method. MF The calculation method is as follows:

[0036]

[0037]

[0038]

[0039] in and These represent the original loss values ​​for inter-modal features and original modal features, respectively. This represents the calculation of binary cross-entropy, where W1, W2, and W3 represent three linear layers, σ represents the sigmoid function, and b... inter and b intraLet b represent the loss compensation values ​​for inter-modal features and original modal features, respectively. inter intra .

[0040] The technical solution adopted by the system of the present invention is: a system for detecting unfriendly events in audio and video, comprising:

[0041] One or more processors;

[0042] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for detecting unfriendly events in audio and video.

[0043] This invention discloses for the first time a method for detecting unfriendly events in audio and video. This method can accurately detect whether an unfriendly event is present in the video and identify the start and end times of the event. It proposes a frame-level clustering comparison method, generating pseudo-labels for frame-level features through unsupervised clustering and constructing positive and negative samples based on these pseudo-labels for comparative learning. This improves the method's ability to identify frame-level unfriendly event features and enhances the accuracy of unfriendly event discrimination boundaries. Simultaneously, this invention discloses a modal ladder-style loss compensation strategy suitable for multimodal scenarios. By allocating higher loss compensation to single-modal features, it avoids the neural network's over-memorization of redundant single-modal information, promoting effective fusion of multimodal information and improving the generalization performance of audio-video fusion features. This method achieves the highest average accuracy on XD-Violence, the largest known dataset of multimodal unfriendly audio and video events, significantly outperforming previous methods. Attached Figure Description

[0044] The technical solutions described herein are further illustrated below using examples and specific implementation methods. Additionally, accompanying drawings are used in the description of the technical solutions. Those skilled in the art can, without any creative effort, obtain other drawings and the intent of the present invention based on these drawings.

[0045] Figure 1 This is a schematic diagram of the method according to an embodiment of the present invention;

[0046] Figure 2 This is the detection score curve for abuse incidents in an embodiment of the present invention; where the solid line is the time-series label ground truth curve of the video to be detected, and the dashed line is the prediction score curve for the video to be detected in an embodiment of the present invention.

[0047] Figure 3 ​This is the detection score curve for a traffic accident event according to an embodiment of the present invention; where the solid line is the time-series label ground truth curve of the video to be detected, and the dashed line is the prediction score curve for the video to be detected according to an embodiment of the present invention.

[0048] Figure 4 This is the detection score curve for an explosion event according to an embodiment of the present invention; where the solid line is the time-series label truth curve of the video to be detected, and the dashed line is the prediction score curve of the video to be detected according to an embodiment of the present invention.

[0049] Figure 5 This is the detection score curve for riot events in an embodiment of the present invention; where the solid line is the time-series label ground truth curve of the video to be detected, and the dashed line is the prediction score curve for the video to be detected in an embodiment of the present invention.

[0050] Figure 6 This is the detection score curve for a beating incident according to an embodiment of the present invention; where the solid line is the time-series label ground truth curve of the video to be detected, and the dashed line is the prediction score curve of the video to be detected according to an embodiment of the present invention.

[0051] Figure 7 The figure shows the detection score curve for shooting incidents in an embodiment of the present invention; the solid line is the ground truth curve of the temporal label of the video to be detected, and the dashed line is the prediction score curve of the video to be detected in an embodiment of the present invention. Detailed Implementation

[0052] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0053] Please see Figure 1 This embodiment provides a method for detecting unfriendly events in audio and video, including the following steps:

[0054] Step 1: Input the video sequence to be detected into the audio pre-training network and the visual pre-training network respectively, and generate pre-trained features p respectively. a p v ;

[0055] In one embodiment, the visual pre-trained network includes sequentially connected 3D convolutional layers (3x3 spatial kernel, 3 temporal kernel, ReLU activation, 64 filters), downsampling layers (3D max pooling method, 1x2x2 spatial kernel, 1 temporal kernel, 1x2x2 stride), Inception module layers, temporal pooling layers, and two-stream architecture layers.

[0056] The Inception module layer consists of three Inception modules connected in sequence; each Inception module consists of several parallel convolutional layers (multiple parallel convolutional layers with different filter sizes, specifically 1x1x1, 3x3x3, 1x1x1 in the implementation), and each convolutional layer is followed by a batch normalization layer and a ReLU activation layer.

[0057] The dual-stream architecture layer consists of a parallel RGB stream processing module (processing RGB frames) and an optical stream processing module (processing optical stream information).

[0058] In one embodiment, the audio pre-trained network consists of a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a third convolutional layer, a fourth convolutional layer, a third max pooling layer, a global average pooling layer, and a fully connected layer connected in sequence.

[0059] It accepts audio frames of length 96 as input. Each audio frame contains 64ms of audio data.

[0060] First layer: The input audio frame passes through a 7x7 convolutional layer with a stride of 2, generating 64 feature maps. The ReLU activation function is used to introduce non-linearity.

[0061] The second layer: This is followed by a 2x2 max pooling layer with a stride of 2, used to reduce the spatial resolution of the feature map.

[0062] The third layer: The next layer is a convolutional layer containing 128 3x3 filters with a stride of 1, also activated by ReLU.

[0063] The fourth layer is followed by a 2x2 max pooling layer with a stride of 2.

[0064] The fifth layer: This is followed by two convolutional layers, each containing 256 3x3 filters with a stride of 1, all activated using ReLU.

[0065] The sixth layer: Then there is a 2x2 max pooling layer with a stride of 2.

[0066] Global average pooling: After the convolutional layers, VGGish uses global average pooling to reduce the dimension of each channel, transforming the entire feature map into a vector with 128 elements.

[0067] The final fully connected layer maps the 128-dimensional output to the final audio embedding space. This fully connected layer does not use the ReLU activation function.

[0068] Step 2: Transfer the pre-trained features p a p vTwo self-attention networks are input separately to learn the time-frequency domain features between modalities, and intra-modal features f are generated respectively. a f v ;

[0069] In one implementation, it consists of a multi-head attention mechanism layer and a feedforward neural network layer;

[0070] The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads;

[0071] The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0072] Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation. The residual connection layer is used to add the sub-layer to the input.

[0073] Input: The audio self-attention network input is the audio features extracted by the audio pre-trained network, the visual self-attention network input is the visual features extracted by the visual pre-trained network, and the audio and visual inputs of the cross-attention network are consistent with the self-attention network inputs of their respective modalities.

[0074] The input features are linearly transformed and mapped to high-dimensional vectors. Specifically, linear layers are used, where the audio self-attention network maps audio features to 256 dimensions, the visual self-attention network maps visual features to 1024 dimensions, and the cross-attention network maps both modality features to 256 dimensions. Each self-attention network and cross-attention network contains four multi-head attention heads.

[0075] Step 3: Transfer the intra-modal features f a f v Input a cross-attention network to learn relevant information between modalities and generate inter-modal features respectively.

[0076] In one implementation, it consists of a multi-head attention mechanism layer and a feedforward neural network layer;

[0077] The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads;

[0078] The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0079] Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation. The residual connection layer is used to add the sub-layer to the input.

[0080] Input: The audio self-attention network input is the audio features extracted by the audio pre-trained network, the visual self-attention network input is the visual features extracted by the visual pre-trained network, and the audio and visual inputs of the cross-attention network are consistent with the self-attention network inputs of their respective modalities.

[0081] The input features are linearly transformed and mapped to high-dimensional vectors. Specifically, linear layers are used, where the audio self-attention network maps audio features to 256 dimensions, the visual self-attention network maps visual features to 1024 dimensions, and the cross-attention network maps both modality features to 256 dimensions. Each self-attention network and cross-attention network contains four multi-head attention heads.

[0082] Step 4: Extract these intermodal features The prediction scores for the two modalities are calculated by passing each modality through a fully connected layer and then by using the sigmoid function. The prediction scores are then directly added together in the temporal dimension to obtain the prediction score for the input video. The nature of the events in the audio and video is determined based on the prediction scores.

[0083] In one implementation, the visual pre-trained network, audio pre-trained network, self-attention network, and cross-attention network constitute an audio-visual event detection network; the audio-visual event detection network is a pre-trained network, and during training, the intra-modal features f a f v The contrast loss L is calculated using a frame-level clustering contrast method. FCC Intermodal features Compared with the original modal features f av The target loss L is calculated using the modal step loss compensation method. MF .

[0084] In one implementation, to construct the semantic gap between unfriendly and friendly frames, this embodiment proposes a frame-level clustering comparison module. This module uses unsupervised clustering to label each unfriendly sample with a pseudo-label, dynamically clustering frames into two centers. A contrastive learning method is then used to control the semantic distance between frames, thereby achieving accurate frame-level modeling. This method includes three steps: unsupervised clustering, similarity measurement, and contrastive learning based on triplet loss.

[0085] The frame-level clustering comparison method calculates the comparison loss L. FCC The specific implementation includes the following sub-steps:

[0086] (1) Unsupervised clustering;

[0087] In a batch, the intra-modal features of the audio and video modalities of non-friendly labeled samples are all unsupervised clustered using K-Means, forming two cluster centers; then, average pooling is performed in the temporal domain, generating two different pooling features e for each modal feature of each non-friendly sample. m cluster1 and e m cluster2 For the friendly-labeled samples, average pooling was performed separately to generate friendly pooling features for each modality, denoted as e. a nor and e v nor ;

[0088] (2) Similarity measurement;

[0089] The similarity metric aims to assign pseudo-labels to two pooled instances in each modality for each non-friendly sample. This method's similarity metric is based on the idea that pseudo-friendly features have lower similarity to pseudo-normal features compared to friendly features. Therefore, the Euclidean distance between the pooled features of each non-friendly and friendly sample is directly calculated, and instances with larger distances are considered pseudo-friendly, while those with smaller distances are considered pseudo-normal. Finally, pseudo-friendly and pseudo-normal instances are generated for each modality, denoted as follows:

[0090] (3) Comparative learning based on triple loss;

[0091] To model the semantic gap between similar frames and the two modalities, the model should reduce the distance between audio-visual pseudo-friendly features while increasing the distance between opposite semantic features, and the same applies to pseudo-normal features. For this purpose, two anchor points are chosen: pseudo-friendly features and pseudo-normal features. and pseudo-normal features Construct two sets of comparison objects, denoted as {anchor point, positive sample, negative sample}, which are represented as follows for audio modalities: and Finally, triplet loss is used to calculate the contrast loss, and the calculation method is as follows:

[0092]

[0093]

[0094]

[0095] Where N represents the number of non-friendly labeled samples in a batch, B represents the batch size, ||·|| represents the Euclidean distance, [...]+ indicates that the operation within the parentheses is equal to its own value when it is greater than 0; otherwise, it is equal to 0; α represents the margin hyperparameter of the ternary loss.

[0096] In one implementation, the optimized target loss L is calculated using the modal step loss compensation method. MF The aim of this method is to encourage models to reduce their reliance on single-modal features and emphasize the use of fused features to aggregate information from two modalities. Previous research has shown that allowing the training loss to fluctuate around a small constant value, enabling the network model to use a non-zero training loss during training, can effectively avoid over-memorizing training techniques, thereby achieving better generalization ability. This embodiment extends this idea to multimodal scenarios, designing a multimodal-based stepped loss compensation mechanism. Since the original modal features are directly concatenated from the intra-modal features of the two modalities, they retain the original information of the single-modal features; while the inter-modal features are calculated from the intra-modal features through a cross-attention mechanism, already considering the inter-modal interaction information during feature calculation. Specifically, a higher compensation value is assigned to the original modal training loss, while a lower compensation value is set for the training loss of the inter-modal features. This method avoids over-memorizing single-modal information, and the inter-modal features with lower loss compensation also avoid overfitting from large-scale networks. Therefore, the network model focuses more on cross-modal feature information and makes inferences based on it. The target loss L is calculated and optimized using the modal step loss compensation method. MF The calculation method is as follows:

[0097]

[0098]

[0099]

[0100] in and These represent the original loss values ​​for inter-modal features and original modal features, respectively. This represents the calculation of binary cross-entropy, where W1, W2, and W3 represent three linear layers, σ represents the sigmoid function, and b... inter and b intra Let b represent the loss compensation values ​​for inter-modal features and original modal features, respectively. inter intra .

[0101] The invention will be further illustrated by specific experiments below.

[0102] ​This experiment achieved an average accuracy of 84.24% on XD-Violence, the largest current dataset of audio and video multimodal unfriendly events, which is higher than previous research methods on this dataset.

[0103] Table 1. Comparison of average accuracy of the method of the present invention and previous research methods on XD-Violence.

[0104] MILss Vision 73.20 RTMF Vision 77.81 HL-Net Audio+Vision 78.64 AVD-Attention Audio+Vision 79.37 ACF Audio+Vision 80.13 S3R Vision 80.26 MACIL-SD Audio+Vision 83.40 The method Audio+Vision 84.24

[0105] Table 1 shows the comparison of the average accuracy of our method with previous studies on XD-Violence. After fully utilizing the audio and visual modal information provided by the dataset, the average accuracy of our method far exceeds that of previous studies.

[0106] To visually demonstrate the performance of this method in detecting unfriendly events, Figures 2-7 The graph shows the predicted scores and ground truth labels for a subset of videos using this method. The dashed lines represent the network model's predicted scores, and the straight lines represent the ground truth labels. The horizontal axis represents time (in seconds), and the vertical axis represents the predicted scores (range: 0-1). From top to bottom, the events are: abuse, car accident, explosion, riot, beating, and shooting.

[0107] Figures 2-7 The results show that the predicted curve closely matches the ground truth label segment. Even when a friendly event occurs between two unfriendly events, this method can accurately predict the start and end times of the unfriendly event. Furthermore, the prediction score for friendly videos remains consistently low, further demonstrating that this method can accurately distinguish between unfriendly and friendly frames.

[0108] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for detecting unfriendly events in audio and video, characterized in that, Includes the following steps: Step 1: Input the video sequence to be detected into the audio pre-training network and the visual pre-training network respectively, and generate pre-trained features p respectively. a p v ; Step 2: Transfer the pre-trained features p a p v Two self-attention networks are input separately to learn the time-frequency domain features between modalities, and intra-modal features f are generated respectively. a f v ; Step 3: Transfer the intra-modal features f a f v Input a cross-attention network to learn relevant information between modalities and generate inter-modal features respectively. ; Step 4: Extract these intermodal features The prediction scores for the two modalities are calculated separately through a fully connected layer; the prediction scores are then directly added together in the temporal dimension to obtain the prediction score for the input video, and the nature of the events in the audio and video is determined based on the prediction scores. The visual pre-trained network, audio pre-trained network, self-attention network, and cross-attention network together form the audio-visual event detection network; the audio-visual event detection network is a pre-trained network, and during training, the intra-modal features f a f v The contrast loss L is calculated using a frame-level clustering contrast method. FCC Intermodal features Compared with the original modal features f av The target loss L is calculated using the modal step loss compensation method. MF ; Original modal features f av Since it is for the intra-modal feature f of two modes a f v It is assembled directly; The frame-level clustering comparison method calculates the comparison loss L. FCC The specific implementation includes the following sub-steps: (1) Unsupervised clustering; In a batch, the intra-modal features of the audio and video modalities of non-friendly labeled samples are all unsupervised clustered using K-Means, forming two cluster centers; then, average pooling is performed in the temporal domain, generating two different pooling features e for each modal feature of each non-friendly sample. m cluster1 and e m cluster2 ; For the friendly-labeled samples, average pooling was performed separately to generate friendly pooling features for each modality, denoted as follows: and ; (2) Similarity measurement; Calculate the Euclidean distance between the pooled features of each non-friendly sample and friendly sample, and consider instances with larger distances as pseudo-friendly and instances with smaller distances as pseudo-normal. Finally, generate pseudo-friendly and pseudo-normal instances for each modality, denoted as follows: , , , ; (3) Contrastive learning based on triple loss; Choose two anchor points, which are pseudo-friendly features. and pseudo-normal features Construct two sets of comparison objects, denoted as {anchor point, positive sample, negative sample}, and represent them respectively for audio modalities as { , , } and{ , Finally, the triplet loss is used to calculate the contrast loss, and the calculation method is as follows: ； Where N represents the number of non-friendly labeled samples in a batch, B represents the number of batches, || || represents Euclidean distance, [ The plus sign indicates that the value within the parentheses equals the value itself if the operation is greater than 0; otherwise, it equals 0. α represents the margin hyperparameter of the ternary loss. The target loss L is calculated and optimized using the modal step loss compensation method. MF The calculation method is as follows: ； in, and These represent the original loss values ​​for inter-modal features and original modal features, respectively. This represents the calculation of binary cross-entropy. These represent three linear layers. This represents the sigmoid function. and These represent the loss compensation values ​​for inter-modal features and original modal features, respectively. .

2. The method for detecting unfriendly events in audio and video according to claim 1, characterized in that: In step 1, the visual pre-trained network includes sequentially connected 3D convolutional layers, downsampling layers, Inception module layers, temporal pooling layers, and two-stream architecture layers; The Inception module layer consists of three Inception modules connected in sequence; each Inception module consists of several parallel convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation layer. The dual-stream architecture layer consists of a parallel RGB stream processing module and an optical stream processing module.

3. The method for detecting unfriendly events in audio and video according to claim 1, characterized in that: In step 1, the audio pre-trained network consists of a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a third convolutional layer, a fourth convolutional layer, a third max pooling layer, a global average pooling layer, and a fully connected layer connected in sequence.

4. The method for detecting unfriendly events in audio and video according to claim 1, characterized in that: In step 2, the self-attention network consists of a multi-head attention mechanism layer and a feedforward neural network layer; The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads; The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function. Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation; the residual connection layer is used to add the sub-layer to the input.

5. The method for detecting unfriendly events in audio and video according to claim 1, characterized in that: In step 3, the cross-attention network consists of a multi-head attention mechanism layer and a feedforward neural network layer; The multi-head attention mechanism layer is a linear layer containing four multi-head attention heads; The feedforward neural network layer consists of two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function. Both the multi-head attention mechanism layer and the feedforward neural network layer are followed by a normalization layer and a residual connection layer. The normalization layer is used to calculate the mean and variance of each sample in the feature dimension and perform normalization calculation; the residual connection layer is used to add the sub-layer to the input.

6. A system for detecting unfriendly events in audio and video, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method for detecting unfriendly events in audio and video as described in any one of claims 1 to 5.

Images