Adverse video classification method and system based on multi-modal alignment and class balancing

Abstract

The application discloses a bad video classification method and system based on multi-modal alignment and category balance, which inputs a data set into a bad video classification model, trains the bad video classification model, and abbreviates the bad video classification model as a model; when the total loss function value of the model no longer decreases, a trained bad video classification model is obtained; in the training process, the model first extracts visual mixed features and audio mixed features from a video; the total loss function of the model includes the weighted sum result of first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; and the third sub-loss function is used for improving the minimum distance between the model decision boundary and sample points and reducing the influence of category imbalance on the training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bad video classification, in particular to a bad video classification method and system based on multi-modal alignment and category balance. BACKGROUND

[0002] Short video platforms have attracted a large number of users and risen rapidly due to their rich content and short duration, but the highly free creative environment inevitably causes problems in the supervision of platform works, making it extremely difficult to effectively supervise short video content. Among them, preventing the spread of sexual suggestion videos has strong practical significance. The reason is that the opposite category of sexual education videos plays an important role in eliminating human sexual shame and focusing on human sexual health, but criminals use various methods to evade supervision, making the boundaries between sexual suggestion videos, sexual education videos and normal videos blurred, increasing the difficulty of identification. Therefore, in order to maintain social interests and public mental health, it is particularly urgent to use artificial intelligence technology to propose an effective and accurate bad video classification method.

[0003] At present, traditional single-modal video classification methods are difficult to cope with bad short video classification, and the reasons can be summarized as follows:

[0004] 1) Single-modal video classification methods mainly rely on the difference of the same modality inside the video to distinguish videos of different categories. For example, in a certain classification task, the model can usually classify according to specific facilities in the scene and human actions with differences, such as identifying different sports activities such as high jump and long jump in a sports video classification task. However, unlike videos of concrete categories, sexual suggestion videos, sexual education videos and normal videos are an abstract concept, which do not have unique scenes and human activities that are easy for algorithm classification, and it is difficult to extract concrete common features. Video creators can complete the popularization of sexual knowledge and the sharing of daily interesting things in any scene and without specific human actions; and the content of normal videos can be divided into various daily activities, virtual special effects, and even animal and plant activities, and the visual content also has a high degree of diversity, so it is extremely difficult to abstract consistency from single-modal information for all normal videos.

[0005] 2) In actual application scenarios, the number of normal videos has a natural advantage over sexual suggestion and sexual education videos, which brings not small challenges to the training of the model. The reason is that in order to achieve the fastest benefit, the model tends to optimize common categories and ignore the learning of rare categories. This biased learning causes the model to ignore the features of rare categories and fail to mine the unique features of different categories, causing the model to overfit to common categories. SUMMARY

[0006] To solve the problems in the prior art, the application provides a bad video classification method and system based on multi-modal alignment and category balance.

[0007] In one aspect, the application provides a bad video classification method based on multi-modal alignment and category balance, comprising:

[0008] obtaining a data set, wherein the data set is a video with a known video classification label;

[0009] inputting the data set into a bad video classification model to train the bad video classification model, wherein the bad video classification model is referred to as a model; stopping the training when a total loss function value of the model no longer decreases or when a number of iterations reaches a set number, to obtain a trained bad video classification model; in the training process, the model first extracts visual mixed features and audio mixed features from the video; the total loss function of the model comprises a weighted sum result of first, second and third sub-loss functions, wherein the first sub-loss function refers to a difference between a Gaussian distribution of the visual mixed features and a Gaussian distribution of the audio mixed features; the second sub-loss function refers to a difference between the visual mixed features and the audio mixed features; and the third sub-loss function is used to increase a minimum distance between a model decision boundary and sample points and to reduce an impact of category imbalance on the training process;

[0010] inputting a video to be classified into the trained bad video classification model to obtain a classification result of the video to be classified.

[0011] In another aspect, the application provides a bad video classification system based on multi-modal alignment and category balance, comprising:

[0012] a obtaining module configured to obtain a data set, wherein the data set is a video with a known video classification label;

[0013] a training module configured to input the data set into a bad video classification model to train the bad video classification model, wherein the bad video classification model is referred to as a model; stop the training when a total loss function value of the model no longer decreases or when a number of iterations reaches a set number, to obtain a trained bad video classification model; in the training process, the model first extracts visual mixed features and audio mixed features from the video; the total loss function of the model comprises a weighted sum result of first, second and third sub-loss functions, wherein the first sub-loss function refers to a difference between a Gaussian distribution of the visual mixed features and a Gaussian distribution of the audio mixed features; the second sub-loss function refers to a difference between the visual mixed features and the audio mixed features; and the third sub-loss function is used to increase a minimum distance between a model decision boundary and sample points and to reduce an impact of category imbalance on the training process;

[0014] The classification module is configured to input the video to be classified into the trained bad video classification model to obtain a classification result of the video to be classified.

[0015] In another aspect, an electronic device is also provided, comprising:

[0016] a memory for non-transitorily storing computer readable instructions; and

[0017] a processor for running the computer readable instructions,

[0018] When the computer readable instructions are run by the processor, the method of the first aspect is executed.

[0019] In another aspect, a storage medium is also provided, which non-transitorily stores computer readable instructions, wherein when the non-transitory computer readable instructions are executed by a computer, the instructions of the method of the first aspect are executed.

[0020] In another aspect, a computer program product is also provided, comprising a computer program for implementing the method of the first aspect when run on one or more processors.

[0021] The above technical solution has the following advantages or beneficial effects:

[0022] In order to enrich the amount of single modal information and solve the problem of insufficient single modal information representation, the application designs a multi-modal representation distribution alignment mechanism. The mechanism reduces the unique information in the visual modal that is irrelevant to the commonality of the category by mapping the representations of different modes to similar feature distributions, and then refines the common features of the videos of the same category.

[0023] In order to solve the problem of training bias caused by the imbalance of the number of categories, the application introduces a label distribution aware training strategy. The strategy learns the best trade-off between the margins of each category by allocating more space to the tail categories, thereby balancing the training process.

[0024] In order to solve the problem of insufficient single modal information representation, the application designs a multi-modal representation distribution alignment mechanism, which can effectively reduce the unique information in the visual modal that is irrelevant to the commonality of the category, and then refine the common features of the videos of the same category. In order to reduce the negative impact of the imbalance of the number of categories on model training, the application introduces a label distribution aware training strategy, which balances the optimization direction of each category by allocating more space to the tail categories. Compared with the previous method, the application improves the classification accuracy of the bad video classification task. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of the application, and their

[0026] Figure 1 The method flowchart of Example One. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0028] Example One

[0029] The present embodiment provides a bad video classification method based on multi-modal alignment and class balancing;

[0030] As shown in the bad video classification method based on multi-modal alignment and class balancing, comprising: Figure 1 S101: obtaining a data set, wherein the data set is a video with known video classification labels;

[0031] S102: inputting the data set into a bad video classification model to train the bad video classification model, wherein the bad video classification model is referred to as a model; when the total loss function value of the model no longer decreases, or the number of iterations reaches a set number of times, the training is stopped, and a trained bad video classification model is obtained;

[0032] During the training process, the model first extracts visual mixed features and audio mixed features from the video; the total loss function of the model includes a weighted sum result of first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; and the third sub-loss function is used to improve the minimum distance between the model decision boundary and the sample points, and to reduce the influence of class imbalance on the training process;

[0033] S103: inputting a video to be classified into the trained bad video classification model to obtain a classification result of the video to be classified.

[0034] Further, the S101: obtaining a data set, wherein the data set is a video with known video classification labels; and the video classification labels include bad video labels and normal video labels.

[0035]

[0036] ​Further, the S102: input the data set into the bad video classification model, train the bad video classification model, the bad video classification model comprises:

[0037] The visual feature extraction module, the audio feature extraction module and the classifier;

[0038] The visual feature extraction module is configured to extract visual mixed features by performing feature extraction on the video.

[0039] The audio feature extraction module is configured to extract audio mixed features by performing feature extraction on the audio in the video.

[0040] The classifier is configured to output a video classification result.

[0041] Further, the visual feature extraction module comprises:

[0042] The visual feature extractor (CLIP, Contrastive Language-Image Pretraining) has the video as an input, and outputs visual feature encoding. The output of the visual feature extractor (CLIP) is connected to the input of the first long short-term memory network (LSTM). The first long short-term memory network (LSTM, Long Short-Term Memory) extracts features from the visual feature encoding to obtain visual global features and visual sequence features. The visual sequence features are subjected to average pooling processing to obtain visual average sequence features. The visual global features and the visual average sequence features are fused to obtain visual mixed features.

[0043] The LSTM is a special type of recurrent neural network, mainly used for processing and predicting sequence data. The visual sequence features come from the hidden state of each time step of the LSTM, which can be understood as a string of features generated at each time step. The visual global features are usually output from the hidden state of the last time step.

[0044] Further, the visual feature extraction module comprises a visual feature processor (CLIP) and a long short-term memory network. The visual feature extractor is configured to extract visual feature encoding. The long short-term memory network is configured to capture dynamic information changing over time in the video.

[0045] Further, the visual feature extraction module comprises:

[0046] The uncut video is denoted as f tis the image of WxH for the t-th frame in the video. First, a visual feature extractor (CLIP) pretrained on the ImageNet dataset is adopted to extract visual feature codes from the uncropped video

[0047] Then, to further improve the model's understanding of the temporal dynamics of the video content and to mine the temporal information in the video, a long short-term memory network is used to obtain global visual features and local visual features

[0048] Finally, the global visual features and the local visual features are fused to obtain the visual mixed features of the video

[0049]

[0050]

[0051] Further, the audio feature extraction module comprises:

[0052] A preprocessing unit is configured to perform short-time Fourier transform, filtering processing, and Hamming window segmentation processing on the audio separated from the video to obtain a frequency spectrum, input the frequency spectrum into an audio feature processor (AST), and obtain audio feature codes. An output end of the audio feature processor is connected to an input end of a second long short-term memory network (LSTM). The second long short-term memory network (LSTM) is configured to extract features from the audio feature codes to obtain audio global features and audio sequence features. The audio sequence features are subjected to average pooling processing to obtain audio average sequence features. The audio global features and the audio average sequence features are fused to obtain audio mixed features.

[0053] The LSTM is a special type of recurrent neural network mainly used for processing and predicting sequence data. The visual sequence features are derived from the hidden state of each time step of the LSTM and can be understood as a string of features generated at each time step. The visual global features are usually output from the hidden state of the last time step.

[0054] Further, the audio feature extraction module comprises a multimedia processing tool (FFMPEG), an audio feature processor (AST), and a long short-term memory network (LSTM). The multimedia processing tool is configured to separate the audio track from the video. The audio feature processor is configured to extract audio feature codes. The long short-term memory network is configured to capture dynamic information that changes over time in the audio.​​

[0055] Further, the audio feature extraction module comprises:

[0056] The audio separated from the video by using the multimedia processing tool FFMPEG is denoted as

[0057] Firstly, the audio sequence of T seconds is converted into a spectrogram which retains both time information and frequency information by short-time Fourier transform (STFT), Mel filter bank and Hamming window operation

[0058] Then, the spectrogram sequence is input into an audio feature processor (AST) which is pre-trained on the AudioSet dataset, and the features in the audio sequence are gradually extracted by the stacked self-attention layers and fully connected layers of the audio feature processor to obtain audio feature encoding

[0059] After that, the global audio feature a of the audio is obtained by using a long short-term memory network g and the local audio feature

[0060] Finally, the global audio feature and the local audio feature are fused to obtain the audio mixed feature of the video

[0061]

[0062]

[0063] Further, the classifier processes the final visual feature to obtain the accurate classification result of the video.

[0064] The classifier is a first linear layer, a second linear layer and an activation function ReLu, which calculates the class probability of the current video by deeply analyzing the information of the visual feature wherein C is the number of classes in the dataset.

[0065]

[0066] Further, the total loss function of the model comprises the weighted sum of the first, second and third sub-loss functions, and the total loss function of the model comprises:

[0067] L = L Mce + λ1L dis + λ2L sim

[0068] ​wherein, λ1 and λ2 are hyper-parameter coefficients to complete the weight distribution of different loss functions, avoid the dominance of a single loss function in the whole training, L Mce represents the first sub-loss function, L dis represents the second sub-loss function, L sim represents the third sub-loss function.

[0069] It should be understood that, in order to avoid the dominance of a single loss function in the whole training process in the multi-loss function training process, by weighing the importance of different loss functions, according to the magnitude of each loss function, the appropriate weight distribution of different loss functions is completed, so as to ensure the balance of the contribution of each loss function in the model training process, thereby realizing more comprehensive model optimization.

[0070] In order to effectively utilize the information hidden in the multi-modal feature space, to weaken the specific information in the visual modal irrelevant to the commonality of the category, and to extract the common information of the category in the visual modal. The distribution alignment mechanism of multi-modal representation completes the accurate alignment of visual modal and audio modal from the two dimensions of statistical distribution and feature representation.

[0071] Further, the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed feature and the Gaussian distribution of the audio mixed feature, including:

[0072] Firstly, the two mapping functions and are used to map the visual mixed feature and the audio mixed feature to the same vector space D, The structure of the mapping function and is a linear layer and a ReLU activation function connected in turn.

[0073] Next, it is assumed that there are two Gaussian kernel functions and respectively from the feature representation v of the visual modal and the feature representation a of the audio modal, the Gaussian distribution of the visual modal and the Gaussian distribution of the audio modal are counted. The specific architecture of the Gaussian kernel function and is a linear layer and a ReLU activation function connected in turn,

[0074]

[0075] Finally, in order to complete the accurate alignment of the visual modal distribution and the audio modal distribution, by optimizing with KL divergence between the two to minimize the difference between them.

[0076]

[0077] Multi-modal feature consistency learning achieves precise alignment of visual and audio modalities in feature representation dimension by aligning the inner similarity between visual feature representation distribution and audio feature representation distribution.

[0078] Further, the second sub-loss function refers to the difference between the visual mixed feature and the audio mixed feature, including:

[0079] First, learn the synthesized feature of the visual modality and the synthesized feature of the audio modality

[0080]

[0081] wherein, follows a normal distribution, the mapping function and is used to map the synthesized feature of the visual modality and the audio modality from the feature space of to the feature space of , the mapping function and The structure of is a linear layer and a ReLU activation function connected in turn.

[0082] Then, the intra-feature similarity matrix of the visual modality and the intra-feature similarity matrix of the audio modality between two videos are solved.

[0083]

[0084] wherein, S v is the cosine similarity matrix between videos, S a is the cosine similarity matrix between audios, is the cosine similarity score of the visual modality feature of the mth video and the nth video, is the cosine similarity score of the audio feature of the mth video and the nth video, are the video local features of the m videos and the nth video, respectively, are the audio local features of the m audios and the nth audio, respectively.

[0085] Finally, in order to complete the precise alignment of the video modality and the audio modality in the feature representation, S v and S aThe distance between the two cosine similarity matrices is used to minimize the gap between them, and M is the total number of videos and audios under the current batch size.

[0086]

[0087] It should be understood that the multi-modal representation distribution alignment mechanism is divided into multi-modal distribution consistency learning and multi-modal feature consistency learning. Multi-modal distribution consistency learning reduces the difference between the Gaussian distribution of visual feature representation and the Gaussian distribution of audio feature representation from a macroscopic perspective. Multi-modal feature consistency learning further reduces the difference between visual feature representation and audio feature representation from the details of feature representation.

[0088] Further, the third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, and to reduce the influence of class imbalance on the training process, including:

[0089] First, obtain the total number of samples of each class in the data set c i is the total number of samples of the i-th class;

[0090] Then, define the hyperparameter coefficient is the maximum boundary value, and the boundary value of each sample is calculated

[0091]

[0092] Finally, the balanced cross-entropy loss with label distribution awareness is as follows:

[0093]

[0094] where β is the hyperparameter coefficient, p i is the possibility size that the current model considers the video to be of the i-th class, and p j is the possibility size that the current model considers the video to be of the j-th class.

[0095] It should be understood that in order to enhance the generalization performance of the model and reduce the adverse effects of class imbalance on the training process, a label distribution aware training strategy is adopted, which replaces the traditional cross-entropy loss function by minimizing the boundary-based generalization error bound. Specifically, this method increases the boundary value of the minority class samples (i.e., increases the minimum distance between the model decision boundary and the sample point), thereby giving these sample classes more marginal space. In order to improve the generalization ability of the minority class and reduce the negative impact of class imbalance on model training, the loss function is adjusted by minimizing the boundary-related generalization bound. On the basis of the standard cross-entropy loss, an additional encouragement for the boundary of the minority class is introduced.

[0096] The application discloses a bad video classification method based on multi-modal alignment and category balance, comprising the following steps: 1) a multi-modal feature extraction module; 2) multi-modal feature consistency learning; and 3) a label distribution-aware training strategy. The application has higher accuracy than existing models in the bad video classification problem.

[0097] Embodiment two

[0098] The embodiment provides a bad video classification system based on multi-modal alignment and category balance, comprising:

[0099] An acquisition module is configured to acquire a data set, wherein the data set is a video with a known video classification label.

[0100] A training module is configured to input the data set into a bad video classification model, train the bad video classification model, and obtain a trained bad video classification model when a total loss function value of the model no longer decreases or when the number of iterations reaches a set number of times. In the training process, the model first extracts visual mixed features and audio mixed features from the video. The total loss function of the model comprises a weighted sum result of a first, a second and a third sub-loss function. The first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features. The second sub-loss function refers to the difference between the visual mixed features and the audio mixed features. The third sub-loss function is used to improve the minimum distance between the model decision boundary and the sample point and reduce the influence of category imbalance on the training process.

[0101] A classification module is configured to input a video to be classified into the trained bad video classification model to obtain a classification result of the video to be classified.

[0102] It should be noted that the acquisition module, the training module and the classification module correspond to steps S101 to S103 in the embodiment one, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment one. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0103] The description of each embodiment in the above embodiments has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0104] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, and the division of the above modules is only a logical functional division. In actual implementation, there can be another division manner, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0105] Embodiment three

[0106] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method in the above embodiment one.

[0107] It should be understood that, in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0108] The memory can include read-only memory and random access memory, and provide instructions and data for the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0109] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0110] The method in the embodiment one can be directly embodied as hardware processor execution completion, or combined execution completion by hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, and other mature storage media in the field. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0111] Those skilled in the art can understand that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0112] Embodiment Four

[0113] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment One.

[0114] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for classifying bad videos based on multimodal alignment and class balance, characterized by: include: Obtain the dataset, which consists of videos with known video classification labels; The dataset is input into the inappropriate video classification model to train the inappropriate video classification model, which is referred to as the model. When the total loss function value of the model no longer decreases, or when the number of iterations reaches the set number, the training is stopped, and the trained inappropriate video classification model is obtained. During training, the model first extracts visual and audio mixture features from the video; The model's total loss function includes: a weighted sum of the first, second, and third sub-loss functions. The first sub-loss function is the difference between the Gaussian distributions of the visual mixture features and the audio mixture features; the second sub-loss function is the difference between the visual mixture features and the audio mixture features; and the third sub-loss function is used to improve the minimum distance between the model's decision boundary and the sample points, thereby mitigating the impact of class imbalance on the training process. The total loss function of the model includes: a weighted sum of the first, second, and third sub-loss functions; the total loss function of the model includes: ； in, and These are hyperparameter coefficients used to assign weights to different loss functions, preventing a single loss function from dominating the entire training process. Denotes the first sub-loss function. Denotes the second sub-loss function. Represents the third sub-loss function; The first sub-loss function refers to the difference between the Gaussian distribution of visual mixture features and the Gaussian distribution of audio mixture features, including: First, using two mapping functions and Visual blending features and audio mixing features Mapped to the same vector space middle, , , , Mapping function and The structures are all sequentially connected linear layers and a ReLU activation function; Next, assume there exist two Gaussian kernel functions. as well as From the feature representation of visual modality respectively and the feature representation of audio modes In the process, the Gaussian distribution of visual modalities was statistically determined. and the Gaussian distribution of audio modes Among them, the Gaussian kernel function as well as The specific architecture consists of a single linear layer and a ReLU activation function connected in series. , ; , ; Finally, to achieve precise alignment of the visual modality distribution and the audio modality distribution, optimization was performed. and The KL divergence between them is used to minimize the difference between them; ； The second sub-loss function refers to the difference between visual blending features and audio blending features, including: First, learn the synthetic features of visual modalities. and the synthesis features of audio modalities ; in, Following a normal distribution, the mapping function and Used to synthesize features from visual and audio modalities Feature space mapping to In the feature space, the mapping function and The structure consists of a linear layer connected in series with a ReLU activation function; Next, the visual modal feature similarity matrix and the audio modal feature similarity matrix between each pair of videos are calculated. in, The cosine similarity matrix between the videos. The cosine similarity matrix between audio tracks. For the first The video and the first Cosine similarity score of visual modal features of each video For the first The video and the first The cosine similarity score of the audio features of each video. They are respectively The video and the first Local features of a video. , They are respectively The audio and the first Local audio features of an audio file; Finally, to achieve accurate alignment of video and audio modalities in feature representation, optimization was performed. and The distance between two cosine similarity matrices is used to minimize the difference between them. This represents the total number of videos and audio files within the current batch size. ； The video to be classified is input into the trained bad video classification model to obtain the classification result of the video to be classified.

2. The bad video classification method based on multimodal alignment and category balancing as described in claim 1, characterized in that, The dataset is input into the inappropriate video classification model, and the inappropriate video classification model is trained. The inappropriate video classification model includes: Visual feature extraction module, audio feature extraction module, and classifier; The visual feature extraction module is used to extract features from the video and extract visual hybrid features; The audio feature extraction module is used to extract features from the audio in the video and extract mixed audio features; the output of the visual feature extraction module is connected to the input of the classifier. The classifier is used to output video classification results.

3. The bad video classification method based on multimodal alignment and category balance as described in claim 2, characterized in that, The visual feature extraction module includes: A visual feature extractor, whose input is video, outputs visual feature codes. The output of the visual feature extractor is connected to the input of a first Long Short-Term Memory (LSTM) network. The LSM network extracts features from the visual feature codes to obtain global visual features and visual sequence features. The visual sequence features are then subjected to average pooling to obtain average visual sequence features. Finally, the global visual features and the average visual sequence features are fused to obtain a mixed visual feature. Alternatively... The visual feature extraction module includes: recording the uncropped video as... , The first in the video frame The images; first, a visual feature extractor pre-trained on an image dataset is used to extract features from uncropped videos. Extract visual feature encoding ; Subsequently, to further enhance the model's ability to understand the temporal dynamics of video content and to extract temporal information from the video, a Long Short-Term Memory (LSTM) network was used to obtain the global visual features of the video. and local visual features Finally, global and local visual features are fused: Obtain visual blending features of the video .

4. The bad video classification method based on multimodal alignment and category balance as described in claim 2, characterized in that, The audio feature extraction module includes: The preprocessing unit performs short-time Fourier transform, filtering, and Hamming window segmentation on the audio separated from the video to obtain a spectrogram. This spectrogram is then input into the audio feature processor to obtain audio feature codes. The output of the audio feature processor is connected to the input of a second long short-term memory network. The second long short-term memory network extracts features from the audio feature codes to obtain global audio features and audio sequence features. The audio sequence features are then subjected to average pooling to obtain average audio sequence features. Finally, the global audio features and the average audio sequence features are fused to obtain mixed audio features; or... The audio feature extraction module includes: separating the audio from the video and denoting it as... ; First, after short-time Fourier transform, Mel filter bank, and Hamming window operation, Convert a second-level audio sequence into a spectrogram that simultaneously retains both time and frequency information. ; Next, using an audio feature processor pre-trained on the dataset, the spectrogram sequence is input into the audio feature processor, and features in the audio sequence are extracted step by step through its stacked self-attention layers and fully connected layers to obtain audio feature encoding. ; Subsequently, a long short-term memory network was used to obtain the global audio features of the audio. and local audio features Finally, global and local audio features are fused: Obtain the audio mixing features of the video .

5. The bad video classification method based on multimodal alignment and category balancing as described in claim 1, characterized in that, The third sub-loss function is used to improve the minimum distance between the model's decision boundary and the sample points, and to mitigate the impact of class imbalance on the training process, including: First, obtain the total number of samples in each category of the dataset. , For the first The total number of samples in each category; Next, define the hyperparameter coefficients. To obtain the maximum boundary value, calculate the boundary value for each sample. : Finally, the label distribution-aware balanced cross-entropy loss is as follows: in, These are hyperparameter coefficients. The current model considers this video to be the first... The probability size of each category This represents the probability that the current model considers the video to belong to the j-th category.

6. A bad video classification system based on multimodal alignment and class balance, characterized by: include: The acquisition module is configured to acquire a dataset, wherein the dataset consists of videos with known video classification labels; The training module is configured to: input the dataset into the inappropriate video classification model, train the inappropriate video classification model (hereinafter referred to as the model); stop training when the total loss function value of the model no longer decreases, or when the number of iterations reaches a set number, and obtain the trained inappropriate video classification model. During training, the model first extracts visual and audio mixture features from the video; The model's total loss function includes: a weighted sum of the first, second, and third sub-loss functions. The first sub-loss function is the difference between the Gaussian distributions of the visual mixture features and the audio mixture features; the second sub-loss function is the difference between the visual mixture features and the audio mixture features; and the third sub-loss function is used to improve the minimum distance between the model's decision boundary and the sample points, thereby mitigating the impact of class imbalance on the training process. The total loss function of the model includes: a weighted sum of the first, second, and third sub-loss functions; the total loss function of the model includes: ； in, and These are hyperparameter coefficients used to assign weights to different loss functions, preventing a single loss function from dominating the entire training process. Denotes the first sub-loss function. Denotes the second sub-loss function. Represents the third sub-loss function; The first sub-loss function refers to the difference between the Gaussian distribution of visual mixture features and the Gaussian distribution of audio mixture features, including: First, using two mapping functions and Visual blending features and audio mixing features Mapped to the same vector space middle, , , , Mapping function and The structures are all sequentially connected linear layers and a ReLU activation function; Next, assume there exist two Gaussian kernel functions. as well as From the feature representation of visual modality respectively and the feature representation of audio modes In the process, the Gaussian distribution of visual modalities was statistically determined. and the Gaussian distribution of audio modes Among them, the Gaussian kernel function as well as The specific architecture consists of a single linear layer and a ReLU activation function connected in series. , ; , ; Finally, to achieve precise alignment of the visual modality distribution and the audio modality distribution, optimization was performed. and The KL divergence between them is used to minimize the difference between them; ； The second sub-loss function refers to the difference between visual blending features and audio blending features, including: First, learn the synthetic features of visual modalities. and the synthesis features of audio modalities ; in, Following a normal distribution, the mapping function and Used to synthesize features from visual and audio modalities Feature space mapping to In the feature space, the mapping function and The structure consists of a linear layer connected in series with a ReLU activation function; Next, the visual modal feature similarity matrix and the audio modal feature similarity matrix between each pair of videos are calculated. in, The cosine similarity matrix between the videos. The cosine similarity matrix between audio tracks. For the first The video and the first Cosine similarity score of visual modal features of each video For the first The video and the first The cosine similarity score of the audio features of each video. They are respectively The video and the first Local features of a video. , They are respectively The audio and the first Local audio features of an audio file; Finally, to achieve accurate alignment of video and audio modalities in feature representation, optimization was performed. and The distance between two cosine similarity matrices is used to minimize the difference between them. This represents the total number of videos and audio files within the current batch size. ； The classification module is configured to input the video to be classified into the trained bad video classification model and obtain the classification result of the video to be classified.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium characterized by being non-transitory. The computer-readable instructions are stored, wherein, when executed by a computer, the instructions of the method according to any one of claims 1-5 are executed.

Images