Bad video classification method and system based on multi-modal alignment and category balance

Abstract

The invention discloses a bad video classification method and system based on multi-modal alignment and category balance, and the method comprises the steps: inputting a data set into a bad video classification model, training the bad video classification model, and enabling the bad video classification model to be called a model for short; when the total loss function value of the model does not decrease any more, a trained bad video classification model is obtained; in the training process, the model firstly extracts visual mixed features and audio mixed features from a video; the total loss function of the model comprises the weighted summation result of a first sub-loss function, a second sub-loss function and a third sub-loss function, and the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixing feature and the audio mixing feature; and the third sub-loss function is used for increasing the minimum distance between a model decision boundary and a sample point and reducing the influence of class imbalance on the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of bad video classification, and in particular to a bad video classification method and system based on multimodal alignment and category balance. Background Art

[0002] Short video platforms have attracted a large number of users with their rich content and short duration, and have risen rapidly. However, their highly free creative environment has inevitably caused regulatory problems for platform works, making it extremely difficult to effectively regulate short video content. Among them, preventing the spread of sexually suggestive videos has a strong practical significance. The reason is that its opposite category, sex education videos, has played an important role in eliminating human sexual shame and paying attention to human sexual health. However, in order to circumvent supervision, criminals have adopted various methods to blur the boundaries between sexually suggestive videos, sex education videos, and normal videos, increasing the difficulty of identification. Therefore, in order to safeguard social interests and public mental health, it is particularly urgent to use artificial intelligence technology to propose an effective and accurate method for classifying bad videos.

[0003] At present, traditional single-modal video classification methods are difficult to deal with bad short video classification. The reasons can be summarized as follows:

[0004] 1) Single-modal video classification methods mainly rely on the differential features of the same modality within the video to distinguish videos of different categories. For example, in a specific classification task, the model can usually classify according to specific facilities in the scene and different human movements, such as identifying different sports activities such as high jump and long jump in the sports video classification task. However, unlike the concrete category of videos, sexually suggestive videos, sex education videos and normal videos are an abstract concept. They do not have unique scenes and human activities that are easy for algorithms to classify, and it is difficult to extract concrete common features. Video creators can complete activities such as popularizing sexual knowledge and sharing daily fun in any scene and without specific human movements; and the content of normal videos can be subdivided into various daily activities, virtual special effects and even activities of animals and plants, etc., and its visual content is also highly diverse. It is extremely difficult to abstract consistency from all normal videos relying only on single modal information.

[0005] 2) In actual application scenarios, the number of normal videos is naturally superior to that of sexually suggestive and sex education videos, which poses a considerable challenge to model training. The reason is that in order to reap the benefits as quickly as possible, the model tends to optimize common categories and ignore the learning of rare categories. This biased learning causes the model to ignore the features of rare categories and fail to explore the unique features of different categories, causing the model to overfit to common categories. Summary of the invention

[0006] In order to solve the deficiencies of the prior art, the present invention provides a method and system for classifying bad videos based on multimodal alignment and category balance;

[0007] On the one hand, a method for classifying bad videos based on multimodal alignment and category balance is provided, including:

[0008] Obtain a data set, wherein the data set is a video with known video classification labels;

[0009] Input the data set into the bad video classification model, train the bad video classification model, and refer to the bad video classification model as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches the set number, stop the training, and obtain the trained bad video classification model; during the training process, the model first extracts the visual mixed features and the audio mixed features from the video; the total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; the third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, and reduce the impact of category imbalance on the training process;

[0010] The video to be classified is input into the trained bad video classification model to obtain the classification result of the video to be classified.

[0011] On the other hand, a bad video classification system based on multimodal alignment and category balance is provided, including:

[0012] An acquisition module is configured to: acquire a data set, wherein the data set is a video with a known video classification label;

[0013] The training module is configured to: input the data set into the bad video classification model, train the bad video classification model, and refer to the bad video classification model as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches the set number, stop the training to obtain the trained bad video classification model; during the training process, the model first extracts the visual mixed features and the audio mixed features from the video; the total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; the third sub-loss function is used to improve the minimum distance between the model decision boundary and the sample point, and reduce the impact of category imbalance on the training process;

[0014] The classification module is configured to: input the video to be classified into the trained bad video classification model to obtain the classification result of the video to be classified.

[0015] On the other hand, there is also provided an electronic device, comprising:

[0016] a memory for non-transitory storage of computer-readable instructions; and

[0017] a processor for executing the computer readable instructions,

[0018] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0019] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0020] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0021] The above technical solution has the following advantages or beneficial effects:

[0022] In order to enrich the amount of information in a single modality and solve the problem of insufficient representation of single modality information, the present invention designs a multimodal representation distribution alignment mechanism. This mechanism maps the representations of different modalities to similar feature distributions, reduces the unique information in the visual modality that is irrelevant to the commonality of the category, and then refines the common features of videos of the same category.

[0023] In order to solve the problem of training bias caused by the imbalanced distribution of the number of categories, this paper introduces a label distribution-aware training strategy, which achieves a balanced training process by allocating more space to the tail categories and learning the optimal trade-off between the margins of each category.

[0024] In order to solve the problem of insufficient representation of single modality information, the present invention designs a multimodal representation distribution alignment mechanism, which can effectively reduce the unique information in the visual modality that is irrelevant to the commonality of the category, and then refine the common features of videos of the same category. In order to reduce the negative impact of the imbalanced distribution of the number of categories on model training, the present invention introduces a label distribution-aware training strategy, which achieves a balance in the optimization direction of each category by allocating more space to the tail category. Compared with previous methods, the present invention improves the classification accuracy of bad video classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0028] Embodiment 1

[0029] This embodiment provides a bad video classification method based on multimodal alignment and category balance;

[0030] like Figure 1 As shown, the bad video classification method based on multimodal alignment and category balance includes:

[0031] S101: Acquire a data set, where the data set is videos with known video classification labels;

[0032] S102: inputting the data set into the bad video classification model, training the bad video classification model, and the bad video classification model is referred to as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches a set number, stopping the training, and obtaining the trained bad video classification model;

[0033] During the training process, the model first extracts visual mixed features and audio mixed features from the video; the total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, where the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; the third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, reducing the impact of category imbalance on the training process;

[0034] S103: Input the video to be classified into the trained bad video classification model to obtain a classification result of the video to be classified.

[0035] Furthermore, the S101: obtaining a data set, the data set is a video with a known video classification label; the video classification label includes: a bad video label and a normal video label.

[0036] Furthermore, the S102: inputting the data set into a bad video classification model to train the bad video classification model, wherein the bad video classification model includes:

[0037] Visual feature extraction module, audio feature extraction module and classifier;

[0038] The visual feature extraction module is used to extract features from the video and extract visual mixed features;

[0039] The audio feature extraction module is used to extract features from the audio in the video and extract audio mixed features; the output end of the visual feature extraction module is connected to the input end of the classifier;

[0040] The classifier is used to output video classification results.

[0041] Furthermore, the visual feature extraction module includes:

[0042] A visual feature extractor (CLIP, Contrastive Language-Image Pretraining), wherein the input end of the visual feature extractor (CLIP) is a video, the output end of the visual feature extractor CLIP outputs a visual feature code, the output end of the visual feature extractor (CLIP) is connected to the input end of a first long short-term memory network (LSTM), the first long short-term memory network (LSTM, Long Short-Term Memory) performs feature extraction on the visual feature code to obtain visual global features and visual sequence features, performs average pooling processing on the visual sequence features to obtain visual average sequence features, and fuses the visual global features with the visual average sequence features to obtain visual mixed features.

[0043] LSTM is a special type of recurrent neural network, which is mainly used to process and predict sequence data. The visual sequence features come from the hidden state of each time step of LSTM, which can be understood as the concatenation of the features generated at each time step. The visual global features are usually output from the hidden state of the last time step.

[0044] Furthermore, the visual feature extraction module includes: a visual feature processor (CLIP) and a long short-term memory network. The visual feature extractor is used to extract visual feature encoding. The long short-term memory network is used to capture dynamic information that changes over time in the video.

[0045] Furthermore, the visual feature extraction module includes:

[0046] The uncut video is recorded as f tis the W×H image of the tth frame in the video. First, a visual feature extractor (CLIP) pre-trained on the image dataset ImageNet is used to extract the feature vector from the uncropped video. Extract visual feature encoding

[0047] Afterwards, in order to further improve the model's ability to understand the temporal dynamics of video content, we mined the temporal information in the video and used the long short-term memory network to obtain the global visual features of the video. And local visual features

[0048] Finally, the global visual features and local visual features are integrated:

[0049]

[0050] Obtaining visual mixed features of the video

[0051] Furthermore, the audio feature extraction module includes:

[0052] A preprocessing unit performs short-time Fourier transform, filtering and Hamming window segmentation on the audio separated from the video to obtain a spectrum graph, and inputs the spectrum graph into an audio feature processor (AST, Audio Spectrogram Transformer) to obtain an audio feature code. The output end of the audio feature processor is connected to the input end of a second long short-term memory network (LSTM). The second long short-term memory network (LSTM) performs feature extraction on the audio feature code to obtain audio global features and audio sequence features, performs average pooling on the audio sequence features to obtain audio average sequence features, and fuses the audio global features with the audio average sequence features to obtain audio mixed features.

[0053] LSTM is a special type of recurrent neural network, which is mainly used to process and predict sequence data. The visual sequence features come from the hidden state of each time step of LSTM, which can be understood as the concatenation of the features generated at each time step. The visual global features are usually output from the hidden state of the last time step.

[0054] Furthermore, the audio feature extraction module includes: a multimedia processing tool (FFMPEG, Fast Forward Moving Picture Experts Group), an audio feature processor (AST) and a long short-term memory network (LSTM). The multimedia processing tool is used to separate the audio track from the video. The audio feature processor is used to extract the audio feature code. The long short-term memory network is used to capture the dynamic information in the audio that changes over time.

[0055] Furthermore, the audio feature extraction module includes:

[0056] The audio separated from the video using the multimedia processing tool FFMPEG is recorded as

[0057] First, the T-second audio sequence is converted into a spectrogram that retains both time and frequency information through short-time Fourier transform (STFT), Mel filter bank, and Hamming window operations.

[0058] Next, the audio feature processor (AST) pre-trained on the AudioSet dataset is used to input the spectrogram sequence into the audio feature processor, and the features in the audio sequence are gradually extracted through its stacked self-attention layer and fully connected layer to obtain the audio feature encoding.

[0059] Afterwards, the long short-term memory network is used to obtain the global audio feature a of the audio g And local audio features

[0060] Finally, the global audio features and local audio features are integrated:

[0061]

[0062] Get the audio mixing characteristics of the video

[0063] Furthermore, the classifier processes the final visual features to obtain an accurate classification result of the video.

[0064] The classifier The first linear layer, the second linear layer and an activation function ReLu calculate the category probability of the current video by deeply analyzing the information of visual features Among them, C is the number of categories in the data set.

[0065]

[0066] Furthermore, the total loss function of the model includes: a weighted sum of the first, second and third sub-loss functions, and the total loss function of the model includes:

[0067] L=L Mce +λ1L dis +λ2L sim

[0068] Among them, λ1 and λ2 are hyperparameter coefficients to complete the weight distribution of different loss functions to avoid a single loss function dominating the entire training. Mce represents the first sub-loss function, L dis represents the second sub-loss function, L sim Represents the third sub-loss function.

[0069] It should be understood that in order to avoid a single loss function dominating the entire training process during the multi-loss function training, the importance of different loss functions is weighed and appropriate weights are assigned to different loss functions according to the magnitude of each loss function to ensure that the contribution of each loss function is balanced during the model training process, thereby achieving a more comprehensive model optimization.

[0070] In order to effectively utilize the information hidden in the multimodal feature space, in order to weaken the unique information in the visual modality that is irrelevant to the commonality of the category, and to extract the common information of the category in the visual modality, the distribution alignment mechanism of multimodal representation completes the precise alignment of the visual modality and the audio modality from the two dimensions of statistical distribution and feature representation.

[0071] Furthermore, the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed feature and the Gaussian distribution of the audio mixed feature, including:

[0072] First, using two mapping functions and Mixing visual features and audio mixing features Mapped into the same vector space D, Mapping Function and The structure is a linear layer and a ReLU activation function connected in series.

[0073] Next, assume that there are two Gaussian kernel functions as well as From the feature representation v of the visual modality and the feature representation a of the audio modality, the Gaussian distribution of the visual modality is statistically calculated. and the Gaussian distribution of the audio mode Among them, the Gaussian kernel function as well as The specific architecture is a series of linear layers and a ReLU activation function.

[0074]

[0075] Finally, in order to achieve accurate alignment of the visual modal distribution and the audio modal distribution, we optimize and The KL divergence between them is used to minimize the difference between them.

[0076]

[0077] Multimodal feature consistency learning achieves precise alignment of the visual modality and the audio modality in the feature representation dimension by aligning the internal similarities between the visual feature representation distribution and the audio feature representation distribution.

[0078] Furthermore, the second sub-loss function refers to the difference between the visual mixed feature and the audio mixed feature, including:

[0079] First, learn synthetic features of the visual modality And the synthetic features of the audio modality

[0080]

[0081] in, Following the normal distribution, the mapping function and It is used to synthesize features of visual and audio modalities from The feature space is mapped to In the feature space, the mapping function and The structure is a series of linear layers and a ReLU activation function.

[0082] Afterwards, the visual modality feature similarity matrix and the audio modality feature similarity matrix between each video are solved.

[0083]

[0084] Among them, S v is the cosine similarity matrix between videos, S a is the cosine similarity matrix between audios, is the cosine similarity score of the visual modality features of the mth video and the nth video, is the cosine similarity score of the audio features of the mth video and the nth video, are the local features of the m videos and the nth video respectively, They are the local audio features of the mth audio and the nth audio respectively.

[0085] Finally, in order to achieve accurate alignment of the video modality and the audio modality in feature representation, by optimizing S v With S aThe distance between the two cosine similarity matrices is used to minimize the gap between the two. M is the total number of videos and audios under the current batch size.

[0086]

[0087] It should be understood that the multimodal representation distribution alignment mechanism is divided into multimodal distribution consistency learning and multimodal feature consistency learning. Multimodal distribution consistency learning reduces the difference between the Gaussian distribution of visual feature representation and the Gaussian distribution of audio feature representation from a macro perspective. Multimodal feature consistency learning further reduces the difference between visual feature representation and audio feature representation from the details of feature representation.

[0088] Furthermore, the third sub-loss function is used to improve the minimum distance between the model decision boundary and the sample point, and reduce the impact of class imbalance on the training process, including:

[0089] First, get the total number of samples for each category in the dataset c i is the total number of samples in the i-th category;

[0090] After that, define the hyperparameter coefficients is the maximum boundary value, calculate the boundary value of each sample

[0091]

[0092] Finally, the label distribution-aware balanced cross entropy loss is as follows:

[0093]

[0094] Among them, β is the hyperparameter coefficient, p i is the probability that the current model considers the video to be of the i-th category, p j is the probability that the current model considers the video to be of the jth category.

[0095] It should be understood that in order to enhance the generalization performance of the model and mitigate the adverse effects of class imbalance on the training process, a label distribution-aware training strategy is adopted to replace the traditional cross-entropy loss function by minimizing the boundary-based generalization error bound. Specifically, this method increases the boundary value of minority class samples (i.e., increases the minimum distance between the model decision boundary and the sample point), thereby giving these sample categories a larger marginal space. In order to improve the generalization ability of minority classes and reduce the negative impact of class imbalance on model training, the loss function is adjusted by minimizing the boundary-related generalization bound, and additional encouragement for the minority class boundary is introduced on the basis of the standard cross-entropy loss.

[0096] The present invention discloses a bad video classification method based on multimodal alignment and category balance, comprising the following steps: 1) a multimodal feature extraction module; 2) multimodal feature consistency learning; 3) a label distribution-aware training strategy. The present invention has a higher accuracy than existing models in the problem of bad video classification.

[0097] Embodiment 2

[0098] This embodiment provides a bad video classification system based on multimodal alignment and category balance, including:

[0099] An acquisition module is configured to: acquire a data set, wherein the data set is a video with a known video classification label;

[0100] The training module is configured to: input the data set into the bad video classification model, train the bad video classification model, and refer to the bad video classification model as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches the set number, stop the training to obtain the trained bad video classification model; during the training process, the model first extracts the visual mixed features and the audio mixed features from the video; the total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed features and the Gaussian distribution of the audio mixed features; the second sub-loss function refers to the difference between the visual mixed features and the audio mixed features; the third sub-loss function is used to improve the minimum distance between the model decision boundary and the sample point, and reduce the impact of category imbalance on the training process;

[0101] The classification module is configured to: input the video to be classified into the trained bad video classification model to obtain the classification result of the video to be classified.

[0102] It should be noted that the acquisition module, training module and classification module described above correspond to steps S101 to S103 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0103] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0105] Embodiment 3

[0106] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0107] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0109] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0110] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0111] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0112] Embodiment 4

[0113] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for classifying bad videos based on multimodal alignment and class balance, characterized by: include: Obtain a data set, wherein the data set is a video with known video classification labels; Input the data set into the bad video classification model, train the bad video classification model, and refer to the bad video classification model as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches a set number, stop the training, and obtain the trained bad video classification model; During the training process, the model first extracts visual mixed features and audio mixed features from the video; The total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed feature and the Gaussian distribution of the audio mixed feature; the second sub-loss function refers to the difference between the visual mixed feature and the audio mixed feature; the third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, and reduce the impact of class imbalance on the training process; The video to be classified is input into the trained bad video classification model to obtain the classification result of the video to be classified.

2. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 1, characterized in that: The data set is input into a bad video classification model to train the bad video classification model, wherein the bad video classification model includes: Visual feature extraction module, audio feature extraction module and classifier; The visual feature extraction module is used to extract features from the video and extract visual mixed features; The audio feature extraction module is used to extract features from the audio in the video and extract audio mixed features; the output end of the visual feature extraction module is connected to the input end of the classifier; The classifier is used to output video classification results.

3. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 2, characterized in that: The visual feature extraction module comprises: A visual feature extractor, wherein the input end of the visual feature extractor is a video, the output end of the visual feature extractor outputs a visual feature code, the output end of the visual feature extractor is connected to the input end of a first long short-term memory network, the first long short-term memory network performs feature extraction on the visual feature code to obtain a visual global feature and a visual sequence feature, performs average pooling processing on the visual sequence feature to obtain a visual average sequence feature, and fuses the visual global feature with the visual average sequence feature to obtain a visual mixed feature; or The visual feature extraction module includes: recording the uncut video as f t is the image of the tth frame W×H in the video; first, a visual feature extractor pre-trained on an image dataset is used to extract the visual feature code from the uncropped video v Afterwards, in order to further improve the model's ability to understand the temporal dynamics of video content, we mined the temporal information in the video and used the long short-term memory network to obtain the global visual features of the video. And local visual features Finally, the global visual features and local visual features are integrated: Obtaining visual mixed features of the video 4. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 2, characterized in that: The audio feature extraction module comprises: A preprocessing unit, wherein the preprocessing unit performs short-time Fourier transform, filtering and Hamming window segmentation processing on the audio separated from the video to obtain a spectrum graph, and inputs the spectrum graph into an audio feature processor to obtain an audio feature code, wherein the output end of the audio feature processor is connected to the input end of a second long short-term memory network, and the second long short-term memory network performs feature extraction on the audio feature code to obtain an audio global feature and an audio sequence feature, performs average pooling processing on the audio sequence feature to obtain an audio average sequence feature, and fuses the audio global feature with the audio average sequence feature to obtain an audio mixed feature; or, The audio feature extraction module includes: recording the audio separated from the video as First, after short-time Fourier transform, Mel filter bank and Hamming window operation, the T-second audio sequence is converted into a spectrogram that retains both time information and frequency information. Next, the audio feature processor pre-trained on the dataset is used to input the spectrogram sequence into the audio feature processor, and the features in the audio sequence are gradually extracted through its stacked self-attention layer and fully connected layer to obtain the audio feature encoding. Afterwards, the long short-term memory network is used to obtain the global audio feature a of the audio g And local audio features Finally, the global audio features and local audio features are integrated: Get the audio mix characteristics of the video 5. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 1, characterized in that: The total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, and the total loss function of the model includes: L=L Mce +λ1L dis +λ2L sim ; Among them, λ1 and λ2 are hyperparameter coefficients to complete the weight distribution of different loss functions to avoid a single loss function dominating the entire training. Mce represents the first sub-loss function, L dis represents the second sub-loss function, L sim represents the third sub-loss function; The first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed feature and the Gaussian distribution of the audio mixed feature, including: First, using two mapping functions and Mixing visual features and audio mixing features Mapped into the same vector space D, Mapping Function and The structure is a series of linear layers and a ReLU activation function. Next, assume that there are two Gaussian kernel functions as well as From the feature representation v of the visual modality and the feature representation a of the audio modality, the Gaussian distribution of the visual modality is statistically calculated. and the Gaussian distribution of the audio mode Among them, the Gaussian kernel function as well as The specific architecture is a series of linear layers and a ReLU activation function. Finally, in order to achieve accurate alignment of the visual modal distribution and the audio modal distribution, we optimize and The KL divergence between them is used to minimize the difference between them.

6. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 1, characterized in that: The second sub-loss function refers to the difference between the visual mixed feature and the audio mixed feature, including: First, learn synthetic features of the visual modality And the synthetic features of the audio modality in, Following the normal distribution, the mapping function and It is used to synthesize features of visual and audio modalities from The feature space is mapped to In the feature space, the mapping function and The structure is a series of linear layers and a ReLU activation function; Afterwards, the visual modality feature similarity matrix and the audio modality feature similarity matrix between each video are solved; Among them, S v is the cosine similarity matrix between videos, S a is the cosine similarity matrix between audios, is the cosine similarity score of the visual modality features of the mth video and the nth video, is the cosine similarity score of the audio features of the mth video and the nth video, are the local features of the m videos and the nth video respectively, are the local audio features of the mth audio and the nth audio respectively; Finally, in order to achieve accurate alignment of the video modality and the audio modality in feature representation, by optimizing S v With S a The distance between the two cosine similarity matrices is used to minimize the gap between the two. M is the total number of videos and audios under the current batch size.

7. The method for classifying bad videos based on multimodal alignment and category balance as claimed in claim 1, characterized in that: The third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, and reduce the impact of class imbalance on the training process, including: First, get the total number of samples for each category in the dataset c i is the total number of samples in the i-th category; After that, define the hyperparameter coefficients is the maximum boundary value, calculate the boundary value of each sample Finally, the label distribution-aware balanced cross entropy loss is as follows: Among them, β is the hyperparameter coefficient, p i is the probability that the current model considers the video to be of the i-th category, p j is the probability that the current model considers the video to be of the jth category.

8. A bad video classification system based on multimodal alignment and class balance, characterized by, include: An acquisition module is configured to: acquire a data set, wherein the data set is a video with a known video classification label; The training module is configured to: input the data set into the bad video classification model, train the bad video classification model, and the bad video classification model is referred to as the model; when the total loss function value of the model no longer decreases, or the number of iterations reaches a set number, stop the training, and obtain the trained bad video classification model; During the training process, the model first extracts visual mixed features and audio mixed features from the video; The total loss function of the model includes: the weighted sum of the first, second and third sub-loss functions, wherein the first sub-loss function refers to the difference between the Gaussian distribution of the visual mixed feature and the Gaussian distribution of the audio mixed feature; the second sub-loss function refers to the difference between the visual mixed feature and the audio mixed feature; the third sub-loss function is used to increase the minimum distance between the model decision boundary and the sample point, and reduce the impact of class imbalance on the training process; The classification module is configured to: input the video to be classified into the trained bad video classification model to obtain the classification result of the video to be classified.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 7 are executed.

Images