A video review method based on multimodal large model

Abstract

The present invention discloses a video review method based on a multimodal large model, including: obtaining a video to be reviewed, performing preprocessing operations such as video format conversion and video quality improvement, and obtaining multimodal data to be reviewed as input data; the input data is the multimodal data to be reviewed, including image sequence Images, audio sequence Audios, and text sequence Texts; preprocessing to obtain a multimodal classification large model network, inputting the input data into the multimodal classification large model network and outputting the classification result and its corresponding score, and when the classification result is a violation label and the corresponding score is greater than a preset threshold, the video to be reviewed is determined to be a violation video. The present invention uses an improved attention multimodal feature fusion method to perform feature fusion on image features, audio features, and text features, and the method can make full use of the complementarity of different modal information to improve the accuracy and robustness of the review.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video review method based on a multimodal large model. Background Art

[0002] In recent years, with the rapid development of online video content, the amount of user-generated content (UGC) on social media and video platforms has surged. However, this also brings a large amount of risk of spreading violence, pornography, hate speech and false information, posing a threat to social order and public safety. Video review has therefore become particularly important. By promptly identifying and filtering illegal content, platforms can protect users from harmful information, maintain a good network environment, and enhance public trust in the platform.

[0003] Video review methods include image content recognition and audio content recognition, and most of them are currently single-modal feature recognition methods. Image content recognition decodes the video into an image sequence and determines whether the video violates the rules by identifying the image sequence. The analysis of static image sequences cannot accurately determine the context. A single image may seem harmless, but it may have a clear violation meaning in a specific context. Audio content recognition converts the audio in the video into text and determines whether the video violates the rules through text recognition technology. When converting speech to text, it is often difficult to accurately understand the context, resulting in missed or misjudgment of the video. Summary of the invention

[0004] Technical purpose: In view of the shortcomings of the existing technology, the present invention proposes a video review method based on a multimodal large model, which fuses image features, audio features, and text features using an improved attention multimodal feature fusion method. This method can make full use of the complementarity of different modal information and improve the accuracy and robustness of the review.

[0005] Technical solution: In order to achieve the above technical objectives, the present invention adopts the following technical solution.

[0006] A video review method based on a multimodal large model comprises the following steps:

[0007] Obtain the video to be reviewed, perform preprocessing operations such as video format conversion and video quality improvement, and obtain the multimodal data to be reviewed as input data; the input data is the multimodal data to be reviewed, including image sequence Images, audio sequence Audios, and text sequence Texts;

[0008] The multimodal classification large model network is obtained through preprocessing. The input data is input into the multimodal classification large model network and the classification result and its corresponding score are output. When the classification result is a violation label and the corresponding score is greater than the preset threshold, the video to be reviewed is judged to be an illegal video.

[0009] Beneficial effects:

[0010] 1. The improved ResNet50 module in the present invention mainly improves and innovates the feature extraction and feature fusion methods in the BTNK2 module. The improved BTNK2 module uses spatial domain global average / maximum pooling to extract the global summary features of the entire input feature map, reducing the number of parameters that need to be processed and the computational complexity, while extracting the global summary features of the entire input feature map; using an innovative attention mechanism calculation method to obtain the attention feature weight Atten, capture the dependency between different spatial positions of the feature map, and significantly enhance the model's ability in semantic understanding and expression;

[0011] 2. Improve the Cross Modal Encoder module to use multiple grouped query vectors Q t , Q a , key vector K and value vector V to achieve group attention feature fusion. This method reduces the amount of parameter calculation and computational complexity when fusion of multimodal features. At the same time, different weight combinations are used to combine text features TF t and image feature TF i Association weight information A ti , audio feature TF a and image feature TF i Association weight information A ai Feature fusion effectively retains the importance of each modal feature, avoids the problem of information loss or over-aggregation, and utilizes the importance difference between different modal information to improve the quality and information richness of the overall feature representation, thereby improving the performance of subsequent tasks.

[0012] 3. Improve the network optimization loss function. The loss function is divided into classification head loss Loss class And feature fusion module loss Loss cross There are two parts, among which the feature fusion module loss Loss cross An improved multimodal feature similarity loss is adopted, and the cosine similarity between different modalities is calculated to evaluate their similarity, which helps the model better learn multimodal data and effectively fuse them. The importance between different modalities is adjusted by introducing hyperparameters ξ and ζ, which is suitable for different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the method of the present invention;

[0014] Figure 2 This is the network structure diagram of the existing ResNet50 model;

[0015] Figure 3 This is a structural diagram of the improved BTNK2 in the improved ResNet50 model network of the present invention;

[0016] Figure 4 It is the improved cross Modal Encoder diagram of the present invention;

[0017] Figure 5 This is the network structure diagram of the multimodal classification large model of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] As attached Figure 1 As shown, a video review method based on a multimodal large model of the present invention comprises the following steps:

[0020] Obtain the video to be reviewed, perform preprocessing operations such as video format conversion and video quality improvement, and obtain the multimodal data to be reviewed as input data; the input data is the multimodal data to be reviewed, including image sequence Images, audio sequence Audios, and text sequence Texts;

[0021] The multimodal classification large model network is obtained through preprocessing. The input data is input into the multimodal classification large model network and the classification result and its corresponding score are output. When the classification result is a violation label and the corresponding score is greater than the preset threshold, the video to be reviewed is judged to be an illegal video.

[0022] The preprocessing process of the multimodal classification large model network includes network construction and network training; the network construction process includes: Figure 5 As shown, the multimodal classification large model network includes a feature extraction layer, a feature embedding sharing layer, a multimodal feature fusion layer and a classification layer connected in sequence;

[0023] The feature extraction layer includes an improved ResNet50 model, a VGGish model, and a Bert model. The improved ResNet50 model is used to calculate and obtain image features (Image Features) for the image sequence Images in the input data. i ); The VGGish model is used to calculate the audio features Audio Features (F a); Bert model is used to calculate the text features (Text Features) for the text sequence Texts in the input data. t );

[0024] The feature embedding shared layer is used to embed the image features according to the image features (F i )、AudioFeatures(F a )、Text Features(F t ) Calculate and obtain the multimodal embedding feature TF i TF a TF t The multimodal feature fusion layer includes an improved multimodal feature fusion layer for embedding the multimodal feature TF according to the multimodal feature i TF a TF t Calculate and obtain the fusion feature F r The classification layer includes a Classifier classification head, which is used to classify the fusion feature F r The classification result and its corresponding score are calculated and obtained. When the classification result is a violation label and the corresponding score is greater than a preset threshold, the video to be reviewed is determined to be a violation video.

[0025] Example:

[0026] As attached Figure 2 and attached Figure 3 As shown, a video review method based on a multimodal large model in this embodiment includes the following steps:

[0027] STEP 1. Obtain the video to be reviewed, perform preprocessing operations such as video format conversion and video quality improvement, and obtain multimodal data to be reviewed as input data;

[0028] The input data is multimodal data to be reviewed, including image sequences Images, audio sequences Audios, and text sequences Texts. The image sequence Images is obtained by preprocessing the video to be reviewed after the video decoding service extracts the video to be reviewed according to the time interval strategy. The image preprocessing operations include image resizing, denoising, and image enhancement; the audio sequence Audios is obtained by preprocessing the video to be reviewed after the audio extraction service extracts the video to be reviewed according to the time interval strategy. The audio preprocessing operations include noise reduction, volume normalization, and removal of silent segments; the text sequence Texts is obtained by transcribing the audio sequence Audios by the ASR service and then preprocessing. The text preprocessing operations include removing punctuation marks, stop words, and spelling check corrections;

[0029] STEP2: For the image sequence Images in the input data, input it into the improved ResNet50 model to obtain the image features Image Features (F i ), that is, the image sequence Images in the input data is used as the INPUT of the improved ResNet50 model, and the image features Im age Features (F i ) as the OUTPUT of the improved ResNet50 model; its calculation formula is as follows:

[0030] Image Features(F i )=∑ImprovedResNet50(X j )

[0031]

[0032] Among them, X j represents the jth image in the image sequence Images, Indicates Image Features(F i ) is n rows d i dimensional feature vector set;

[0033] Improved ResNet50 model structure diagram as follows Figure 2 As shown in the figure, the present invention mainly optimizes the BTNK2 module in the existing ResNet50, and obtains the optimized BTNK2 module by integrating the improved transformer attention module into the BTNK2 module. The optimized BTNK2 module not only solves the gradient vanishing problem by adopting the residual structure, but also significantly improves the generalization and robustness of the expression features by optimizing the feature fusion method.

[0034] The construction process of the improved ResNet50 model includes: based on the existing ResNet50, all BTNK2 modules are optimized to obtain the optimized BTNK2 modules. The specific steps for constructing the optimized BTNK2 modules are as follows:

[0035] STEP2.1. For the input feature X(X∈R H*W*C ), where H, W, and C are the height, width, and number of channels of the input feature X, respectively. A 1*1 convolution operation is performed to obtain the first feature map f1. The 1*1 convolution operation significantly reduces the number of parameters and the amount of computation required to improve the ResNet50 model.

[0036] STEP2.2, using the first feature map f1, perform a 3*3 convolution operation to obtain the second feature map f2, The convolution operation obtains input pixel information in a larger range, thereby expanding the receptive field of the feature map and capturing the local features of the image more comprehensively.

[0037] STEP2.3, using the obtained second feature map f2, after global average pooling in the spatial domain and global maximum pooling in the spatial domain, and then flatten and unsqueeze operations in turn, the third feature map f3 and the fourth feature map f4 are obtained. The calculation formula is as follows:

[0038] f3=unsqueeze(flatten(AvgPool(f2))),f3∈R HW*l

[0039] f4=unsqueeze(flatten(MaxPool(f2))),f4∈R HW*1

[0040] Among them, the spatial domain global average pooling AvgPool calculates the average value of all the values ​​at the same position (h, w) in channel C to obtain the feature map (H, W, 1); the spatial domain global maximum pooling MaxPool calculates the maximum value of all the values ​​at the same position (h, w) in channel C to obtain the feature map (H, W, 1).

[0041] Through step S2.3, the second feature map f2 is converted into the input feature map of the improved transformer attention module, namely the third feature map f3 and the fourth feature map f4. Compared with the traditional method of directly reshaping the second feature map f2 to obtain the (H*W, C / 4) feature map, the present invention uses the third feature map f3 and the fourth feature map f4 converted from the second feature map f2 as the input of the improved transformer attention module, which has the following advantages: the global summary features of the entire input feature map are extracted using the spatial domain global average and maximum pooling. These global features provide the transformer with more informative input and reduce the number of parameters and computational complexity that need to be processed. The improved transformer attention module includes a parameter (f2, f3, f4) acquisition unit for attention module calculation, two parallel improved attention units (q1, k1, v), (q2, k2, v) and a two-dimensional convolution unit;

[0042] STEP2.4, using the obtained third feature map f3 and fourth feature map f4, respectively, through the improved attention units (q1, k1, v) and (q2, k2, v) to obtain the attention feature weight Atten, the calculation formula is:

[0043]

[0044]

[0045] Atten=Softmax(a1+β*a2),Atten∈R HW*HW

[0046] v=f2·W v

[0047] Among them, d k is the feature vector dimension of the first retrieval feature q1, k1 is the first comparison feature, a1 is the feature map with weight information obtained from the third feature map f3, q2 is the second retrieval feature, k2 is the second comparison feature, a2 is the feature map with weight information obtained from the fourth feature map f4, Atten is the fusion attention feature weight, β is the hyperparameter, β = 0.47; v is the original feature, W v is the learning weight;

[0048] The present invention improves the existing transformer attention module and the attention feature weight Atten obtained by fusing multiple features a1 and a2, so as to capture the dependency between different spatial positions of the feature map and effectively fuse multiple information at different positions, thereby enhancing the comprehensive expression ability of the features and especially highlighting the key information.

[0049] STEP2.5, using the obtained fused attention feature weight Atten, the second feature map f2 is calculated with the original feature v in the improved attention unit (q1, k1, v) and (q2, k2, v) to obtain the total feature map F with fused weight information. The calculation formula is as follows:

[0050] f=reshape(Conv(f2))·v,f∈R HW*C

[0051] F=reshape(Atten·f)·v,F∈R H*W*C

[0052] Wherein, f is the spatial position feature map obtained from the second feature map f2;

[0053] Compared with the feature map F obtained by the existing BTNK2 module before the improvement, the total feature map F with fusion weight information has the following advantages: it more accurately captures the association and dependency between different spatial positions in the feature map, making the feature representation more detailed and comprehensive; it has a larger receptive field and can capture more semantic information across a wider spatial range, significantly enhancing the model's ability in semantic understanding and expression.

[0054] STEP2.6. Use the obtained total feature map F and the input feature X to perform an add operation to obtain the output feature X of the improved BTNK2 module. The calculation formula is as follows:

[0055] X′=F+X

[0056] The improved BTNK2 module uses spatial domain global average / maximum pooling to extract the global summary features of the entire input feature map, reducing the number of parameters that need to be processed and the computational complexity, while extracting the global summary features of the entire input feature map; an innovative attention mechanism calculation method is used to obtain the attention feature weight Atten, capturing the dependency between different spatial positions of the feature map, significantly enhancing the model's ability in semantic understanding and expression.

[0057] STEP 3. Obtain multimodal audio features (Audio Features) for the audio sequence Audios in the input data. a ); Audio sequences Audios are extracted through the VGGish model to obtain multimodal audio features Audio Features (F a ), the calculation formula is as follows:

[0058] Audio Features(F a )=∑VGGish(A j )

[0059]

[0060] in, Indicates Audio Features(F a ) is n rows d a A is a set of feature vectors of dimension A. j is the jth audio in the audio sequence Audios;

[0061] STEP 4: Get multimodal text features (Text Features) for the text sequence Texts in the input data t ); Text sequence Texts is extracted through the Bert model to obtain multimodal text features (Text Features) t ), the calculation formula is as follows:

[0062] Text Features(F t )=∑Bert(T j )

[0063]

[0064] in, Indicates Text Features (Ft ) is n rows d t The feature vector set of dimension, T j is the jth text in the text sequence Texts;

[0065] STEP 5: Obtain multimodal image features (F i )、Audio Features(F a )、Text Features(F t ), input to the shared feature embedding layer, and obtain the multimodal embedding feature TF in turn i TF a TF t ; The calculation formula is as follows:

[0066]

[0067] TF x ∈R n*d , x∈(a, t, i)

[0068] Among them, d x are the different dimensions of the multimodal features, d is the dimension of the output feature of the shared feature embedding layer, and R n*d TF x is a set of n-row, d-dimensional feature vectors;

[0069] The feature embedding shared layer first changes the number of feature channels through a 1*1 convolution operation on the features of different modes, and then performs a two-dimensional convolution operation on the features of different modes. The fused features of different channels are fused to obtain fused features, and the different dimensions of multimodal features are mapped to the same output feature dimension d, so that the dot product in the cross-modal attention fusion large model block is available. This feature embedding shared layer can capture the temporal dependencies in sequence data, especially when the convolution kernel spans multiple time steps, it can identify dependencies over a long time range.

[0070] STEP6: Based on the acquired multimodal embedding feature TF i TF a TF t , after improving the multi-modal feature fusion layer (Cross Modal Encoder), the fusion feature F is obtained r , improve the Cross Modal Encoder module such as Figure 4 As shown, the calculation formula is as follows:

[0071] F r =∑ Improved CModel_Encoder(TF i , TF a , TFt ), F r ∈R n*d

[0072] Improvements to the Cross Modal Encoder module building process include:

[0073] STEP 6.1. Multimodal embedding feature TF i TF a TF t To improve the input of the Cross Modal Encoder module, multiple group query vectors Q are calculated separately. t , Q a , key vector K and value vector V; the calculation formula includes:

[0074] Q t =W t *TF t

[0075] Q a =W a *TF a

[0076] K=W k *TF i

[0077] V=W v *TF i

[0078] Among them, Q t , Q a , K,

[0079] Using multiple group query vectors Q t , Q a , key vector K and value vector V for multimodal embedding feature TF i TF a TF t Compared with the traditional method of directly concatenating multi-modal embedded features and then fusing multi-head attention features, the grouped attention feature fusion retains the integrity of multi-modal information and improves the ability to capture complex interactions between different modal features.

[0080] STEP 6.2. Query vector Q through multiple groups t , Q a , key vector K and value vector V and multimodal embedding features TF i TF a TF t Calculate and obtain the weight A of multimodal association information ti , A ai, A tt , the calculation formula includes:

[0081]

[0082] A tt =Soft max(α1*A ti +β1*A ai ), A tt ∈R n*n

[0083] Among them, α1 and β1 are hyperparameters, representing the weight information of different features, and α1+β1=1, A ti is the key information weight feature of text features and image features, A ai is the key information weight feature of audio features and image features, A tt is the multimodal fusion associated weight feature, q t ,q a is the query vector, k is the key vector, d is the feature dimension, ε is the number of attention heads, and Concat represents the concatenation operation;

[0084] Use the improved group attention mechanism to obtain text features TF separately t and image feature TF i Association weight feature A ti , audio feature TF a and image feature TF i Association weight feature A ai , multimodal fusion associated weight feature A tt Compared with the traditional pairwise attention fusion mechanism, the parameter calculation amount and model complexity are significantly reduced. This method not only effectively considers the importance difference between different modal information, but also can efficiently realize the fusion of multimodal features.

[0085] STEP 6.3. Weight A of multimodal association information ti , A ai , A tt Respectively with its multimodal embedding feature TF i TF a TF t Perform a dot multiplication operation to obtain a feature map F with multimodal association weight information t ′、F′ a 、F i ′, the calculation formula includes:

[0086] F t ′=A ti ·TF t , F t ′∈R n*d

[0087] F′ a =A ai ·TF a , F′ a ∈R n*d

[0088] F i ′=A tt ·TF i , F i ′∈R n*d

[0089] STEP6.4: Based on the feature map F with multimodal association weight information t , F′ a 、F i Perform FC and other operations to obtain the fusion feature F output by the improved Cross Modal Encoder r , the calculation formula includes:

[0090] F r =FC(Concat(F t ′、F′ a 、F i ′))

[0091] TF t ′, TF′ a , TF i ′=Split(F r ), TF t ′, TF′ a , TF i ′∈R n*d

[0092] Among them, TF t ′, TF′ a , TF i ′ represents the output fusion features of a single improved Cross Modal Encoder module, Split represents the segmentation operation; the fusion feature F r It is a multi-modal fusion feature.

[0093] In STEP6.3 and STEP6.4, multimodal feature maps are used as input, and the output is a feature map with multimodal association weight information. This method effectively retains the importance of each modal feature in the multimodal fusion process and avoids the problem of information loss or over-aggregation. The introduction of multimodal association weight information can make the final feature map more powerful and comprehensive in representation ability, thereby improving the performance of subsequent tasks.

[0094] Improve the Cross Modal Encoder module to use multiple grouped query vectors Q t , Q a , key vector K and value vector V to achieve group attention feature fusion. This method reduces the amount of parameter calculation and computational complexity when fusion of multimodal features. At the same time, different weight combinations are used to combine text features TF t and image feature TF i Association weight information A ti , audio feature TF a and image feature TF i Association weight information A ai Feature fusion effectively retains the importance of each modal feature, avoids the problem of information loss or over-aggregation, and utilizes the importance difference between different modal information to improve the quality and information richness of the overall feature representation, thereby improving the performance of subsequent tasks.

[0095] STEP7: The fusion feature F obtained in STEP6 r Input to the Classifier classification head to obtain each classification label and its corresponding score. When the classification result is a violation label and the corresponding score is greater than the preset threshold, the video to be reviewed is judged to be a violation video. The model outputs classification labels of five categories: pornography, politics, violence, advertising, and normal. The label with the highest score is selected as the model output classification label for subsequent judgment of whether the video is compliant; among them, the classification labels of pornography, politics, violence, and advertising are all considered violation labels, and the corresponding score range is (0, 1). The preset threshold is a dynamic threshold. Users can dynamically adjust the threshold according to the different degrees of compliance requirements for video content in different periods. The value range is generally (0.75, 0.9). When the corresponding score is greater than the preset threshold, the video to be reviewed is judged to be a violation video.

[0096] The Classifier classification head consists of two FC (fully connected layers), Relu activation function and SoftMax. The calculation formula is as follows:

[0097] Labels, Confs=Softmax(FC2(FC1+Relu)(Concat(F r )))

[0098] Among them, Labels and Confs represent the model network output labels and their corresponding scores respectively; FC1 is the processing of the first fully connected layer, FC2 is the processing of the second fully connected layer, and Concat represents the concatenation operation;

[0099] The network training process includes:

[0100] Manual review data of several videos were collected as training data, and the training data was divided into training set, validation set and test set in a ratio of 8:1:1. During the model training process, we introduced an improved loss function to tune the model. The loss function calculation formula is as follows:

[0101] Loss=Loss class +Loss cross

[0102]

[0103] Among them, Loss represents the total loss function of the multimodal classification large model network, Loss class is the classification head loss, Loss cross is the feature fusion module loss, 1≤e≤E, E represents the number of classification labels, and the value of this invention is 5, y e represents the true label of the data in the training set, p e It represents the predicted probability of the data in the training set, that is, the score output by the multimodal classification model network, with a value range of (0, 1). Cos represents the calculation of the cosine similarity of two variables. r TF is the fusion feature. i TF a TF t is a multimodal embedding feature, ξ and ζ are hyperparameters, representing the weights of data loss of different modalities, ξ+ζ = 1. In some embodiments of the present invention, ξ = 0.65, ζ = 0.35.

[0104] The loss function in this invention is divided into two parts: classification head loss Loss class And feature fusion module loss Loss cross . Classification head loss Loss class Using cross entropy loss, this loss can effectively optimize the predicted probability of the model output to make it close to the distribution of the true label. Feature fusion module loss Loss cross An improved multimodal feature similarity loss is adopted, and the cosine similarity between different modalities is calculated to evaluate their similarity, which helps the model better learn multimodal data and effectively fuse them. The importance between different modalities is adjusted by introducing hyperparameters ξ and ζ, which is suitable for different application scenarios.

[0105] The loss function of the present invention comprehensively considers the accuracy of classification and the generalization and stability of the improved Cross Modal Encoder fusion module to improve the accuracy and reliability of video content compliance judgment. The optimized model test set indicators have an accuracy rate of 98%, a precision rate of 97.5%, and a recall rate of 96%. This method not only improves the model's ability to identify various types of illegal content, but also optimizes its generalization performance in practical applications.

[0106] The present invention also discloses an electronic device, which includes: a memory for storing a computer program; and a processor for executing the computer program so that the device executes the aforementioned video review method based on a multimodal large model.

[0107] The present invention also provides a computer storage medium, on which a computer program is stored. When the computer program is executed, the device executing the computer program implements the aforementioned video review method based on a multimodal large model.

[0108] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, and the memory can be various types of memory, which can be a random access memory, a read-only memory, a flash memory, etc., such as a read-only memory (English: read-only memory, ROM) / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0109] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video review method based on a multimodal large model, characterized in that: The following steps are involved: Obtain the video to be reviewed, perform video format conversion and video quality improvement preprocessing operations, and obtain the multimodal data to be reviewed as input data; the input data is the multimodal data to be reviewed, including image sequence Images, audio sequence Audios, and text sequence Texts; Preprocessing obtains a multimodal classification large model network, inputs the input data into the multimodal classification large model network, and outputs the classification result and its corresponding score. When the classification result is a violation label and the corresponding score is greater than a preset threshold, the video to be reviewed is determined to be a violation video; The preprocessing process of the multimodal classification large model network includes network construction and network training; the network construction process includes: the multimodal classification large model network includes a feature extraction layer, a feature embedding sharing layer, a multimodal feature fusion layer and a classification layer connected in sequence; The feature extraction layer includes an improved ResNet50 model, a VGGish model, and a Bert model. The improved ResNet50 model is used to calculate and obtain image features (Image Features) for the image sequence Images in the input data. i ); The VGGish model is used to calculate and obtain multimodal audio features Audio Features (F a ); Bert model is used to calculate and obtain multimodal text features TextFeatures (F t ); The feature embedding shared layer is used to embed the image features according to the image features (F i ), Multimodal audio features AudioFeatures(F a ), Multimodal Text Features(F t ) Calculate and obtain the multimodal embedding feature TF i TF a TF t The multimodal feature fusion layer includes an improved multimodal feature fusion layer for embedding the multimodal feature TF according to the multimodal feature i TF a TF t Calculate and obtain the fusion feature F r The classification layer includes a Classifier classification head, which is used to classify the fusion feature F r Calculate and obtain the classification result and its corresponding score. When the classification result is a violation label and the corresponding score is greater than a preset threshold, the video to be reviewed is determined to be a violation video; During the network training process, the loss function calculation formula includes: Loss=Loss class +Loss cross Among them, Loss represents the total loss function of the multimodal classification large model network, Losss class is the classification head loss, Loss cross is the feature fusion module loss, 1≤e≤E, E represents the number of classification labels, y e represents the true label of the data in the training set, p e represents the predicted probability of the data in the training set, that is, the score output by the multimodal classification model network, cos represents the calculation of the cosine similarity of two variables, F r TF is the fusion feature. i TF a TF t is the multimodal embedding feature, ξ and ζ are hyperparameters, representing the loss weights of different modal data, ξ+ζ=1.

2. The video review method based on a multimodal large model according to claim 1, characterized in that: In the improved ResNet50 model, the BTNK2 module is optimized, and the optimized BTNK2 module is obtained by integrating the improved transformer attention module into the BTNK2 module; The improved ResNet50 model is used to calculate the image features (F i ),include: For the input feature X, a 1*1 convolution operation is performed to obtain the first feature map V; Using the first feature map f1 obtained, a 3*3 convolution operation is performed to obtain the second feature map f2; The obtained second feature map f2 is used to perform global average pooling in the spatial domain and global maximum pooling in the spatial domain, and then to perform flatten and unsqueeze operations in sequence to obtain the third feature map f3 and the fourth feature map f4; Using the obtained third feature map f3 and fourth feature map f4, the attention feature weight Atten is obtained by improving the attention units (q1, k1, v) and (q2, k2, v) respectively; The obtained attention feature weight Atten and the second feature map f2 are used to calculate the total feature map F with fusion weight information through the improved attention units (q1, k1, v) and v features in (q2, k2, v); The obtained total feature map F is added to the input feature X to obtain the output X′ of the improved BTNK2 module.

3. The video review method based on a multimodal large model according to claim 2 is characterized in that: The calculation formulas of the third characteristic map f3 and the fourth characteristic map f4 include: <h2 style=";text-align:left;direction:ltr">f3=unsqueeze(flatten(AvgPool(f2))),f3∈R<h2 style=";text-align:left;direction:ltr"> HW*1 f4=unsqueeze(flatten(MaxPool(f2))),f4∈R HW*1 Among them, the spatial domain global average pooling AvgPool calculates the average value of all the values ​​at the same position (h, w) in channel C to obtain the feature map (H, W, 1); the spatial domain global maximum pooling MaxPool calculates the maximum value of all the values ​​at the same position (h, w) in channel C to obtain the feature map (H, W, 1); H, W, C are the height, width, and number of channels of the input feature X respectively; The calculation formula of the attention feature weight Atten includes: Atten=Soft max(a1+β*a2),Atten∈R HW*HW v=f2·W v Among them, d k is the feature vector dimension of the first retrieval feature q1, k1 is the first comparison feature, a1 is the feature map with weight information obtained from the third feature map f3, q2 is the second retrieval feature, k2 is the second comparison feature, a2 is the feature map with weight information obtained from the fourth feature map f4, Atten is the fusion attention feature weight, β is the hyperparameter, β = 0.47; v is the original feature, W v is the learning weight; The calculation formula of the total feature map F with fusion weight information includes: f=reshape(Conv(f2))·v,f∈R HW*C F=reshape(Atten·f)·v,F∈R H*W*C Wherein, f is the spatial position feature map obtained from the second feature map f2; The calculation formula of the output feature X′ includes: X′=F+X Among them, X is the input feature.

4. The video review method based on a multimodal large model according to claim 1, characterized in that: The feature embedding shared layer is used to embed the image features according to the image features (F i ), Multimodal Audio Features(F a ), Multimodal Text Features(F t ) Calculate and obtain the multimodal embedding feature TF i TF a TF t ; The calculation formula includes: TF x ∈R n*d ,x∈(a,t,i) Among them, d x are the different dimensions of the multimodal features, d is the dimension of the output feature of the shared feature embedding layer, and R n*d TF x is a set of n-row and d-dimensional feature vectors.

5. The video review method based on a multimodal large model according to claim 1 is characterized in that: The multimodal feature fusion layer includes an improved multimodal feature fusion layer for embedding the multimodal feature TF according to the multimodal feature i TF a TF t Calculate and obtain the fusion feature F r ,include: Embedding features TF in multimodal i TF a TF t To improve the input of the Cross Modal Encoder module, multiple group query vectors Q are calculated separately. t , Q a , key vector K and value vector V; Query vector Q by multiple groups t , Q a , key vector K and value vector V and multimodal embedding features TF i TF a TF i Calculate and obtain the weight A of multimodal association information ti , A ai , A tt ; Weight A of multimodal correlation information ti , A ai , A tt Respectively with its multimodal embedding feature TF i TF a TF t Perform a dot multiplication operation to obtain a feature map F with multimodal association weight information t ′、F a ′、F i ′; According to the feature map F with multimodal association weight information t ′、F′ a 、F i ′ Perform FC operation to obtain fusion feature F r .

6. The video review method based on a multimodal large model according to claim 5 is characterized in that: Multimodal correlation information weight A ti , A ai , A tt , the calculation formula includes: A tt =Soft max(α1*A ti +β1*A ai ),A tt ∈R n*n Among them, α1 and β1 are hyperparameters, representing the weight information of different features, and α1+β1=1, A ti is the key information weight feature of text features and image features, A ai is the key information weight feature of audio features and image features, A tt is the multimodal fusion associated weight feature, q t ,q a is the query vector, k is the key vector, d is the feature dimension, ε is the number of attention heads, Concat represents the concatenation operation, R n*n Represented as a set of n-row n-dimensional feature vectors.

7. The video review method based on a multimodal large model according to claim 6 is characterized in that: Feature map F with multimodal association weight information t ′、F a ′、F i ′, the calculation formula includes: F t ′=A ti ·TF t ,F t ′∈R n*d F′ a =A ai ·TF a ,F′ a ∈R n*d F i ′=A tt ·TF i ,F i ′∈R n*d Fusion feature F r The calculation formula includes: F r =FC(Concat(F t ′、F′ a 、F i ′)) TF t ′,TF′ a ,TF i ′=Split(F r ),TF t ′,TF′ a ,TF i ′∈R n*d Among them, TF t ′, TF′ a , TF i ′ represents the output fusion features of a single improved Cross Modal Encoder module, and Split represents the segmentation operation.

8. The video review method based on a multimodal large model according to claim 1 is characterized in that: The classification layer includes a Classifier classification head, which is used to classify the fusion feature F r Calculate and obtain the classification results and their corresponding scores to obtain the video review results of the video to be reviewed. The calculation formula includes: Labels,Confs=Soft max(FC2(FC1+Relu)(Concat(F r ))) Among them, Labels and Confs represent the model network output labels and their corresponding scores respectively; FC1 is the processing of the first fully connected layer, FC2 is the processing of the second fully connected layer, and Concat represents the concatenation operation.

Images