Unstructured video data automatic classification method

Abstract

The invention relates to the technical field of data security, in particular to an unstructured video data automatic classification method, which comprises the following steps: S1, video data preprocessing: standardizing video data to a uniform format and size, and carrying out noise processing; s2, video data feature extraction and multi-mode feature fusion: feature extraction is performed on the preprocessed video data from three modes of audio, video and text, time sequence processing is performed, and multi-mode feature fusion is performed; s3, video data same-category matching: classifying the fused multi-modal features to complete video classification; according to the invention, multi-modal information in a video can be captured more accurately, and automatic processing of video data is realized; high processing efficiency and classification accuracy are kept in large-scale data, meanwhile, the degree of automation is improved, the necessity of manual participation is reduced, and the overall process is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data security, and in particular to an automatic classification method for unstructured video data. Background Art

[0002] Data is one of the most important assets of an enterprise. The business of an enterprise often involves a large amount of data and sensitive information. Data classification and grading is an important part of data security management. Its purpose is to better protect the organization's data assets and ensure data security and compliance. Through data classification, it is possible to identify which data is sensitive or has high value, such as personal information, financial data, etc., and then give sensitive data a higher level of protection, and take stricter access control and encryption measures. Through data classification and grading, enterprises can not only protect their data assets, but also ensure compliance with relevant laws and regulations, improve the efficiency of data management and use, reduce security risks, use classified data for deeper analysis, obtain more valuable insights, and support the sustainable development of the business.

[0003] With the rapid development and popularization of Internet technology, the amount of video data has exploded. Video data has been widely used in various fields, including entertainment, education, security, medical treatment, industrial manufacturing, etc. How to efficiently and accurately classify and grade video data has become an urgent problem to be solved. The existing video data classification work mainly relies on manual identification and classification, which has problems such as low classification efficiency, low accuracy, and difficulty in processing large-scale video data. Summary of the invention

[0004] The invention provides a method for automatically classifying unstructured video data.

[0005] A method for automatically classifying unstructured video data comprises the following steps:

[0006] S1, video data preprocessing: standardize the video data to a unified format and size, and perform noise processing;

[0007] S2, video data feature extraction and multimodal feature fusion: extract features from the preprocessed video data in the three modes of audio, video and text, perform time series processing, and perform multimodal feature fusion;

[0008] S3, video data same category matching: classify the fused multimodal features to complete the video classification.

[0009] Optionally, the video data preprocessing in S1 includes:

[0010] S11, video format conversion: use OpenCV to convert the video to a unified MP4 format;

[0011] S12, adjust the size of the video frame: by scaling or cropping, use Opencv to uniformly adjust to 720*576 standard definition size;

[0012] S13, denoising: using a filter to remove noise from the video.

[0013] Optionally, the video data feature extraction and multimodal feature fusion in S2 includes:

[0014] S21, audio processing of video data: convert the audio file to mono and resample it, generate a spectrogram using short-time Fourier transform (STFT), and convert it to Log-Me l spectrum through Mel filter. After normalization, the spectrum is serialized by time step and position coding is added;

[0015] S22, audio data text processing: convert audio into text through automatic speech recognition (ASR), extract word embedding using the BERT model, and capture the temporal features of the text through the LSTM model;

[0016] S23, video image processing: decode the video into frame sequences and standardize the size and pixel values, use the LSTM network to process the frame sequences, learn the temporal relationship between video frames, and generate video feature representation;

[0017] S24, multimodal temporal information modeling: compressing the features of audio, video, and text and introducing position coding for temporal modeling;

[0018] S25, multimodal feature fusion: The features of audio, video, and text modalities are fused through Transformer and multi-head attention mechanism to generate a comprehensive feature representation.

[0019] Optionally, the audio processing of the video data in S21 includes:

[0020] S211, Audio conversion and mono adjustment: Convert audio files to a unified format and adjust to mono I

[0021] S212, Audio Samples: Resample the audio at a sampling rate of 22050 Hz;

[0022] S213, short-time Fourier transform: perform short-time Fourier transform (STFT) using a Hamming window of size 1024 and a jump length of 512 to obtain an amplitude spectrum diagram;

[0023] S214, Amplitude-spectrum conversion: converting the amplitude spectrum into a Log-Mel spectrum through a Mel filter bank;

[0024] S215, normalization processing: adjusting the range of the characteristic value by normalization processing;

[0025] S216, position encoding: Use Sinusoidal position encoding to add position information to each time step through sine and cosine functions. For time step t, its position encoding is:

[0026]

[0027]

[0028] Where i is the index of the encoding dimension, d is the total encoding dimension size, and t is the time step position in the sequence;

[0029] S217, Dimension expansion: By increasing the batch dimension (batch_size dimension), it is expressed as:

[0030] X′=reshape(X)=X[np.newaxis,:,:];

[0031] Among them, np.newaxis is used to insert a dimension with a value of 1 in the first dimension (batch dimension) of the matrix, which is equivalent to upgrading the data from two dimensions to three dimensions.

[0032] Optionally, the audio data text processing in S22 includes:

[0033] S221, audio preprocessing: convert the speech in the audio file into text through automatic speech recognition (ASR), and preprocess the text, including text cleaning and word segmentation;

[0034] S222, vectorization: Input the preprocessed text into the pre-trained BERT model to obtain the context embedding vector of each word:

[0035] S223, feature extraction: Capture the temporal features of the text through the LSTM model, including:

[0036] Input initialization: The vectorized result is used as input, expressed as X = [x1, x2, ..., X T ], where T is the number of time steps, and the input x at each time step t is t is a vector;

[0037] Forget gate calculation: After the forget gate calculation, the proportion of the memory part forgotten at different time steps is obtained, expressed as:

[0038] f t =σ(w f +[h t-1 , x t ]+bf );

[0039] Among them, w f and b f They are the weight and bias of the forget gate respectively;

[0040] Input gate and candidate memory calculation: The value of the input gate determines the amount of memory written, and the tanh activation function is used to generate candidate memory content, expressed as:

[0041] i t =σ(W i ·[h t-1 , x t ]+b i );

[0042]

[0043] Among them, W i , b i , W c , b C are the corresponding weight matrices and biases;

[0044] Memory state update: The results of the forget gate and the input gate jointly determine the memory state of the current time step, expressed as:

[0045]

[0046] Output gate and hidden state update: Output gate o t Control extracts information from the memory cell to update the hidden state, and uses the output gate and the current memory state to calculate the new hidden state, expressed as:

[0047] o t =σ(W o ·[h t-1 , x t ]+b o );

[0048] h t =o t tanh(C t );

[0049] Hidden state fusion: Use the hidden state sequence [h1, h2, ..., h T ] Input Transformer for feature fusion.

[0050] Optionally, the video image processing in S23 includes:

[0051] S231, video processing: use the opencv suite to decode the video file into a frame sequence, select non-repeated frame images, adjust them to a uniform size, and normalize the pixel values ​​of the images;

[0052] S232, LSTM modeling and video feature extraction: Process feature sequences through LSTM networks and learn the temporal relationship between video frames, including:

[0053] Determine the LSTM network structure: Set the structure of the LSTM network according to actual needs, including the number of LSTM layers (this project is set to 1 layer), the number of units in each LSTM layer, and the configuration of the output layer;

[0054] Set the input layer: use the processed feature sequence as the input of the LSTM network. The format of the input data is three-dimensional (number of samples, time steps, number of features).

[0055] LSTM time step processing: At each time step, the LSTM network receives the input of the current step and the cell state of the previous time step. The forget gate determines the information to be forgotten, the input gate determines the amount of new information to be added, and the output gate controls the information of the current step to be output to the hidden state.

[0056] Learning the temporal relationship between video frames: During the training process, the temporal relationship between video frames is learned through the LSTM network, and its weights are updated to minimize the loss function;

[0057] Generate video feature representation: The output of LSTM contains the contextual information and temporal features of the video sequence;

[0058] S233, feature fusion and output: The output of the LSTM network contains the overall feature representation of the entire video frame sequence.

[0059] Optionally, the multimodal time series information modeling in S24 includes:

[0060] S241, feature compression: high-dimensional video features X v , audio feature X a , text feature X t Compress to low-dimensional space D;

[0061] S242, Multimodal Modeling: Introducing positional encodings and learning positional encodings with temporal context w, shared between audio and video and text modalities;

[0062] S243, Modal Coding Fusion: Modal Coding Add it to the learned vector and we get the vector:

[0063]

[0064]

[0065]

[0066] in, Represents the projection of video features and audio features from dimensions dy, da, and dt to a fully connected layer on a lower dimension D, position encoding

[0067] S244, feature concatenation: Two learnable classification tags are added to the end of the video features, audio features, and text features, one for marking verbs, expressed as The other is used to mark nouns, expressed as And add the position coding information to get the tag, and fuse the video, audio, and text features into a coding representation including video input features with position information and modality information, audio input features, text input features, and gerund summary embedding with tags, expressed as:

[0068]

[0069]

[0070]

[0071] S245, multi-head attention mechanism processing: for X e After slicing, the obtained video features with gerund tags, audio features, and text features are input into the Transformer and processed by the multi-head attention mechanism, which is expressed as:

[0072] MultiHead=Concat(h1, h2..., hN);

[0073]

[0074] Among them, Concat(·) represents the concatenation operation on the feature dimension, N represents the number of attention heads, and h i represents the self-attention mechanism, Q, K, and V represent Query, Key, and Value respectively, and d k Represents the dimension I of K

[0075] S246, vector processing: the vector is sent to the feedforward neural network layer. The activation function of the first layer is the ReLU activation function, and the second layer is the linear activation function, which is expressed as:

[0076] ReLU(X)=max(0,X);

[0077] FFN(X)=max(0,W1X+b1)W2+b2;

[0078] S247, normalization: normalize the feature vector through the normalization layer.

[0079] Optionally, the multimodal feature fusion in S25 includes:

[0080] S251, feature combination: video features and audio features with temporal information after Transformer processing Connect and get the joint feature representation as:

[0081]

[0082] Among them, d is the dimension of the joint feature, and T is the transpose;

[0083] S252, calculating the joint video feature matrix: by using the video feature X v And the joint feature representation J is calculated to obtain the joint correlation matrix C v , expressed as:

[0084]

[0085] in, Indicates the connection video feature X v The combined feature representation J is a learnable weight matrix, and tanh represents the activation function;

[0086] S253, calculating the joint audio feature matrix: by using the audio feature X a And the joint feature representation J is calculated to obtain the joint correlation matrix C a , expressed as:

[0087]

[0088] in, Indicates the connection audio feature X a The combined feature representation J is a learnable weight matrix, and tanh represents the activation function;

[0089] S254, calculate the joint text feature matrix: by using the text feature X t And the joint feature representation J is calculated to obtain the joint correlation matrix C t , expressed as:

[0090]

[0091] in, Represents the connection text feature X tThe combined feature representation J is a learnable weight matrix, and tanh represents the activation function;

[0092] S255, calculate the attention weights of multiple modalities: calculate the attention weights of the joint video feature, joint audio feature, and joint text feature modalities, expressed as:

[0093]

[0094]

[0095]

[0096] Among them, ReLu is the activation function, W cv , W ca , W v , W a , Calculate H v , H a , H t They are the attention map of the video modality and the attention map of the audio modality respectively;

[0097] S256, calculate fusion features: calculate the video feature X that integrates the audio information att，v , calculate the audio feature X that integrates the video information att，a , text feature X att，t , expressed as:

[0098] X att，v =H v W hv +X v ;

[0099] X att,a =H a W ha +X a ;

[0100] X att,t =H t W ht +X t ;

[0101] S257, Computational Audiovisual and Text Features: Fusion X att，v , X att，a , X att,t Get feature X att , expressed as:

[0102] X att =[X att,v ;X att,a ;X att,t ].

[0103] Optionally, the same category matching of the video data in S3 includes:

[0104] S31, feature vector fusion and representation: The fused feature vector represents the comprehensive representation of video and audio information as well as text information;

[0105] S32, fully connected layer mapping: Use the fully connected layer to map X att Process it and map it to the probability distribution of different categories, expressed as:

[0106] O = softmax(W c X att +b c );

[0107] Among them, W c and b c They are the category x-dimensional weight matrix and bias vector of the fully connected layer respectively;

[0108] S33, Application of Positional Encoding: Use positional encoding to provide Transformer with the position information of each input element in the sequence;

[0109] S34, calculate category probability: Use the Softmax function to convert the output value of the fully connected layer into a probability distribution, expressed as:

[0110]

[0111] Among them, O i is the output score of category i, and C is the total number of categories;

[0112] S35, select the prediction category: according to the probability output by Softmax, select the category with the highest probability as the prediction result, indicating that the probability that the video belongs to this category is the highest;

[0113] S36, calculate the loss function: use the cross entropy loss function to calculate the difference between the output and the true label, expressed as:

[0114]

[0115] Among them, y i is the encoding of the true category, p i is the probability distribution of the model output;

[0116] S37, Back Propagation and Model Optimization: By comparing the differences between the predicted results and the true labels, the back propagation algorithm is used to optimize the model parameters.

[0117] Beneficial effects of the present invention:

[0118] The present invention improves the accuracy of video classification by introducing multimodal feature extraction and fusion methods. Compared with existing manual classification methods, it can more accurately capture multimodal information (audio, video, text) in the video, and comprehensively understand the contextual relationship between video frames through temporal modeling, thereby reducing misclassification in complex scenes. In addition, by optimizing the algorithm and model structure, it can realize automatic processing of video data, significantly improve the classification speed, and meet the rapidly growing video data processing needs.

[0119] The present invention, through the automated classification method, can greatly improve the processing capability of large-scale video data. Through automated model processing, it reduces manual intervention, reduces labor costs and time consumption. With the help of advanced models such as LSTM, Transformer, and multi-head attention mechanism, it can maintain high processing efficiency and classification accuracy in large-scale data. At the same time, the improvement of the degree of automation reduces the necessity of manual participation and optimizes the overall process. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0121] Figure 1 A schematic diagram of a classification method flow chart of an embodiment of the present invention;

[0122] Figure 2 Schematic diagram of video data feature extraction and multimodal feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0123] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0124] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0125] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0126] like Figure 1-Figure 2 As shown, a method for automatically classifying unstructured video data includes the following steps:

[0127] 1. Video data preprocessing:

[0128] Video preprocessing is an important step in video analysis and machine learning tasks, which directly affects the effect and efficiency of subsequent processing. The goal of preprocessing is to convert the video into a form suitable for analysis, remove noise, and extract useful information.

[0129] This part of the work includes standardizing the size and format of the input video and removing noise to make it meet the requirements of the model:

[0130] The main steps of this stage are:

[0131] a. Video format conversion: Use OpenCV to convert videos to a unified MP4 format to ensure compatibility.

[0132] b. Adjust the size of the video frame: usually by scaling or cropping it to meet the model input requirements. In this solution, Opencv is used to uniformly adjust it to 720*576 standard definition size;

[0133] c. De-noising: Use filters to remove noise from the video and improve video quality.

[0134] 2. Video data feature extraction and feature fusion:

[0135] The purpose of this part is to extract features from the preprocessed video data from three aspects: audio, video, and text, and perform time series processing on them respectively, and finally perform multi-modal fusion output on the features.

[0136] a. Audio processing stage of video data:

[0137] The audio file is first converted to a unified format and adjusted to mono. Then, the audio is resampled at a sampling rate of 22050 Hz and a short-time Fourier transform (STFT) is performed using a Hamming window of size 1024 and a jump length of 512 to obtain an amplitude spectrum. Finally, the amplitude spectrum is converted to a Log-Me l spectrum through a Mel filter bank.

[0138] The frequency spectrum of an audio signal usually has a large difference in the numerical range. Normalization processing can adjust all eigenvalues ​​to a similar range to prevent some eigenvalues ​​from being too large or too small and having an adverse effect on the training of the model.

[0139]

[0140] x is the original eigenvalue, μ is the mean of the feature, and σ is the standard deviation of the feature.

[0141] The normalized spectrogram is serialized by time step, so that the frequency feature of each time step becomes a feature vector.

[0142] Use Sinusoidal position encoding to add position information to each time step through sine and cosine functions to help Transformer perceive time order.

[0143] For time step t, its position encoding is:

[0144]

[0145]

[0146] Where i is the index of the encoding dimension, d is the total encoding dimension size, and t is the time step position in the sequence.

[0147] To meet the input requirements of Transformer, the dimension needs to be expanded (increase batch_size dimension). The matrix after the above processing is X By adding a batch dimension at the beginning, it becomes

[0148] X′=reshape(X)=X[np.newaxis,:,:];

[0149] np.newaxis is used to insert a dimension with a value of 1 into the first dimension (batch dimension) of the matrix, which is equivalent to promoting the data from two dimensions to three dimensions.

[0150] b. Audio data text processing:

[0151] Ⅰ. Audio preprocessing:

[0152] Audio sampling, converting the speech in the audio file into text through automatic speech recognition (ASR). Text preprocessing, including text cleaning and word segmentation, removing unnecessary punctuation marks, special characters, etc., to ensure the text is clean.

[0153] Ⅱ. Vectorization:

[0154] Input the preprocessed text into the pre-trained BERT model to obtain the context embedding vector of each word.

[0155] III. LSTM model for feature extraction:

[0156] LSTM takes the vectorized result as input, usually expressed as X = [x1, x2, ......., X T ], where T is the number of time steps, and the input x at each time step t is t is a vector.

[0157] The proportion of certain memory parts that should be forgotten at different time steps is obtained through calculation of the forget gate:

[0158] f t =σ(w f +[h t-1 , x t ]+b f );

[0159] W f and b f are the weight and bias of the forget gate;

[0160] The output gate calculation consists of two parts, the value of the input gate and the candidate memory state:

[0161] The value of the input gate determines how much is written to memory:

[0162] i t =σ(W i ·[h t-1 , x t ]+b i );

[0163] Candidate memory state: Use the tanh activation function to generate candidate memory content:

[0164]

[0165] W i , b i , W c , b C are the corresponding weight matrices and biases.

[0166] Arrives at the memory update unit to calculate the memory state C of the current time step t . C t By the memory C of the previous time step t-1 Together with the current candidate memory, decide:

[0167]

[0168] Output gate o t Controls how much information to extract from the memory cell to update the hidden state:

[0169] o t =σ(W o ·[h t-1 , x t ]+b o );

[0170] The output hidden state h t It depends on the current memory state and the value of the output gate:

[0171] h t =o t tanh(C t );

[0172] Using the hidden state sequence [h1, h2, ..., h T ] Input Transformer for feature fusion.

[0173] c. Video image processing of video data:

[0174] The main tasks of this stage are: video processing, LSTM modeling and video feature extraction, feature fusion and output.

[0175] Ⅰ. Video Processing:

[0176] Use the opencv suite to decode the video file into a frame sequence. Select non-repeated frame images and resize them to a uniform size so that the LSTM network can process them.

[0177] Next, the pixel values ​​of the image are normalized and scaled to a specific range (0 to 1) to speed up model training and improve model performance.

[0178] II. LSTM modeling and video feature extraction:

[0179] First, determine the LSTM network structure: the number of LSTM layers, the number of units in each LSTM layer, and the configuration of the output layer are adjusted according to the actual situation. This project sets 1 LSTM layer followed by a fully connected layer (Dense). The second step is to determine the input layer: the preprocessed feature sequence is used as the input of the LSTM network for training. The input data is three-dimensional, that is, (number of samples, time step, number of features).

[0180] The LSTM network receives an input and the cell state of the previous time step at each time step. Through the control of the forget gate, input gate and output gate, LSTM determines whether some information is forgotten, some new information is added, and some information is output to the hidden state.

[0181] During the training process, the temporal relationship between video frames is learned through the LSTM network and its weights are updated to minimize the loss function.

[0182] The output of the LSTM layer (especially the hidden state) contains the contextual information and temporal features of the video sequence. These outputs can be regarded as feature representations of the video.

[0183] III. Feature fusion and output:

[0184] The output of the LSTM network contains the overall feature representation of the video frame sequence.

[0185] d. Multimodal temporal information modeling stage:

[0186] This stage takes as input the output of the previous two stages. The visual input of the video is The audio input sequence is The text input sequence of the video is w represents the number of consecutive actions from this video, and these action sequences are defined as temporal context, d a ,d v ,d t Represents the input dimension of the audio modality, video modality, and text modality sequence. The modality temporal information is modeled as follows:

[0187] I. Feature Compression:

[0188] The high-dimensional video feature X v , audio feature X a , text feature X t Compress it into a low-dimensional space D to remove the noise mixed in the original data and save the cost of extracting unnecessary features;

[0189] II. Multimodal Modeling:

[0190] Introduce position encoding p and use temporal context w to learn absolute position encoding, which is shared between audio, video and text modalities, modeling the corresponding information from the three modalities and maintaining the continuity of previous and next information;

[0191] III. Modal Coding Fusion:

[0192] Modality Coding Add it to the learned vector and we get the vector:

[0193]

[0194]

[0195]

[0196] in, Represents the projection of video features and audio features from dimensions dv, da, and dt to a fully connected layer on a lower dimension D, position encoding

[0197] IV. Feature stitching:

[0198] Two learnable classification tags are added to the end of the video features, audio features, and text features, one for marking verbs, expressed as The other is used to mark nouns, expressed as And add the position coding information to get the tag:

[0199]

[0200]

[0201] The video, audio, and text features are fused into an encoded representation that includes video input features with position information and modality information, audio input features, text input features, and labeled gerund summary embeddings:

[0202]

[0203] V. Multi-head attention mechanism processing:

[0204] X e After slicing, the obtained video features with gerund tags, audio features, and text features are input into the Transformer respectively, and then processed by the multi-head attention mechanism:

[0205] MultiHead=Concat(h1, h2..., hN);

[0206]

[0207] Concat(·) represents the concatenation operation on the feature dimension, N represents the number of attention heads, and h i represents the self-attention mechanism, Q, K, and V represent Query, Key, and Value respectively, and d k represents the dimension of K;

[0208] VI. Vector Processing:

[0209] The vector is fed into the feedforward neural network layer. The activation function of the first layer is the ReLU activation function, and the second layer is the linear activation function:

[0210] ReLU(X)=max(0,X);

[0211] FFN(X)=max(0,W1X+b1)W2+b2;

[0212] VII. Normalization:

[0213] The feature vector is normalized through the normalization layer.

[0214] e. Multimodal feature fusion stage:

[0215] Visual Transformer and Auditory Transformer only use two modalities, video and audio. It is also necessary to increase the mutual influence and utilization between different modalities and use multimodal feature fusion methods. The method is as follows:

[0216] I. Feature Union:

[0217] Video features and audio features with temporal information after Transformer processing Connect and get the joint feature representation as:

[0218]

[0219] Among them, d is the dimension of the joint feature, and T is the transpose;

[0220] II. Calculate the joint video feature matrix:

[0221] By utilizing video feature X v And the joint feature representation J is calculated to obtain the joint correlation matrix C v :

[0222]

[0223] Indicates the connection video feature X vand the combined feature representation J is a learnable weight matrix, and tanh represents the activation function I

[0224] III. Calculate the joint audio feature matrix:

[0225] By utilizing the audio feature X a And the joint feature representation J is calculated to obtain the joint correlation matrix C a :

[0226]

[0227] Indicates the connection audio feature X a and the combined feature representation J is a learnable weight matrix, and tanh represents the activation function I

[0228] IV. Calculate the joint text feature matrix:

[0229] By utilizing the text feature X t And the joint feature representation J is calculated to obtain the joint correlation matrix C t :

[0230]

[0231] Represents the connection text feature X t and the combined feature representation J is a learnable weight matrix, and tanh represents the activation function I

[0232] V. Calculate the attention weights of the three modalities:

[0233]

[0234]

[0235]

[0236] Among them, ReLu is the activation function, W cv , W ca , W v , W a , Calculate H v , H a , H t They are the attention map of the video modality and the attention map of the audio modality respectively;

[0237] VI. Calculate fusion features:

[0238] Calculate the video feature X that integrates the audio information att，v , calculate the audio feature X that integrates the video informationatt，a , text feature X att，t The calculation process is:

[0239] X att，v =H v W hv +X v ;

[0240] X att，a =H a W ha +X a ;

[0241] X att，t =H t W ht +X t ;

[0242] VII. Computing audio-visual and text features:

[0243] Fusion X att，v , X att，a , X att，t Get feature X att :

[0244] X att =[X att，v ;X att，a ;X att，t ].

[0245] 3. Video data of the same category is matched:

[0246] After processing and fusion, the obtained feature vector already contains multimodal information of video, audio, and text, and the contextual relationship of each moment in the sequence is obtained through time series modeling. Next, these feature vectors need to be classified to complete the video classification task.

[0247] First, the fused feature vector is expressed as, the feature vector X att It can be regarded as a comprehensive representation of video, audio and text information. In order to perform classification tasks, a fully connected layer can be used to att Process it and map it to the probability distribution of different categories. The calculation formula of the fully connected layer is as follows:

[0248] O = softmax(W c X att +b c );

[0249] Among them, W c and b care the category x-dimensional weight matrix and bias vector of the fully connected layer. Initially, W c The parameters of W are usually randomly initialized. As the training process progresses, c According to the input feature X att and the true label y i The error between them (i.e., the loss function) is continuously updated, and the parameters that can effectively map the input features to the correct categories are gradually learned. The role of positional encoding is to provide the model with the position of each input element in the sequence, so that the model can consider the order relationship of the elements in the sequence. O is the output category probability distribution vector, and the Softmax function is used to convert the output value of the linear combination into a probability value. The formula is as follows:

[0250]

[0251] Among them, O i is the output score of category i, and C is the total number of categories.

[0252] In practical applications, the video category corresponding to the category with the highest probability in O indicates that the model predicts that the video belongs to this category with the highest probability. Finally, by comparing the predicted category with the true label, the loss function is calculated, and the model is optimized through back propagation. The commonly used loss function is the cross entropy loss function, which is defined as follows:

[0253]

[0254] Among them, y i is the encoding of the true category, p i is the probability distribution of the model output.

[0255] Through the above process, the model can correctly classify video clips into corresponding categories.

[0256] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0257] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for automatic classification of unstructured video data, characterized in that: The following steps are involved: S1, video data preprocessing: standardize the video data to a unified format and size, and perform noise processing; S2, video data feature extraction and multimodal feature fusion: extract features from the preprocessed video data in the three modes of audio, video and text, perform time series processing, and perform multimodal feature fusion; S3, video data same category matching: classify the fused multimodal features to complete the video classification.

2. The method for automatic classification of unstructured video data according to claim 1, characterized in that: The video data preprocessing in S1 includes: S11, video format conversion: use OpenCV to convert the video to a unified MP4 format; S12, adjust the size of the video frame: by scaling or cropping, use Opencv to uniformly adjust to 720*576 standard definition size; S13, denoising: using a filter to remove noise from the video.

3. The method for automatic classification of unstructured video data according to claim 2, characterized in that: The video data feature extraction and multimodal feature fusion in S2 include: S21, audio processing of video data: convert the audio file to mono and resample it, generate a spectrogram using short-time Fourier transform, and convert it to Log-Mel spectrum through Mel filter. After normalization, the spectrum is serialized by time step and position coding is added; S22, audio data text processing: convert audio into text through automatic speech recognition, extract word embedding using the BERT model, and capture the temporal characteristics of the text through the LSTM model; S23, video image processing: decode the video into frame sequences and standardize the size and pixel values, use the LSTM network to process the frame sequences, learn the temporal relationship between video frames, and generate video feature representation; S24, multimodal temporal information modeling: compressing the features of audio, video, and text and introducing position coding for temporal modeling; S25, multimodal feature fusion: The features of audio, video, and text modalities are fused through Transformer and multi-head attention mechanism to generate a comprehensive feature representation.

4. The method for automatic classification of unstructured video data according to claim 3, characterized in that: The audio processing of the video data in S21 includes: S211, Audio conversion and mono adjustment: Convert audio files to a unified format and adjust to mono I S212, Audio Samples: Resample the audio at a sampling rate of 22050 Hz; S213, short-time Fourier transform: perform short-time Fourier transform using a Hamming window of size 1024 and a jump length of 512 to obtain an amplitude spectrum diagram; S214, Amplitude-spectrum conversion: converting the amplitude spectrum into a Log-Mel spectrum through a Mel filter bank; S215, normalization processing: adjusting the range of the characteristic value by normalization processing; S216, position encoding: Use Sinusoidal position encoding to add position information to each time step through sine and cosine functions. For time step t, its position encoding is: Where i is the index of the encoding dimension, d is the total encoding dimension size, and t is the time step position in the sequence; S217, Dimension expansion: By increasing the batch dimension (batch_size dimension), it is expressed as: X′=reshape(X)=X[np.newaxis,:,:]; Among them, np.newaxis is used to insert a dimension with a value of 1 in the first dimension of the matrix, which is equivalent to upgrading the data from two dimensions to three dimensions.

5. The method for automatic classification of unstructured video data according to claim 4, characterized in that: The audio data text processing in S22 includes: S221, audio preprocessing: converting the speech in the audio file into text through automatic speech recognition, and preprocessing the text, including text cleaning and word segmentation; S222, vectorization: input the preprocessed text into the pre-trained BERT model to obtain the context embedding vector of each word; S223, feature extraction: Capture the temporal features of the text through the LSTM model, including: Input initialization: The vectorized result is used as input, expressed as X = [x1, x2, ..., X T ], where T is the number of time steps, and the input x at each time step t is t is a vector; Forget gate calculation: After the forget gate calculation, the proportion of the memory part forgotten at different time steps is obtained, expressed as: f t =σ(w f +[h t-1 ,x t ]+b f ); Among them, w f and b f They are the weight and bias of the forget gate respectively; Input gate and candidate memory calculation: The value of the input gate determines the amount of memory written, and the tanh activation function is used to generate candidate memory content, expressed as: i t =σ(W i ·[h t-1 ,x t ]+b i ); Among them, W i 、b i , W c 、b C are the corresponding weight matrices and biases; Memory state update: The results of the forget gate and the input gate jointly determine the memory state of the current time step, expressed as: Output gate and hidden state update: Output gate o t Control extracts information from the memory cell to update the hidden state, and uses the output gate and the current memory state to calculate the new hidden state, expressed as: the t =σ(W o ·[h t-1 ,x t ]+b o ); h t =o t ·tanh(C t ); Hidden state fusion: Use the hidden state sequence [h1, h2, …, h T ] Input Transformer for feature fusion.

6. The method for automatic classification of unstructured video data according to claim 5, characterized in that: The video image processing in S23 includes: S231, video processing: use the opencv suite to decode the video file into a frame sequence, select non-repeated frame images, adjust them to a uniform size, and normalize the pixel values ​​of the images; S232, LSTM modeling and video feature extraction: Process feature sequences through LSTM networks and learn the temporal relationship between video frames, including: Determine the LSTM network structure: Set the structure of the LSTM network according to actual needs, including the number of LSTM layers, the number of units in each LSTM layer, and the configuration of the output layer; Set the input layer: Use the processed feature sequence as the input of the LSTM network. The format of the input data is three-dimensional I LSTM time step processing: At each time step, the LSTM network receives the input of the current step and the cell state of the previous time step. The forget gate determines the information to be forgotten, the input gate determines the amount of new information to be added, and the output gate controls the information of the current step to be output to the hidden state. Learning the temporal relationship between video frames: During the training process, the temporal relationship between video frames is learned through the LSTM network, and its weights are updated to minimize the loss function; Generate video feature representation: The output of LSTM contains the contextual information and temporal features of the video sequence; S233, feature fusion and output: The output of the LSTM network contains the overall feature representation of the entire video frame sequence.

7. The method for automatic classification of unstructured video data according to claim 6, characterized in that: The multimodal time series information modeling in S24 includes: S241, feature compression: high-dimensional video features X v , audio feature X a , text feature X t Compress to low-dimensional space D; S242, Multimodal Modeling: Introducing positional encodings and learning positional encodings with temporal context w, shared between audio and video and text modalities; S243, Modal Coding Fusion: Modal Coding Add it to the learned vector and we get the vector: in, Represents the projection of video features and audio features from dimensions dy, da, and dt to a fully connected layer on a lower dimension D, position encoding S244, feature concatenation: Two learnable classification tags are added to the end of the video features, audio features, and text features, one for marking verbs, expressed as The other is used to mark nouns, expressed as And add the position coding information to get the tag, and fuse the video, audio, and text features into a coding representation including video input features with position information and modality information, audio input features, text input features, and gerund summary embedding with tags, expressed as: S245, multi-head attention mechanism processing: for X e After slicing, the obtained video features with gerund tags, audio features, and text features are input into the Transformer respectively, and processed by the multi-head attention mechanism, which is expressed as: MultiHead=Concat(h1, h2..., hN); Among them, Concat(·) represents the concatenation operation on the feature dimension, N represents the number of attention heads, and h i represents the self-attention mechanism, Q, K, and V represent Query, Key, and Value respectively, and d k Represents the dimension I of K S246, vector processing: the vector is sent to the feedforward neural network layer. The activation function of the first layer is the ReLU activation function, and the second layer is the linear activation function, which is expressed as: ReLU(X)=max(0,X); FFN(X)=max(0,W1X+b1)W2+b2; S247, normalization: normalize the feature vector through the normalization layer.

8. The method for automatic classification of unstructured video data according to claim 7, characterized in that: The multimodal feature fusion in S25 includes: S251, feature combination: video features and audio features with temporal information after Transformer processing Connect and get the joint feature representation as: Among them, d is the dimension of the joint feature, and T is the transpose; S252, calculating the joint video feature matrix: by using the video feature X v And the joint feature representation J is calculated to obtain the joint correlation matrix C v , expressed as: in, Indicates the connection video feature X v The combined feature representation J is a learnable weight matrix, and tanh represents the activation function; S253, calculating the joint audio feature matrix: by using the audio feature X a And the joint feature representation J is calculated to obtain the joint correlation matrix C a , expressed as: in, Indicates the connection audio feature X a The combined feature representation J is a learnable weight matrix, and tanh represents the activation function; S254, calculate the joint text feature matrix: by using the text feature X t And the joint feature representation J is calculated to obtain the joint correlation matrix C t , expressed as: in, Represents the connection text feature X t The combined feature representation J is a learnable weight matrix, and tanh represents the activation function; S255, calculate the attention weights of multiple modalities: calculate the attention weights of the joint video feature, joint audio feature, and joint text feature modalities, expressed as: Among them, ReLu is the activation function, Calculate H v , H a , H t They are the attention map of the video modality and the attention map of the audio modality respectively; S256, calculate fusion features: calculate the video feature X that integrates the audio information att,v , calculate the audio feature X that integrates the video information att，a , text feature X att,t , expressed as: X att，v =H v W hv +X v ; X att,a =H a W ha +X a ; X att,t =H t W ht +X t ; S257, Computational Audiovisual and Text Features: Fusion X att，v , X att,a , X att,t Get feature X att , expressed as: X att =[X att,v ;X att,a ;X att,t ]。 9. The method for automatic classification of unstructured video data according to claim 8, characterized in that: The same category matching of the video data in S3 includes: S31, Feature vector fusion and representation: The fused feature vector represents the comprehensive representation of video and audio information as well as text information I S32, fully connected layer mapping: Use the fully connected layer to map X att Process it and map it to the probability distribution of different categories, expressed as: O=softmax(W c X att +b c ); Among them, W c and b c They are the category x-dimensional weight matrix and bias vector of the fully connected layer respectively; S33, Application of Positional Encoding: Use positional encoding to provide Transformer with the position information of each input element in the sequence; S34, calculate category probability: Use the Softmax function to convert the output value of the fully connected layer into a probability distribution, expressed as: Among them, O i is the output score of category i, and C is the total number of categories; S35, select the prediction category: according to the probability output by Softmax, select the category with the highest probability as the prediction result, indicating that the probability that the video belongs to this category is the highest; S36, calculate the loss function: use the cross entropy loss function to calculate the difference between the output and the true label, expressed as: Among them, y i is the encoding of the true category, p i is the probability distribution of the model output; S37, Back Propagation and Model Optimization: By comparing the differences between the predicted results and the true labels, the back propagation algorithm is used to optimize the model parameters.

Images