Model training and content auditing management system based on intelligent data acquisition

Abstract

The invention relates to the technical field of intelligent data processing, in particular to a model training and content auditing management system based on intelligent data acquisition, which comprises an acquisition processing unit, a model training unit and a content auditing unit. The model training unit constructs a multi-modal fusion model, a joint training mechanism of the model learns modal association, adaptive learning rate strategy optimization training and early stop technology anti-overfitting, the trained model is used for content auditing, the content auditing unit receives multimedia content, extracts key frames, audio voice content and subtitle text, and performs feature representation and fusion to obtain a subtitle text. Whether violation occurs or not is judged through the trained model, all the units cooperate with one another, the collecting and processing unit provides a data basis for model training, the model training unit provides a precise model for content auditing, effective auditing of multimedia content is jointly achieved, and legitimacy and health of the content are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent data processing technology, and in particular to a model training and content review management system based on intelligent data standard collection. Background Art

[0002] Intelligent data processing is an important technology. With the rapid development of the Internet and multimedia technology, various multimedia content such as text, images, audio and video in the network environment have shown an explosive growth trend. In the current information dissemination ecology, these multimedia contents are widely circulated in social platforms, online entertainment, online education and other fields. In this process, it is very important to ensure the legality and health of the content, especially to strictly prevent the spread of content containing violence, pornography, infringement and other illegal information.

[0003] However, the current content review methods have obvious limitations. Most review methods only focus on single-modal data. For example, in text review, they often simply perform keyword matching. For image review, they are limited to the recognition of specific visual features. This single-modal review method is inadequate when faced with complex multimedia content and cannot comprehensively and accurately review it. In order to solve this technical problem, we provide a model training and content review management system based on intelligent data sampling. Summary of the invention

[0004] The purpose of the present invention is to provide a model training and content review management system based on intelligent data collection to solve the problems raised in the above-mentioned background technology.

[0005] The present invention extracts and fuses video key frames and acoustic features in multimedia audio to obtain comprehensive features, and transmits the comprehensive features to a multimodal fusion model based on deep learning. It repeatedly trains and adjusts model parameters to obtain a classifier, and audits and identifies multimodal data based on the classifier.

[0006] To achieve the above purpose, a model training and content review management system based on intelligent data collection is provided, including a data collection unit, a model training unit and a content review unit;

[0007] The acquisition processing unit is used to acquire multimodal data from multiple data sources, wherein the multimodal data includes text, image, audio and video data, and to add metadata including timestamp and source information to each data during the acquisition process, and to dynamically adjust acquisition parameters and pre-process the data according to the characteristics of different modal data;

[0008] The model training unit constructs a multimodal fusion model based on deep learning according to the multimodal data, including sub-networks for processing data of different modalities respectively, and the sub-networks cooperate and share information with each other through a joint training mechanism, and the joint mechanism is realized by introducing a correlation term between modalities in a loss function. The model training unit also includes an adaptive learning rate adjustment strategy and an early stopping technology;

[0009] The content review unit is used to receive multimedia content to be reviewed, extract video key frames, audio voice content and subtitle text therein, and use a multimodal fusion model to represent and fuse their features, and finally use a classifier to determine whether the content contains illegal information. The classifier is trained based on multimodal data.

[0010] As a further improvement of the technical solution, the acquisition processing unit dynamically adjusts the acquisition parameters and preprocesses the data according to the characteristics of different modal data, as follows:

[0011] Normalize, crop and denoise the image data;

[0012] Perform noise reduction and sampling rate adjustment on audio data;

[0013] For video data, the acquisition frequency is dynamically adjusted and key frames are extracted according to its frame rate and resolution, and the same preprocessing operations as image data are performed on the key frames;

[0014] For text data, the collection time interval is determined according to the information update speed, and lexical and syntactic analysis and encoding conversion processing are performed.

[0015] As a further improvement of the technical solution, the joint training mechanism in the model training unit is as follows:

[0016] When calculating the loss function, the prediction loss of each sub-network for its own modal data and the loss term of the interaction information between modalities are combined to calculate the minimized total loss. The model parameters are optimized by minimizing the total loss to obtain the correlation between different sub-network modalities. The loss term of the interaction information between modalities is obtained by calculating the similarity between different modal feature vectors.

[0017] As a further improvement of the technical solution, the adaptive learning rate adjustment strategy in the model training unit is as follows:

[0018] During the training process, the learning rate is adjusted according to the changes in the model performance indicators on the training sequence and the validation set. If the loss value on the validation set shows a monotonically increasing trend for several consecutive times, the learning rate is increased. If the loss value on the validation set shows a monotonically decreasing trend for several consecutive times and the rate of change is less than the preset threshold, the learning rate is reduced, and the adjustment range of the learning rate is within the preset range.

[0019] As a further improvement of the technical solution, the early stopping technology of the model training unit specifically includes:

[0020] The dataset is divided into training set, validation set and test set. After each training round, the model performance is evaluated on the validation set. When the validation set performance index begins to decline within consecutive preset rounds, the training is paused and the model is evaluated on the test set. If the test set performance also shows a downward trend, the training is stopped.

[0021] As a further improvement of the technical solution, when extracting video key frames, the content review unit extracts key frames based on key action change points of the video content, and increases the extraction frequency of key frames for video content whose dynamic changes exceed a threshold, as follows:

[0022] The optical flow method is used to analyze the continuous frames of the video. By calculating the optical flow vector of each pixel, the movement direction and speed information of the object in the video are obtained. The background difference method is combined to highlight the movement of the foreground object. By establishing a background model, the current frame is subtracted from the background model to obtain the mask of the foreground object.

[0023] Based on the motion information obtained from optical flow and background difference, the change rate and direction of the object's motion speed between adjacent frames are calculated. When the change rate exceeds the preset threshold, it is determined to be a key action change point. The machine learning model is used to learn and identify the action change pattern. When the model detects that a specific type of key action occurs, the corresponding frame is determined as a key frame.

[0024] Perform pixel-by-pixel difference calculation on adjacent frames of the video, add and average the absolute values ​​of the differences of all pixels, and obtain the average absolute value of the inter-frame differences. If the average absolute value exceeds the set threshold, increase the acquisition frequency of the key frames according to the preset ratio.

[0025] As a further improvement of the technical solution, when extracting the audio voice content, the content review unit uses speech recognition technology to convert the audio into voice content in text form, and extracts the acoustic features of the voice at the same time, as follows:

[0026] The pre-processed audio signal is framed and windowed, the audio is divided into time segments of a set length and multiplied by a window function, then the discrete Fourier transform of each frame of audio is calculated to obtain the spectrum, and then the spectrum is mapped to the Mel frequency scale, and finally the MFCC features are obtained through discrete cosine transform;

[0027] The MFCC features are input into the preset speech recognition based on hidden Markov model and Gaussian mixture model for training and recognition. In the training stage, the parameters of the hidden Markov model and Gaussian mixture model are estimated using the labeled speech data. In the recognition stage, the MFCC features are input into the trained model to calculate the probability of each word. Finally, the text output is obtained through the decoding algorithm, and the fundamental frequency of the speech signal is calculated using the average amplitude difference function as the acoustic feature.

[0028] As a further improvement of the technical solution, the content review unit adopts a weighted fusion method when using the multimodal fusion model to represent and fuse the extracted information, as follows:

[0029] Initial weights are set based on prior knowledge and experience, and experiments are conducted on labeled data sets to observe the recognition accuracy and recall rate of the model for various types of violation information under different weight combinations. Finally, the correlation of different modal data in different types of violation information is analyzed. If it is found that the correlation of a certain modal data with the violation label in a certain violation type exceeds the preset threshold, its weight is increased proportionally.

[0030] According to the determined weights, each dimension of the acquired acoustic features of the video key frame and the audio speech content is multiplied to obtain the weighted feature vectors and record them separately. Finally, the corresponding dimensions of the weighted feature vectors are added to obtain the fused feature vector.

[0031] As a further improvement of the technical solution, the classifier in the content review unit identifies illegal information by adjusting the hyperparameters in the deep learning classification network, as follows:

[0032] When training the classifier, the fused feature vector is input into the deep learning classification network, and the deep learning classification network outputs the probability value of the sample belonging to each violation category. For each training sample, its cross entropy loss is calculated, and then the loss values ​​of all samples are averaged to obtain the loss value of the entire training batch;

[0033] Based on the calculated loss value, the network parameters are updated through the back-propagation algorithm. Starting from the output layer, the gradient of the loss function for each parameter is calculated, and then the weights and biases of each layer in the network are adjusted according to the gradient. Finally, the final number of network layers is determined by experiments on the training set and the validation set, and the violation information is identified based on the deep learning classification network corresponding to the final number of network layers.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] In the model training and content review management system based on intelligent data collection, the model training unit uses a joint training mechanism to accurately learn modal associations, an adaptive learning rate strategy to efficiently optimize training, and early stopping technology to prevent overfitting, providing a powerful and accurate multimodal fusion model foundation for the content review unit. Based on this model, the content review unit uses optical flow and background difference methods as well as speech recognition technology to comprehensively and accurately extract key video information, deeply understand audio content, and effectively integrate multimodal features to make violation judgments, which effectively ensures the legality and health of content in the network environment and adapts to large-scale content review needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is an overall block diagram of the present invention.

[0037] The meaning of each number in the figure is:

[0038] 1. Data collection unit; 2. Model training unit; 3. Content review unit. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] With the rapid development of the Internet and multimedia technology, multimedia content in the network environment has exploded, and the problem of content dissemination containing illegal information such as violence, pornography, and infringement has become prominent. However, existing content review methods mostly focus on single-modal data, which has limitations when facing complex multimedia content and cannot conduct comprehensive and accurate review.

[0041] The present invention provides a model training and content review management system based on intelligent data collection. Figure 1 As shown, it includes a data collection unit 1, a model training unit 2 and a content review unit 3;

[0042] The acquisition processing unit 1 is used to acquire multimodal data from multiple data sources, where the multimodal data includes text, image, audio and video data, and to add metadata including timestamp and source information to each data during the acquisition process, and to dynamically adjust acquisition parameters and pre-process the data according to the characteristics of different modal data;

[0043] Data of different modalities have their own unique properties. Image data has different pixel value ranges, edge information, and noise conditions. Audio data has noise interference and sampling rate differences. Video data has different frame rates and resolutions, so the amount of data and the distribution of key information also vary. Text data has different update speeds and requires a unified format and grammatical structure analysis to be better processed. Processing these characteristics can make the data more in line with the requirements of subsequent analysis and model training.

[0044] The acquisition processing unit 1 dynamically adjusts the acquisition parameters and preprocesses the data according to the characteristics of different modal data, as follows:

[0045] The image data is normalized, cropped and denoised, the audio data is denoised and the sampling rate is adjusted, the video data acquisition frequency is adjusted according to the frame rate and resolution, key frames are extracted and preprocessed like image data, the text data acquisition interval is determined according to the update speed, and lexical and syntactic analysis and encoding conversion are performed to improve data quality, enhance processing efficiency, and enhance system adaptability. Data collection is more accurate and efficient, the improved data quality after preprocessing is conducive to subsequent processing, and the overall system operation is more stable and efficient to meet the needs of large-scale multimedia content processing and review.

[0046] The model training unit 2 constructs a multimodal fusion model based on deep learning according to the multimodal data, including sub-networks for processing data of different modalities respectively. The sub-networks cooperate with each other and share information through a joint training mechanism. The joint mechanism is realized by introducing correlation terms between modalities in the loss function. The model training unit 2 also includes an adaptive learning rate adjustment strategy and an early stopping technology.

[0047] Data of different modalities often have inherent correlations in real scenarios. For example, in video content, images and audio appear at the same time and cooperate with each other. Images of violent scenes are usually accompanied by intense sounds. By considering the interactive information between modalities through joint training, the model can better capture this cross-modal correlation, thereby improving the understanding and classification capabilities of the content.

[0048] The joint training mechanism in model training unit 2 is as follows:

[0049] Corresponding subnetworks are constructed for different modal data. For image and video key frame data, a convolutional neural network is constructed. The convolutional neural subnetwork is composed of several convolutional layers, pooling layers and fully connected layers. The convolution kernel size, number and step size of the convolutional layer are set according to the image data. In this way, the local features and hierarchical structure of the image data can be effectively extracted, which is conducive to the subsequent accurate classification of the image content. It can make full use of the two-dimensional structural characteristics of the image to extract representative visual features. For audio and speech content, a recurrent neural subnetwork is constructed. After preprocessing the audio signal, the audio feature sequence is input into the recurrent neural subnetwork to adapt to the temporal characteristics of the audio and capture the changes in speech in the time dimension. It performs better in tasks such as audio sentiment analysis and speech content understanding, and can more keenly perceive the illegal speech information contained in the audio. For text data, a recurrent neural subnetwork is also constructed, and the text is vectorized and preprocessed before input, which helps to understand the grammatical and semantic structure of the text.

[0050] Each subnetwork will generate prediction loss during the training process. Taking the image subnetwork as an example, in the image classification task, the cross entropy loss function is used to calculate the difference between the predicted result and the true label. For a batch of training images, the image is input into the convolutional neural subnetwork to obtain the predicted category probability distribution, which is compared with the true category label to calculate the cross entropy loss. This method directly measures the gap between the predicted result and the actual situation, provides a clear optimization direction for model training, and enables the model to converge quickly in the direction of reducing prediction errors, improve the accuracy of image classification, and then more accurately judge whether the image contains illegal content in image content review. The audio and text subnetworks also use the same method to calculate the prediction loss in their respective tasks. This ensures that each modal subnetwork is trained in the direction of accurately predicting its own modal data, laying the foundation for better performance after joint training and improving the overall system's ability to handle complex multimodal content.

[0051] In each training batch, a feature vector is extracted from the last convolutional layer output of the convolutional neural network subnetwork or the last hidden layer output of the recurrent neural network. For the image subnetwork, the dimension of the extracted feature vector is set to n1, the dimension of the feature vector extracted by the audio subnetwork is set to n2, and the dimension of the feature vector extracted by the text subnetwork is set to n3. This method of extracting feature vectors at a specific layer can obtain the abstract features of each modality at different levels, providing a suitable data basis for accurately calculating the similarity between modalities, so that the model can better learn that the association between modalities is based on meaningful feature representation.

[0052] For the feature vectors of the image and audio sub-networks, calculate the cosine similarity between them. Let the image feature vector be v1 and the audio feature vector be v2. Their cosine similarity By calculating the cosine similarity between each pair of feature vectors of different modalities, the degree of similarity between them can be measured. Cosine similarity can effectively measure the consistency of the vector in direction, is insensitive to the length of the feature vector, accurately reflects the semantic correlation between different modal features, and helps the model learn the intrinsic connection between modalities.

[0053] According to the calculated similarity, the loss term of the mutual information between modalities is constructed and s 12 As part of the interactive information loss, this way of constructing loss terms guides model training according to the expectations of the relationship between modalities, and flexibly adjusts the model's learning direction of the relationship between modalities according to the characteristics of multimodal data and task requirements. For multimodal situations, the interactive information loss terms between all modalities are added together to obtain the total interactive information loss between modalities. This comprehensively considers the correlation information between all modalities, improves the performance of the model in complex multimodal scenarios, and can more accurately and comprehensively judge whether multiple modal information contains illegal content in content review, avoiding misjudgment caused by only considering partial modal relationships.

[0054] The prediction loss of each sub-network is added to the loss of the inter-modal interactive information to obtain the total loss. Let the prediction loss of the image sub-network be L1, the prediction loss of the audio sub-network be L2, the prediction loss of the text sub-network be L3, and the inter-modal interactive information loss be L int , then the total loss L=L1+L2+L3+L int This comprehensive calculation method of total loss unifies the intra-modal prediction and inter-modal association, balances the independent training of each modal sub-network and multi-modal fusion training, so that the model does not ignore the relationship between modalities while learning its own modal knowledge. After calculating the gradient according to the total loss, the model parameters are updated using the adaptive learning rate calculation method, and the parameters are adjusted in the direction of minimizing the total loss. During the training process, the model continuously learns the association between modalities and accurately predicts the data of each modality. The gradient-based optimization algorithm can quickly adjust the model parameters according to the changes in the loss function, accelerate the model training process, and improve the efficiency of model training.

[0055] During the model training process, as the training progresses, changes in data distribution and model parameters will cause the optimal value of the learning rate to change. If a fixed learning rate is always used, the learning speed may be too slow in the early stage of training, or it may not converge to the optimal solution in the later stage of training due to the learning rate being too large. By adjusting the learning rate according to the training rounds and the performance of the validation set, you can better adapt to this complex training dynamics.

[0056] The adaptive learning rate adjustment strategy in model training unit 2 is as follows:

[0057] During the model training process, a counter is set to record the training rounds. After each round of traversal of the entire training data set, the counter is increased by 1. This allows you to clearly track the progress of training and understand the performance of the model at different stages. At the same time, after each training round, the validation set data is input into the current model for evaluation, and the loss value and accuracy on the validation set are calculated. By evaluating the validation set after each round, the performance of the model on unseen data can be obtained in a timely manner, enabling the model to have better processing capabilities for new data in practical applications, thereby improving the accuracy of the model in tasks such as content review.

[0058] To judge the changing trend of the loss value of the validation set, a window size u is set to observe the changes in the loss value of u consecutive rounds. This window-based observation method can smooth short-term fluctuations, more accurately judge the long-term trend of the loss value, make the learning rate adjustment decision more stable and reasonable, avoid frequent adjustment of the learning rate due to local fluctuations, and improve the stability and efficiency of model training.

[0059] When the loss value on the validation set is found to be monotonically increasing for u consecutive times, the learning rate increase operation is triggered. Suppose the current learning rate is η, and the increase multiple is m1, then the new learning rate η new =η×m1. Increasing the learning rate can help the model escape from the possible local optimal solution or speed up the learning speed, so that the model can find better parameter values ​​more quickly and reduce the training time. In order to avoid the learning rate growing too fast, a learning rate upper limit η can be set. max , if η new >η max , then set the learning rate to η max Setting an upper limit can prevent the learning rate from being too large and causing model instability, avoid divergence of model parameters due to excessive learning rate, ensure that the model can converge stably, improve the reliability of model training, and ensure the stability of the content review model during long-term training.

[0060] If the loss value on the validation set shows a monotonically decreasing trend for u consecutive times, calculate the rate of change of the loss value for these u times. The formula for calculating the rate of change is: This method of calculating the rate of change can accurately quantify the degree of change in the loss value, making the learning rate adjustment more accurate, which is conducive to more fine-tuning the parameters when the model is close to the optimal solution and improving the model performance. dq is the loss value of the current round, loss ed is the loss value of the previous round. If the rate of change is less than the preset threshold ε, the learning rate is reduced. The reduction factor is m2, and the new learning rate η new =η×m2; Reducing the learning rate when close to convergence helps to adjust the parameters more finely and improve the performance indicators of the model on the validation set and test set. At the same time, setting a lower limit of the learning rate η min , if ηnew <η min , then set the learning rate to η min ,Setting a lower limit can prevent the learning rate from being too small and causing training to stagnate, avoid the training from falling into an infinitely slow state, ensure that the model completes training within a reasonable time, and improve training efficiency and model practicality.

[0061] As the number of training rounds increases, the model may over-learn the features of the training set, resulting in a decrease in performance on the validation set and test set, which is called overfitting. Through early stopping technology, when the performance of the validation set begins to decline, training is paused in time to avoid the model from continuing to develop in the direction of overfitting and retain the model's generalization ability on new data.

[0062] The early stopping technique of model training unit 2 includes:

[0063] Determine the total number of sample data sets and divide the data sets into training sets, validation sets, and test sets. When dividing the data, use random sampling to ensure that the data distribution of each subset is as similar as possible to the distribution of the original data set. For data with time series characteristics, it is necessary to pay attention to dividing them in chronological order to avoid data leakage problems, make the data set division more in line with the actual situation, improve the accuracy of model evaluation, and effectively train models on data of various sizes and types to avoid model evaluation bias caused by unreasonable division. Improve the content review model's ability to process data of different modalities. For example, if the data is video, audio, or text data collected in chronological order, use the front part as the training set, the middle part as the validation set, and the back part as the test set. This scientific data division method ensures the independence and representativeness of the data, improves the generalization ability of the model, and enables the model to have better processing capabilities for new multimedia data during content review, reducing misjudgments caused by improper data division.

[0064] During the model training process, each time all samples in the training set are traversed, the validation set data is input into the model for evaluation. This regular evaluation method can timely grasp the performance of the model on unseen data, help to quickly adjust the training strategy, avoid overfitting of the model to the training data, and improve the content review model's recognition accuracy of different types of illegal content. When the accuracy of the validation set begins to decline within consecutive preset rounds, the training is suspended, and the performance indicator value of each round on the validation set is recorded. A counter count is set to record the rounds in which the accuracy rate decreases continuously. Initially, count = 0. This recording and counting method can clearly track the changing trend of the model performance, accurately determine whether the model is beginning to show a performance decline trend, and provide a reliable basis for early stopping of training. When the accuracy of the validation set is found to be lower than that of the previous round in a certain round, count is increased by 1. This timely response to performance changes can quickly Capture model anomalies and reduce the waste of training resources. If the count reaches the preset round, the training process is suspended. Reasonable preset rounds can balance the adequacy of model training and avoid overfitting, and reduce misjudgment. After pausing training, input the test set data into the current model for evaluation, calculate the accuracy, avoid erroneous stopping of training due to accidental fluctuations in the validation set, ensure that the model does have performance degradation when stopping training, and improve the stability of model training. If the test set accuracy also shows a downward trend, stop training to prevent the model from continuing to train when the performance has deteriorated, and improve the final performance of the model in content review tasks. If the test set accuracy does not decrease, choose to continue training, but you need to reinitialize count to 0, continue to observe the validation set accuracy, avoid stopping training too early, and enable the model to fully learn the data features, further improving the accuracy and generalization ability of the model in content review.

[0065] The content review unit 3 is used to receive multimedia content to be reviewed, extract video key frames, audio voice content and subtitle text, and use a multimodal fusion model to represent and fuse their features. Finally, a classifier is used to determine whether the content contains illegal information. The classifier is trained based on a deep neural network multimodal fusion model.

[0066] When extracting video key frames, the content review unit 3 extracts key frames based on key action change points of the video content, and increases the extraction frequency of key frames for video content whose dynamic changes exceed the threshold, as follows:

[0067] The optical flow method is used to analyze the continuous frames of the video. By calculating the optical flow vector of each pixel, the movement direction and speed information of the object in the video are obtained. The optical flow method is based on the assumption of constant brightness and small motion. The movement of pixels is calculated by solving the optical flow constraint equation. Suppose in two adjacent frames I(x, y, t) and I(x+Δx, y+Δy, t+Δt), according to the optical flow constraint equation Ix u+I y v+I t =0; where I x ,I y is the partial derivative of the image in the x and y directions, u and v are the components of the optical flow vector in the x and y directions, and I t It is the time partial derivative. This method can accurately capture the subtle movements of objects. In content review, for illegal content involving actions, it can more comprehensively extract key frames and improve the detection ability of such illegal content.

[0068] The background difference method is combined to highlight the movement of foreground objects. By building a background model, the current frame is subtracted from the background model to obtain the mask of the foreground object. For a surveillance video scene, the background is relatively stable. The mixed Gaussian model can be used to model the background well. When an object enters, the contour of the object can be clearly obtained by subtracting the current frame from the background model, highlighting the moving foreground object, reducing background interference, focusing on key motion information, and improving the pertinence of key frame extraction. The extracted key frames can better reflect the important actions in the video, which helps to improve the efficiency of content review, especially for video content review with complex backgrounds;

[0069] According to the motion information obtained by optical flow and background difference, the change rate and direction of the object's motion speed between adjacent frames are calculated. When the change rate exceeds the preset threshold, it is determined to be a key action change point. Assuming that the object's motion speeds in two adjacent frames are v1 = (u1, v1) and v2 = (u2, v2), the formula for the change rate of motion speed is: By setting thresholds based on actual data and application scenarios, drastic changes in motion can be detected. This method based on speed change rate can accurately capture sudden changes in motion. In content review, for illegal content containing rapid motion changes, key frames can be extracted more timely, thereby enhancing the accuracy of the review of such content.

[0070] Use machine learning models to learn and identify action change patterns. When the model detects that a specific type of key action has occurred, the corresponding frame is identified as a key frame. A convolutional neural network model can be used to classify actions. For example, a convolutional neural network model is trained to identify specific actions such as fighting and running. When the model determines that a frame contains these key actions, it is marked as a key frame. This model-based method can use a large amount of data to learn complex action patterns, identify complex action types, and improve the accuracy and intelligence of key frame extraction. For various types of action scenes, whether common or field-specific actions, key frames can be extracted more accurately. In content review, it can more widely adapt to different types of video content and reduce key frame omissions due to complex action types.

[0071] Perform pixel-by-pixel difference calculation on adjacent frames of the video, add and average the absolute values ​​of the differences of all pixels, and obtain the average absolute value of the inter-frame differences. If the average absolute value exceeds the set threshold, increase the acquisition frequency of the key frame according to the preset ratio, as follows:

[0072] Assume that two adjacent frames are F1(x, y) and F2(x, y), and the calculation formula of the average absolute value of the inter-frame difference is:

[0073]

[0074] Where N is the total number of image pixels. For a sports game video with large dynamic changes, the average absolute value of the inter-frame difference will be large. When it exceeds the threshold, the key frame acquisition frequency is increased. This method based on inter-frame difference can effectively measure the dynamic degree of the video. When processing videos with different dynamic degrees, it can ensure that the number and distribution of key frames are reasonable. For videos with large dynamic changes, more details can be captured. In content review, key information will not be lost due to large dynamic changes, thereby improving the review ability of various types of video content.

[0075] As an important part of multimedia content, audio may contain key information about illegal content. Converting audio into text and extracting acoustic features can enable the content review system to better understand the audio content and thus more accurately determine whether there is illegal information. This helps to improve the comprehensiveness and accuracy of content review and avoid missing illegal clues in the audio by relying solely on visual information.

[0076] When extracting the audio speech content, the content review unit 3 uses speech recognition technology to convert the audio into speech content in text form, and extracts the acoustic features of the speech, as follows:

[0077] The pre-processed audio signal is framed and windowed. For each frame of audio signal x(n), n represents the sampling point index within the frame. Each frame is N sampling points long, where N is determined by the frame length and the sampling rate. This framing operation divides the audio signal into multiple small segments in time. Each small segment can be independently subjected to subsequent frequency analysis to prepare for discrete Fourier transform. Then, the discrete Fourier transform of each frame of audio is calculated to obtain the spectrum using the formula:

[0078]

[0079] Where x(n) is the audio signal sequence of the current frame, X(k) is the frequency domain spectrum value, k = 0, 1, ..., N-1, N is the number of points of the discrete Fourier transform. Through the discrete Fourier transform, the time domain audio signal is converted into a frequency domain representation, showing the energy distribution of the audio signal at different frequencies, which facilitates the analysis of frequency information. In speech recognition, it can better identify the characteristics of different speech units, improve the accuracy of speech recognition, and thus improve the content reviewer's ability to understand the language information in the audio.

[0080] Then map the spectrum to the Mel frequency scale, using the formula Mel(f) = 2595log 10 (1+f / 700); where f is the linear frequency. The Mel frequency scale is a nonlinear frequency scale based on human auditory perception. It is more in line with human perception of sounds of different frequencies. Through this mapping, the extracted features can be closer to human perception of speech. In speech recognition, it can better process various frequency components in speech signals, especially for different speakers and speech in different environments. It has better adaptability, improves the performance of speech recognition in complex scenarios, and is conducive to the accurate judgment of speech content by content review. Finally, the MFCC features are obtained through discrete cosine transform, as follows:

[0081]

[0082] where x(n) is the input sequence, here the spectral coefficients on the Mel frequency scale, C(k) is the discrete cosine transform coefficient, Discrete cosine transform can compress and decorrelate the spectral coefficients on the Mel frequency scale to obtain a set of more compact and representative features. These MFCC features can effectively describe the spectral envelope information of speech. In different speech pronunciations, MFCC features will have obvious differences and can be used to distinguish different speech content. In the speech recognition model, it reduces the computational complexity while retaining the key information of the speech, improves the efficiency and accuracy of speech recognition, and provides more efficient speech content processing for content review.

[0083] The MFCC features are input into the preset speech recognition based on hidden Markov model and Gaussian mixture model for training and recognition. In the training stage, the parameters of the hidden Markov model and Gaussian mixture model are estimated using the labeled speech data. The hidden Markov model is used to model the temporal structure of speech. It assumes that speech is generated by a series of hidden states according to a certain probability transfer relationship, while the Gaussian mixture model is used to model the distribution of MFCC features in each hidden state. For example, for the state corresponding to a certain phoneme, the probability distribution of the MFCC features in the state can be learned using the Gaussian mixture model through the MFCC features in a large amount of labeled data. In the recognition stage, the MFCC features are input into the trained model, the probability of each word is calculated, and finally the text output is obtained through the decoding algorithm. In speech recognition, the speech content can be accurately identified based on the MFCC features, which improves the accuracy and robustness of speech recognition and provides reliable text information for content review.

[0084] The fundamental frequency of the speech signal is calculated using the average amplitude difference function and used as the acoustic feature, as follows:

[0085]

[0086] Where x(n) is the speech signal sequence, τ is the delay time, and the fundamental frequency F0 is determined by finding the value of τ that minimizes AMDF(τ). The fundamental frequency is an important acoustic parameter of the speech signal, which is related to the pitch of the speech. In content review, in addition to text information, acoustic features can assist in judging the emotional tendency of the audio content, emphasize content, etc., and further improve the understanding of the audio content and the accuracy of review.

[0087] Data of different modalities have different importance and roles in reflecting violation information. For example, in violent violation content, video keyframes may more intuitively display violent actions, but shouting in audio voice content also plays an auxiliary role in judgment. Through weighted fusion, appropriate weights can be assigned according to their relevance in different violation types, and information of each modality can be better utilized. This helps to improve the recognition ability of complex multimodal violation content and avoid over-reliance on or neglect of certain modal information.

[0088] When the content review unit 3 uses the multimodal fusion model to represent and fuse the extracted information, a weighted fusion method is adopted, as follows:

[0089] Initial weights are set based on prior knowledge and experience, and experiments are conducted on annotated data sets to observe the recognition recall rate of the model for various types of violation information under different weight combinations. Finally, the correlation of different modal data in different types of violation information is analyzed. If it is found that the correlation of a certain modal data with the violation label in a certain violation type exceeds the preset threshold, its weight is increased proportionally, so that the model can better utilize each modal information when facing new data or changes in data distribution, improve the recognition ability of different types of violation content, and reduce misjudgments and missed judgments;

[0090] According to the determined weights, each dimension of the acoustic features of the acquired video key frame and audio speech content is multiplied to obtain the weighted feature vectors and record them separately. Suppose the feature vector of the video key frame is V = (v1, v2, ..., v n ) whose weight is ω1, and the acoustic feature vector of the audio speech content is A=(a1,a2,...,a n ), whose weight is ω2, then the weighted video key frame feature vector is V jq =(ω1v1,ω1v2,...,ω1v n );The weighted audio speech content feature vector is A jq =(ω2a1,ω2a2,...,ω2a n ); This multiplication operation can integrate the weight information into the features of each dimension, highlight the features of important modes, and in the fusion process, make more relevant modal features play a greater role in subsequent processing, improve the model's pertinence to different types of violations, such as in the review of violent violations, highlight the action features in the key frames of the video, and finally add the corresponding dimensions of the weighted feature vectors to obtain the fused feature vector, that is, the fused feature vector F = (ω1v1+ω2a1, ω1v2+ω2a2, ..., ω1v n +ω2a n ); This addition operation realizes the fusion of different modal features and integrates multimodal information into a vector, which is convenient for subsequent model processing and analysis. It provides a comprehensive feature representation for the multimodal content review model, enabling the model to use video and audio information to judge illegal content at the same time, thereby improving the accuracy and comprehensiveness of the review.

[0091] Deep learning classification networks have powerful feature learning and representation capabilities, and can automatically mine complex patterns and relationships from the input fused feature vectors. In content review, the manifestations of illegal information are diverse and complex. Deep learning can effectively capture this information, improve the ability to identify different types of illegal information, and adapt to various violation scenarios under multimodal data.

[0092] The classifier in content review unit 3 identifies illegal information by adjusting the hyperparameters in the deep learning classification network, as follows:

[0093] When training the classifier, the fused feature vector is input into the deep learning classification network, and the deep learning classification network outputs the probability value of the sample belonging to each violation category. The deep learning classification network can process the feature vector after multimodal fusion, and map the input features to the probability distribution of each violation category through nonlinear transformation of multiple hidden layers. For example, for violation categories such as violence, pornography, infringement, false information, malicious guidance, etc., the network outputs a probability vector (p1, p2, ..., p k ), which indicates the possibility that the sample belongs to each category. This probability-based output method can handle situations where the category is uncertain. For each training sample, multiple violation types can be fully considered in content review, thereby improving the recognition coverage of different types of violation information and reducing the omission of certain violation types.

[0094] For each training sample, calculate its cross entropy loss, and then average the loss values ​​of all samples to get the loss value of the entire training batch. The cross entropy loss function is often used for multi-classification problems, and its formula is where y i is the true label of the sample. If the sample belongs to the i-th category, then y i =1, otherwise y i =0, p i It is the probability that the sample output by the network belongs to the i-th category, which gives the model training a clear direction, adjusts the parameters in the direction of reducing the loss value, and improves the accuracy of the model in classifying illegal information.

[0095] According to the calculated loss value, the parameters of the network are updated through the back propagation algorithm. Starting from the output layer, the gradient of the loss function for each parameter is calculated, and then the weights and biases of each layer in the network are adjusted according to the gradient. The back propagation algorithm is based on the chain rule and can efficiently calculate the gradient of the loss function for each parameter in the network, accelerate the convergence of the model, and enable the model to learn the characteristics of illegal information in a shorter time, improve the efficiency of content review, and process large amounts of multimedia content in a timely manner.

[0096] Finally, the final number of network layers is determined by conducting experiments on the training set and the validation set, as follows:

[0097] Starting from a neural network structure with 2-3 hidden layers, gradually increase to 5-6 layers. For each selected number of network layers, build a corresponding deep learning classification network model and keep the initial value of the learning rate the same, providing a comprehensive candidate solution for finding the optimal number of network layers, improving the possibility of determining the optimal network structure, and thus better identifying illegal information.

[0098] For each model with different number of network layers, training is performed using the training set data. During the training process, the loss value is calculated according to the method mentioned above and the network parameters are updated through the back propagation algorithm. Each model is trained for 10-20 rounds until the loss value of the model on the training set reaches the preset training stop condition. This allows models with different number of network layers to learn data features under the same conditions and be compared fairly, providing a reliable basis for subsequent evaluation on the validation set and helping to identify the learning ability of models with different number of network layers for violation information features.

[0099] After completing the training of the training set, the model corresponding to each network layer is evaluated on the validation set, and the accuracy is calculated. The performance of the model on undetected data is measured by the accuracy. For content review tasks, it can better judge the model's ability to identify illegal information on new data and avoid one-sided evaluation caused by focusing on a single indicator.

[0100] Compare the accuracies of models with different numbers of network layers on the validation set. If the model accuracy shows a trend of first increasing and then decreasing with the increase of the number of network layers, select the number of network layers corresponding to the moment of highest accuracy as the final number of network layers, and determine the optimal number of network layers in a data-driven manner to improve model performance, so that the final deep learning classification network can more accurately identify illegal information in content review, and identify illegal information based on the deep learning classification network corresponding to the final number of network layers.

[0101] In the present invention, multimodal data is collected and preprocessed by the collection and processing unit to provide high-quality data for the subsequent process. The model training unit constructs a multimodal fusion model. Its joint training mechanism learns modal associations, the adaptive learning rate strategy optimizes training, and the early stopping technology prevents overfitting. The trained model is used for content review. The content review unit receives multimedia content, extracts key frames, audio and voice content, and subtitle text, represents and fuses them through features, and uses the trained model to determine whether there is a violation. The units cooperate with each other to jointly achieve effective review of multimedia content and ensure the legality and health of the content.

[0102] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. Model training and content review management system based on intelligent data standard collection, characterized by: It includes a data collection unit (1), a model training unit (2) and a content review unit (3); The acquisition processing unit (1) is used to acquire multimodal data from multiple data sources, wherein the multimodal data includes text, image, audio and video data, and to add metadata including a timestamp and source information to each data during the acquisition process, and to dynamically adjust acquisition parameters and pre-process the data according to the characteristics of different modal data; The model training unit (2) constructs a multimodal fusion model based on deep learning according to the multimodal data, including subnetworks for processing data of different modalities respectively, and the subnetworks cooperate with each other and share information through a joint training mechanism, and the joint mechanism is realized by introducing a correlation term between modalities in a loss function. The model training unit (2) also includes an adaptive learning rate adjustment strategy and an early stopping technology; The content review unit (3) is used to receive multimedia content to be reviewed, extract video key frames, audio and speech content and subtitle text therein, and use a multimodal fusion model to perform feature representation and fusion on them, and finally use a classifier to determine whether the content contains illegal information, and the classifier is trained based on a deep neural network multimodal fusion model.

2. The model training and content review management system based on intelligent data standard collection according to claim 1 is characterized by: The acquisition processing unit (1) dynamically adjusts acquisition parameters and pre-processes the data according to the characteristics of different modal data, specifically as follows: Normalize, crop and denoise the image data; Perform noise reduction and sampling rate adjustment on audio data; For video data, the acquisition frequency is dynamically adjusted and key frames are extracted according to its frame rate and resolution, and the same preprocessing operations as image data are performed on the key frames; For text data, the collection time interval is determined according to the information update speed, and lexical and syntactic analysis and encoding conversion processing are performed.

3. The model training and content review management system based on intelligent data standard collection according to claim 1 is characterized by: The joint training mechanism in the model training unit (2) is specifically as follows: When calculating the loss function, the prediction loss of each sub-network for its own modal data and the loss term of the interaction information between modalities are combined to calculate the minimized total loss. The model parameters are optimized by minimizing the total loss to obtain the correlation between different sub-network modalities. The loss term of the interaction information between modalities is obtained by calculating the similarity between different modal feature vectors.

4. The model training and content review management system based on intelligent data standard collection according to claim 3 is characterized by: The adaptive learning rate adjustment strategy in the model training unit (2) is as follows: During the training process, the learning rate is adjusted according to the changes in the model performance indicators on the training sequence and the validation set. If the loss value on the validation set shows a monotonically increasing trend for several consecutive times, the learning rate is increased. If the loss value on the validation set shows a monotonically decreasing trend for several consecutive times and the rate of change is less than the preset threshold, the learning rate is reduced, and the adjustment range of the learning rate is within the preset range.

5. The model training and content review management system based on intelligent data standard collection according to claim 3 is characterized by: The early stopping technology of the model training unit (2) specifically includes: The dataset is divided into training set, validation set and test set. After each training round, the model performance is evaluated on the validation set. When the validation set performance index begins to decline within consecutive preset rounds, the training is paused and the model is evaluated on the test set. If the test set performance also shows a downward trend, the training is stopped.

6. The model training and content review management system based on intelligent data standard collection according to claim 3 is characterized by: When extracting video key frames, the content review unit (3) extracts key frames based on key action change points of the video content, and increases the extraction frequency of key frames for video content whose dynamic changes exceed a threshold, as follows: The optical flow method is used to analyze the continuous frames of the video. By calculating the optical flow vector of each pixel, the movement direction and speed information of the object in the video are obtained. The background difference method is combined to highlight the movement of the foreground object. By establishing a background model, the current frame is subtracted from the background model to obtain the mask of the foreground object. Based on the motion information obtained from optical flow and background difference, the change rate and direction of the object's motion speed between adjacent frames are calculated. When the change rate exceeds the preset threshold, it is determined to be a key action change point. The machine learning model is used to learn and identify the action change pattern. When the model detects that a specific type of key action occurs, the corresponding frame is determined as a key frame. Perform pixel-by-pixel difference calculation on adjacent frames of the video, add and average the absolute values ​​of the differences of all pixels, and obtain the average absolute value of the inter-frame differences. If the average absolute value exceeds the set threshold, increase the acquisition frequency of the key frames according to the preset ratio.

7. The model training and content review management system based on intelligent data standard collection according to claim 3 is characterized by: When extracting the audio speech content, the content review unit (3) uses speech recognition technology to convert the audio into speech content in text form, and extracts the acoustic features of the speech, as follows: The pre-processed audio signal is framed and windowed, the audio is divided into time segments of a set length and multiplied by a window function, then the discrete Fourier transform of each frame of audio is calculated to obtain the spectrum, and then the spectrum is mapped to the Mel frequency scale, and finally the MFCC features are obtained through discrete cosine transform; The MFCC features are input into the preset speech recognition based on hidden Markov model and Gaussian mixture model for training and recognition. In the training stage, the parameters of the hidden Markov model and Gaussian mixture model are estimated using the labeled speech data. In the recognition stage, the MFCC features are input into the trained model to calculate the probability of each word. Finally, the text output is obtained through the decoding algorithm, and the fundamental frequency of the speech signal is calculated using the average amplitude difference function as the acoustic feature.

8. The model training and content review management system based on intelligent data standard collection according to claim 3 is characterized by: The content review unit (3) uses a weighted fusion method when performing feature representation and fusion on the extracted information using a multimodal fusion model, as follows: Initial weights are set based on prior knowledge and experience, and experiments are conducted on labeled data sets to observe the recognition accuracy and recall rate of the model for various types of violation information under different weight combinations. Finally, the correlation of different modal data in different types of violation information is analyzed. If it is found that the correlation of a certain modal data with the violation label in a certain violation type exceeds the preset threshold, its weight is increased proportionally. According to the determined weights, each dimension of the acquired acoustic features of the video key frame and the audio speech content is multiplied to obtain weighted feature vectors and record them separately. Finally, the corresponding dimensions of the weighted feature vectors are added to obtain a fused feature vector.

9. The model training and content review management system based on intelligent data standard collection according to claim 1 is characterized by: The classifier in the content review unit (3) identifies illegal information by adjusting the hyperparameters in the deep learning classification network, as follows: When training the classifier, the fused feature vector is input into the deep learning classification network, and the deep learning classification network outputs the probability value of the sample belonging to each violation category. For each training sample, its cross entropy loss is calculated, and then the loss values ​​of all samples are averaged to obtain the loss value of the entire training batch; Based on the calculated loss value, the network parameters are updated through the back-propagation algorithm. Starting from the output layer, the gradient of the loss function for each parameter is calculated, and then the weights and biases of each layer in the network are adjusted according to the gradient. Finally, the final number of network layers is determined by experiments on the training set and the validation set, and the violation information is identified based on the deep learning classification network corresponding to the final number of network layers.

Images