Content management method, device and equipment based on deep integration

Abstract

The application provides a content management method, device and equipment based on deep integration, which comprises the following steps: processing to-be-detected data into K pieces of data of different dimensions; for any piece of data of a dimension among the K pieces of data of different dimensions, using a trained pre-processing model of the dimension to pre-process the data of the dimension to obtain a feature vector of the dimension; taking the K feature vectors of different dimensions as inputs of a trained content detection model, using the trained content detection model to determine a content detection result of the to-be-detected data, and performing content management processing on the to-be-detected data according to the content detection result. The method can improve the accuracy of content management.

Description

Technical Field

[0001] This application relates to the fields of network security and data security, and in particular to a content management method, apparatus and device based on deep integration. Background Technology

[0002] Content control refers to using technical means to detect content from multiple dimensions such as images, text, and voice in order to identify and filter abnormal content and ensure that the content of platforms, applications, or websites complies with relevant regulations.

[0003] Content control technology is an important tool for ensuring a healthy and safe internet environment, and how to accurately control content has become a popular research direction. Summary of the Invention

[0004] In view of this, this application provides a content management method, apparatus and device based on deep integration.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this application, a content management method based on deep integration is provided, comprising:

[0007] Traffic data for asset identification is obtained through traffic monitoring.

[0008] The data to be detected is processed into K different dimensions; where K ≥ 2.

[0009] For any dimension of the K different dimensions of data, the data of that dimension is preprocessed using the preprocessing model trained for that dimension to obtain the feature vector of that dimension;

[0010] The K feature vectors of different dimensions are used as inputs to a trained content detection model. The trained content detection model is then used to determine the content detection result of the data to be detected. Based on the content detection result, content control processing is performed on the data to be detected. The content detection model includes a deep ensemble learning model, or a student model generated from the deep ensemble learning model using model distillation. The deep ensemble learning model includes multiple base models of different dimensions. The content detection result of the deep ensemble learning model on the data to be detected is determined based on the detection results of the base models of different dimensions on the feature vectors of the corresponding dimensions. For base models that are not the first base model in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension and the output of the feature output layer of the previous base model.

[0011] According to a second aspect of the embodiments of this application, a content management device based on deep integration is provided, comprising:

[0012] The first processing unit is used to process the data to be detected into K different dimensions; where K≥2;

[0013] The second processing unit is used to preprocess the data of any dimension among the K different dimensions of data using the trained preprocessing model of that dimension to obtain the feature vector of that dimension.

[0014] The content control unit is used to take the K feature vectors of different dimensions as input to a trained content detection model, use the trained content detection model to determine the content detection result of the data to be detected, and perform content control processing on the data to be detected based on the content detection result. The content detection model includes a deep ensemble learning model, or a student model generated based on the deep ensemble learning model using model distillation. The deep ensemble learning model includes multiple base models of different dimensions. The content detection result of the deep ensemble learning model on the data to be detected is determined based on the detection results of the base models of different dimensions on the feature vectors of the corresponding dimensions. For a non-first base model in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension and the output of the feature output layer of the previous base model.

[0015] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor and a memory, wherein...

[0016] Memory, used to store computer programs;

[0017] The processor, when executing a program stored in memory, implements the method provided in the first aspect.

[0018] According to a fourth aspect of the embodiments of this application, a computer program product is provided, wherein the computer program product stores a computer program, and the computer program, when executed by a processor, implements the method provided in the first aspect.

[0019] The content control method based on deep integration in this application involves processing the data to be detected into K different dimensions, and using pre-trained pre-processing models for each of the corresponding dimensions to preprocess the K different dimensions of data, obtaining K feature vectors. These K feature vectors are then input into the base models of different dimensions of the deep integration learning model. The deep integration learning model determines the content detection result of the data to be detected, and content control processing is performed based on the determined detection result. By fusing multimodal detection results, the accuracy of content detection is improved, thereby enhancing the precision of content control. Alternatively, the K different dimension feature vectors can be input into the student model of the deep integration learning model, which determines the content detection result of the data to be detected, and content control processing is performed based on the determined detection result. By reducing the resource requirements of content control, content control is achieved on resource-scarce devices, expanding the applicability of internal control. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an exemplary embodiment of a content control method based on deep integration.

[0021] Figure 2 This is a schematic diagram illustrating a preprocessing model as an exemplary embodiment of this application;

[0022] Figure 3 A schematic diagram illustrating a deep ensemble learning model as an exemplary embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating a model distillation as an exemplary embodiment of this application;

[0024] Figure 5 This is a schematic diagram illustrating the structure of a content management device based on deep integration, as shown in an exemplary embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the hardware structure of an electronic device as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some terms involved in the embodiments of this application will be briefly explained below.

[0027] 1. Ensemble learning: In machine learning or deep learning, learning tasks are accomplished by building and combining multiple learners. For example, an algorithm or model can generate multiple sub-models, and these models can be combined together in some way to solve a specific problem. Common examples include bagging, boosting, stacking, etc.

[0028] 2. Deep Ensemble Learning: Similar to ensemble learning, it integrates multiple sub-models to solve a specific problem. The difference is that it combines the differences in deep learning model construction and training to propose new ensemble ideas, including dropout, test dataset expansion TTA, and snapshot.

[0029] 3. Model distillation: Model distillation can distill a larger model into a smaller model. By training a teacher model and a student model, the smaller student model can imitate the teacher model and minimize the difference loss between the two models.

[0030] For example, model distillation can be implemented in ways including but not limited to: online distillation, offline distillation, soft-label distillation, intermediate layer feature distillation, and adversarial distillation.

[0031] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0032] Please see Figure 1 This is a flowchart illustrating a content management method based on deep integration, as provided in an embodiment of this application. Figure 1 As shown, this content management method based on deep integration may include the following steps:

[0033] Step S100: Process the data to be detected into K different dimensions; where K≥2.

[0034] For example, the data to be detected may include data that needs to be detected for content control processing.

[0035] For example, the data to be detected may include, but is not limited to, videos, images, audio recordings, or text to be published.

[0036] In this embodiment of the application, in order to improve the accuracy of content control, content detection can be performed on the data to be detected from multiple different dimensions.

[0037] Accordingly, the data to be detected can be processed into K (K≥2) different dimensions of data.

[0038] For example, the different dimensions mentioned above may include image dimension, text dimension, or voice dimension.

[0039] Step S110: For any dimension of the data in the K different dimensions, use the preprocessing model of that dimension to preprocess the data in that dimension to obtain the feature vector of that dimension.

[0040] In this embodiment of the application, preprocessing models of different dimensions can be pre-trained to preprocess data of different dimensions and obtain feature vectors of the corresponding dimensions.

[0041] For example, the preprocessing model can use different models for different dimensions.

[0042] For example, for the text dimension, the preprocessing model can include, but is not limited to, LSTM (Long Short-Term Memory) models or transformer models.

[0043] For the image dimension, the preprocessing model can include, but is not limited to, CNN (Convolutional Neural Network) models or ResNet (Residual Network) models.

[0044] For the speech dimension, the preprocessing model can include, but is not limited to: Conformer model (CNN+transformer model, i.e., convolution-enhanced transformer model) or transformer model, etc.

[0045] Accordingly, for any of the K different dimensions of data mentioned above, the pre-processing model for that dimension can be used to preprocess the data in that dimension to obtain the feature vector of that dimension.

[0046] For example, assuming that the data to be detected is processed to obtain K dimensions of data, including image dimension data and text dimension data, then a pre-trained image dimension pre-processing model can be used to pre-process the image dimension data to obtain the image dimension feature vector; and a pre-trained text dimension pre-processing model can be used to pre-process the text dimension data.

[0047] For example, in order to improve the processing efficiency of the preprocessing model in processing data of the corresponding dimension, the data of the corresponding dimension can be encoded before using the preprocessing model to process the data of the corresponding dimension.

[0048] For example, data can be encoded separately for text, image, and speech dimensions:

[0049] 1) Text dimension: The text data is processed into an N*1 vector using word segmentation and embedding techniques;

[0050] 2) Image Dimensions: Image data is processed into M*M dimensional tensors using resizing (such as scaling, cropping, and padding) and normalization techniques.

[0051] 3) Speech dimension: Convert speech data into MFCC (Mel Frequency Cepstral Coefficients) features, or process it into a P*1 vector using an audio feature extractor.

[0052] The calculation process of MFCC may include the following steps:

[0053] Preprocessing includes pre-emphasis and framing. Pre-emphasis is used to boost high-frequency components, while framing divides the continuous speech signal into multiple short time segments (frames) for processing.

[0054] Fast Fourier Transform (FFT): Perform FFT on each frame of the speech signal to obtain the spectrum.

[0055] Mel filter bank: A set of Mel filters is used to filter the spectrum, with each filter corresponding to a frequency range. The outputs of these filters represent the energy of different frequency bands.

[0056] Logarithmic energy calculation: Take the logarithm of the output of each filter to obtain the logarithmic energy.

[0057] Discrete Cosine Transform (DCT): The logarithmic energy is subjected to a discrete cosine transform to obtain the MFCC coefficients.

[0058] For example, for the obtained MFCC coefficients, a portion of the MFCC coefficients, such as the previously preset number of coefficients, can be used as feature vectors. Step S120: Use K feature vectors of different dimensions as input to the trained content detection model, use the trained content detection model to determine the content detection result of the data to be detected, and perform content control processing on the data to be detected based on the content detection result; wherein, the content detection model includes a deep ensemble learning model, or a student model generated based on the deep ensemble learning model using model distillation; the deep ensemble learning model includes multiple base models of different dimensions, and the deep ensemble learning model determines the content detection result of the data to be detected based on the detection results of the feature vectors of the corresponding dimensions of the base models of different dimensions; for non-first base models in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension, and the output of the feature output layer of the previous base model.

[0059] In this embodiment of the application, a deep ensemble learning model can be used to improve the accuracy of content detection.

[0060] For example, deep ensemble learning models can include base models with multiple different dimensions.

[0061] For example, a deep ensemble learning model may include at least two of the following: a base model for text, a base model for images, and a base model for speech.

[0062] In one example, a deep ensemble learning model may include a base model for the text dimension, a base model for the image dimension, and a base model for the speech dimension.

[0063] For any dimension of the base model, it can be used to perform content detection on the data to be detected based on the feature vector of the corresponding dimension.

[0064] For example, for a basic model based on the text dimension, content detection can be performed on the data to be detected based on the feature vectors of the text dimension.

[0065] Deep ensemble learning models can determine the final content detection result of the data to be detected based on the content detection results of the data to be detected in each dimension of the basic model.

[0066] For example, the content detection result may include normal or abnormal.

[0067] For example, when the content detection result is abnormal, the abnormality type can be further distinguished (the deep ensemble learning model needs to be trained accordingly).

[0068] For example, in order to further improve the accuracy of content detection, for non-first base models in deep ensemble learning models, the input of the base model can include not only the feature vector of the corresponding dimension, but also the output of the feature output layer of the previous base model. That is, when performing content detection on the data to be detected, the non-first base model can also refer to the features output by the previous base model.

[0069] In one example, the feature output layer of the base model can be the penultimate layer of the base model.

[0070] In this embodiment of the application, it is considered that in real-world scenarios, some devices with relatively scarce resources may also have content management needs, such as servers, industrial control computers, or edge terminal devices with low resources. For these types of devices, directly deploying the aforementioned deep ensemble learning model may result in insufficient resources.

[0071] Correspondingly, for devices with relatively scarce resources and content control needs, model distillation can be used to generate student models using the aforementioned deep ensemble learning model as the teacher model. The student models can then be deployed on the devices for content control. This can reduce resource consumption while lowering accuracy to some extent, thus meeting the content control needs of resource-scarce devices.

[0072] It can be seen that, in Figure 1 In the illustrated method, the data to be detected is processed into K different dimensions. Preprocessing models of the corresponding dimensions are then used to preprocess each of the K different dimensions, resulting in K feature vectors. These feature vectors are then input into the base models of different dimensions of a deep ensemble learning model. The deep ensemble learning model determines the content detection result of the data to be detected, and content control processing is performed based on the determined detection result. By fusing multimodal detection results, the accuracy of content detection is improved, thereby enhancing the precision of content control. Alternatively, the K different feature vectors can be input into a student model of the deep ensemble learning model. This student model determines the content detection result of the data to be detected, and content control processing is performed based on the determined detection result. By reducing the resource requirements for content control, content control is achieved on resource-scarce devices, expanding the applicability of internal control.

[0073] In some embodiments, the K different dimensions include at least two of the image dimension, text dimension, and voice dimension;

[0074] For any of the K different dimensions, if data for that dimension cannot be generated based on the data to be detected, the data for that dimension obtained by processing the data to be detected is empty, and the feature vector of the data to be detected corresponding to that dimension is all 0.

[0075] For example, in practical applications, it may be impossible to generate data with preset dimensions for the data to be detected.

[0076] For example, for image data, it can generate image-dimensional data and text-dimensional data (e.g., text information obtained from an image through OCR (Optical Character Recognition)), but it typically cannot generate speech-type data. For speech data, it can generate speech-dimensional data and text-dimensional data (e.g., text information obtained from speech recognition), but it typically cannot generate image-dimensional data.

[0077] To accommodate the above situation, for any of the K different dimensions, if data for that dimension cannot be generated based on the data to be detected, the data for that dimension obtained by processing the data to be detected is empty, and the feature vector of the data to be detected corresponding to that dimension is all 0.

[0078] For example, when the data to be detected is image data, and the above K different dimensions include the speech dimension, the speech dimension data obtained by processing the data to be detected is empty, and the feature vector of the speech dimension of the data to be detected is all 0.

[0079] When the data to be detected is speech data, and there are K different dimensions including the image dimension, the data of the image dimension obtained by processing the data to be detected is empty, and the feature vector of the image dimension corresponding to the data to be detected is all 0;

[0080] When the data to be detected is text data and has K different dimensions including the image dimension, the image dimension data obtained by processing the data to be detected is empty, and the feature vector of the image dimension of the data to be detected is all 0; and / or, when the data to be detected is text data and has K different dimensions including the speech dimension, the speech dimension data obtained by processing the data to be detected is empty, and the feature vector of the speech dimension of the data to be detected is all 0.

[0081] For example, assuming the data to be detected is text data, and the above K different dimensions include image dimension and speech dimension, then the image dimension data and speech dimension data obtained by processing the data to be detected are both empty, and the feature vectors of the image dimension and the corresponding speech dimension of the data to be detected are both 0.

[0082] In some embodiments, the above preprocessing model can be trained in the following ways:

[0083] For any training sample file, process the training sample file into training data of K different dimensions;

[0084] For any dimension of training data in K different dimensions, the training data of that dimension is preprocessed using the preprocessing model to be trained, and the training feature vector of that dimension is obtained.

[0085] Based on the training feature vector of this dimension, the predicted label of the training data of this dimension is determined, and the preprocessed model of this dimension to be trained is optimized based on the predicted label and the labeled label of the training data of this dimension; wherein, the labeled label of the training data of different dimensions corresponding to the same training sample is the same.

[0086] For example, in order to train a preprocessed model, training sample files can be obtained for training the preprocessed model.

[0087] For example, training sample files can be labeled using manual annotation, model-assisted annotation, or automatic model annotation.

[0088] For any training sample file, it can be processed into K different dimensions of data (which can be called training data), and each of the K different dimensions of training data can be used to train the preprocessed model of the corresponding dimension.

[0089] For example, the labeled data of different dimensions corresponding to the same training sample have the same label.

[0090] For any dimension of training data among K different dimensions, the training data of that dimension can be preprocessed using the preprocessing model to be trained to obtain the feature vector of that dimension (which can be called the training feature vector).

[0091] During the training process of the preprocessing model, a label prediction part can be set in the preprocessing model, such as a softmax layer. This label prediction part can determine the predicted label of the training data in this dimension based on the training feature vector obtained in the above manner. Then, based on the predicted label and the labeled label of the training data in this dimension, the preprocessing model to be trained in this dimension can be fine-tuned.

[0092] For example, the training loss (such as cross-entropy loss) can be determined based on the difference between the predicted label and the labeled label of the training data in that dimension, and the preprocessed model can be fine-tuned based on the determined training loss.

[0093] It should be noted that in practical applications, when deploying the pre-trained model, it is not necessary to deploy the label prediction part mentioned above. That is, when applying the pre-trained model, it is not necessary to perform label prediction. Instead, the feature vectors of the data in the corresponding dimensions can be obtained in the manner described above.

[0094] In some embodiments, deep ensemble learning models can be trained in the following ways:

[0095] For any training sample file, process the training sample file into training data of K different dimensions;

[0096] For any dimension of training data in K different dimensions, the training data of that dimension is preprocessed using the preprocessing model of that dimension to obtain the training feature vector of that dimension.

[0097] K training feature vectors of different dimensions are used as inputs to the base model of the deep ensemble learning model to be trained in the corresponding dimension. The deep ensemble learning model to be trained is used to determine the predicted labels of the training sample files. Based on the predicted labels and the labeled labels of the training sample files, the deep ensemble learning model to be trained is optimized.

[0098] For example, in order to train a preprocessed model, training sample files can be obtained for training the preprocessed model.

[0099] For example, training sample files can be labeled using manual annotation, model-assisted annotation, or automatic model annotation.

[0100] For any training sample file, it can be processed into data of K different dimensions (which can be called training data), and these K different dimensions of training data can be used to train a deep ensemble learning model.

[0101] In one example, the training data used to train a deep ensemble learning model can be the same batch of training data used to train a preprocessed model.

[0102] For example, K training feature vectors of different dimensions can be used as inputs to the base model of the corresponding dimension of the deep ensemble learning model to be trained, and the predicted labels of the training sample files can be determined based on the output results of the base models of different dimensions.

[0103] For example, for any training sample, the K training feature vectors of different dimensions corresponding to the training sample can be input into the base model of the corresponding dimension of the deep ensemble learning model to be trained, and the predicted label of the training sample file can be determined by weighted averaging based on the output results of the base models of different dimensions.

[0104] For example, the deep ensemble learning model to be trained can be fine-tuned based on the predicted labels and labeled labels of the training sample files.

[0105] In one example, where the content detection model is a student model generated from a deep ensemble learning model using model distillation, the content detection model training also includes:

[0106] Using the pre-trained deep ensemble learning model as the teacher model, and based on K different-dimensional training feature vectors and their corresponding labels, the student model is trained using model distillation to obtain the pre-trained student model.

[0107] For example, when the content detection model is a student model of a deep ensemble learning model, after obtaining the trained deep ensemble learning model in the above manner, the trained deep ensemble learning model can also be used as a teacher model. Based on the above K training feature vectors of different dimensions and the corresponding labels, the constructed student model can be trained using the model distillation method to obtain the trained student model.

[0108] As an example, the above-mentioned training of the constructed student model using model distillation based on K training feature vectors of different dimensions and their corresponding labels can include:

[0109] Based on K training feature vectors of different dimensions, the output probability distributions of the deep ensemble learning model and the student model corresponding to the same training samples are determined respectively. Based on the output probability distributions of the deep ensemble learning model and the student model, the first loss is determined.

[0110] as well as,

[0111] Based on K training feature vectors of different dimensions, the predicted labels of the student model for the training samples are determined, and based on the predicted labels and the labeled labels of the training samples, the second loss is determined.

[0112] The student model is then fine-tuned based on the first and second losses.

[0113] For example, during the model distillation process, the output probability distribution of the teacher model can be used as a "soft label" to train the school model.

[0114] For example, in the model distillation process, on the one hand, the output probability distributions of the deep ensemble learning model (i.e., the teacher model) and the student model corresponding to the same training samples can be determined based on the K training feature vectors of different dimensions, and the corresponding loss (which can be called the first loss) can be determined based on the output probability distribution of the deep ensemble learning model and the output probability distribution of the student model.

[0115] For example, the first loss can be the KL divergence.

[0116] On the other hand, based on K different dimensions of training feature vectors, the predicted labels of the student model for the training samples can be determined, and based on the predicted labels and the labeled labels of the training samples, the corresponding loss (which can be called the second loss) can be determined.

[0117] For example, the second loss could be the cross-entropy loss.

[0118] Once the first loss and the second loss are determined, the comprehensive loss of the student model can be determined based on the first loss and the second loss, and the student model can be fine-tuned based on the feedback of the comprehensive loss.

[0119] For example, the weighted average of the first loss and the second loss can be determined as the comprehensive loss.

[0120] It should be noted that a temperature parameter (T) can be introduced in the process of determining the output probability distribution of the teacher model. During training, the T value can be increased to amplify the differences between categories, so that the output probability distribution of the teacher model can convey knowledge more richly.

[0121] In some embodiments, where the base model of the deep ensemble learning model includes a text-dimensional base model, the first base model in the deep ensemble learning model is the text-dimensional base model, and the input of the first base model of the deep ensemble learning model is a text-dimensional feature vector.

[0122] For example, considering that various types of raw data (such as the data to be detected mentioned above) can typically generate text-dimensional data, setting the first base model in a deep ensemble learning model as a text model can effectively prevent the input of the first base model of the deep ensemble learning model from being empty.

[0123] Since the inputs of non-first base models in a deep ensemble learning model include the output of the feature output layer of the previous base model, the inputs of each base model will not be empty if the input of the first base model is not empty. This can improve the content detection accuracy of the deep ensemble learning model.

[0124] Based on this, when the base model of a deep ensemble learning model includes a base model in the text dimension, the first base model in the deep ensemble learning model is the base model in the text dimension, and the input of this base model is the feature vector in the text dimension.

[0125] In some embodiments, when the base models of the deep ensemble learning model include a base model for the text dimension, a base model for the image dimension, and a base model for the speech dimension, the order of the base models for each dimension in the deep ensemble learning model is as follows: base model for the text dimension, base model for the image dimension, and base model for the speech dimension.

[0126] For example, in a deep ensemble learning model, the input to a non-first base model includes the output of the feature output layer of the previous base model.

[0127] Furthermore, in real-world scenarios, there is usually more raw data that can generate image dimensions than raw data that can generate speech dimensions.

[0128] Based on this, when the base models of a deep ensemble learning model include base models for text, images, and speech, the order of the base models for each dimension in the deep ensemble learning model is as follows: base model for text, base model for images, and base model for speech. This can effectively increase the amount of input information for the later base models in the deep ensemble learning model and improve the content detection accuracy of the deep ensemble learning model.

[0129] It should be noted that, in the embodiments of this application, the number of base models of the same dimension in the deep ensemble learning model can be multiple.

[0130] For example, suppose a deep ensemble learning model includes two base models for text (which can be denoted as A1 and A2), two base models for image (which can be denoted as B1 and B2), and one base model for speech (which can be denoted as C). Then the order of the base models in the deep ensemble learning model can be A1, A2 (the order of A1 and A2 can be adjusted), B1, B2 (the order of B1 and B2 can be adjusted), and C.

[0131] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0132] This application provides a deep learning-based content control method. In the process of solving the content detection target, a deep ensemble learning framework capable of handling multimodal data input is designed to train and generate a teacher model. Furthermore, through model distillation, a large-scale teacher model can be distilled into a smaller-scale student model, thereby achieving high accuracy and multimodal data support for content detection and control.

[0133] The implementation process of the content management scheme based on deep integration provided in the embodiments of this application is described below.

[0134] 1. Multimodal content data processing.

[0135] In this embodiment, training can be performed based on labeled data. The original training sample files include sample files from dimensions such as video, text, images, and audio, labeled as normal and abnormal (taking anomaly type as an example without specifically distinguishing them). For each type of original training sample file, data preprocessing is performed from three dimensions: text, image, and audio. Specifically:

[0136] 1.1 Process all types of original training sample files into three dimensions: text, image, and speech.

[0137] For example, video files can generate data in the three dimensions mentioned above; image files can generate data in the two dimensions of image and text; and audio files can generate data in the two dimensions of audio and text.

[0138] 1.2 Data Encoding: Data in the text, image, and speech dimensions are encoded separately for use as preprocessing input for model training and testing.

[0139] 1.2.1) Text dimension: The text data is processed into an N*1 vector using word segmentation and embedding techniques.

[0140] 1.2.2) Image Dimensions: Image data is processed into M*M dimensional tensors by using resizing (such as scaling, cropping, and padding) and normalization techniques.

[0141] 1.2.3) Speech dimension: Convert speech data into MFCC features, or process it into a P*1 vector using an audio feature extractor.

[0142] 1.3 Preprocessing model construction and pretraining.

[0143] For example, preprocessed data processed in the text, image, and speech dimensions are used to train a model, which is then used as input for a subsequent deep ensemble learning model.

[0144] For example, during training, a schematic diagram of the preprocessed model can be shown as follows: Figure 2 As shown. In practical applications, the trained preprocessing model may not need to output predicted labels; instead, it can output the extracted data features as input to the content detection model.

[0145] 1.3.1) Text dimension: Construct a text feature extraction model (also known as a text preprocessing model, i.e., a text dimension preprocessing model). Use the N*1 vectors obtained in 1.2.1 and the corresponding labels to train the text preprocessing model. After the model converges, solidify the model parameters to obtain the text preprocessing model Mw.

[0146] For example, the text preprocessing model can use an LSTM or transformer model, where the second and penultimate layers are fully connected layers with X neurons, and the output of the penultimate layer (i.e., the fully connected layer) is the extracted text feature vector.

[0147] 1.3.2) Image Dimension: Construct an image feature extraction model (also known as an image preprocessing model, i.e., an image dimension preprocessing model). Use the M*M dimension tensor obtained in 1.2.2 and the corresponding labels to train the image preprocessing model. After the model converges, solidify the model parameters to obtain the image preprocessing model Mp.

[0148] For example, the image preprocessing model can use CNN or ResNet, with the second and penultimate layers being fully connected layers with X neurons. The output of the penultimate layer (i.e., the fully connected layer) is the extracted image feature vector.

[0149] 1.3.3) Speech dimension: Construct a speech feature extraction model (also known as a speech preprocessing model, i.e., a speech dimension preprocessing model). Use the MFCC features obtained in 1.2.3, or P*1 vectors and corresponding labels to train the speech preprocessing model. After the model converges, solidify the model parameters to obtain the speech preprocessing model Mv.

[0150] For example, the speech preprocessing model can adopt the transformer model, where the second and penultimate layers are fully connected layers with X neurons, and the output of the penultimate layer (i.e., the fully connected layer) is the extracted speech feature vector.

[0151] II. Construction and Training of Deep Integrated Learning Framework

[0152] For example, after step one, all the original training sample files can be processed into feature vectors of size X*1 for use in training the deep ensemble learning framework in this step.

[0153] 2.1 Basic Model Construction.

[0154] For example, a base model is built for data learning in each dimension (speech dimension, text dimension, image dimension).

[0155] 2.2.1) Basic model for text dimension: The text preprocessing model in step one is adopted, or the transformer model with a second and penultimate layer of fully connected layers containing X neurons is adopted.

[0156] 2.2.2) Image dimension basic model: The image preprocessing model in step one is adopted, or the ResNet model with a second and penultimate fully connected layer of X neurons is adopted.

[0157] 2.2.3) Basic model for speech dimension: The speech preprocessing model in step one is adopted, or the Conformer model with a second and penultimate fully connected layer of X neurons is adopted.

[0158] 2.2 Construction of a deep ensemble learning framework.

[0159] For example, a deep ensemble learning base model trains multiple base models simultaneously, and the output of each base model is weighted and averaged to obtain the final output.

[0160] For example, the base model for each dimension may include one or more.

[0161] For example, when there is only one base model in each dimension, the schematic diagram of a deep ensemble learning model can be shown as follows: Figure 3 As shown.

[0162] 2.2.1) For each original training sample file, it is processed from three dimensions: text dimension, image dimension and speech dimension.

[0163] For example, data of different dimensions generated from the same original training sample file (which may be a file of video, text, voice, image, etc.) share the same label (a label can be added, which can be normal or abnormal). The preprocessing model in step one processes the original data into a feature vector of dimension X*1.

[0164] 2.2.2) Each training session uses the three-dimensional feature vectors of an original training sample file to train a deep ensemble learning model.

[0165] For example, if a dimension is missing, it can be padded with zeros. For instance, if the original training sample file is an image, the feature vector for the speech dimension could be a vector consisting of X zeros.

[0166] 2.2.3) The outputs Fw, Fp and Fv of each basic model are weighted and averaged to obtain the model prediction value (i.e. the predicted label). The difference between the predicted value and the true label (i.e. the labeled label above) is calculated to obtain the LOSS. The entire deep ensemble learning model is trained by minimizing the LOSS.

[0167] III. Model distillation.

[0168] For example, deep ensemble learning can effectively integrate the data features of the original samples across three dimensions, enabling multimodal feature extraction and content detection. However, the overall model size increases significantly, leading to reduced model efficiency. This solution can mitigate the model size increase problem caused by deep ensemble learning by employing model distillation.

[0169] 3.1 Student Model Construction: The student model can be a completely new model (such as the BERT model), or it can be a transformer model.

[0170] For example, taking the transformer model as an example, a fully connected layer with X*3 neurons can be introduced into the input layer as a student model (Mstudent).

[0171] 3.2 Online Distillation: Online distillation is used to distill the model, simultaneously training the teacher model (i.e., the aforementioned deep ensemble learning model Mteacher) and the student model using the training data from step one.

[0172] 3.3 Soft label distillation: Various distillation methods can be used (such as intermediate feature distillation, counter-distillation, dynamic distillation, etc.).

[0173] In this embodiment, a soft-label distillation method is used, where Mteacher outputs the probability distribution of samples to train Mstudent to fit the teacher model's distribution. A temperature parameter can also be introduced during this process to amplify differences between categories. A schematic diagram is shown below. Figure 4 As shown.

[0174] For example, the loss function can be set as a joint cross-entropy loss (determined based on the true labels) and a KL divergence loss (determined based on the soft labels).

[0175] IV. Model Output

[0176] After the above three steps, the final model is output:

[0177] 4.1 Mw, Mp, and Mv: These are used to perform feature processing on the original sample in three dimensions, respectively.

[0178] 4.2 Mteacher: Used for content management on high-resource servers, such as central-end content management.

[0179] 4.3 Mstudent: Used for content management on servers, industrial control computers, or terminals with lower resources, such as edge-based content management.

[0180] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0181] Please see Figure 5 This is a schematic diagram of the structure of a content management device based on deep integration provided in an embodiment of this application, as shown below. Figure 5 As shown, the content management device based on deep integration may include:

[0182] The first processing unit is used to process the data to be detected into K different dimensions; where K≥2;

[0183] The second processing unit is used to preprocess the data of any dimension among the K different dimensions of data using the trained preprocessing model of that dimension to obtain the feature vector of that dimension.

[0184] The content control unit is used to take the K feature vectors of different dimensions as input to a trained content detection model, use the trained content detection model to determine the content detection result of the data to be detected, and perform content control processing on the data to be detected based on the content detection result. The content detection model includes a deep ensemble learning model, or a student model generated based on the deep ensemble learning model using model distillation. The deep ensemble learning model includes multiple base models of different dimensions. The content detection result of the deep ensemble learning model on the data to be detected is determined based on the detection results of the base models of different dimensions on the feature vectors of the corresponding dimensions. For a base model that is not the first base model in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension and the output of the previous base model.

[0185] For example, the specific implementation process of the first processing unit, the second processing unit, and the content control unit in implementing content control based on deep integration can be found in the relevant descriptions in the above method embodiments.

[0186] This application also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; and the processor is used to implement the content management method based on deep integration described above when executing the program stored in the memory.

[0187] Please see Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 can communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to the deeply integrated content management logic in the memory 602, the processor 601 can execute the deeply integrated content management method described above.

[0188] The memory 602 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0189] In some embodiments, a machine-readable storage medium, such as Figure 6The memory 602 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the deep integration-based content management method described above. For example, the machine-readable storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0190] This application also provides a computer program product that stores a computer program, and when a processor executes the computer program, it causes the processor to execute the content management method based on deep integration described above.

Claims

1. A content management method based on deep integration, characterized in that, include: The data to be detected is processed into K different dimensions; where K ≥ 2. For any dimension of the K different dimensions of data, the data of that dimension is preprocessed using the preprocessing model trained for that dimension to obtain the feature vector of that dimension; The K feature vectors of different dimensions are used as input to a trained content detection model. The trained content detection model is used to determine the content detection result of the data to be detected, and content control processing is performed on the data to be detected based on the content detection result. The content detection model includes a deep ensemble learning model, or a student model generated based on the deep ensemble learning model using model distillation. The deep ensemble learning model includes multiple base models of different dimensions, and the number of base models of the same dimension in the deep ensemble learning model is one or more. The content detection result of the deep ensemble learning model on the data to be detected is determined based on the detection results of the base models of different dimensions on the feature vectors of the corresponding dimensions. For a non-first base model in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension and the output of the feature output layer of the previous base model. The K different dimensions include at least two of the following: image dimension, text dimension, and voice dimension; For any of the K different dimensions, if data for that dimension cannot be generated based on the data to be detected, the data for that dimension obtained by processing the data to be detected is empty, and the feature vector of the data to be detected corresponding to that dimension is all 0; In the case where the base model of the deep ensemble learning model includes a text-dimensional base model, the first base model in the deep ensemble learning model is the text-dimensional base model, and the input of the first base model of the deep ensemble learning model is the text-dimensional feature vector.

2. The method according to claim 1, characterized in that, When the data to be detected is image data, and the K different dimensions include the speech dimension, the speech dimension data obtained by processing the data to be detected is empty, and the feature vector of the speech dimension corresponding to the data to be detected is all 0; When the data to be detected is speech data, and the K different dimensions include the image dimension, the data of the image dimension obtained by processing the data to be detected is empty, and the feature vector of the image dimension corresponding to the data to be detected is all 0; When the data to be detected is text data, and the K different dimensions include image dimension and / or voice dimension, the data of the image dimension and / or voice dimension obtained by processing the data to be detected is empty, and the feature vector of the data to be detected corresponding to the image dimension and / or voice dimension is all 0.

3. The method according to claim 1, characterized in that, The preprocessing model is trained in the following manner: For any training sample file, process the training sample file into training data of K different dimensions; For any dimension of the training data in the K different dimensions, the training data of that dimension is preprocessed using the preprocessing model to be trained to obtain the training feature vector of that dimension. Based on the training feature vector of this dimension, the predicted label of the training data of this dimension is determined, and the preprocessed model of this dimension to be trained is optimized based on the predicted label and the labeled label of the training data of this dimension; wherein, the labeled label of the training data of different dimensions corresponding to the same training sample is the same.

4. The method according to claim 1, characterized in that, The deep ensemble learning model is trained in the following manner: For any training sample file, process the training sample file into training data of K different dimensions; For any dimension of the training data in the K different dimensions, the training data of that dimension is preprocessed using the preprocessing model of that dimension to obtain the training feature vector of that dimension. The K training feature vectors of different dimensions are used as inputs to the base model of the corresponding dimension of the deep ensemble learning model to be trained. The predicted labels of the training sample files are determined using the deep ensemble learning model to be trained. Based on the predicted labels and the labeled labels of the training sample files, the deep ensemble learning model to be trained is optimized.

5. The method according to claim 4, characterized in that, When the content detection model is a student model generated based on the deep ensemble learning model using model distillation, the content detection model training further includes: Using the trained deep ensemble learning model as the teacher model, and based on the K different-dimensional training feature vectors and their corresponding labels, the constructed student model is trained using model distillation to obtain the trained student model.

6. The method according to claim 5, characterized in that, The step of training the constructed student model using model distillation based on the K training feature vectors of different dimensions and their corresponding labels includes: Based on the K training feature vectors of different dimensions, the output probability distributions of the deep ensemble learning model and the student model corresponding to the same training samples are determined respectively, and the first loss is determined based on the output probability distributions of the deep ensemble learning model and the student model. as well as, Based on the K training feature vectors of different dimensions, the predicted label of the student model for the training sample is determined, and based on the predicted label and the labeled label of the training sample, the second loss is determined. The student model is then fine-tuned based on the first loss and the second loss.

7. The method according to any one of claims 1-6, characterized in that, When the base models of a deep ensemble learning model include a base model for the text dimension, a base model for the image dimension, and a base model for the speech dimension, the order of the base models for each dimension in the deep ensemble learning model is as follows: base model for the text dimension, base model for the image dimension, and base model for the speech dimension.

8. A content management and control device based on deep integration, characterized in that, include: The first processing unit is used to process the data to be detected into K different dimensions; where K≥2; The second processing unit is used to preprocess the data of any dimension among the K different dimensions of data using the trained preprocessing model of that dimension to obtain the feature vector of that dimension. A content control unit is used to take the K feature vectors of different dimensions as input to a trained content detection model, use the trained content detection model to determine the content detection result of the data to be detected, and perform content control processing on the data to be detected based on the content detection result. The content detection model includes a deep ensemble learning model, or a student model generated based on the deep ensemble learning model using model distillation. The deep ensemble learning model includes multiple base models of different dimensions, and the number of base models of the same dimension in the deep ensemble learning model is one or more. The content detection result of the deep ensemble learning model on the data to be detected is determined based on the detection results of the base models of different dimensions on the feature vectors of the corresponding dimensions. For a non-first base model in the deep ensemble learning model, the input of the base model includes the feature vector of the corresponding dimension and the output of the feature output layer of the previous base model. The K different dimensions include at least two of the following: image dimension, text dimension, and voice dimension; For any of the K different dimensions, if data for that dimension cannot be generated based on the data to be detected, the data for that dimension obtained by processing the data to be detected is empty, and the feature vector of the data to be detected corresponding to that dimension is all 0; In the case where the base model of the deep ensemble learning model includes a text-dimensional base model, the first base model in the deep ensemble learning model is the text-dimensional base model, and the input of the first base model of the deep ensemble learning model is the text-dimensional feature vector.

9. An electronic device, characterized in that, Including processor and memory, among which, Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product contains a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

Images