A method and apparatus for video review, an electronic device, and a storage medium

Abstract

Embodiments of the present application provide a video auditing method and device, electronic equipment and storage medium, the method comprising: obtaining video data, performing frame extraction processing on the video data to obtain an image to be audited; distilling a video auditing model with a deep network to obtain a distilled student model; inputting the image to be audited into the distilled student model to determine whether the video data is qualified. The scheme provided by the embodiments of the present application can improve the video auditing speed while ensuring the accuracy of the audit.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for video review. Background Technology

[0002] With the development of the internet, the number of videos is growing rapidly, and the number of videos requiring review is also increasing. Current technologies often employ video review models for review, but due to the increased volume of video data, especially after frame extraction, the data volume becomes even larger. This places high demands on the review speed of video review models, leading to a preference for lightweight models for deployment. However, lightweight models often cannot achieve the precision of deeper networks, resulting in inaccurate video review results. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for video review, so as to improve the speed of video review while ensuring the accuracy of the review. The specific technical solution is as follows:

[0004] In a first aspect of this invention, a video review method is provided, the method comprising:

[0005] Acquire video data, perform frame extraction on the video data, and obtain the image to be reviewed;

[0006] Distillation is performed on a video moderation model with deep networks to obtain a distilled student model;

[0007] The image to be reviewed is input into the distilled student model to determine whether the video data is qualified.

[0008] Optionally, the distillation of the video moderation model with deep networks to obtain the distilled student model includes:

[0009] Obtain the trained teacher model and the student model to be trained;

[0010] At least one feature layer is selected from the pre-trained teacher model and the student model to be trained as the alignment feature layer;

[0011] The first sample image is obtained and input into the pre-trained teacher model and the student model to be trained, respectively, and the feature loss of each alignment feature layer is calculated.

[0012] Based on the feature loss of each aligned feature layer, the total loss of the two models is obtained.

[0013] The parameters of the student model to be trained are adjusted based on the total loss.

[0014] Determine whether the total loss meets the preset training termination condition. If yes, obtain the distilled student model; otherwise, obtain the second sample image again and repeat the training steps.

[0015] Optionally, the alignment feature layer includes: the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained, and the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, wherein the dimensions of the one-to-one corresponding alignment feature layers are different.

[0016] The calculation of the feature loss for each aligned feature layer includes:

[0017] Feature extraction is performed using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and the feature loss of each alignment feature layer is calculated.

[0018] The total loss of the two models is obtained based on the feature loss of each aligned feature layer, including:

[0019] The network output layer of the pre-trained teacher model outputs the output probability value of the teacher model; the network output layer of the student model to be trained outputs the output probability value of the student model.

[0020] Using a preset loss function, the output loss of the network output layer is calculated based on the output probability values ​​of the teacher model and the student model.

[0021] The total loss of the two models is obtained by using the feature loss of each aligned feature layer and the output loss of the network output layer.

[0022] Optionally, the alignment feature layer includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer has a higher level than the first alignment feature layer, and the second alignment feature layer has a higher level than the third alignment feature layer.

[0023] The step of extracting features using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and calculating the feature loss of each alignment feature layer, includes:

[0024] After the first sample image is input into the teacher model, the teacher model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the teacher model, the first feature on the teacher side is generated. After the first sample image is input into the student model to be trained, the student model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the student model to be trained, the first feature on the student side is generated.

[0025] After performing preset processing on the teacher-side first feature, a teacher-side first alignment feature is obtained; after performing preset processing on the student-side first feature, a student-side first alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side first alignment feature, a student-side first dimensionality-upgraded feature is obtained.

[0026] The first feature loss is obtained by calculating the feature loss between the first aligned feature on the teacher side and the first upgraded feature on the student side using a preset loss function.

[0027] After the teacher-side first alignment feature is extracted by the second alignment feature layer of the teacher model, the teacher-side second feature is generated; after the student-side first alignment feature is extracted by the second alignment feature layer of the student model to be trained, the student-side second feature is generated.

[0028] After performing preset processing on the teacher-side second feature, a teacher-side second alignment feature is obtained; after performing preset processing on the student-side second feature, a student-side second alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side second alignment feature, a student-side second dimensionality-upgraded feature is obtained.

[0029] The feature loss of the teacher-side second alignment feature and the student-side second up-dimensional feature after up-dimensionalization is calculated using a preset loss function to obtain the second feature loss.

[0030] After the teacher-side second alignment feature is extracted by the third alignment feature layer of the teacher model, the teacher-side third feature is generated; after the student-side second alignment feature is extracted by the third alignment feature layer of the student model to be trained, the student-side third feature is generated.

[0031] After performing preset processing on the teacher-side third feature, a teacher-side third alignment feature is obtained; after performing preset processing on the student-side third feature, a student-side third alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side third alignment feature, a student-side third dimensionality-upgraded feature is obtained.

[0032] The feature loss of the third alignment feature on the teacher side and the third up-dimensional feature on the student side after dimensionality increase are calculated using a preset loss function to obtain the third feature loss.

[0033] Optionally, the step of using the feature loss of each aligned feature layer and the output loss of the network output layer to obtain the total loss of the two models includes:

[0034] Using preset weights, the first feature loss, the second feature loss, the third feature loss, and the output loss are weighted and calculated to obtain the total loss of the two models.

[0035] Optionally, the step of inputting each image to be reviewed into the distilled student model to determine whether the video data is qualified includes:

[0036] The images to be reviewed are input into the distilled student model to obtain the probability that each image contains unqualified content;

[0037] Determine whether the probability of each image to be reviewed containing unqualified content reaches a preset probability;

[0038] If the target is reached, the images to be reviewed that reach the preset probability will be considered as unqualified images.

[0039] Determine whether the proportion of unqualified images to the total number of images to be reviewed reaches a preset proportion;

[0040] If the preset ratio is reached, the video data is considered unqualified.

[0041] If the preset ratio is not reached, the video data is considered qualified.

[0042] In a second aspect of the present invention, a video review device is also provided, the device comprising:

[0043] The image acquisition module is used to acquire video data and perform frame extraction processing on the video data to obtain the image to be reviewed;

[0044] The student model acquisition module is used to distill the video review model with deep networks to obtain the distilled student model.

[0045] The video review result determination module is used to input the image to be reviewed into the distilled student model to determine whether the video data is qualified.

[0046] Optionally, the student model acquisition module includes:

[0047] The model acquisition submodule is used to acquire the trained teacher model and the student model to be trained.

[0048] The alignment feature layer selection submodule is used to select at least one feature layer as the alignment feature layer from the trained teacher model and the student model to be trained, respectively.

[0049] The feature loss calculation submodule is used to acquire the first sample image, input it into the trained teacher model and the student model to be trained respectively, and calculate the feature loss of each aligned feature layer.

[0050] The total loss calculation submodule is used to obtain the total loss of the two models based on the feature loss of each aligned feature layer;

[0051] The model tuning submodule is used to adjust the parameters of the student model to be trained based on the total loss;

[0052] The condition judgment submodule is used to determine whether the total loss meets the preset training termination condition. If so, the distilled student model is obtained; otherwise, the second sample image is obtained again, and the training steps are repeated.

[0053] Optionally, the alignment feature layer selected by the alignment feature layer selection submodule includes: the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained, and the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, and the dimensions of the one-to-one corresponding alignment feature layers are different.

[0054] The feature loss calculation submodule is specifically used for:

[0055] Feature extraction is performed using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and the feature loss of each alignment feature layer is calculated.

[0056] The total loss calculation submodule is specifically used for:

[0057] The network output layer of the pre-trained teacher model outputs the output probability value of the teacher model; the network output layer of the student model to be trained outputs the output probability value of the student model.

[0058] Using a preset loss function, the output loss of the network output layer is calculated based on the output probability values ​​of the teacher model and the student model.

[0059] The total loss of the two models is obtained by using the feature loss of each aligned feature layer and the output loss of the network output layer.

[0060] Optionally, the alignment feature layer selected by the alignment feature layer selection submodule includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer has a higher level than the first alignment feature layer, and the second alignment feature layer has a higher level than the third alignment feature layer.

[0061] The feature loss calculation submodule is specifically used for:

[0062] After the first sample image is input into the teacher model, the teacher model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the teacher model, the first feature on the teacher side is generated. After the first sample image is input into the student model to be trained, the student model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the student model to be trained, the first feature on the student side is generated.

[0063] After performing preset processing on the teacher-side first feature, a teacher-side first alignment feature is obtained; after performing preset processing on the student-side first feature, a student-side first alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side first alignment feature, a student-side first dimensionality-upgraded feature is obtained.

[0064] The first feature loss is obtained by calculating the feature loss between the first aligned feature on the teacher side and the first upgraded feature on the student side using a preset loss function.

[0065] After the teacher-side first alignment feature is extracted by the second alignment feature layer of the teacher model, the teacher-side second feature is generated; after the student-side first alignment feature is extracted by the second alignment feature layer of the student model to be trained, the student-side second feature is generated.

[0066] After performing preset processing on the teacher-side second feature, a teacher-side second alignment feature is obtained; after performing preset processing on the student-side second feature, a student-side second alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side second alignment feature, a student-side second dimensionality-upgraded feature is obtained.

[0067] The feature loss of the teacher-side second alignment feature and the student-side second up-dimensional feature after up-dimensionalization is calculated using a preset loss function to obtain the second feature loss.

[0068] After the teacher-side second alignment feature is extracted by the third alignment feature layer of the teacher model, the teacher-side third feature is generated; after the student-side second alignment feature is extracted by the third alignment feature layer of the student model to be trained, the student-side third feature is generated.

[0069] After performing preset processing on the teacher-side third feature, a teacher-side third alignment feature is obtained; after performing preset processing on the student-side third feature, a student-side third alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side third alignment feature, a student-side third dimensionality-upgraded feature is obtained.

[0070] The feature loss of the third alignment feature on the teacher side and the third up-dimensional feature on the student side after dimensionality increase are calculated using a preset loss function to obtain the third feature loss.

[0071] Optionally, the total loss calculation submodule is specifically used for:

[0072] Using preset weights, the first feature loss, the second feature loss, the third feature loss, and the output loss are weighted and calculated to obtain the total loss of the two models.

[0073] Optional, a video review result determination module, specifically used for:

[0074] The images to be reviewed are input into the distilled student model to obtain the probability that each image contains unqualified content;

[0075] Determine whether the probability of each image to be reviewed containing unqualified content reaches a preset probability;

[0076] If the target is reached, the images to be reviewed that reach the preset probability will be considered as unqualified images.

[0077] Determine whether the proportion of unqualified images to the total number of images to be reviewed reaches a preset proportion;

[0078] If the preset ratio is reached, the video data is considered unqualified.

[0079] If the preset ratio is not reached, the video data is considered qualified.

[0080] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the video review methods described above.

[0081] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video review methods described above.

[0082] This invention provides a method, apparatus, electronic device, and storage medium for video review. The method involves acquiring video data, performing frame extraction on the video data to obtain an image to be reviewed, distilling a video review model with deep networks to obtain a distilled student model, and inputting the image to be reviewed into the distilled student model to determine whether the video data is acceptable. Since the distilled student model is obtained using a video review model with deep networks, which is characterized by high review accuracy, the distilled student model also possesses high review accuracy and is more lightweight. Therefore, using the distilled student model to review video data can improve review speed while ensuring accuracy. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0084] Figure 1 This is a first flowchart of the video review method provided in this embodiment of the invention;

[0085] Figure 2 This is a flowchart of obtaining a distilled student model provided by an embodiment of the present invention;

[0086] Figure 3 This is a flowchart of calculating the feature loss of three aligned feature layers provided by an embodiment of the present invention;

[0087] Figure 4 This is a flowchart of generating the feature loss of each aligned feature layer and the output loss of the network output layer, provided by an embodiment of the present invention;

[0088] Figure 5 This is a schematic diagram of the video review device provided in an embodiment of the present invention;

[0089] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0090] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0091] This invention provides a video review method, see [link to relevant documentation]. Figure 1 , Figure 1 The method shown specifically includes the following steps S110-S130.

[0092] S110: Acquire video data, perform frame extraction on the video data, and obtain the image to be reviewed.

[0093] The video review method provided in this embodiment of the invention can be implemented using a server or a client terminal, specifically a desktop computer, a laptop, a smartphone, or a smart tablet.

[0094] The video data mentioned above is the video data that needs to be reviewed. The images to be reviewed are the video frames obtained after frame extraction. Frame extraction can be performed on the video according to a preset standard, and the extracted frames are used as images to be reviewed. In one example, according to a standard of 24 frames per second, frames are extracted from a one-minute video, resulting in 1440 frames, which are then used as images to be reviewed. In another example, according to a standard of 12 frames per second, frames are extracted from a three-minute video, resulting in 1960 frames, which are then used as images to be reviewed.

[0095] S120, the video moderation model with deep network is distilled to obtain the distilled student model.

[0096] The video review model with deep networks mentioned above is a heavyweight video review model, which has the characteristics of high review accuracy, but its review speed is lower compared to the lightweight model.

[0097] The aforementioned distilled student model is a lightweight video review model obtained by distilling a heavyweight video review model with high precision. In other words, the distilled student model is obtained by distilling a lightweight model with high precision using a heavyweight video review model with deep networks.

[0098] S130: Input the image to be reviewed into the distilled student model to determine whether the video data is qualified.

[0099] Whether the aforementioned video data is acceptable is determined based on the content that is unacceptable contained in each image to be reviewed within the video data. Unacceptable content may be content that does not conform to social norms, public opinion, or other social rules. An image to be reviewed may contain one or more unacceptable pieces of content.

[0100] In this embodiment of the invention, the distilled student model is obtained by distilling a video review model with deep networks. Since the video review model with deep networks has the characteristic of high review accuracy, the distilled student model also has the characteristic of high review accuracy and is more lightweight. Thus, using the distilled student model to review video data can improve the speed of video review while ensuring the accuracy of the review.

[0101] In one possible implementation, see Figure 2The above step S120 may specifically include the following steps S1201-S1206.

[0102] S1201: Obtain the trained teacher model and the student model to be trained.

[0103] The aforementioned teacher model is a pre-trained video review model with a deep network, characterized by high review accuracy. The student model to be trained is a lightweight model. In one example, SE-ResNeXt-50 is selected as the teacher model, and the lightweight MobileNet v3 is selected as the student model.

[0104] S1202, select at least one feature layer from the trained teacher model and the student model to be trained as the alignment feature layer.

[0105] In this process, selecting at least one layer can avoid choosing the first few layers of the model. This is because the first few layers may not have extracted many effective features from the image, so selecting this layer as the alignment feature layer will not benefit subsequent model distillation.

[0106] Specifically, based on the structures of the teacher and student models, at least one shallow, at least one mid-level, and at least one deep feature layer can be selected from multiple feature layers of the model as the model's alignment feature layer. This alignment feature layer is used for feature alignment. Specifically, it involves subsequently increasing the dimensionality of the student model's alignment features on this layer, ensuring that the dimensionality of the student model's features after the increase is the same as the alignment feature dimension of the teacher model. Then, a loss is calculated based on these two features with the same dimension, and the student model is adjusted based on this loss, enabling the student model to learn the feature distribution of the teacher model at each alignment feature layer.

[0107] The terms "shallow," "middle," and "deep" are merely hierarchical concepts, without specifying which layer should be considered shallow, middle, or deep.

[0108] As the model deepens, the features it learns become increasingly complex and sophisticated. Therefore, the deeper the model, the more detailed and complete the feature representations it acquires. In one example, shallow layers could be the first few layers of the model, primarily used to extract edge and local features of an image; middle layers could be the middle layers; and deep layers could be the last few layers, capable of representing the entire object's outline and detailed information.

[0109] S1203, acquire the first sample image, input it into the pre-trained teacher model and the student model to be trained respectively, and calculate the feature loss of each aligned feature layer.

[0110] The first sample image can be a video frame of a certain video data.

[0111] Specifically, the first sample image is input into the pre-trained teacher model and the student model to be trained. The first sample image is processed through the feature layers of the teacher model and the student model for feature extraction. When the extracted features are input into each alignment feature layer, the feature loss at each alignment feature layer is calculated based on the difference between the features extracted by the two models at the alignment feature layer.

[0112] In addition, the existing technology is to process the image into the model and perform feature calculations through each feature layer in sequence. Specifically, the existing technology is to input the features extracted from the previous layer into the next layer as the input for feature extraction. This will not be described in detail.

[0113] S1204, based on the feature loss of each aligned feature layer, obtains the total loss of the two models.

[0114] The total loss mentioned above includes not only the feature loss of the two models at each aligned feature layer, but also the output loss of the two models at the network output layer.

[0115] Specifically, the total loss of the two models can be calculated based on the feature loss of each aligned feature layer and the output loss between the output layers of the two models' networks.

[0116] S1205, adjusts the parameters of the student model to be trained based on the total loss.

[0117] Specifically, based on the total loss, the parameters of the student model to be trained are adjusted using the backpropagation principle of the neural network. The overall parameters include the parameters corresponding to each aligned feature layer in the student model.

[0118] S1206, determine whether the total loss meets the preset training termination condition. If yes, obtain the distilled student model; otherwise, obtain the second sample image again and repeat the training steps.

[0119] The aforementioned preset training termination condition refers to the total loss converging to a preset convergence value. When the total loss decreases to a certain value and then tends to stabilize, that value is the preset convergence value.

[0120] The second sample image is different from the first sample image.

[0121] In this embodiment of the invention, by selecting aligned feature layers in the teacher model and the student model, calculating feature loss on each aligned feature layer, and then adjusting the student model based on the total loss obtained from each feature loss, the adjusted student model can learn the feature distribution of the teacher model on each aligned feature layer, thereby gradually enhancing the feature extraction capability of the student model and making the review accuracy of the learning model closer to that of the teacher model.

[0122] In one possible implementation, the alignment feature layer includes: an alignment feature layer of a trained teacher model and an alignment feature layer of a student model to be trained, wherein the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, and the dimensions of the one-to-one corresponding alignment feature layers are different.

[0123] The dimension of the alignment feature layer is the feature dimension of the feature layer, which refers to the number of channels in the feature layer. The deeper the layer, the more channels there are, and the higher the feature dimension.

[0124] In order for the student model to learn the feature distribution of the teacher model during the distillation process, the high-dimensional features in the teacher model need to be distilled into the low-dimensional features in the student model. Therefore, when selecting the alignment feature layers, the dimensionality of the teacher model is higher than that of the student model in each of the corresponding alignment feature layers.

[0125] Specifically, the following rules can be used for selection: Feature layers with dimensions proportional to those of the student and teacher models are selected as alignment feature layers. That is, the feature dimensions of each alignment feature layer in the student model are proportional to the feature dimensions of each alignment feature layer in the teacher model. When there is no obvious proportional relationship, the student model can select feature layers with dimensions close to those proportional to those of the teacher model's alignment feature layers. By selecting feature layers with proportional or near-proportional dimensions as alignment feature layers, it is easier to subsequently upgrade the dimensions of the student model's alignment features to those of the teacher model. Loss is calculated on each selected alignment feature layer based on alignment features of the same dimension, and the student model is adjusted based on this loss, allowing the student model to learn the feature distribution of the teacher model across multiple alignment feature layers.

[0126] In one example, the feature dimensions of each alignment feature layer in the student model and teacher model are proportional. The teacher model contains 30 feature layers, which are divided into layers 1-10 as shallow layers, layers 11-20 as middle layers, and layers 21-30 as deep layers according to the model structure. Layers 10, 20, and 30 are selected as the alignment feature layers of the teacher model structure, where the dimensions of layer 10 are 256, layer 20 are 512, and layer 30 are 1024. The student model contains 10 feature layers, where the dimensions of layer 2 are 128, layer 5 are 256, and layer 9 are 512. In this case, layers 2, 5, and 9 of the student model are selected as the alignment feature layers of the student model structure. In this model, the 10th layer of the teacher model corresponds to the 2nd layer of the student model, the 20th layer of the teacher model corresponds to the 5th layer of the student model, and the 30th layer of the teacher model corresponds to the 9th layer of the student model. The dimensions of the one-to-one corresponding alignment feature layers are different, and the alignment feature layers of the teacher model have more dimensions than the alignment feature layers of the student model to which they correspond.

[0127] In one example, the dimensions of the alignment feature layers in the student model and teacher model are disproportionate. The teacher model contains 30 feature layers, divided into shallow layers (1-10), mid-layers (11-20), and deep layers (21-30) according to the model structure. Layers 10, 20, and 30 are selected as the alignment feature layers for the teacher model structure, with dimensions of 256 for layer 10, 512 for layer 20, and 1024 for layer 30. The student model contains 10 feature layers, with dimensions of 72 for layer 2, 240 for layer 5, and 672 for layer 9. Layers 2, 5, and 9 are selected as the alignment feature layers for the student model structure. Layer 10 of the teacher model corresponds to layer 2 of the student model, layer 20 corresponds to layer 5 of the student model, and layer 30 corresponds to layer 9 of the student model, with different dimensions for each corresponding layer.

[0128] In this embodiment of the invention, the calculation of the feature loss for each aligned feature layer may specifically include:

[0129] Step X involves extracting features using the alignment feature layers of the teacher model and the alignment feature layers of the student model to be trained, and calculating the feature loss for each alignment feature layer.

[0130] Feature extraction is performed separately for the aligned feature layers of the teacher model and the student model. Based on the difference in feature loss between the two models, the feature loss of the two models at the aligned feature layer is determined.

[0131] Step S1205 above, based on the feature loss of each aligned feature layer, obtains the total loss of the two models, which can specifically include:

[0132] Step A1: The output layer of the trained teacher model outputs the output probability value of the teacher model.

[0133] Step A2: The network output layer of the student model to be trained outputs the output probability value of the student model;

[0134] Step A3: Using a preset loss function, calculate the output loss of the network output layer based on the output probability values ​​of the teacher model and the student model.

[0135] Step A4: Using the feature loss of each aligned feature layer and the output loss of the network output layer, obtain the total loss of the two models.

[0136] Specifically, using the cross-entropy loss function, the output loss of the two models at the network output layer is calculated based on the output probability values ​​of the teacher model and the student model. Weights are assigned to the feature loss of each alignment feature layer and the output loss of the network output layer, with the weight of the feature loss of the alignment feature layer being less than the weight of the output loss of the network output layer. Based on the weights of the feature loss of each alignment feature layer and the output loss of the network output layer, a weighted average is calculated to obtain the total loss.

[0137] During the model distillation process, due to the structural differences between the teacher model and the student model, the features extracted by each layer of the teacher model and the student model will also differ significantly. Therefore, if only the output loss is used as a constraint, it is generally difficult to achieve good distillation results.

[0138] In this embodiment of the invention, not only is the output loss calculated for the network output layer, but also the feature loss of each alignment feature layer is calculated. The feature losses of the alignment feature layers and the output loss are used together as constraints for training the student model, guiding the student model distillation process. This allows the performance of the teacher model to be distilled to the student model, achieving performance levels comparable to the teacher model. Since the student model is lightweight, it can improve video review speed while ensuring accuracy, thereby reducing resource utilization, saving costs, and minimizing resource consumption. Furthermore, because the feature losses of the alignment feature layers and the output loss are used together as constraints for student model training, the student model learns not only the distribution of the output layer but also the feature distributions of each alignment feature layer during training. This allows the student model to better learn the feature distributions of the teacher model. During the constraint process of the alignment feature layer feature distributions, the student model's output more easily converges to the teacher model's output, simplifying the distillation process and resulting in a better distillation effect than simply learning the output layer distribution.

[0139] In one possible implementation Z, the alignment feature layer includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer has a higher level than the first alignment feature layer, and the second alignment feature layer has a higher level than the third alignment feature layer.

[0140] Specifically, the first alignment feature layer is located in the shallow layer, the second alignment feature layer is located in the middle layer, and the third alignment feature layer is located in the deep layer.

[0141] In this embodiment, three feature layers are selected as alignment feature layers to allow the student model to learn the feature distribution of the teacher model from shallow, medium, and deep layers respectively. This ensures that the student model can learn the features of the teacher model more comprehensively after distillation, thereby making the performance of the student model closer to that of the teacher model. However, no more than three feature layers can also be selected as alignment feature layers, but this method will result in lower accuracy of the distilled student model compared to the case of three layers or at least three layers.

[0142] Specifically, the shallow alignment feature layers of the selected teacher model correspond to the shallow alignment feature layers of the student model, the middle alignment feature layers of the teacher model correspond to the middle alignment feature layers of the student model, and the deep alignment feature layers of the teacher model correspond to the deep alignment feature layers of the student model.

[0143] In an embodiment of the present invention, see Figure 3 , Figure 3 This is a flowchart of calculating the feature loss of three aligned feature layers provided by an embodiment of the present invention. The above step X may specifically include the following steps X1-X9.

[0144] X1. After the first sample image is input into the teacher model, the teacher model extracts features from the first sample image. After the feature extraction operation is performed in the first alignment feature layer of the teacher model, the first feature on the teacher side is generated. After the first sample image is input into the student model to be trained, the student model extracts features from the first sample image. After the feature extraction operation is performed in the first alignment feature layer of the student model to be trained, the first feature on the student side is generated.

[0145] The first alignment feature layer is a shallow feature layer among the alignment feature layers.

[0146] Since alignment feature layers located in the shallow, middle, and deep layers are selected in this embodiment, during the feature extraction process in the model, the alignment feature layer located in the shallow layer will be passed first, then the alignment feature layer located in the shallow layer will be passed second, and finally the alignment feature layer located in the shallow layer will be passed last. Therefore, this feature layer is regarded as the first alignment feature layer, the alignment feature layer located in the middle layer is regarded as the second alignment feature layer, and the alignment feature layer located in the deep layer is regarded as the third alignment feature layer.

[0147] X2, after pre-processing the first feature on the teacher side, we obtain the first alignment feature on the teacher side; after pre-processing the first feature on the student side, we obtain the first alignment feature on the student side; after dimensionality-upgrading the first alignment feature on the student side, we obtain the first dimensionality-upgraded feature on the student side.

[0148] The aforementioned pre-processing may include feature enhancement processing and normalization processing. When enhancing features, attention mechanisms can be used to achieve feature enhancement, such as channel attention mechanisms and / or spatial attention mechanisms.

[0149] The channel attention mechanism processes features as follows: First, the input features are processed in parallel using global average pooling and global max pooling, preserving the channel dimensions to obtain two channel features. For example, a feature with a height of 10, a width of 10, and 2048 channels is denoted as a 10x10x2048 feature. After pooling, the resulting feature is a 1x1x2048 feature, preserving the channel characteristics. Then, the channel features are fused together to obtain richer, deeper features. Finally, an overactivation function is used to calculate the channel attention weights based on the fused features. Based on these weights and the input features, a weighted, channel-enhanced feature is obtained, thus achieving channel attention enhancement. The input features described above correspond to the first feature on the student's side and the first feature on the teacher's side.

[0150] The spatial attention mechanism operates similarly to the above process, except that it operates on different feature locations. It pools the input features to 10x10x1, retaining spatial features, and then obtains spatial features. Similarly, it uses an activation function to obtain the weight coefficients of spatial attention. Based on the weight coefficients of spatial attention and the input features, it obtains weighted spatially enhanced features, thereby achieving spatial attention enhancement.

[0151] The two attention mechanisms mentioned above can be performed sequentially to achieve feature recoding, thereby enhancing features and suppressing background and irrelevant information interference.

[0152] The above normalization process is to transform the data of each layer to a state with a mean of zero and a variance of 1, so that the data distribution of each layer is basically consistent, which can speed up the training and convergence of the model and prevent gradient explosion.

[0153] Specifically, the aforementioned dimensionality upscaling process can be performed using a 1x1 convolution kernel.

[0154] Specifically, feature enhancement can be performed on the teacher-side first feature to obtain the enhanced teacher-side first feature. The enhanced teacher-side first feature is then normalized through a Batch Normalization (BN) layer to obtain the teacher-side first aligned feature. Feature enhancement can also be performed on the student-side first feature to obtain the enhanced student-side first feature. The enhanced student-side first feature is then normalized through a BN layer to obtain the student-side first aligned feature. Finally, the student-side first aligned feature is convolved using a 1x1 convolution kernel to obtain the student-side first upscaled feature.

[0155] X3 uses a preset loss function to calculate the feature loss of the first aligned feature on the teacher side and the first upgraded feature on the student side after dimensionality upgrade, thus obtaining the first feature loss.

[0156] Specifically, the mean squared error loss function can be used to calculate the feature loss between the first aligned feature on the teacher's side and the first upgraded feature on the student's side, thus obtaining the first feature loss.

[0157] X4 generates a second feature on the teacher side after the first alignment feature of the teacher model is extracted by the second alignment feature layer; and generates a second feature on the student side after the first alignment feature of the student model is extracted by the second alignment feature layer.

[0158] Among them, the second alignment feature layer mentioned above is the feature layer located in the middle layer of the alignment feature layer.

[0159] X5, after pre-processing the second feature on the teacher side, obtains the second alignment feature on the teacher side; after pre-processing the second feature on the student side, obtains the second alignment feature on the student side; after dimensionality-upgrading the second alignment feature on the student side, obtains the second dimensionality-upgraded feature on the student side.

[0160] Specifically, the teacher-side second feature can be enhanced and normalized to obtain the teacher-side second alignment feature; the student-side second feature can be enhanced and normalized to obtain the student-side second alignment feature; and the student-side second alignment feature can be convolved with a 1x1 convolution kernel to obtain the student-side second up-dimensional feature.

[0161] X6 uses a preset loss function to calculate the feature loss of the second alignment feature on the teacher side and the second upgraded feature on the student side after dimensionality upgrade, thus obtaining the second feature loss.

[0162] Specifically, the feature loss is calculated using the mean squared error loss function to obtain the second feature loss by comparing the second aligned feature on the teacher's side with the second upgraded feature on the student's side after dimensionality upgrade.

[0163] X7 generates a third feature on the teacher side after the second alignment feature is extracted from the teacher side by the third alignment feature layer in the teacher model structure; and generates a third feature on the student side after the second alignment feature is extracted from the student side by the third alignment feature layer in the student model structure.

[0164] Among them, the third alignment feature layer mentioned above is a feature layer located in the deep layer of the alignment feature layers.

[0165] X8, after pre-processing the third feature on the teacher side, obtains the third alignment feature on the teacher side; after pre-processing the third feature on the student side, obtains the third alignment feature on the student side; after dimensionality-upgrading the third alignment feature on the student side, obtains the third dimensionality-upgraded feature on the student side.

[0166] Specifically, the teacher-side third feature can be enhanced and normalized to obtain the teacher-side third alignment feature; the student-side third feature can be enhanced and normalized to obtain the student-side third alignment feature; and the student-side third alignment feature can be convolved with a 1x1 convolution kernel to obtain the student-side second up-dimensional feature.

[0167] X9 uses a preset loss function to calculate the feature loss of the third alignment feature on the teacher side and the third upgraded feature on the student side after dimensionality upgrade, thus obtaining the third feature loss.

[0168] Specifically, the feature loss of the third alignment feature on the teacher side and the third upgraded feature on the student side after dimensionality increase are calculated using the mean squared error loss function, thus obtaining the third feature loss.

[0169] As seen above, the alignment feature layers achieve alignment in two ways: First, hierarchical alignment, where the shallow alignment feature layers of the teacher model correspond to the shallow alignment feature layers of the student model, the mid-level alignment feature layers of the teacher model correspond to the mid-level alignment feature layers of the student model, and the deep alignment feature layers of the teacher model correspond to the deep alignment feature layers of the student model. Second, dimensional alignment, where the student model upscales the alignment features at the alignment feature layers to the same dimension as the teacher model's alignment features, ensuring that the dimensions of the student model's upscaled features are identical to those of the teacher model's alignment features at the alignment feature layers. This allows for subsequent loss calculation based on the two features with the same dimension, and subsequent adjustments to the student model based on the loss. Both of these aspects aim to enable the student model to learn the feature distributions of each alignment feature layer in the teacher model at the selected alignment feature layers, thereby improving the convergence speed of the student model.

[0170] In this embodiment of the invention, before calculating the feature loss of each alignment feature layer, the features extracted by the alignment feature layer are first subjected to pre-processing. Specifically, this can be feature enhancement and normalization. Feature enhancement improves the feature representation capability of the alignment feature layer, ensuring a high-precision student model for subsequent training. Normalization ensures that the data distribution of each layer is basically consistent, thereby accelerating the training of the student model and improving its convergence speed. Furthermore, by performing dimensionality upscaling on the normalized features on the student side, the alignment features on the student side and the alignment features on the teacher side are made to be in the same dimension. Therefore, the loss is calculated based on the alignment features of both sides at each selected alignment feature layer, and the student model is trained based on the loss. This allows the student model to continuously learn the feature distribution of the teacher model at each alignment feature layer, thereby improving the distillation speed of the student model.

[0171] In one possible implementation, corresponding to the above implementation Z, the above-mentioned method of obtaining the total loss of the two models by utilizing the feature loss of each aligned feature layer and the output loss of the network output layer may specifically include: using preset weights to perform weighted calculations on the first feature loss, the second feature loss, the third feature loss, and the output loss to obtain the total loss of the two models.

[0172] During model training, the main constraint is on the network output layer. Therefore, a larger preset weight is set for the output loss, and a smaller preset weight is set for the loss of each alignment feature layer. Furthermore, when setting the preset weights for the loss of each alignment feature layer, the principle of increasing weight for larger layers can be followed. For example, the preset weight for the output loss is set to 0.6, the preset weight for the first feature loss is set to 0.1, the preset weight for the second feature loss is set to 0.13, and the preset weight for the third feature loss is set to 0.17.

[0173] In this embodiment of the invention, by setting weights for the feature losses calculated at different alignment feature layers, and calculating the total loss based on the weights, the total loss can better reflect the difference between the two models, thereby making the subsequent adjustments to the student model based on the total loss more accurate and improving the training speed of the student model.

[0174] In addition, before performing feature enhancement, the features obtained after feature extraction through the alignment feature layer can be normalized. In one example, after obtaining the features after feature extraction through the alignment feature layer, the features are normalized, and then feature enhancement and normalization are performed. A total of two normalization processes are performed to make the data distribution more consistent and facilitate subsequent processing.

[0175] In one possible implementation, the determination of the output probability values ​​of the teacher model and the student model to be trained is achieved through the following steps: the network output layer of the teacher model extracts features based on the previous layer of its network output layer to obtain features, and outputs the output probability value of the teacher model; the network output layer of the student model to be trained extracts features based on the previous layer of its network output layer to obtain features, and outputs the output probability value of the student model to be trained.

[0176] In one example, if the third alignment feature layer of the teacher model is the layer preceding the output layer of the teacher model network, then the output probability value of the teacher model is output based on the third alignment feature of the teacher side; if the third alignment feature layer of the student model to be trained is the layer preceding the output layer of the student model network to be trained, then the output probability value of the student model to be trained is output based on the third alignment feature of the student side.

[0177] In one example, if the third alignment feature layer of the teacher model is not the layer preceding the output layer of the teacher model network, then the third alignment feature is input to the layer below the third alignment feature layer as the input to the next layer. Feature extraction is then performed sequentially on the subsequent layers until the feature is extracted to the layer preceding the network output layer. Based on the feature extraction performed on the layer preceding the network output layer, the output probability value of the teacher model is obtained and output. Similarly, if the third alignment feature layer of the student model to be trained is not the layer preceding the output layer of the student model to be trained, then the third alignment feature is input to the layer below the third alignment feature layer as the input to the next layer. Feature extraction is then performed sequentially on the subsequent layers until the feature is extracted to the layer preceding the network output layer. Based on the feature extraction performed on the layer preceding the network output layer, the output probability value of the student model to be trained is obtained and output.

[0178] In one possible implementation, step S130 may specifically include the following steps S1301-S1306.

[0179] S1301, Input the image to be reviewed into the distilled student model to obtain the probability that each image to be reviewed contains unqualified content.

[0180] In Example 1, three images to be reviewed are input into the distilled student model. The three images to be reviewed are Image 1, Image 2, and Image 3. The probabilities of Image 1 containing unqualified content are 30%, Image 2 containing unqualified content is 10%, and Image 3 containing unqualified content is 90%.

[0181] S1302, determine whether the probability that each image to be reviewed contains unqualified content reaches a preset probability.

[0182] The preset probability can be set according to actual needs, and there are no restrictions here.

[0183] Corresponding to Example 1 above, if the preset probability is set to 80%, then the probability that Image 1 does not reach the preset probability, Image 2 does not reach the preset probability, and Image 3 contains unqualified content is 90%, which reaches the preset probability of 80%.

[0184] S1303, if the target is reached, the image to be reviewed that reaches the preset probability will be regarded as an unqualified image.

[0185] If the requirement is not met, it means that the image to be reviewed is not an unqualified image. The image to be reviewed most likely does not contain unqualified content and is a normal image.

[0186] Corresponding to Example 1 above, Image 3 reaches the preset probability of 80%, therefore Image 3 is a substandard image.

[0187] S1304, determine whether the proportion of the number of unqualified images to the total number of images to be reviewed reaches a preset proportion.

[0188] S1305, if the preset ratio is reached, the video data is unqualified.

[0189] S1306 If the preset ratio is not reached, the video data is considered qualified.

[0190] The above preset ratios are set according to actual requirements and are not restricted here.

[0191] In one example, the preset ratio is 60%. 350 unqualified images are identified from the video data. The total number of images to be reviewed is 500. At this point, the 350 unqualified images account for 70% of the total 500 images to be reviewed, reaching the preset ratio of 60%. Therefore, the video data is determined to be unqualified.

[0192] In addition, when it is determined that the probability of an image to be reviewed containing unqualified content reaches a preset probability, the video data can be directly determined to be unqualified. In other words, as long as it is determined that there is an image to be reviewed that reaches the preset probability, the video data is determined to be unqualified.

[0193] Both of the above methods can determine whether the video data is qualified, and you can choose according to your actual needs.

[0194] In this embodiment of the invention, the probability that each image to be reviewed in the video data contains unqualified content is obtained using a distilled student model. When this probability reaches a preset probability, the image to be reviewed is determined to be unqualified. Then, the proportion of unqualified images in the video data is determined. When this proportion reaches a preset proportion, the video data is determined to be unqualified. The pass / fail status of the video data can be intuitively determined based on preset probabilities and proportions. Furthermore, the preset probabilities and proportions can be adjusted as needed, allowing the determination of video data pass / fail status to be adjusted according to requirements.

[0195] In one example, the following feature layers are selected from the teacher model structure: a shallow feature layer with a dimension of 256 as the first alignment feature layer, a middle feature layer with a dimension of 512 as the second alignment feature layer, a middle feature layer with a dimension of 1024 as the third alignment feature layer, and a deep feature layer with a dimension of 2048 as the fourth alignment feature layer. Similarly, in the student model structure, the following feature layers are selected: a shallow feature layer with a dimension of 72 as the first alignment feature layer, a middle feature layer with a dimension of 240 as the second alignment feature layer, a middle feature layer with a dimension of 672 as the third alignment feature layer, and a deep feature layer with a dimension of 960 as the fourth alignment feature layer. Based on this, see [further details omitted]. Figure 4 The following explains the process of generating the feature loss of each aligned feature layer and the output loss of the network output layer:

[0196] Step 1: After inputting the sample image into the teacher model, the teacher model extracts features from the first sample image. After feature extraction through the first alignment feature layer of the teacher model, the first feature on the teacher side is obtained. Then, after inputting the sample image into the student model to be trained, the student model extracts features from the first sample image. After feature extraction through the first alignment feature layer of the student model, the first feature on the student side is obtained. The teacher-side first feature is processed through channel attention, spatial attention, and BN layers to obtain the teacher-side first aligned feature. The student-side first feature is also processed through channel attention, spatial attention, and BN layers to obtain the student-side first aligned feature. The student-side first aligned feature is then subjected to dimensionality upscaling using a 1x1 convolution kernel to obtain the student-side first dimensionality upscaling feature. The feature loss of the teacher-side first aligned feature and the student-side first dimensionality upscaling feature is calculated using the mean squared error loss function to obtain the first feature loss.

[0197] Step two: After extracting features from the teacher-side first alignment features through the second alignment feature layer of the teacher model, a second feature on the teacher side is generated; similarly, after extracting features from the student-side first alignment features through the second alignment feature layer of the student model to be trained, a second feature on the student side is generated. The teacher-side second feature is processed through channel attention, spatial attention, and BN layers to obtain the teacher-side second alignment feature; the student-side second feature is also processed through channel attention, spatial attention, and BN layers to obtain the student-side second alignment feature. The student-side second alignment feature is then subjected to dimensionality upscaling using a 1x1 convolution kernel to obtain the student-side second dimensionality upscaling feature. The feature loss of the teacher-side second alignment feature and the student-side second dimensionality upscaling feature is calculated using the mean squared error loss function to obtain the second feature loss.

[0198] Step 3: After extracting features from the teacher-side second alignment features through the third alignment feature layer of the teacher model, a third feature on the teacher side is generated. Similarly, after extracting features from the student-side second alignment features through the third alignment feature layer of the student model to be trained, a third feature on the student side is generated. The teacher-side third feature is processed through channel attention, spatial attention, and BN layers to obtain the teacher-side third alignment feature. The student-side third feature is also processed through channel attention, spatial attention, and BN layers to obtain the student-side third alignment feature. The student-side third alignment feature is then subjected to dimensionality upscaling using a 1x1 convolutional kernel to obtain the student-side third dimensionality upscaling feature. The feature loss of the teacher-side third alignment feature and the student-side third dimensionality upscaling feature is calculated using the mean squared error loss function to obtain the third feature loss.

[0199] Step four: After extracting features from the teacher-side third alignment feature through the fourth alignment feature layer of the teacher model, a fourth feature on the teacher side is generated; after extracting features from the student-side third alignment feature through the fourth alignment feature layer of the student model to be trained, a fourth feature on the student side is generated. The teacher-side fourth feature is processed through channel attention, spatial attention, and BN layers to obtain the teacher-side fourth alignment feature; the student-side fourth feature is processed through channel attention, spatial attention, and BN layers to obtain the student-side fourth alignment feature. The student-side fourth alignment feature is then subjected to dimensionality upscaling using a 1x1 convolution kernel to obtain the student-side fourth dimensionality upscaling feature. The feature loss of the teacher-side fourth alignment feature and the student-side fourth dimensionality upscaling feature is calculated using the mean squared error loss function to obtain the fourth feature loss.

[0200] Step 5: Obtain the output probability value of the teacher model through the output layer of the teacher model network; obtain the output probability value of the student model through the output layer of the student model network; calculate the output loss of the output probability values ​​of the teacher model and the student model using the cross-entropy loss function, and obtain the output loss of the network output layer.

[0201] In this embodiment of the invention, not only is the loss of the network output layer calculated, but the feature loss is also calculated for each aligned feature layer, providing a basis for adjusting the parameters of the student model based on the loss.

[0202] The present invention also provides a video review device, see [link to relevant documentation]. Figure 5 The device includes:

[0203] Image acquisition module 51 is used to acquire video data, perform frame extraction on the video data, and obtain the image to be reviewed;

[0204] The student model acquisition module 52 is used to distill the video review model with deep networks to obtain the distilled student model.

[0205] The video review result determination module 53 is used to input the image to be reviewed into the distilled student model to determine whether the video data is qualified.

[0206] In this embodiment of the invention, the distilled student model is obtained by distilling a video review model with deep networks. Since the video review model with deep networks has the characteristic of high review accuracy, the distilled student model also has the characteristic of high review accuracy and is more lightweight. Thus, using the distilled student model to review video data can improve the speed of video review while ensuring the accuracy of the review.

[0207] In one possible implementation, the student model acquisition module includes:

[0208] The model acquisition submodule is used to acquire the trained teacher model and the student model to be trained.

[0209] The alignment feature layer selection submodule is used to select at least one feature layer as the alignment feature layer from the trained teacher model and the student model to be trained, respectively.

[0210] The feature loss calculation submodule is used to acquire the first sample image, input it into the pre-trained teacher model and the student model to be trained respectively, and calculate the feature loss of each aligned feature layer.

[0211] The total loss calculation submodule is used to obtain the total loss of the two models based on the feature loss of each aligned feature layer;

[0212] The model tuning submodule is used to adjust the parameters of the student model to be trained based on the total loss;

[0213] The condition judgment submodule is used to determine whether the total loss meets the preset training termination condition. If so, the distilled student model is obtained; otherwise, the second sample image is obtained again, and the training steps are repeated.

[0214] In one possible implementation, the alignment feature layer selected by the alignment feature layer selection submodule includes: the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained, and the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, and the dimensions of the one-to-one corresponding alignment feature layers are different.

[0215] The aforementioned feature loss calculation submodule is specifically used for:

[0216] Feature extraction is performed using the alignment feature layers of the teacher model and the student model to be trained, and the feature loss of each alignment feature layer is calculated.

[0217] The above-mentioned total loss calculation submodule is specifically used for:

[0218] The output layer of the pre-trained teacher model outputs the output probability value of the teacher model; the output layer of the student model to be trained outputs the output probability value of the student model.

[0219] Using a preset loss function, the output loss of the network output layer is calculated based on the output probability values ​​of the teacher model and the student model.

[0220] The total loss of the two models is obtained by using the feature loss of each aligned feature layer and the output loss of the network output layer.

[0221] In one possible implementation, the alignment feature layer selected by the alignment feature layer selection submodule includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer is at a higher level than the first alignment feature layer, and the second alignment feature layer is at a higher level than the third alignment feature layer.

[0222] The aforementioned feature loss calculation submodule is specifically used for:

[0223] After the first sample image is input into the teacher model, the teacher model extracts features from the first sample image. After the feature extraction operation is performed in the first alignment feature layer of the teacher model, the first feature on the teacher side is generated. After the first sample image is input into the student model to be trained, the student model extracts features from the first sample image. After the feature extraction operation is performed in the first alignment feature layer of the student model to be trained, the first feature on the student side is generated.

[0224] After pre-processing the first feature on the teacher side, the first alignment feature on the teacher side is obtained; after pre-processing the first feature on the student side, the first alignment feature on the student side is obtained; after dimensionality-upgrading the first alignment feature on the student side, the first dimensionality-upgraded feature on the student side is obtained.

[0225] The first feature loss is obtained by calculating the feature loss of the first aligned feature on the teacher's side and the first upgraded feature on the student's side using a preset loss function.

[0226] After extracting features from the first alignment features on the teacher side through the second alignment feature layer of the teacher model, the second feature on the teacher side is generated; after extracting features from the first alignment features on the student side through the second alignment feature layer of the student model to be trained, the second feature on the student side is generated.

[0227] After pre-processing the second feature on the teacher side, the second alignment feature on the teacher side is obtained; after pre-processing the second feature on the student side, the second alignment feature on the student side is obtained; after dimensionality-upgrading the second alignment feature on the student side, the second dimensionality-upgraded feature on the student side is obtained.

[0228] The feature loss of the second alignment feature on the teacher's side and the second up-dimensional feature on the student's side after up-dimensionalization are calculated using a preset loss function to obtain the second feature loss.

[0229] After extracting features from the second alignment features on the teacher side through the third alignment feature layer of the teacher model, the third feature on the teacher side is generated; after extracting features from the second alignment features on the student side through the third alignment feature layer of the student model to be trained, the third feature on the student side is generated.

[0230] After pre-processing the third feature on the teacher side, the third alignment feature on the teacher side is obtained; after pre-processing the third feature on the student side, the third alignment feature on the student side is obtained; after dimensionality-upgrading the third alignment feature on the student side, the third dimensionality-upgraded feature on the student side is obtained.

[0231] The feature loss of the third alignment feature on the teacher side and the third up-dimensional feature on the student side after dimensionality increase are calculated using a preset loss function to obtain the third feature loss.

[0232] In one possible implementation, the above-mentioned total loss calculation submodule is specifically used for:

[0233] Using preset weights, the first feature loss, second feature loss, third feature loss, and output loss are weighted and calculated to obtain the total loss of the two models.

[0234] In one possible implementation, the video review result determination module is specifically used for:

[0235] The images to be reviewed are input into the distilled student model to obtain the probability that each image contains unqualified content;

[0236] Determine whether the probability of each image to be reviewed containing unqualified content reaches a preset probability;

[0237] If the target is reached, the images to be reviewed that reach the preset probability will be considered as unqualified images.

[0238] Determine whether the proportion of unqualified images to the total number of images to be reviewed reaches a preset proportion;

[0239] If the preset ratio is reached, the video data is considered unqualified.

[0240] If the preset ratio is not reached, the video data is considered qualified.

[0241] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0242] This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 61, a communication interface 62, a memory 63, and a communication bus 64, wherein the processor 61, the communication interface 62, and the memory 63 communicate with each other through the communication bus 64.

[0243] Memory 63 is used to store computer programs;

[0244] When the processor 61 executes the program stored in the memory 63, it implements any of the above-mentioned video review methods.

[0245] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0246] The communication interface is used for communication between the aforementioned terminal and other devices.

[0247] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0248] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0249] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the video review method described in any of the above embodiments.

[0250] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video review methods described in the above embodiments.

[0251] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0252] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0253] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0254] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A video review method, characterized in that, The method includes: Acquire video data, perform frame extraction on the video data, and obtain the image to be reviewed; Obtain the trained teacher model and the student model to be trained; At least one feature layer is selected from the pre-trained teacher model and the student model to be trained as an alignment feature layer. The alignment feature layer includes at least one shallow layer, at least one medium layer, and at least one deep layer. The deeper the layer, the more delicate and complete the detailed information of the learned feature representation. The shallow layer is used to extract the edge features and local features of the image, and the deep layer represents the outline and detailed information of the entire object. The first sample image is obtained and input into the pre-trained teacher model and the student model to be trained, respectively, and the feature loss of each alignment feature layer is calculated. Based on the feature loss of each aligned feature layer, the total loss of the two models is obtained. The parameters of the student model to be trained are adjusted based on the total loss. Determine whether the total loss meets the preset training termination condition. If yes, obtain the distilled student model; otherwise, obtain the second sample image again and repeat the training steps. The images to be reviewed are input into the distilled student model, and the video data is determined to be qualified based on the non-compliant content contained in each image to be reviewed in the video data. The alignment feature layer is selected based on the following rule: the feature layers whose dimensions are proportional or nearly proportional in the student model and the teacher model are selected as the alignment feature layer; The alignment feature layer includes: the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained, and the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, and the dimensions of the one-to-one corresponding alignment feature layers are different. The calculation of the feature loss for each aligned feature layer includes: Feature extraction is performed using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and the feature loss of each alignment feature layer is calculated. The alignment feature layer includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer is at a higher level than the first alignment feature layer, and the third alignment feature layer is at a higher level than the second alignment feature layer.

2. The method according to claim 1, characterized in that, The total loss of the two models is obtained based on the feature loss of each aligned feature layer, including: The network output layer of the pre-trained teacher model outputs the output probability value of the teacher model; the network output layer of the student model to be trained outputs the output probability value of the student model. Using a preset loss function, the output loss of the network output layer is calculated based on the output probability values ​​of the teacher model and the student model. The total loss of the two models is obtained by using the feature loss of each aligned feature layer and the output loss of the network output layer.

3. The method according to claim 1, characterized in that, The step of extracting features using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and calculating the feature loss of each alignment feature layer, includes: After the first sample image is input into the teacher model, the teacher model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the teacher model, the first feature on the teacher side is generated. After the first sample image is input into the student model to be trained, the student model extracts features from the first sample image. After the feature extraction operation is performed by the first alignment feature layer of the student model to be trained, the first feature on the student side is generated. After performing preset processing on the teacher-side first feature, a teacher-side first alignment feature is obtained; after performing preset processing on the student-side first feature, a student-side first alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side first alignment feature, a student-side first dimensionality-upgraded feature is obtained. The first feature loss is obtained by calculating the feature loss between the first aligned feature on the teacher side and the first upgraded feature on the student side using a preset loss function. After the teacher-side first alignment feature is extracted by the second alignment feature layer of the teacher model, the teacher-side second feature is generated; after the student-side first alignment feature is extracted by the second alignment feature layer of the student model to be trained, the student-side second feature is generated. After performing preset processing on the teacher-side second feature, a teacher-side second alignment feature is obtained; after performing preset processing on the student-side second feature, a student-side second alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side second alignment feature, a student-side second dimensionality-upgraded feature is obtained. The feature loss of the teacher-side second alignment feature and the student-side second up-dimensional feature after up-dimensionalization is calculated using a preset loss function to obtain the second feature loss. After the teacher-side second alignment feature is extracted by the third alignment feature layer of the teacher model, the teacher-side third feature is generated; after the student-side second alignment feature is extracted by the third alignment feature layer of the student model to be trained, the student-side third feature is generated. After performing preset processing on the teacher-side third feature, a teacher-side third alignment feature is obtained; after performing preset processing on the student-side third feature, a student-side third alignment feature is obtained; after performing dimensionality-upgrading processing on the student-side third alignment feature, a student-side third dimensionality-upgraded feature is obtained. The feature loss of the third alignment feature on the teacher side and the third up-dimensional feature on the student side after dimensionality increase are calculated using a preset loss function to obtain the third feature loss.

4. The method according to claim 3, characterized in that, The total loss of the two models is obtained by utilizing the feature loss of each aligned feature layer and the output loss of the network output layer, including: Using preset weights, the first feature loss, the second feature loss, the third feature loss, and the output loss are weighted and calculated to obtain the total loss of the two models.

5. The method according to claim 1, characterized in that, The step of inputting the image to be reviewed into the distilled student model, and determining whether the video data is acceptable based on the non-compliant content contained in each image to be reviewed in the video data, includes: The images to be reviewed are input into the distilled student model to obtain the probability that each image contains unqualified content; Determine whether the probability of each image to be reviewed containing unqualified content reaches a preset probability; If the target is reached, the images to be reviewed that reach the preset probability will be considered as unqualified images. Determine whether the proportion of unqualified images to the total number of images to be reviewed reaches a preset proportion; If the preset ratio is reached, the video data is considered unqualified. If the preset ratio is not reached, the video data is considered qualified.

6. A video review device, characterized in that, The device includes: The image acquisition module is used to acquire video data and perform frame extraction processing on the video data to obtain the image to be reviewed; A student model acquisition module is used to acquire a pre-trained teacher model and a student model to be trained; select at least one feature layer from the pre-trained teacher model and the student model to be trained as an alignment feature layer, wherein the alignment feature layer includes at least one shallow layer, at least one middle layer, and at least one deep layer; the deeper the layer, the more detailed and complete the learned feature representation information. The shallow layer is used to extract edge features and local features of the image, and the deep layer represents the outline and detail information of the entire object; acquire a first sample image and input it into the pre-trained teacher model and the student model to be trained respectively, and calculate the feature loss of each alignment feature layer; obtain the total loss of the two models based on the feature loss of each alignment feature layer; adjust the parameters of the student model to be trained based on the total loss; determine whether the total loss meets the preset training termination condition. If yes, obtain the distilled student model; otherwise, acquire a second sample image again and repeat the training steps. The video review result determination module is used to input the image to be reviewed into the distilled student model, and determine whether the video data is qualified based on the non-compliant content contained in each image to be reviewed in the video data; The alignment feature layer is selected based on the following rule: the feature layers whose dimensions are proportional or nearly proportional in the student model and the teacher model are selected as the alignment feature layer; The alignment feature layer includes: the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained, and the alignment feature layer of the trained teacher model and the alignment feature layer of the student model to be trained correspond one-to-one, and the dimensions of the one-to-one corresponding alignment feature layers are different. The student model acquisition module is specifically used to extract features using the alignment feature layer of the teacher model and the alignment feature layer of the student model to be trained, and to calculate the feature loss of each alignment feature layer. The alignment feature layer includes a first alignment feature layer, a second alignment feature layer, and a third alignment feature layer, wherein the second alignment feature layer is at a higher level than the first alignment feature layer, and the third alignment feature layer is at a higher level than the second alignment feature layer.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.

Images