Training method and device of video classification model, electronic equipment and storage medium

Abstract

The application relates to a video classification model training method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. In the embodiment of the application, a video classification model is trained through a sample video, a task of classifying the sample video based on image features of sample video images is constructed in the training process, and a task of classifying the sample video based on video features of the sample video is constructed. Through the two classification tasks, the video classification model learns both the content at the video image level and the content at the video level, thereby improving the accuracy of the video classification model when performing video classification. Using the video classification model to classify videos can improve the efficiency of video classification while ensuring accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technology field, and particularly relates to a video classification model training method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of computer technology, more and more users will relax and entertain through watching videos. A video providing platform will audit a video before publishing the video, so as to classify the video, such as classifying the video into a video meeting relevant regulations and a video not meeting relevant regulations.

[0003] In the related art, a video auditor classifies the content of a video to be published, so as to timely find a video not meeting relevant regulations. However, when the video auditor classifies the content of the video, the video auditor needs to completely watch the video, which leads to low efficiency of video classification. Therefore, a more intelligent method is needed to improve the efficiency of video classification, such as training a video classification model based on machine learning technology, and classifying the video through the video classification model. SUMMARY

[0004] The present application provides a video classification model training method and device, electronic equipment and a storage medium to improve the efficiency of video classification. The technical solutions of the present application are as follows:

[0005] In one aspect, a video classification model training method is provided, comprising:

[0006] inputting a sample video into a video classification model, and obtaining image features of a plurality of sample video images of the sample video through a feature extraction sub-model in the video classification model;

[0007] encoding the image features of the plurality of sample video images through an attention encoding sub-model in the video classification model to obtain a video feature of the sample video;

[0008] processing the image features of the plurality of sample video images through the video classification model to obtain a first predicted video type of the sample video, and processing the video feature of the sample video to obtain a second predicted video type of the sample video;

[0009] training the video classification model based on first difference information and second difference information, the first difference information being difference information between a labeled video type of the sample video and the first predicted video type, and the second difference information being difference information between the labeled video type and the second predicted video type.

[0010] In a possible implementation, the training of the video classification model based on the first difference information and the second difference information comprises:

[0011] training the feature extraction sub-model in the video classification model based on the first difference information and the second difference information;

[0012] training the attention encoding sub-model in the video classification model based on the second difference information.

[0013] In a possible implementation, the training of the feature extraction sub-model in the video classification model based on the first difference information and the second difference information comprises:

[0014] In a case where the plurality of sample video images comprise at least two sample video images with different image contents, the feature extraction sub-model is trained based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training, the second weight being used to represent an influence degree of the second loss function on the model parameters of the video classification model during training, and the second weight being greater than the first weight;

[0015] The training of the attention encoding sub-model in the video classification model based on the second difference information comprises:

[0016] training the attention encoding sub-model based on the second loss function and the second weight.

[0017] In a possible implementation, the training of the feature extraction sub-model in the video classification model based on the first difference information and the second difference information comprises:

[0018] In a case where the plurality of sample video images are sample video images with the same image content, the feature extraction sub-model is trained based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training, the second weight being used to represent an influence degree of the second loss function on the model parameters of the video classification model during training, and the second weight being less than the first weight;

[0019] The training of the attention encoding sub-model in the video classification model based on the second difference information comprises:

[0020] training the attention encoding sub-model based on the second loss function and the second weight.

[0021] In a possible implementation, the encoding, by the attention encoding sub-model in the video classification model, of the image features of the plurality of sample video images to obtain the video feature of the sample video comprises:

[0022] encoding, by the attention encoding sub-model in the video classification model, of the image features of each two adjacent sample video images in the plurality of sample video images to obtain attention features of the plurality of sample video images; and fusing the attention features of the plurality of sample video images to obtain the video feature of the sample video.

[0023] In a possible implementation, the processing of the image features of the plurality of sample video images to obtain the first predicted video type of the sample video comprises:

[0024] pooling the image features of the plurality of sample video images to obtain image classification features of the sample video;

[0025] fully connecting and normalizing the image classification features to obtain first classification parameters of the sample video;

[0026] determining the first predicted video type of the sample video based on the first classification parameters.

[0027] In a possible implementation, the pooling of the image features of the plurality of sample video images to obtain the image classification features of the sample video comprises:

[0028] concatenating the image features of the plurality of sample video images to obtain sample concatenated features;

[0029] performing maximum value pooling on each column of the sample concatenated features to obtain the image classification features of the sample video.

[0030] In a possible implementation, the processing of the video feature of the sample video to obtain the second predicted video type of the sample video comprises:

[0031] fully connecting and normalizing the video feature of the sample video to obtain second classification parameters of the sample video;

[0032] determining the second predicted video type of the sample video based on the second classification parameters.

[0033] In a possible implementation, the method further comprises:

[0034] inputting a target video into the video classification model, acquiring image features of a plurality of target video images of the target video through the feature extraction sub-model in the video classification model;

[0035] encoding the image features of the plurality of target video images through an attention encoding sub-model in the video classification model to obtain video features of the target video;

[0036] processing the video features of the target video through the video classification model to obtain a video type of the target video.

[0037] In a possible implementation, before the sample video is input into the video classification model, the method further includes:

[0038] frame extraction from the initial video at a target interval to obtain the sample video.

[0039] In a possible implementation, the frame extraction from the initial video at a target interval to obtain the sample video includes:

[0040] frame extraction from the initial video at the target interval to obtain N sample video images, N being a positive integer;

[0041] In the case where N is less than M, K reference video images are acquired, the reference video images being the first video image of the initial video, M and K being positive integers, and the sum of K and N being M;

[0042] splicing the N sample video images and the K reference video images into the sample video.

[0043] In an aspect, a training device of a video classification model is provided, including:

[0044] an image feature acquisition unit configured to perform inputting a sample video into a video classification model, acquiring image features of a plurality of sample video images of the sample video through a feature extraction sub-model in the video classification model;

[0045] an attention encoding unit configured to perform encoding the image features of the plurality of sample video images through an attention encoding sub-model in the video classification model to obtain video features of the sample video;

[0046] a classification unit configured to perform processing the image features of the plurality of sample video images through the video classification model to obtain a first predicted video type of the sample video, and processing the video features of the sample video to obtain a second predicted video type of the sample video.

[0047] a training unit configured to train the video classification model based on first difference information and second difference information, the first difference information being difference information between a labeled video type of the sample video and a first predicted video type, the second difference information being difference information between the labeled video type and a second predicted video type.

[0048] In a possible implementation, the training unit is configured to train the feature extraction sub-model in the video classification model based on the first difference information and the second difference information, and train the attention encoding sub-model in the video classification model based on the second difference information.

[0049] In a possible implementation, the training unit is configured to, in a case where the plurality of sample video images include at least two sample video images with different image contents, train the feature extraction sub-model based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training, the second weight being used to represent an influence degree of the second loss function on the model parameters of the video classification model during training, the second weight being greater than the first weight, and train the attention encoding sub-model based on the second loss function and the second weight.

[0050] In a possible implementation, the training unit is configured to, in a case where the plurality of sample video images are sample video images with same image content, train the feature extraction sub-model based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training, the second weight being used to represent an influence degree of the second loss function on the model parameters of the video classification model during training, the second weight being less than the first weight, and train the attention encoding sub-model based on the second loss function and the second weight.

[0051] In a possible implementation, the attention encoding unit is configured to perform encoding, by the attention encoding sub-model in the video classification model, of image features of each two adjacent sample video images in the plurality of sample video images to obtain attention features of the plurality of sample video images; and fusing the attention features of the plurality of sample video images to obtain the video feature of the sample video.

[0052] In a possible implementation, the classification unit is configured to perform pooling of the image features of the plurality of sample video images to obtain image classification features of the sample video; full connection and normalization of the image classification features to obtain first classification parameters of the sample video; and determining, based on the first classification parameters, a first predicted video type of the sample video.

[0053] In a possible implementation, the classification unit is configured to perform splicing of the image features of the plurality of sample video images to obtain sample splicing features; maximum pooling of each column of the sample splicing features to obtain image classification features of the sample video.

[0054] In a possible implementation, the base classification unit is configured to perform full connection and normalization of the video feature of the sample video to obtain second classification parameters of the sample video; and determining, based on the second classification parameters, a second predicted video type of the sample video.

[0055] In a possible implementation, the apparatus further includes:

[0056] The type determination unit is configured to perform inputting of a target video into the video classification model, acquiring, by the feature extraction sub-model in the video classification model, image features of a plurality of target video images of the target video, encoding, by the attention encoding sub-model in the video classification model, the image features of the plurality of target video images to obtain a video feature of the target video, and processing, by the video classification model, the video feature of the target video to obtain a video type of the target video.

[0057] In a possible implementation, the apparatus further includes:

[0058] The sample video acquisition unit is configured to perform frame extraction from an initial video at a target interval to obtain the sample video.

[0059] In a possible implementation, the sample video acquisition unit is configured to perform frame extraction from the initial video at the target interval to obtain N sample video images, N being a positive integer; in a case where N is less than M, K reference video images are acquired, the reference video images being the first video image of the initial video, M and K being positive integers, and the sum of K and N being M; and the N sample video images and the K reference video images are spliced into the sample video.

[0060] In an aspect, an electronic device is provided, comprising:

[0061] a processor;

[0062] a memory for storing processor-executable instructions;

[0063] wherein the processor is configured to execute the instructions to implement the method for training the video classification model.

[0064] In an aspect, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for training the video classification model.

[0065] In an aspect, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, the method for training the video classification model is implemented.

[0066] The embodiments of the present application provide at least the following beneficial effects:

[0067] In the embodiments of the present application, a video classification model is trained by using sample videos, in the training process, a task of classifying sample videos based on image features of sample video images and a task of classifying sample videos based on video features of sample videos are constructed, through the two classification tasks, the video classification model learns both the content at the video image level and the content at the video level, thereby improving the accuracy of the video classification model when classifying videos. Using the video classification model to classify videos can improve the efficiency of video classification while ensuring accuracy.

[0068] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0069] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application, and are not an undue limitation on the present application.

[0070] Figure 1is a schematic diagram of an implementation environment of a training method of a video classification model according to an exemplary embodiment.

[0071] Figure 2 is a flowchart of a training method of a video classification model according to an exemplary embodiment.

[0072] Figure 3 is a flowchart of another training method of a video classification model according to an exemplary embodiment.

[0073] Figure 4 is a flowchart of yet another training method of a video classification model according to an exemplary embodiment.

[0074] Figure 5 is a flowchart of a video classification method according to an exemplary embodiment.

[0075] Figure 6 is a block diagram of a training device of a video classification model according to an exemplary embodiment.

[0076] Figure 7 is a block diagram of a terminal according to an exemplary embodiment.

[0077] Figure 8 is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] In order to make the ordinary person skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings.

[0079] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0080] Some terms related to the embodiments of the present application are introduced below.

[0081] Normalization: mapping a number series with different value ranges to the interval (0, 1) for easy data processing. In some cases, the normalized numerical value can be directly implemented as a probability.

[0082] Embedded Coding: Embedded coding mathematically represents a corresponding relationship, that is, mapping data on X space to Y space through a function F, where the function F is a single function, the mapping result is structure preservation, the single function means that the mapped data is uniquely corresponding to the data before mapping, and the structure preservation means that the size relationship of the data before mapping is the same as that of the data after mapping, for example, there are data X1 and X2 before mapping, and Y1 corresponding to X1 and Y2 corresponding to X2 are obtained after mapping. If the data X1 > X2 before mapping, then the data Y1 after mapping is greater than Y2 accordingly. For words, it is to map words to another space, which is convenient for subsequent machine learning and processing.

[0083] Attention weight: can represent the importance of certain data in the training or prediction process, the importance represents the size of the influence of the input data on the output data. The data with high importance has a higher value of the corresponding attention weight, and the data with low importance has a lower value of the corresponding attention weight. In different scenarios, the importance of data is not the same, and the process of training attention weight of the model is also the process of determining the importance of data.

[0084] It should be noted that the information (including but not limited to user device information, user personal information, etc.) data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0085] Figure 1 is an implementation environment schematic diagram of a video classification model training method provided by an embodiment of the present application, referring to Figure 1 The implementation environment includes a terminal 101 and a server 102.

[0086] The terminal 101 can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer and a notebook computer. The terminal 101 can be installed and run with an application program supporting video classification model training. The user can log in to the application program through the terminal 101, train the video classification model through the application program, and in some embodiments, the user's account is logged in on the application program.

[0087] The terminal 101 can be used to represent one of a plurality of terminals, and the embodiment is only used to illustrate the terminal 101. Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminal 101 can be only a few, or the above terminal 101 can be dozens or hundreds, or more, and the number of the terminal 101 and the type of the equipment are not limited in the embodiment. The terminal 101 can be connected to the server 102 through a wireless network or a wired network.

[0088] The server 102 can be at least one of a server, a plurality of servers, a cloud computing platform and a virtualization center. The server 102 provides a background service for an application program running on the terminal 101.

[0089] In some embodiments, the number of the above server 102 can be more or less, and the embodiment is not limited. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0090] After introducing the implementation environment of the embodiment, the application scenario of the embodiment will be introduced in combination with the above implementation environment. In the following description, the terminal is the terminal 101 in the above implementation environment, and the server is the server 102 in the above implementation environment.

[0091] The technical scheme provided by the embodiment can be applied to various scenes requiring video classification, such as scenes of determining whether a video is a rule violation video or scenes of adding a label to a video, and the embodiment is not limited.

[0092] Taking the training method of the video classification model provided by the embodiment as an example, the technical personnel collect sample videos used for training the video classification model through the terminal, and add labels to the sample videos and sample video images in the sample videos. The label is used to indicate whether the sample video and the sample video image in the sample video are rule violation videos. The terminal uploads the collected sample videos and corresponding labels to the server, and the server trains the video classification model based on the sample videos and corresponding labels by using the technical scheme provided by the embodiment. After the training is completed, the video classification model is deployed on the server, and the technical personnel can call the video classification model through the terminal, input a target video to be classified into the video classification model, and output the type of the target video from the video classification model. The type is used to indicate whether the target video is a rule violation video.

[0093] It should be noted that the above is an example of the technical solution provided by the embodiments of the present application applied in the scenario of determining whether a video is a rule violation video. The technical solution provided by the embodiments of the present application is applied in other scenarios of classifying videos, and the above description belongs to the same inventive concept, and details are not repeated.

[0094] In addition, the above is an example of the server training the video classification model. In other possible implementations, the video classification model can also be trained by a terminal or other electronic device, and the embodiments of the present application do not limit this.

[0095] After introducing the implementation environment and application scenario of the embodiments of the present application, the technical solution provided by the embodiments of the present application is described, referring to Figure 2 Taking the server as an execution subject, the method comprises:

[0096] In step S201, the server inputs a sample video into a video classification model, and obtains image features of a plurality of sample video images of the sample video through a feature extraction sub-model in the video classification model.

[0097] The video classification model is a model for classifying videos, and the sample video is a video used to train the video classification model. In the process of training the video classification model, the server will use a plurality of sample videos to train the video classification model. The sample video image is also referred to as a video image in the sample video, and the image feature of the sample video image is used to represent the characteristics of the sample video image. In some embodiments, the image feature of the sample video image is an image feature vector of the sample video image. The video classification model includes a feature extraction sub-model, which is used to extract image features of video images.

[0098] In step S202, the server encodes the image features of the plurality of sample video images based on an attention mechanism through an attention encoding sub-model in the video classification model, to obtain a video feature of the sample video.

[0099] The attention encoding sub-model can process image features based on an attention mechanism. When processing the image features of the plurality of sample video images based on the attention mechanism, the information in the image features can be fully utilized, so that the determined video feature can more accurately represent the sample video.

[0100] In step S203, the server processes the image features of the plurality of sample video images through the video classification model, to obtain a first predicted video type of the sample video. The video feature of the sample video is processed to obtain a second predicted video type of the sample video.

[0101] The first predicted video type is determined based on image features, and the second predicted video type is determined based on video features.

[0102] In step S204, the server trains the video classification model based on first difference information and second difference information, the first difference information being difference information between a labeled video type of the sample video and the first predicted video type, and the second difference information being difference information between the labeled video type and the second predicted video type.

[0103] The labeled video type can be regarded as a true label of the sample video, serving as supervision when training the video classification model.

[0104] In the embodiments of the present application, a video classification model is trained by using sample videos, and in the training process, a task of classifying sample videos based on image features of sample video images and a task of classifying sample videos based on video features of sample videos are constructed, so that the video classification model learns both video image level content and video level content through the two classification tasks, thereby improving the accuracy of the video classification model when classifying videos. Using the video classification model to classify videos can improve the efficiency of video classification while ensuring accuracy.

[0105] In a possible implementation, the training of the video classification model based on the first difference information and the second difference information includes:

[0106] The feature extraction sub-model in the video classification model is trained based on the first difference information.

[0107] The feature extraction sub-model and the attention encoding sub-model in the video classification model are trained based on the second difference information.

[0108] In this implementation, different sub-models of the video classification model can be trained based on the first difference information and the second difference information, thereby realizing the overall training of the video classification model.

[0109] In a possible implementation, the training of the feature extraction sub-model in the video classification model based on the first difference information includes:

[0110] In the case where the plurality of sample video images include at least two sample video images with different image contents, the feature extraction sub-model is trained based on a first loss function corresponding to the first difference information and a first weight, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training.

[0111] The training of the feature extraction sub-model and the attention encoding sub-model in the video classification model based on the second difference information includes:

[0112] The training of the feature extraction sub-model and the attention encoding sub-model based on a second loss function corresponding to the second difference information and a second weight, the second weight being used to represent an influence degree of the second loss function on model parameters of the video classification model during training, the second weight being greater than the first weight.

[0113] In this implementation, the plurality of sample video images includes at least two different sample video images, which means that the sample video is a real video, and the attention encoding sub-model can be focused on training to improve the accuracy of the video features output by the attention encoding sub-model.

[0114] In a possible implementation, the training of the feature extraction sub-model in the video classification model based on the first difference information includes:

[0115] In the case where the plurality of sample video images are sample video images of the same image content, the training of the feature extraction sub-model based on a first loss function corresponding to the first difference information and a first weight, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training.

[0116] The training of the feature extraction sub-model and the attention encoding sub-model in the video classification model based on the second difference information includes:

[0117] The training of the feature extraction sub-model and the attention encoding sub-model based on a second loss function corresponding to the second difference information and a second weight, the second weight being used to represent an influence degree of the second loss function on model parameters of the video classification model during training, the second weight being less than the first weight.

[0118] In this implementation, the plurality of sample video images are the same sample video images, which means that the sample video is a synthetic video, and the video features are actually the same as the image features, and the feature extraction sub-model can be focused on training to improve the accuracy of the image features output by the feature extraction sub-model.

[0119] In a possible implementation, the encoding of the image features of the plurality of sample video images by the attention encoding sub-model in the video classification model to obtain the video features of the sample video includes:

[0120] The image features of each two adjacent sample video images in the plurality of sample video images are encoded by the attention encoding submodel in the video classification model to obtain attention features of the plurality of sample video images. The attention features of the plurality of sample video images are fused to obtain the video feature of the sample video.

[0121] In this implementation, the image features of the plurality of sample video images can be encoded based on the attention mechanism by the video classification model to obtain the attention features of the plurality of sample video images. The video feature of the sample video can be obtained by fusing the attention features of the plurality of sample video images.

[0122] In a possible implementation, the processing of the image features of the plurality of sample video images to obtain the first predicted video type of the sample video includes:

[0123] The image features of the plurality of sample video images are pooled to obtain image classification features of the sample video.

[0124] The image classification features are fully connected and normalized to obtain first classification parameters of the sample video.

[0125] The first predicted video type of the sample video is determined based on the first classification parameters.

[0126] In this implementation, the image features of the plurality of sample video images are converted into image classification features by pooling, and the image classification features fuse the contents of the image features of the plurality of sample video images. The image classification features can represent the sample video at the granularity of video images.

[0127] In a possible implementation, the pooling of the image features of the plurality of sample video images to obtain the image classification features of the sample video includes:

[0128] The image features of the plurality of sample video images are spliced to obtain sample splicing features.

[0129] Each column of the sample splicing features is maximum-pooled to obtain the image classification features of the sample video.

[0130] In a possible implementation, the processing of the video feature of the sample video to obtain the second predicted video type of the sample video includes:

[0131] The video feature of the sample video is fully connected and normalized to obtain second classification parameters of the sample video.

[0132] The second predicted video type of the sample video is determined based on the second classification parameters.

[0133] In a possible implementation, the method further includes:

[0134] inputting the target video into the video classification model, and obtaining image features of a plurality of target video images of the target video through the feature extraction sub-model in the video classification model.

[0135] encoding the image features of the plurality of target video images through the attention encoding sub-model in the video classification model to obtain video features of the target video.

[0136] processing the video features of the target video through the video classification model to obtain a video type of the target video.

[0137] In this implementation, the target video can be quickly classified through the video classification model, without the need for manual viewing of the target video and subsequent classification, thereby greatly improving the efficiency of video classification.

[0138] In a possible implementation, before inputting the sample video into the video classification model, the method further includes:

[0139] obtaining the sample video by frame extraction from the initial video at a target interval.

[0140] In a possible implementation, the obtaining the sample video by frame extraction from the initial video at a target interval includes:

[0141] obtaining N sample video images from the initial video at the target interval, where N is a positive integer.

[0142] In a case where N is less than M, obtaining K reference video images, the reference video images being the first video image of the initial video, M and K are both positive integers, and the sum of K and N is M.

[0143] splicing the N sample video images and the K reference video images into the sample video.

[0144] The steps S201-S204 are a simple introduction to the technical solution provided by the embodiments of the application. The technical solution provided by the embodiments of the application will be described more clearly in combination with some examples, with reference to Figure 3 For example, the execution subject is a server, and the method includes:

[0145] In step S301, the server obtains a sample video.

[0146] The sample video is a video used for training the video classification model. In some embodiments, the sample video is a real video or a synthetic video. The real video refers to a video obtained by a video producer through shooting, editing, animation production, or the like. The synthetic video refers to a video synthesized by the server based on one or more images.

[0147] In a possible implementation, the server frames the initial video at a target interval to obtain the sample video.

[0148] The initial video is a real video, that is, a video obtained by a video producer through shooting, editing, animation production, or the like, such as a movie, a television series, and a short video. The target interval is set by a technician according to actual conditions, for example, 5 frames, 10 frames, or the like, which is not limited in the embodiments of the present application. In the technical scheme provided by the embodiments of the present application applied to a scenario of determining whether a video is a rule violation video, the initial video is a rule violation video or a normal video. In this case, the server can also obtain a label of the initial video, which is also a label of the sample video, used to indicate whether the sample video is a rule violation video or a normal video.

[0149] In this implementation, the server obtains the sample video by framing the initial video, so that the number of video images in the sample video is less than that in the initial video, thereby reducing the operation amount of processing a single video in the process of training the video classification model and improving the efficiency of training the video classification model.

[0150] For example, the server frames the initial video at the target interval to obtain N sample video images, where N is a positive integer. In the case where N is less than M, the server obtains K reference video images, the reference video image is the first video image of the initial video, M and K are positive integers, and the sum of K and N is M. The server splices the N sample video images and the K reference video images into the sample video. M is a preset number of video images. Through the technical scheme provided in the above example, different sample videos each include the same number M of video images, thereby avoiding the interference of the difference in the number of video images on model training and improving the accuracy of the trained video classification model in video classification.

[0151] In a possible implementation, the server obtains a target image, which is a rule violation image or a normal image. The server duplicates the target image multiple times to obtain multiple target images. The server fuses the multiple target images into the sample video. In some embodiments, the server determines the label of the target image as the label of the sample video.

[0152] In this implementation, the server can generate the sample video based on the target image, greatly reducing the difficulty of obtaining the sample video, so that more sample videos can be used when training the video classification model, thereby improving the training effect of the sample video.

[0153] It should be noted that the server can obtain the sample video in any of the above manners, or obtain multiple sample videos in combination with the above two implementations, and the embodiments of the present application do not limit this. In addition to the above two implementations, the embodiments of the present application further provide another manner of obtaining a sample video.

[0154] In a possible implementation, in response to the operation on the sample video, the terminal sends the sample video to the server. In this implementation, the technician can control the terminal to send the sample video to the server through the operation on the sample video, and the technician can select the sample video by himself / herself, so that the efficiency of human-computer interaction is higher.

[0155] For example, the terminal displays a video selection page, and the video selection page includes multiple candidate videos. In response to a click operation on a sample video in the multiple candidate videos, the terminal sends the sample video to the server, and the server obtains the sample video. In this case, the multiple candidate videos are videos stored on the terminal. In the case where the multiple candidate videos are videos stored on the server, in response to a click operation on a sample video on the video selection page, the terminal sends a video selection instruction to the server, and the video selection instruction carries an identifier of the sample video. After receiving the video selection instruction, the server obtains the identifier of the sample video from the video selection instruction. The server queries based on the identifier of the sample video and obtains the sample video.

[0156] It should be noted that step S301 is an optional step. In the case where there is a sample video generated in advance, the server directly performs the following step S302. In the following steps S302-S306, the server is taken as an example for performing one iteration training of the video classification model, and the implementation process of the server in other iteration training processes belongs to the same inventive concept as the above steps S302-S306.

[0157] In step S302, the server inputs the sample video into the video classification model, and obtains image features of multiple sample video images of the sample video through a feature extraction sub-model in the video classification model.

[0158] The video classification model is a model used for classifying videos, and the image features are used to represent deep features of the sample video images. The feature extraction sub-model in the video classification model is used for feature extraction on video images to obtain image features.

[0159] In a possible implementation, the server inputs the sample video into the video classification model, and performs convolution on sample video images of the sample video by a feature extraction sub-model in the video classification model to obtain image features of the sample video images.

[0160] In this implementation, the image features of the sample video images can be extracted by the feature extraction sub-model, so that abstract expression of the sample video images is realized, and the subsequent operation efficiency is improved.

[0161] For example, after the server inputs the sample video into the video classification model, for any sample video image in the sample video images, the feature extraction sub-model is used to perform feature extraction on the sample video image, that is, a convolution kernel on a convolution layer of the feature extraction sub-model is used to slide on the sample video image, and convolution is performed on the covered area in the sliding process to obtain image features of the sample video image. In some embodiments, the image features of the sample video image are also referred to as feature maps of the sample video image. In the case where the sample video image includes multiple color channels, the feature extraction sub-model is used to perform convolution on the multiple color channels of the sample video image respectively, that is, multiple convolution kernels are used to slide on the multiple color channels of the sample video image respectively, and convolution is performed on the covered area in the sliding process to obtain multiple channel image features corresponding to the multiple color channels, where the multiple convolution kernels correspond to the multiple color channels one by one. The feature extraction sub-model is used to fuse the multiple channel image features to obtain the image features of the sample video image. In some embodiments, the feature extraction sub-model further includes a pooling layer, and after the convolution layer of the feature extraction sub-model is used to perform convolution on the sample video image, the feature maps obtained by the convolution are input into the pooling layer, and the pooling layer is used to pool the feature maps obtained by the convolution to obtain the image features of the sample video image. The pooling layer is used to pool the feature maps obtained by the convolution, so that the data amount in the feature maps is reduced, and the efficiency of subsequent training of the video classification model is improved.

[0162] It should be noted that the above is an example of one convolution of the sample video image by the feature extraction sub-model in the server, and in other possible implementations, the server can perform multiple convolutions on the sample video image by the feature extraction sub-model, which is not limited in the embodiments of the present application. In some embodiments, the feature extraction sub-model is Resent-50 (Residual Network 50) or Inception-V3 (Asymmetric Convolution), and of course, as science and technology develop, the feature extraction sub-model can be replaced by a more advanced feature extraction model, which is not limited in the embodiments of the present application.

[0163] In a possible implementation, the server inputs the sample video into the video classification model, performs time-frequency transformation on a plurality of sample video images of the sample video by a feature extraction sub-model in the video classification model, to obtain a plurality of frequency domain sample video images corresponding to the plurality of sample video images.

[0164] In this implementation, the server can extract the spatial domain image feature and the frequency domain image feature of the sample video image by the feature extraction sub-model, and the image feature obtained based on the spatial domain image feature and the frequency domain image feature can more accurately represent the sample video image, thereby improving the accuracy of subsequent classification by the video classification model.

[0165] For example, after the server inputs the sample video into the video classification model, for any sample video image in the plurality of sample video images, the server performs Fourier transformation on the sample video image by the feature extraction sub-model, to obtain a frequency domain sample video image corresponding to the sample video image. The server performs convolution on the sample video image by a spatial domain convolution layer of the feature extraction sub-model, to obtain a spatial domain image feature of the sample video image. The server performs convolution on the frequency domain sample video image by a frequency domain convolution layer of the feature extraction sub-model, to obtain a frequency domain image feature of the sample video image. The server fuses the spatial domain image feature and the frequency domain image feature of the sample video image by the feature extraction sub-model, to obtain an image feature of the sample video image.

[0166] In a possible implementation, the server inputs the sample video into the video classification model, encodes a plurality of sample video images of the sample video based on an attention mechanism by a feature extraction sub-model in the video classification model, to obtain image features of the plurality of sample video images.

[0167] In this implementation, the server can encode the plurality of sample video images based on the attention mechanism through the feature extraction sub-model in the video classification model, so as to reference other sample video images in the sample video when obtaining the image features of each sample video image, improve the accuracy of the determined image features, and improve the accuracy of subsequent video classification based on the image features.

[0168] For example, the server inputs the plurality of sample video images into the video classification model, embeds and encodes the plurality of sample video images through the feature extraction sub-model of the video classification model to obtain a plurality of first embedding features, one first embedding feature corresponding to one sample video image, and the first embedding feature being used to represent the position of each sample video image in the plurality of sample video images and the content of each sample video image. The server inputs the plurality of first embedding features into the video classification model, linearly transforms the plurality of first embedding features through the three linear transformation matrices of the feature extraction sub-model to obtain a query vector, a key vector and a value vector corresponding to each sample video image of the plurality of sample video images. The server obtains the attention weights of the plurality of sample video images based on the query vector and the key vector corresponding to the plurality of sample video images through the feature extraction sub-model. The server obtains the attention encoding vector of the plurality of sample video images based on the attention weight of each sample video image of the plurality of sample video images and the value vector of each sample video image of the plurality of sample video images through the feature extraction sub-model, and the attention encoding vector is also the image feature of the sample video image.

[0169] It should be noted that the server can obtain the image features of the plurality of sample video images in any of the above manners, and the embodiments of the present application do not limit the same.

[0170] In step S303, the server encodes the image features of the plurality of sample video images through the attention encoding sub-model in the video classification model to obtain the video feature of the sample video.

[0171] The attention encoding sub-model can process the image features based on the attention mechanism, and when processing the image features of the plurality of sample video images based on the attention mechanism, the information in the image features can be fully utilized, so that the determined video feature can more accurately represent the sample video.

[0172] In a possible implementation, the server encodes image features of each two adjacent sample video images in the plurality of sample video images by an attention encoding sub-model in the video classification model to obtain attention features of the plurality of sample video images. The server fuses the attention features of the plurality of sample video images to obtain the video feature of the sample video.

[0173] In this implementation, the server can encode image features of the plurality of sample video images by an attention encoding sub-model in the video classification model to obtain attention features of the plurality of sample video images. The video feature of the sample video can be obtained by fusing the attention features of the plurality of sample video images, and the accuracy of the obtained video feature is high.

[0174] For example, the attention encoding sub-model is used to obtain a video feature of a video. The server inputs image features of the plurality of sample video images into the attention encoding sub-model in the video classification model, and encodes the image features of the plurality of sample video images by the attention encoding sub-model to obtain a plurality of second embedding features. Each second embedding feature corresponds to an image feature of a sample video image, and is used to represent a position of the image feature of the sample video image in the image features of the plurality of sample video images and a content of the image feature of the sample video image. The server inputs the plurality of second embedding features into the video classification model, and linearly transforms the plurality of second embedding features by three linear transformation matrices of the attention encoding sub-model to obtain a query vector, a key vector, and a value vector corresponding to the image feature of each sample video image in the plurality of sample video images. The server obtains attention weights of the image features of the plurality of sample video images based on the query vector and the key vector corresponding to the image features of the plurality of sample video images by the attention encoding sub-model. The server obtains attention features of the plurality of sample video images based on the attention weights and the value vector of the image features of the plurality of sample video images by the video classification model. The server fuses the attention features of the plurality of sample video images to obtain the video feature of the sample video. In some embodiments, the video classification model is a semantic feature encoder, such as a Bidirectional Encoder Representations from Transformers (BERT) encoder.

[0175] For example, the server multiplies each second embedding feature with three linear transformation matrices respectively through the attention encoding sub-model to obtain a query vector, a key vector and a value vector corresponding to the image feature of each sample video image. For a first sample video image in the plurality of sample video images, the server determines a plurality of attention weights of the image features of the plurality of other sample video images on the image feature of the first sample video image based on the query vector of the image feature of the first sample video image and the key vectors of the image features of the plurality of other sample video images through the attention encoding sub-model. The server performs weighted summation on the attention weights of the image features of the plurality of other sample video images on the image feature of the first sample video image and the value vectors of the image features of the plurality of other sample video images through the attention encoding sub-model to obtain the attention feature of the first sample video image. It should be noted that the above is described by taking the server encoding the first sample video image in the plurality of sample video images through the video classification model to obtain the attention feature of the first sample video image as an example. The encoding manner of the server on other sample video images in the plurality of sample video images belongs to the same inventive concept as the encoding method of the first sample video image, and the implementation process is described above and will not be described here.

[0176] In step S304, the server processes the image features of the plurality of sample video images through the video classification model to obtain a first predicted video type of the sample video.

[0177] The first predicted video type is a video type determined based on the image features, and the second predicted type is a video type determined based on the video features. The granularities of the features used to determine the two video types are different.

[0178] In a possible implementation, the server pools the image features of the plurality of sample video images through the video classification model to obtain image classification features of the sample video. The server performs full connection and normalization on the image classification features through the video classification model to obtain first classification parameters of the sample video. The server determines a first predicted video type of the sample video based on the first classification parameters through the video classification model. In some embodiments, the video classification model outputs a label, which is used to indicate the video type of the sample video. For example, the video classification model outputs a first predicted label, which is used to represent the first predicted video type.

[0179] In this implementation, the server converts the image features of the plurality of sample video images into image classification features by pooling, and the image classification features fuse the contents of the image features of the plurality of sample video images, and the image classification features can represent the sample video at the granularity of the video images.

[0180] For example, the server splices the image features of the plurality of sample video images by the video classification model to obtain sample spliced features. The server performs maximum value pooling on each column of the sample spliced features to obtain the image classification features of the sample video. For example, in the case where the sample spliced features are an MxD-dimensional feature matrix, the server performs maximum value pooling on each column of the MxD-dimensional feature matrix by the video classification model, that is, retains the maximum value of each column of the MxD-dimensional feature matrix to obtain a D-dimensional feature vector, and the D-dimensional feature vector is the image classification features, M and D are positive integers, and in the case where the image features are a 1xD-dimensional feature vector, M is the number of sample video images in the sample video. The server performs at least one full connection on the image classification features by the video classification model to obtain first full connection features. The server performs normalization on the first full connection features by the video classification model to obtain first classification parameters. The first classification parameters are probability sequences, and the probability sequences include a plurality of probability values, and each probability value corresponds to a video type. The server determines the video type corresponding to the maximum probability value in the first classification parameters as the first predicted video type by the video classification model. In some embodiments, the video classification model includes a first classifier, and the first classifier is configured to perform the above steps, and the server determines the first predicted video type based on the image features of the plurality of sample video images by the first classifier.

[0181] In the case where the training method of the video classification model provided in the embodiments of the present application is applied to the scenario of determining whether a video is a violation video, the number of probability values in the first classification parameters is two or three. In the case where the number of probability values in the first classification parameters is two, the two probability values correspond to two video types, which are a violation video and a normal video, respectively. In the case where the number of probability values in the first classification parameters is three, the three probability values correspond to three video types, which are a violation video, a normal video, and a pending video, respectively, and the pending video is a video that cannot be accurately classified by the video classification model and needs to be confirmed by a video reviewer.

[0182] In step S305, the server processes the video features of the sample video by the video classification model to obtain a second predicted video type of the sample video.

[0183] In a possible implementation, the server performs full connection and normalization on the video features of the sample video by using the video classification model to obtain second classification parameters of the sample video. The server determines a second predicted video type of the sample video based on the second classification parameters by using the video classification model.

[0184] For example, the server performs at least one time of full connection on the video features of the sample video by using the video classification model to obtain second full connection features. The server performs normalization on the second full connection features by using the video classification model to obtain second classification parameters. The second classification parameters are probability sequences, and each probability value in the probability sequences corresponds to a video type. The server determines a video type corresponding to a maximum probability value in the second classification parameters as the second predicted video type by using the video classification model. In some embodiments, the video classification model includes a second classifier, and the second classifier is used to perform the above steps. The server determines the second predicted video type based on the video features by using the second classifier. In some embodiments, the video classification model outputs a label indicating the video type of the sample video. For example, the video classification model outputs a second predicted label indicating the second predicted video type.

[0185] In the case where the training method of the video classification model provided in the embodiments of the present application is applied to the scenario of determining whether a video is a violation video, the number of probability values in the second classification parameters is two or three. In the case where the number of probability values in the second classification parameters is two, the two probability values correspond to two video types, i.e., a violation video and a normal video. In the case where the number of probability values in the second classification parameters is three, the three probability values correspond to three video types, i.e., a violation video, a normal video, and a pending video. The pending video is a video that cannot be accurately classified by the video classification model and needs to be confirmed by a video reviewer.

[0186] In order to make the technical solutions provided in the embodiments of the present application clearer, the following will be combined with Figure 4 The above steps S302-S305 will be described.

[0187] Referring to Figure 4The video classification model includes a feature extraction sub-model 401, an attention encoding sub-model 402, a first classifier 403, and a second classifier 404. The server inputs a sample video into the video classification model, performs feature extraction on a plurality of sample video images 4011 of the sample video by the feature extraction sub-model 401 of the video classification model, and obtains image features 4012 of the plurality of sample video images 4011. The server performs maximum value pooling on a sample splicing feature by the video classification model, and obtains image classification features 405 of the sample video, the sample splicing feature being obtained by splicing the image features 4012 of the plurality of sample video images 4011. The server inputs the image features 4012 of the plurality of sample video images 4011 into the attention encoding sub-model 402, encodes the image features 4012 of the plurality of sample video images 4011 based on an attention mechanism by the attention encoding sub-model 402, and obtains video features 4021 of the sample video. The server inputs the image classification features 405 of the sample video into the first classifier 403, performs full connection and normalization on the image classification features 405 by the first classifier 403, and obtains first classification parameters. The server obtains a first predicted video type of the sample video based on the first classification parameters. The server inputs the video features 4021 of the sample video into the second classifier 404, performs full connection and normalization on the video features 4021 by the second classifier 404, and obtains second classification parameters. The server obtains a second predicted video type of the sample video based on the second classification parameters.

[0188] In step S306, the server trains the video classification model based on the first difference information and the second difference information, the first difference information being difference information between the labeled video type of the sample video and the first predicted video type, and the second difference information being difference information between the labeled video type and the second predicted video type.

[0189] The labeled video type can be regarded as a true label of the sample video, and serves as supervision when training the video classification model.

[0190] In a possible implementation, the video classification model includes a feature extraction sub-model and an attention encoding sub-model, the server trains the feature extraction sub-model based on the first difference information and the second difference information. The server trains the attention encoding sub-model based on the second difference information.

[0191] In this implementation, the server can train different sub-models of the video classification model based on the first difference information and the second difference information, thereby realizing overall training of the video classification model.

[0192] To make the above-mentioned embodiments more clear, the following two examples are used to illustrate the above-mentioned embodiments.

[0193] In the case where the plurality of sample video images include at least two sample video images with different image contents, the server trains the feature extraction sub-model based on a first loss function corresponding to the first difference information and a first weight, and a second loss function corresponding to the second difference information and a second weight, the first weight representing an influence degree of the first loss function on model parameters of the video classification model during training, the second weight representing an influence degree of the second loss function on the model parameters of the video classification model during training, and the second weight being greater than the first weight. The server trains the attention encoding sub-model based on the second loss function and the second weight.

[0194] In the case where the plurality of sample video images include at least two different sample video images, the plurality of sample video images are obtained by the first embodiment in step S301. The second weight being greater than the first weight means that the attention encoding sub-model is mainly trained based on the video features of the sample videos during training. The model parameters of the video classification model are internal parameters of the video classification model.

[0195] In this embodiment, the plurality of sample video images include at least two different sample video images, which means that the sample videos are real videos, and thus the attention encoding sub-model can be mainly trained to improve the accuracy of the video features output by the attention encoding sub-model.

[0196] For example, in the case where the plurality of sample video images include at least two sample video images with different image contents, the server constructs a first loss function based on the first difference information, the first loss function being a cross-entropy loss function. The server multiplies a loss value of the first loss function by a first weight to obtain a first target loss value. The server constructs a second loss function based on the second difference information, the second loss function being a cross-entropy loss function. The server multiplies a loss value of the second loss function by a second weight to obtain a second target loss value. The server trains the feature extraction sub-model by using a gradient descent method based on the first target loss value and the second target loss value, i.e., adjusts model parameters of the feature extraction sub-model. The server trains the attention encoding sub-model by using the gradient descent method based on the second target loss value, i.e., adjusts model parameters of the attention encoding sub-model. The form of the first loss function is shown in the following formula (1), and the form of the second loss function is shown in the following formula (2).

[0197]

[0198]

[0199] wherein, L1 is a first loss function, L2 is a second loss function, is a label corresponding to a labeled video type, y1 is a label corresponding to a first predicted video type, and y2 is a label corresponding to a second predicted video type.

[0200] In some embodiments, the first loss function L1 and the second loss function L2 described above can be combined into a joint loss function L by a first weight and a second weight c , and the server trains the feature extraction sub-model and the attention encoding sub-model based on the joint loss function L c The form of the joint loss function L c is shown in the following formula (3).

[0201] L c = w1L1 + w2L2 (3)

[0202] wherein, w1 is a first weight, and w2 is a second weight. In some embodiments, in the case where the plurality of sample video images include at least two different sample video images, w1 = 0.4, and w2 = 1.

[0203] In example 2, in the case where the plurality of sample video images are sample video images of the same image content, the server trains the feature extraction sub-model based on the first loss function corresponding to the first difference information and the first weight, and the second loss function corresponding to the second difference information and the second weight, the first weight is used to represent the influence degree of the first loss function on the model parameters of the video classification model during training, the second weight is used to represent the influence degree of the second loss function on the model parameters of the video classification model during training, and the second weight is less than the first weight. The server trains the attention encoding sub-model based on the second loss function and the second weight.

[0204] wherein, in the case where the plurality of sample video images include at least two different sample video images, the plurality of sample video images are obtained by the second embodiment in the above step S301. The second weight being less than the first weight means that the feature extraction sub-model is emphasized during training.

[0205] In this embodiment, the plurality of sample video images are the same sample video images, which means that the sample video is a synthetic video, and the video features and the image features are actually the same. Therefore, the feature extraction sub-model can be emphasized for training to improve the accuracy of the image features output by the feature extraction sub-model.

[0206] For example, in a case where the plurality of sample video images are sample video images of the same image content, the server constructs a first loss function based on the first difference information, the first loss function being a cross-entropy loss function. The server multiplies a loss value of the first loss function by a first weight to obtain a first target loss value. The server constructs a second loss function based on the second difference information, the second loss function being a cross-entropy loss function. The server multiplies a loss value of the second loss function by a second weight to obtain a second target loss value. The server trains the feature extraction sub-model using the gradient descent method based on the first target loss value and the second target loss value, that is, adjusts the model parameters of the feature extraction sub-model. The server trains the attention encoding sub-model using the gradient descent method based on the second target loss value, that is, adjusts the model parameters of the attention encoding sub-model. The form of the first loss function is shown in the above formula (1), and the form of the second loss function is shown in the above formula (2). In this case, the first weight w1=1 and the second weight w2=0 in the above formula (3).

[0207] In some embodiments, after obtaining the sample video, the server adds a video source label to the sample video, the video source label being used to indicate whether the sample video is a real video or a synthetic video.

[0208] It should be noted that the above steps S302-S306 are described by taking an example of one iteration training of the video classification model by the server, and the implementation process of the server in other iteration training processes belongs to the same inventive concept as the above steps S302-S306.

[0209] In some embodiments, the flag indicating that the server ends the training of the video classification model includes that the joint loss function converges to a target value or the number of times of training the video classification model reaches a number threshold, wherein the target value and the number threshold are set by a technician according to actual conditions, and the embodiments of the present application do not limit this.

[0210] In the embodiments of the present application, a video classification model is trained by using sample videos, and in the training process, a task of classifying sample videos based on image features of sample video images and a task of classifying sample videos based on video features of sample videos are constructed, so that the video classification model learns both the content at the video image level and the content at the video level through the two classification tasks, thereby improving the accuracy of the video classification model in video classification. Using the video classification model to classify videos can improve the efficiency of video classification while ensuring accuracy.

[0211] In addition to the steps S301-S306 described above, the embodiment of the present application also provides a video classification method. Referring to Figure 5 Taking the server as an example, the method comprises the following steps of:

[0212] S501, the server inputs a target video into the video classification model, and obtains image features of a plurality of target video images of the target video through a feature extraction sub-model in the video classification model.

[0213] The target video is a video to be classified, and the video classification model is a video classification model trained by the steps S301-S306 described above. The method of obtaining the image features of the target video images in the step S501 belongs to the same inventive concept as the method of obtaining the image features of the sample video images in the step S302, and the implementation process is described with reference to the step S302.

[0214] S502, the server encodes the image features of the plurality of target video images through an attention encoding sub-model in the video classification model, to obtain video features of the target video.

[0215] The method of obtaining the video features of the target video in the step S502 belongs to the same inventive concept as the method of obtaining the video features of the sample video in the step S303, and the implementation process is described with reference to the step S303.

[0216] S503, the server processes the video features of the target video through the video classification model, to obtain a video type of the target video.

[0217] The method of determining the video type of the target video in the step S503 belongs to the same inventive concept as the method of obtaining the second predicted video type of the sample video in the step S305, and the implementation process is described with reference to the step S305.

[0218] Through the steps S501-S503 described above, the server can quickly classify the target video through the video classification model, without manually watching the target video and then classifying it, thereby greatly improving the efficiency of video classification.

[0219] Figure 6 is a block diagram of a video classification model training device according to an example embodiment. Referring to Figure 6 The device comprises an image feature acquisition unit 601, an attention encoding unit 602, a classification unit 603, and a training unit 604.

[0220] The image feature acquisition unit 601 is configured to perform inputting a sample video into a video classification model, and acquiring image features of a plurality of sample video images of the sample video by a feature extraction sub-model in the video classification model.

[0221] The attention encoding unit 602 is configured to perform encoding the image features of the plurality of sample video images by an attention encoding sub-model in the video classification model, to obtain a video feature of the sample video.

[0222] The classification unit 603 is configured to perform processing the image features of the plurality of sample video images by the video classification model, to obtain a first predicted video type of the sample video, and processing the video feature of the sample video, to obtain a second predicted video type of the sample video.

[0223] The training unit 604 is configured to perform training the video classification model based on first difference information and second difference information, the first difference information being difference information between a labeled video type of the sample video and the first predicted video type, and the second difference information being difference information between the labeled video type and the second predicted video type.

[0224] In a possible implementation, the training unit 604 is configured to perform training the feature extraction sub-model in the video classification model based on the first difference information and the second difference information. The attention encoding sub-model in the video classification model is trained based on the second difference information.

[0225] In a possible implementation, the training unit 604 is configured to perform, when the plurality of sample video images include at least two sample video images with different image contents, training the feature extraction sub-model based on a first loss function corresponding to the first difference information and a first weight, and a second loss function corresponding to the second difference information and a second weight, the first weight being used to represent an influence degree of the first loss function on model parameters of the video classification model during training, the second weight being used to represent an influence degree of the second loss function on the model parameters of the video classification model during training, and the second weight being greater than the first weight. The attention encoding sub-model is trained based on the second loss function and the second weight.

[0226] In one possible implementation, the training unit 604 is configured to train the feature extraction sub-model based on a first loss function and first weight corresponding to the first difference information, and a second loss function and second weight corresponding to the second difference information, when the plurality of sample video images contain the same image content. The first weight represents the degree of influence of the first loss function on the model parameters of the video classification model during training, and the second weight represents the degree of influence of the second loss function on the model parameters of the video classification model during training. The second weight is less than the first weight. The attention encoding sub-model is then trained based on the second loss function and the second weight.

[0227] In one possible implementation, the attention encoding unit 602 is configured to encode the image features of every two adjacent sample video images in the plurality of sample video images through the attention encoding sub-model in the video classification model, thereby obtaining the attention features of the plurality of sample video images. The attention features of the plurality of sample video images are then fused to obtain the video features of the sample video.

[0228] In one possible implementation, the classification unit 603 is configured to perform pooling of image features from the plurality of sample video images to obtain image classification features for the sample video. The image classification features are then fully connected and normalized to obtain first classification parameters for the sample video. Based on these first classification parameters, a first predicted video type for the sample video is determined.

[0229] In one possible implementation, the classification unit 603 is configured to concatenate the image features of the plurality of sample video images to obtain sample concatenated features. Max pooling is then performed on each column of the sample concatenated features to obtain the image classification features of the sample video.

[0230] In one possible implementation, the base classification unit 603 is configured to perform fully connected and normalized processing on the video features of the sample video to obtain second classification parameters for the sample video. Based on these second classification parameters, a second predicted video type for the sample video is determined.

[0231] In one possible implementation, the device further includes:

[0232] The type determination unit is configured to input the target video into the video classification model, obtain image features of multiple target video images through the feature extraction sub-model in the video classification model, encode the image features of the multiple target video images through the attention encoding sub-model in the video classification model to obtain the video features of the target video, and process the video features of the target video through the video classification model to obtain the video type of the target video.

[0233] In one possible implementation, the device further includes:

[0234] The sample video acquisition unit is configured to extract frames from the initial video at target intervals to obtain the sample video.

[0235] In one possible implementation, the sample video acquisition unit is configured to extract frames from the initial video at the target interval to obtain N sample video images, where N is a positive integer. If N is less than M, K reference video images are acquired, where each reference video image is the first video image of the initial video, and both M and K are positive integers, with the sum of K and N equal to M. The N sample video images and the K reference video images are then concatenated to form the sample video.

[0236] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0237] In this embodiment, a video classification model is trained using sample videos. During training, two tasks are constructed: classifying sample videos based on image features and classifying them based on video features. These two classification tasks enable the video classification model to learn both image-level and video-level content, thereby improving its accuracy in video classification. Using a video classification model to classify videos improves efficiency while maintaining accuracy.

[0238] In this embodiment of the application, the electronic device can be implemented as a terminal, and the structure of the terminal is described below:

[0239] Figure 7 This is a block diagram illustrating a terminal 700 according to an exemplary embodiment. The terminal 700 can be a terminal used by a user. Typically, the terminal 700 includes a processor 701 and a memory 702.

[0240] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0241] The memory 702 may include one or more storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash storage devices.

[0242] In some embodiments, the terminal 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708, and a power supply 709.

[0243] Peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0244] The radio frequency (RF) circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0245] Display screen 705 is used to display a UI (User Interface). This UI may include images, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 705, which serves as the front panel of terminal 700; in other embodiments, there may be at least two display screens 705, respectively disposed on different surfaces of terminal 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of terminal 700. Furthermore, display screen 705 may be configured as a non-rectangular, irregular image, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0246] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0247] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0248] The positioning component 708 is used to determine the current geographic location of the terminal 700 in order to enable navigation or LBS (Location Based Service). The positioning component 708 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the EU's Galileo system.

[0249] The power supply 709 is used to power the various components in the terminal 700. The power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0250] In some embodiments, the terminal 700 further includes one or more sensors 170. The one or more sensors 170 include, but are not limited to: an accelerometer 711, a gyroscope 712, a pressure sensor 713, a fingerprint sensor 714, an optical sensor 715, and a proximity sensor 716.

[0251] Accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 700. For example, accelerometer 711 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 701 can control display screen 705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 711. Accelerometer 711 can also be used for games or for acquiring user motion data.

[0252] The gyroscope sensor 712 can detect the orientation and rotation angle of the terminal 700. The gyroscope sensor 712, in conjunction with the accelerometer sensor 711, can collect 3D motion data from the user on the terminal 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0253] The pressure sensor 713 can be disposed on the side bezel of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 713 is disposed on the side bezel of the terminal 700, it can detect the user's grip signal on the terminal 700, and the processor 701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is disposed on the lower layer of the display screen 705, the processor 701 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0254] The fingerprint sensor 714 is used to collect a user's fingerprint. The processor 701 identifies the user based on the fingerprint collected by the fingerprint sensor 714, or vice versa. When the user's identity is identified as trusted, the processor 701 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 714 can be located on the front, back, or side of the terminal 700. When the terminal 700 has a physical button or manufacturer logo, the fingerprint sensor 714 can be integrated with the physical button or manufacturer logo.

[0255] An optical sensor 715 is used to collect ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 based on the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity collected by the optical sensor 715.

[0256] The proximity sensor 716, also known as a distance sensor, is typically located on the front panel of the terminal 700. The proximity sensor 716 is used to detect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from a screen-on state to a screen-off state; when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from a screen-off state to a screen-on state.

[0257] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on terminal 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0258] The aforementioned electronic devices can also be implemented as servers. The structure of a server is described below:

[0259] Figure 8 This is a block diagram of a server provided in an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. The one or more memories 802 store at least one computer program, which is loaded and executed by the one or more processors 801 to implement the methods provided in the various method embodiments described above. Of course, the server 800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 800 may also include other components for implementing device functions, which will not be elaborated upon here.

[0260] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 701 of the terminal 700 or the processor 801 of the server 800 to complete the training method of the video classification model described above. Optionally, the storage medium can be a non-transitory storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0261] In an exemplary embodiment, a computer program product is also provided, including a computer program that can be executed by a processor of an electronic device to implement the training method of the video classification model described above.

[0262] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0263] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a video classification model, characterized in that, include: The sample video is input into the video classification model, and the image features of multiple sample video images of the sample video are obtained through the feature extraction sub-model in the video classification model. The image features of the multiple sample video images are encoded using the attention encoding sub-model in the video classification model to obtain the video features of the sample video. The image features of the multiple sample video images are processed using the video classification model to obtain the first predicted video type of the sample video; The video features of the sample video are processed to obtain the second predicted video type of the sample video; The feature extraction sub-model in the video classification model is trained based on the first difference information and the second difference information. Based on the second difference information, the attention encoding sub-model in the video classification model is trained; the first difference information is the difference information between the labeled video type and the first predicted video type of the sample video, and the second difference information is the difference information between the labeled video type and the second predicted video type.

2. The training method for the video classification model according to claim 1, characterized in that, The step of training the feature extraction sub-model in the video classification model based on the first difference information and the second difference information includes: When the plurality of sample video images include at least two sample video images with different image content, the feature extraction sub-model is trained based on the first loss function and first weight corresponding to the first difference information, and the second loss function and second weight corresponding to the second difference information. The first weight is used to represent the degree of influence of the first loss function on the model parameters of the video classification model during training, and the second weight is used to represent the degree of influence of the second loss function on the model parameters of the video classification model during training. The second weight is greater than the first weight. The step of training the attention encoding sub-model in the video classification model based on the second difference information includes: The attention encoding sub-model is trained based on the second loss function and the second weight.

3. The training method for the video classification model according to claim 1, characterized in that, The step of training the feature extraction sub-model in the video classification model based on the first difference information and the second difference information includes: When the multiple sample video images are sample video images with the same image content, the feature extraction sub-model is trained based on the first loss function and first weight corresponding to the first difference information, and the second loss function and second weight corresponding to the second difference information. The first weight is used to represent the degree of influence of the first loss function on the model parameters of the video classification model during training, and the second weight is used to represent the degree of influence of the second loss function on the model parameters of the video classification model during training. The second weight is less than the first weight. The step of training the attention encoding sub-model in the video classification model based on the second difference information includes: The attention encoding sub-model is trained based on the second loss function and the second weight.

4. The training method for the video classification model according to claim 1, characterized in that, The step of encoding the image features of the multiple sample video images using the attention encoding sub-model in the video classification model to obtain the video features of the sample videos includes: The attention encoding sub-model in the video classification model encodes the image features of every two adjacent sample video images in the plurality of sample video images to obtain the attention features of the plurality of sample video images; the attention features of the plurality of sample video images are then fused to obtain the video features of the sample video.

5. The training method for the video classification model according to claim 1, characterized in that, The process of processing the image features of the plurality of sample video images to obtain the first predicted video type of the sample video includes: Pooling is performed on the image features of the multiple sample video images to obtain the image classification features of the sample videos; The image classification features are fully connected and normalized to obtain the first classification parameters of the sample video; Based on the first classification parameters, the first predicted video type of the sample video is determined.

6. The training method for the video classification model according to claim 5, characterized in that, The process of pooling the image features of the multiple sample video images to obtain the image classification features of the sample videos includes: The image features of the multiple sample video images are stitched together to obtain sample stitched features; Max pooling is performed on each column of the sample splicing features to obtain the image classification features of the sample video.

7. The training method for the video classification model according to claim 1, characterized in that, The process of processing the video features of the sample video to obtain the second predicted video type of the sample video includes: The video features of the sample video are fully connected and normalized to obtain the second classification parameters of the sample video; Based on the second classification parameter, the second predicted video type of the sample video is determined.

8. The training method for the video classification model according to claim 1, characterized in that, The method further includes: The target video is input into the video classification model, and the image features of multiple target video images of the target video are obtained through the feature extraction sub-model in the video classification model. The image features of the multiple target video images are encoded using the attention encoding sub-model in the video classification model to obtain the video features of the target video; The video classification model is used to process the video features of the target video to obtain the video type of the target video.

9. A training device for a video classification model, characterized in that, include: The image feature acquisition unit is configured to input the sample video into the video classification model and, through the feature extraction sub-model in the video classification model, acquire the image features of multiple sample video images of the sample video. The attention encoding unit is configured to encode the image features of the plurality of sample video images through the attention encoding sub-model in the video classification model to obtain the video features of the sample video; The classification unit is configured to perform image feature processing on the plurality of sample video images through the video classification model to obtain a first predicted video type of the sample video; The video features of the sample video are processed to obtain the second predicted video type of the sample video; The training unit is configured to train the feature extraction sub-model in the video classification model based on the first difference information and the second difference information. Based on the second difference information, the attention encoding sub-model in the video classification model is trained; the first difference information is the difference information between the labeled video type and the first predicted video type of the sample video, and the second difference information is the difference information between the labeled video type and the second predicted video type.

10. The training apparatus for the video classification model according to claim 9, characterized in that, The training unit is configured to train the feature extraction sub-model based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, when the plurality of sample video images include at least two sample video images with different image content. The first weight is used to represent the degree of influence of the first loss function on the model parameters of the video classification model during training, and the second weight is used to represent the degree of influence of the second loss function on the model parameters of the video classification model during training. The second weight is greater than the first weight. The attention encoding sub-model is trained based on the second loss function and the second weight.

11. The training apparatus for the video classification model according to claim 9, characterized in that, The training unit is configured to train the feature extraction sub-model based on a first loss function and a first weight corresponding to the first difference information, and a second loss function and a second weight corresponding to the second difference information, when the plurality of sample video images are sample video images with the same image content. The first weight is used to represent the degree of influence of the first loss function on the model parameters of the video classification model during training, and the second weight is used to represent the degree of influence of the second loss function on the model parameters of the video classification model during training. The second weight is less than the first weight. The attention encoding sub-model is trained based on the second loss function and the second weight.

12. The training apparatus for the video classification model according to claim 9, characterized in that, The attention encoding unit is configured to encode the image features of every two adjacent sample video images in the plurality of sample video images through the attention encoding sub-model in the video classification model, so as to obtain the attention features of the plurality of sample video images; The attention features of the multiple sample video images are fused to obtain the video features of the sample video.

13. The training apparatus for the video classification model according to claim 9, characterized in that, The classification unit is configured to perform pooling on the image features of the plurality of sample video images to obtain the image classification features of the sample videos; The image classification features are fully connected and normalized to obtain the first classification parameters of the sample video; based on the first classification parameters, the first predicted video type of the sample video is determined.

14. The training apparatus for the video classification model according to claim 13, characterized in that, The classification unit is configured to perform image feature concatenation on the plurality of sample video images to obtain sample concatenation features; and to perform max pooling on each column of the sample concatenation features to obtain the image classification features of the sample video.

15. The training apparatus for the video classification model according to claim 9, characterized in that, The classification unit is configured to perform fully connected and normalized processing on the video features of the sample video to obtain a second classification parameter for the sample video; and to determine a second predicted video type for the sample video based on the second classification parameter.

16. The training apparatus for the video classification model according to claim 9, characterized in that, The device further includes: The type determination unit is configured to input the target video into the video classification model, obtain image features of multiple target video images of the target video through the feature extraction sub-model in the video classification model, encode the image features of the multiple target video images through the attention encoding sub-model in the video classification model to obtain the video features of the target video, and process the video features of the target video through the video classification model to obtain the video type of the target video.

17. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the video classification model as described in any one of claims 1 to 8.

18. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a training method for a video classification model as described in any one of claims 1 to 8.

Images