Model training method, video classification method, device, equipment and storage medium

Abstract

The present application provides a model training method, a video classification method, an apparatus, a device, and a storage medium. The method includes obtaining a first video, a classification label, and title information of the first video; performing frame sampling on the first video to obtain at least one video frame of the first video; performing text recognition on the at least one video frame of the first video to obtain text information of each of the at least one video frame; processing the text information of each of the at least one video frame to obtain a first word vector of the first video, processing the title information of the first video to obtain a second word vector of the first video; and training a video classification model based on the first word vector, the second word vector, and the at least one video frame of the first video. Because text information can better reflect video content, the video classification model trained with the text information of the video frame, the video frame, and the title information of the video is more optimized, thereby improving video classification accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a model training method, a video classification method, an apparatus, a device, and a storage medium. Background Art

[0002] Video classification is widely used on major video websites, TV stations, and short video sharing platforms, and is mainly used for video archiving and content recommendation.

[0003] Currently, video classification is performed based on the video frames and video title information. Specifically, Figure 1 A schematic diagram of existing video classification methods, such as Figure 1 As shown, the electronic device can obtain a video and the title information of the video, perform frame sampling on the video to extract several video frames, input the several video frames into the image neural network to obtain the image features of each video frame; and perform word vector conversion on the title information of the video, input the converted word vector into the text neural network to obtain the text features of the video; finally, input the image features of each video frame and the text features of the video into the classifier to obtain the classification label of the video.

[0004] However, the existing video classification method only considers the video frame and video title information, resulting in low video classification accuracy. Based on this, how to train a video classification model to improve video classification accuracy is a technical problem that needs to be urgently solved in this application. Summary of the Invention

[0005] The present application provides a model training method, a video classification method, an apparatus, a device and a storage medium. The present application introduces text information of video frames, and the text information can better reflect the video content, so that the video classification model trained with the text information of video frames, video frames and video title information can improve the video classification accuracy.

[0006] In a first aspect, a model training method is provided, including: obtaining a first video, a classification label of the first video, and title information of the first video; performing frame sampling on the first video to obtain at least one video frame of the first video; performing text recognition on at least one video frame of the first video to obtain text information of at least one video frame; processing the text information of at least one video frame to obtain a first word vector of the first video, and processing the title information of the first video to obtain a second word vector of the first video; training a video classification model based on the first word vector, the second word vector, and at least one video frame of the first video.

[0007] In a second aspect, a video classification method is provided, including: obtaining a second video and title information of the second video; performing frame sampling on the second video to obtain at least one video frame of the second video; performing text recognition on at least one video frame of the second video to obtain text information of at least one video frame; processing the text information of at least one video frame to obtain a third word vector of the second video, and processing the title information of the second video to obtain a fourth word vector of the second video; inputting the third word vector, the fourth word vector and at least one video frame of the second video into the video classification model trained by the model training method provided by the first aspect to obtain a classification label for the second video.

[0008] In a third aspect, a model training device is provided, comprising: an acquisition module, a sampling module, a text recognition module, a processing module and a training module, wherein the acquisition module is used to acquire a first video, a classification label of the first video and the title information of the first video; the sampling module is used to perform frame sampling on the first video to obtain at least one video frame of the first video; the text recognition module is used to perform text recognition on at least one video frame of the first video to obtain text information of at least one video frame; the processing module is used to process the text information of at least one video frame to obtain a first word vector of the first video, and process the title information of the first video to obtain a second word vector of the first video; the training module is used to train a video classification model based on the first word vector, the second word vector and at least one video frame of the first video.

[0009] In a fourth aspect, a video classification device is provided, comprising: an acquisition module, a sampling module, a text recognition module, a processing module and an input module, wherein the acquisition module is used to acquire a second video and the title information of the second video; the sampling module is used to perform frame sampling on the second video to obtain at least one video frame of the second video; the text recognition module is used to perform text recognition on at least one video frame of the second video to obtain text information of at least one video frame; the processing module is used to process the text information of at least one video frame to obtain a third word vector of the second video, and to process the title information of the second video to obtain a fourth word vector of the second video; the input module is used to input the third word vector, the fourth word vector and at least one video frame of the second video into the video classification model trained by the model training method provided by the first aspect to obtain a classification label for the second video.

[0010] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method as in the first aspect, the second aspect or their respective implementations.

[0011] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method as in the first aspect, the second aspect or any implementation thereof.

[0012] In a seventh aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method in the first aspect, the second aspect or any implementation thereof.

[0013] In an eighth aspect, a computer program is provided, which enables a computer to execute the method in the first aspect, the second aspect, or each implementation thereof.

[0014] Through the technical solution provided by this application, electronic devices can combine the text information of video frames, video frames and video title information to train video classification models. Compared with the model training method in the prior art that only combines video frames and video title information, this application introduces the text information of video frames, and this text information can better reflect the video content, so that the video classification model trained with the text information of video frames, video frames and video title information can improve the video classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 Schematic diagram of existing video classification methods;

[0017] Figure 2A A video recommendation scene graph provided in an embodiment of the present application;

[0018] Figure 2B A video retrieval scene graph provided in an embodiment of the present application;

[0019] Figure 2C A video archiving scene graph provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0021] Figure 4 A flowchart of a text recognition method provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a text recognition process provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a video classification model provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of image processing using ResNet provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the text information processing process provided in an embodiment of the present application;

[0026] Figure 9 A schematic diagram of the process of processing title information provided in an embodiment of the present application;

[0027] Figure 10 A flowchart of a video classification method provided in an embodiment of the present application;

[0028] Figure 11 A schematic diagram of a model training device 1100 provided in an embodiment of the present application;

[0029] Figure 12 A schematic diagram of a video classification device 1200 provided in an embodiment of the present application;

[0030] Figure 13 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Before introducing the technical solution of this application, the following will explain the relevant knowledge of the technical solution of this application:

[0034] The residual neural network (ResNet) is mainly used to solve the problems of gradient diffusion, gradient explosion, and network degradation (that is, the deeper the network layers, the worse the performance on the data set) caused by too deep networks.

[0035] The Bidirectional Encoder Representations from Transformers (BERT) model is an autoencoding language model, and two tasks are designed to pre-train the model. The first task is to train the BERT model using a masked language model. In layman's terms, when inputting a sentence, some words to be predicted are randomly selected and replaced with a special mask symbol. The BERT model then learns the words to fill in these places based on the given labels. The second task adds a sentence-level continuity prediction task based on the bidirectional language model. This task predicts whether the two text segments input to BERT are continuous. Introducing this task allows the model to better learn the relationship between continuous text segments.

[0036] Transformer: The Transformer uses an encoder-decoder structure. The encoder can be composed of N = 6 identical units, and the decoder is almost the same as the encoder.

[0037] Average Pooling: There are four common pooling operations: mean-pooling (average pooling), max-pooling, stochastic-pooling, and global average pooling. The pooling layer has a significant purpose: reducing the size of feature maps, which reduces computation and memory requirements. Average pooling simply averages the feature points within a neighborhood.

[0038] Fully connected layers (FCs) act as classifiers within a convolutional neural network. While operations like convolutional layers, pooling layers, and activation functions map raw data into the hidden feature space, fully connected layers map the learned distributed feature representations into the sample label space. In practice, fully connected layers are implemented using convolution operations.

[0039] Layer Normalization (LN) is a neural network layer.

[0040] A key feature of a multilayer perceptron (MLP) is its multi-layer structure. The first layer is called the input layer, the last layer is called the output layer, and the layers in between are called hidden layers. MLPs do not specify the number of hidden layers, so you can choose the appropriate number based on your needs. There is also no limit on the number of neurons in the output layer.

[0041] Multi-head Self-Attention (MSA) does not calculate attention only once, but calculates attention on multiple subspaces in parallel multiple times. Finally, the attention on multiple subspaces is simply connected and linearly converted to the expected dimension.

[0042] Adaptive Moment Estimation (Adam): is a gradient descent algorithm used to train neural networks.

[0043] Cross-entropy: A loss function used to supervise the convergence of neural networks.

[0044] The following is an explanation of the technical problems and inventive concepts to be solved by the technical solution of this application:

[0045] As mentioned above, the existing video classification method only considers the video frame and video title information, resulting in low video classification accuracy. Based on this, how to train a video classification model to improve video classification accuracy is a technical problem that needs to be solved urgently in this application.

[0046] In order to solve the above technical problems, the present application can combine the text information of the video frame, the title information of the video frame and the video to train the video classification model to improve the video classification accuracy.

[0047] The technical solution of this application can be applied to video recommendation scenarios, video retrieval scenarios, video archiving scenarios, etc., and this application does not impose any restrictions on this.

[0048] For example, Figure 2A A video recommendation scene diagram provided in an embodiment of the present application, such as Figure 2AAs shown, users can upload videos to a video platform through video clients, mini-programs, etc. R&D personnel can retrieve these videos from the video platform's corresponding database and classify and annotate them. Furthermore, these videos and their classification annotations can be sent to a backend server, which can use these videos and their classification annotations to train a video classification model. Furthermore, the backend server can retrieve videos uploaded by a user and classify them based on the model to obtain classification annotations for the videos. Based on these classification annotations, the server can then recommend videos to users who are interested in these videos.

[0049] For example, Figure 2B A video retrieval scene graph provided in an embodiment of the present application, such as Figure 2B As shown, each user can upload videos to a video platform through a video client, mini-program, etc. R&D personnel can obtain these videos from the corresponding database of the video platform and classify and annotate them. Furthermore, these videos and their classification annotations can be sent to the backend server, which can train a video classification model based on each video and its classification annotations. Furthermore, the backend server can obtain videos uploaded by each user and classify them based on the model to obtain the classification annotations of the video. The backend server can also obtain search requests from any user. The search request may include: the classification annotations to be retrieved, such as food, etc. Based on the classification annotations to be retrieved, the backend server searches for such videos in the video platform and displays the search results to the user.

[0050] For example, Figure 2C A video archiving scene graph provided in an embodiment of the present application, such as Figure 2C As shown, each user can upload videos to a video platform through a video client, mini-program, etc. R&D personnel can obtain these videos from the corresponding database of the video platform and classify and annotate them. Furthermore, these videos and their classification annotations can be sent to the backend server, which can use these videos and their classification annotations to train a video classification model. Furthermore, the backend server can obtain the videos uploaded by each user and classify and archive them based on the model, for example: grouping food videos into one category, car videos into another category, movie and TV videos into another category, and news videos into another category.

[0051] Optionally, the above-mentioned background server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, which is not limited in this application.

[0052] It should be understood that Figures 2A to 2CThe model training process and the application process based on video classification, such as video recommendation, video retrieval, and video archiving, shown in the figure are all implemented by the background server. In fact, the model training process and the application process based on video classification can be implemented by different devices.

[0053] The technical solution of this application can be applied to the field of AI technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, giving them the capabilities of perception, reasoning, and decision-making.

[0054] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0055] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0056] The technical solution of this application will be described in detail below:

[0057] Figure 3 This is a flowchart of a model training method provided in an embodiment of the present application. The method can be executed by an electronic device such as a desktop computer, a laptop computer, a handheld computer, etc. For example, the electronic device can be Figures 2A to 2C In addition, the electronic device may also be a processor, etc., which is not limited in this application. Figure 3 As shown, the method includes:

[0058] S310: Obtaining a first video, a classification label of the first video, and title information of the first video;

[0059] S320: Perform frame sampling on the first video to obtain at least one video frame of the first video;

[0060] S330: Perform text recognition on at least one video frame of the first video to obtain text information of the at least one video frame;

[0061] S340: Processing text information of at least one video frame to obtain a first word vector of the first video, and processing title information of the first video to obtain a second word vector of the first video;

[0062] S350: Training a video classification model according to the first word vector, the second word vector, and at least one video frame of the first video.

[0063] Optionally, the number of the above-mentioned first videos can be multiple. In other words, when training the video classification model, the video classification model can be trained through multiple videos and the text information, title information and classification labels of these videos.

[0064] Optionally, the first video can be any video used to train a video classification model. The first video can be any video uploaded to a video platform by any user through a client or mini-program, etc., or any video received by an electronic device from other devices, or any video automatically generated by an electronic device. In short, this application does not impose any restrictions on the source, form, or content of the first video.

[0065] Optionally, R&D personnel or other personnel may select the first video from a database of the video platform and manually annotate the first video to form a classification annotation for the first video.

[0066] Optionally, the database of the video platform may exist in the above-mentioned electronic device, or in other devices, or in a cloud server, and this application does not impose any restrictions on this.

[0067] Optionally, R&D personnel or other personnel may analyze the content of the first video and classify and label the first video according to some predefined classification labels, such as food, news, etc.

[0068] It should be understood that the classification label of the first video can also be referred to as the classification result of the first video, and the classification result can be a category such as food, cars, movies and television, and news.

[0069] It should be understood that the title information of the first video is also referred to as the title of the first video.

[0070] Optionally, when acquiring the first video, the electronic device may extract the title information of the first video to obtain the title information of the first video.

[0071] Optionally, when performing frame sampling on the first video, the electronic device may perform frame sampling using a sampling frequency corresponding to the video length of the first video.

[0072] Optionally, there may be a correspondence between the video length interval and the sampling frequency. For example, for a video of moderate length, frame sampling may generally be performed at a sampling frequency of 1 frame per second. For a video of shorter length, frame sampling may be performed at a sampling frequency of more frames per second. Conversely, for a video of longer length, frame sampling may be performed at a sampling frequency of one frame every few seconds.

[0073] It should be understood that this application does not impose any restrictions on the specific correspondence between the video length interval range and the sampling frequency, the division method of the video length interval, etc.

[0074] Optionally, the correspondence between the video length interval range and the sampling frequency may be predefined, or indicated by the electronic device, or indicated by other devices, and this application does not impose any restrictions on this.

[0075] Optionally, the electronic device usually only needs to acquire 16 video frames of the first video according to the sampling frequency.

[0076] Optionally, the electronic device may use a text recognition algorithm to perform text recognition on at least one video frame of the first video to obtain text information of each video frame.

[0077] Optionally, Figure 4 A flowchart of a text recognition method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes:

[0078] S410: Perform text detection on at least one video frame to obtain a text position of each of the at least one video frame;

[0079] S420: intercepting a local image at a text position of each of the at least one video frame to obtain a intercepted image of each of the at least one video frame;

[0080] S430: Perform text recognition on the captured image of at least one video frame to obtain text information of at least one video frame.

[0081] Optionally, the electronic device may use a text detection model to perform text detection on each video frame in at least one video frame. The text detection model may be a faster-based region convolutional neural network (Faster Region-Convolutional Neural Networks, Faster R-CNN), a region convolutional neural network (Region-Convolutional Neural Networks, R-CNN), etc. This application does not impose any restrictions on this.

[0082] Optionally, the electronic device may determine an area with the text position as the center and the target radius as the radius as the above-mentioned partial image.

[0083] Optionally, the target radius may be predefined, or indicated by an electronic device, or indicated by other devices, and this application does not impose any limitation on this.

[0084] Optionally, the electronic device may use a text recognition model to recognize text in each captured image. The text recognition model may be a convolutional recurrent neural network (CRNN) structure, etc., and this application does not impose any restrictions on this.

[0085] For example, Figure 5 A schematic diagram of a text recognition process provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device can perform text detection on the current video frame to obtain the text position of at least one video frame; intercept the local image at the text position of the current video frame to obtain a intercepted image including "I'm afraid my brothers will quit." Further, the electronic device uses a text recognition model to perform text recognition on the intercepted image to obtain the final text information, namely, "I'm afraid my brothers will quit."

[0086] Optionally, if a video frame contains multiple lines of text, the electronic device can splice the recognized text together in order from top to bottom and from left to right according to the position of the text as the text information of the picture frame.

[0087] Optionally, since some video frames may carry text, if there are multiple video frames carrying text in at least one of the above-mentioned video frames, then after obtaining the text information of each of these video frames, the electronic device can splice the text information of at least one video frame to obtain spliced text information, and finally convert the spliced text information into the first word vector of the first video.

[0088] For example, assuming that there are multiple video frames carrying text in the at least one video frame mentioned above, namely: a video frame including "I'm afraid my brothers won't do this anymore" and a video frame including "Don't worry". After the electronic device performs text detection and recognition on the two image frames, the text information obtained from the two video frames is: "I'm afraid my brothers won't do this anymore" and "Don't worry". Based on this, the electronic device can splice the two text information together with commas, and the final spliced text information is "I'm afraid my brothers won't do this anymore, don't worry".

[0089] Optionally, the electronic device may use a Word2Vec method to convert the concatenated text information into a first word vector, but is not limited thereto.

[0090] Optionally, the electronic device may also use a Word2Vec method to convert the title information of the first video into a second word vector, but is not limited thereto.

[0091] Optionally, Figure 6 A schematic diagram of a video classification model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the video classification model includes: an image neural network, an average pooling layer, a first text neural network, a first fully connected layer, a second text neural network, a second fully connected layer, a transformer and a classifier, wherein the image neural network is used to process at least one video frame respectively to obtain the first image feature of at least one video frame; the average pooling layer is used to process the first image feature of at least one video frame respectively to obtain the second image feature of at least one video frame respectively; the first text neural network is used to process the first word vector of the first video to obtain the first text feature of the first video; the first fully connected layer is used to process the first text feature of the first video to obtain the second text feature of the first video; the second text neural network is used to process the second word vector of the first video to obtain the third text feature of the first video; the second fully connected layer is used to process the third text feature of the first video to obtain the fourth text feature of the first video; the transformer is used to perform feature fusion on the second image feature of at least one video frame, the second text feature and the fourth text feature of the first video to obtain the fused feature of the first video; the classifier is used to classify and label the first video according to the fused feature of the first video.

[0092] Optionally, before training a video classification model based on the first word vector, the second word vector and at least one video frame of the first video, the electronic device may process at least one video frame to adapt the at least one video frame to the image neural network. This processing process may be referred to as a pre-processing process for the video frame.

[0093] It should be understood that the image neural network has specification requirements for video frames. For example, if the image neural network requires the video frame size to be 1028*1028, then if the size of a video frame is not 1028*1028, the electronic device can pre-process the video frame to make the size of the video frame 1028*1028. However, this application does not limit how the electronic device pre-processes the video frame.

[0094] Optionally, the image neural network can be a classic convolutional neural network such as Resnet, which is not limited in this application.

[0095] For example, Figure 7 A schematic diagram of image processing by Resnet provided in an embodiment of the present application is shown in FIG. Figure 7 As shown in the figure, a video frame usually goes through 5 convolution stages, and each stage is used to realize feature extraction of the video frame. Among them, the size of the video frame is reduced through each stage, and the number of channels is increased. For example, a video frame format is 224*224*3, 224*224 represents the height*width of the video frame, that is, the size of the video frame, and 3 represents the number of channels. After the first stage processing, the video frame format is updated to 112*112*64; after the second stage processing, the video frame format is updated to 56*56*64; after the third stage processing, the video frame format is updated to 28*28*128; after the fourth stage processing, the video frame format is updated to 14*14*256; after the fifth stage processing, the video frame format is updated to 7*7*512.

[0096] It should be understood that the purpose of the average pooling layer is to reduce the dimension of the first image feature obtained by the image neural network to 1*1*c, where c is the feature length of the input subsequent transformer.

[0097] Figure 8 A schematic diagram of the text information processing process provided in the embodiment of the present application is shown as follows: Figure 8 As shown, the electronic device can process the text information of at least one video frame of a video to obtain a first word vector of the video. Furthermore, the electronic device can input the first word vector into a first text neural network to obtain a first text feature of the video, and then input the first text feature into a first fully connected layer to obtain a second text feature of the video.

[0098] Optionally, the first text neural network may be a Bert model, which is not limited in this application.

[0099] Optionally, the first fully connected layer can map the first text feature to the same dimension as the second image feature to obtain a second text feature, that is, the dimension of the second text feature is c, and c is the feature length input to the subsequent transformer.

[0100] Figure 9 A schematic diagram of the process of processing title information provided in the embodiment of the present application is shown as follows: Figure 9 As shown, the electronic device can process the title information of a video to obtain a second word vector of the video. Furthermore, the electronic device can input the second word vector into the second text neural network to obtain a third text feature of the video, and then input the third text feature into the second fully connected layer to obtain a fourth text feature of the video.

[0101] Optionally, the second text neural network may be a Bert model, which is not limited in this application.

[0102] Optionally, the second fully connected layer can map the third text feature to the same dimension as the second image feature to obtain a fourth text feature, that is, the dimension of the fourth text feature is c, and c is the feature length input to the subsequent converter.

[0103] Optionally, the transformer mainly functions to perform feature fusion on the second image feature of each of the at least one video frame, the second text feature of the first video, and the fourth text feature.

[0104] Exemplarily, the transformer may perform feature concatenation on the second image feature of each of the at least one video frame, the second text feature of the first video, and the feature of the fourth text feature video frame to obtain a (N+2)*c-dimensional feature, and the feature may be concatenated again with the c-dimensional class token to obtain a (N+3)*c-dimensional feature, and the concatenated feature is added to the position code of the same dimension to obtain the input x of the transformer encoder. pos , where x pos ∈R (N+3)*c The transformer is composed of L layers of encoders connected sequentially, and each layer of encoder consists of three modules: LN, MSA and MLP. That is, the L-layer encoder calculates the fusion feature z of the first video through formulas (1) to (4).

[0105] y0=x pos (1)

[0106]

[0107]

[0108]

[0109] Optionally, the electronic device may input the first c features of the fused features into a classifier, and finally obtain a classification label for the first video.

[0110] Optionally, the classifier can perform category labeling with a number of categories being d. The classifier can be a fully connected layer.

[0111] Optionally, when training the video classification model, the electronic device may adopt any form of loss function, such as a cross-entropy loss function.

[0112] Optionally, when training the video classification model, the electronic device may use any gradient descent algorithm for the loss function, for example, the Adam algorithm may be used.

[0113] In summary, the present application provides a model training method, in which an electronic device can combine the text information of video frames, video frames and video title information to train a video classification model. Compared with the model training method in the prior art that only combines the video frames and video title information, the present application introduces the text information of video frames, and the text information can better reflect the video content, thereby making the video classification model trained with the text information of video frames, video frames and video title information better, thereby improving the video classification accuracy.

[0114] Furthermore, the training method for the video classification model of the present application is an end-to-end training method. In other words, the video classification model is treated as a whole. During the model training process, the gradient calculation results can be back-propagated to any unit in the model, such as the image neural network, text neural network and classifier, thereby reducing the loss of the gradient calculation results during the model training process, and further improving the video classification accuracy.

[0115] Furthermore, the video classification model provided in this application performs feature fusion through a transformer rather than simply feature splicing. This fusion method can improve the classification accuracy of the video classification model, thereby further improving the video classification accuracy.

[0116] Figure 10 This is a flowchart of a video classification method provided in an embodiment of the present application. The method can be executed by an electronic device such as a desktop computer, a laptop computer, a handheld computer, etc. For example, the electronic device can be Figures 2A to 2C In addition, the electronic device may also be a processor, etc., which is not limited in this application. Figure 10 As shown, the method includes:

[0117] S1010: Acquire a second video and title information of the second video;

[0118] S1020: Perform frame sampling on the second video to obtain at least one video frame of the second video;

[0119] S1030: Perform text recognition on at least one video frame of the second video to obtain text information of the at least one video frame;

[0120] S1040: Processing text information of at least one video frame to obtain a third word vector of the second video, and processing title information of the second video to obtain a fourth word vector of the second video;

[0121] S1050: Input the third word vector, the fourth word vector, and at least one video frame of the second video into a video classification model to obtain a classification label for the second video.

[0122] Optionally, the second video can be any video to be classified. The second video can be any video uploaded to the video platform by any user through a client or mini-program, etc., or any video received by the electronic device from other devices, or any video automatically generated by the electronic device. In short, this application does not limit the source, form and content of the second video.

[0123] It should be understood that the title information of the second video is also referred to as the title of the second video.

[0124] Optionally, when acquiring the second video, the electronic device may extract the title information of the second video to obtain the title information of the second video.

[0125] Optionally, when performing frame sampling on the second video, the electronic device may perform frame sampling using a sampling frequency corresponding to the video length of the second video.

[0126] Optionally, there may be a correspondence between the video length interval and the sampling frequency. For example, for a video of moderate length, frame sampling may generally be performed at a sampling frequency of 1 frame per second. For a video of shorter length, frame sampling may be performed at a sampling frequency of more frames per second. Conversely, for a video of longer length, frame sampling may be performed at a sampling frequency of one frame every few seconds.

[0127] It should be understood that this application does not impose any restrictions on the specific correspondence between the video length interval range and the sampling frequency, the division method of the video length interval, etc.

[0128] Optionally, the correspondence between the video length interval range and the sampling frequency may be predefined, or indicated by the electronic device, or indicated by other devices, and this application does not impose any restrictions on this.

[0129] Optionally, the electronic device usually only needs to acquire 16 video frames of the second video according to the sampling frequency.

[0130] Optionally, the electronic device may use a text recognition algorithm to perform text recognition on at least one video frame of the second video to obtain text information of each video frame.

[0131] It should be understood that the text recognition process of the electronic device for the second video can refer to the above-mentioned text recognition process for the first video, and this application will not go into details.

[0132] Optionally, since some video frames may carry text, if there are multiple video frames carrying text in at least one video frame of the second video, then after obtaining the text information of each of these video frames, the electronic device can splice the text information of at least one video frame to obtain spliced text information, and finally convert the spliced text information into the third word vector of the second video.

[0133] It should be understood that the splicing method for splicing the text information of at least one video frame of the second video can refer to the above-mentioned splicing method for splicing the text information of at least one video frame of the first video, and this application will not go into details.

[0134] Optionally, the electronic device may use a Word2Vec method to convert the concatenated text information into a third word vector, but is not limited thereto.

[0135] Optionally, the electronic device may also use the Word2Vec method to convert the title information of the second video into a fourth word vector, but is not limited thereto.

[0136] Optionally, the video classification model includes: an image neural network, an average pooling layer, a first text neural network, a first fully connected layer, a second text neural network, a second fully connected layer, a transformer and a classifier, wherein the image neural network is used to process at least one video frame of the second video separately to obtain the third image feature of at least one video frame; the average pooling layer is used to process the third image feature of at least one video frame to obtain the fourth image feature of at least one video frame; the first text neural network is used to process the third word vector of the second video to obtain the fifth text feature of the second video; the first fully connected layer is used to process the fifth text feature of the second video to obtain the sixth text feature of the second video; the second text neural network is used to process the fourth word vector of the second video to obtain the seventh text feature of the second video; the second fully connected layer is used to process the seventh text feature of the second video to obtain the eighth text feature of the second video; the transformer is used to perform feature fusion on the fourth image feature of at least one video frame, the sixth text feature and the eighth text feature of the second video to obtain the fused feature of the second video; the classifier is used to classify and label the second video according to the fused feature of the second video.

[0137] Optionally, before inputting at least one video frame of the second video into the image neural network, the electronic device may process at least one video frame of the second video to make the at least one video frame adapt to the image neural network. This processing process may be referred to as a pre-processing process of the video frame.

[0138] It should be understood that the image neural network has specification requirements for video frames. For example, if the image neural network requires the video frame size to be 1028*1028, then if the size of a video frame is not 1028*1028, the electronic device can pre-process the video frame to make the size of the video frame 1028*1028. However, this application does not limit how the electronic device pre-processes the video frame.

[0139] It should be understood that the functions of each unit in the video classification model, such as the image neural network, text neural network, transformer, and classifier, as well as the processing process of the second video can be referred to above, and this application will not go into details.

[0140] The following is a comparison of the technical solution provided by this application with the existing technical solution through experimental data:

[0141] Compared with existing technical solutions, this application, on the one hand, places the image neural network and the text neural network in the same model for end-to-end training, which facilitates the management and deployment of the model. On the other hand, this application also improves the classification accuracy of the model. On the short video test set of 15,000 videos constructed, the test set is divided into 35, 61 and 234 categories respectively. Compared with existing technical solutions, the technical solutions provided by this application can improve the classification accuracy, as shown in Table 1.

[0142] Table 1 Comparison of the effects of the present technical solution and the existing technical solution on the video classification test set

[0143]

[0144] In summary, the present application provides a video classification method, in which the electronic device can combine the text information of the video frame, the video frame and the title information of the video to perform video classification. Compared with the method in the prior art that only combines the video frame and the title information of the video to perform video classification, the present application introduces the text information of the video frame, and the text information can better reflect the video content, thereby improving the video classification accuracy.

[0145] Figure 11 A schematic diagram of a model training device 1100 provided in an embodiment of the present application, wherein the device 1100 includes: an acquisition module 1110, a sampling module 1120, a text recognition module 1130, a processing module 1140 and a training module 1150, wherein the acquisition module 1110 is used to acquire a first video, a classification label of the first video and title information of the first video; the sampling module 1120 is used to perform frame sampling on the first video to obtain at least one video frame of the first video; the text recognition module 1130 is used to perform text recognition on at least one video frame of the first video to obtain text information of at least one video frame; the processing module 1140 is used to process the text information of at least one video frame to obtain a first word vector of the first video, and process the title information of the first video to obtain a second word vector of the first video; the training module 1150 is used to train a video classification model based on the first word vector, the second word vector and at least one video frame of the first video.

[0146] Optionally, the video classification model includes: an image neural network, an average pooling layer, a first text neural network, a first fully connected layer, a second text neural network, a second fully connected layer, a converter and a classifier, wherein the image neural network is used to process at least one video frame separately to obtain the first image feature of at least one video frame; the average pooling layer is used to process the first image feature of at least one video frame to obtain the second image feature of at least one video frame; the first text neural network is used to process the first word vector of the first video to obtain the first text feature of the first video; the first fully connected layer is used to process the first text feature of the first video to obtain the second text feature of the first video; the second text neural network is used to process the second word vector of the first video to obtain the third text feature of the first video; the second fully connected layer is used to process the third text feature of the first video to obtain the fourth text feature of the first video; the converter is used to perform feature fusion on the second image feature of at least one video frame, the second text feature and the fourth text feature of the first video to obtain the fused feature of the first video; the classifier is used to classify and label the first video according to the fused feature of the first video.

[0147] Optionally, the processing module 1140 is also used to process at least one video frame before the training module 1150 trains the video classification model based on the first word vector, the second word vector and at least one video frame of the first video, so as to adapt the at least one video frame to the image neural network.

[0148] Optionally, the text recognition module 1130 is specifically used to: perform text detection on at least one video frame to obtain the text position of at least one video frame; capture the local image at the text position of at least one video frame to obtain the captured image of at least one video frame; perform text recognition on the captured image of at least one video frame to obtain the text information of at least one video frame.

[0149] Optionally, the processing module 1140 is specifically configured to: splice text information of each of at least one video frame to obtain spliced text information; and convert the spliced text information into a first word vector of the first video.

[0150] It should be understood that the device embodiment and the model training method embodiment may correspond to each other, and similar descriptions can refer to the model training method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 11 The device 1100 shown can execute the above-mentioned model training method embodiment, and the aforementioned and other operations and / or functions of each module in the device 1100 are respectively for realizing the corresponding processes in the above-mentioned model training method. For the sake of brevity, they will not be repeated here.

[0151] The above describes the device 1100 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the various steps of the model training method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the model training method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or can be completed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned model training method embodiment in conjunction with its hardware.

[0152] Figure 12 A schematic diagram of a video classification device 1200 provided in an embodiment of the present application is shown as follows: Figure 12 As shown, the device 1200 includes: an acquisition module 1210, a sampling module 1220, a text recognition module 1230, a processing module 1240 and an input module 1250, wherein the acquisition module 1210 is used to acquire the second video and the title information of the second video; the sampling module 1220 is used to perform frame sampling on the second video to obtain at least one video frame of the second video; the text recognition module 1230 is used to perform text recognition on at least one video frame of the second video to obtain text information of at least one video frame; the processing module 1240 is used to process the text information of at least one video frame to obtain a third word vector of the second video, and process the title information of the second video to obtain a fourth word vector of the second video; the input module 1250 is used to input the third word vector, the fourth word vector and at least one video frame of the second video into the video classification model trained by the above-mentioned model training method to obtain the classification label of the second video.

[0153] It should be understood that the device embodiment and the video classification method embodiment may correspond to each other, and similar descriptions may refer to the video classification method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 12 The device 1200 shown can execute the above-mentioned video classification method embodiment, and the aforementioned and other operations and / or functions of each module in the device 1200 are respectively for implementing the corresponding processes in the above-mentioned video classification method. For the sake of brevity, they are not repeated here.

[0154] The above text describes the device 1200 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules. Specifically, the various steps of the video classification method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the video classification method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned video classification method embodiment in conjunction with its hardware.

[0155] Figure 13 It is a schematic block diagram of an electronic device provided in an embodiment of the present application.

[0156] like Figure 13 As shown, the electronic device may include:

[0157] The memory 1310 and the processor 1320 are configured to store computer programs and transmit the program code to the processor 1320. In other words, the processor 1320 can call and execute the computer program from the memory 1310 to implement the method in the embodiment of the present application.

[0158] For example, the processor 1320 may be configured to execute the above method embodiments according to instructions in the computer program.

[0159] In some embodiments of the present application, the processor 1320 may include but is not limited to:

[0160] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0161] In some embodiments of the present application, the memory 1310 includes but is not limited to:

[0162] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0163] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 1310 and executed by the processor 1320 to implement the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0164] like Figure 13 As shown, the electronic device may further include:

[0165] The transceiver 1330 may be connected to the processor 1320 or the memory 1310 .

[0166] The processor 1320 may control the transceiver 1330 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices. The transceiver 1330 may include a transmitter and a receiver. The transceiver 1330 may further include an antenna, which may be one or more.

[0167] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0168] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0169] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0170] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0172] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0173] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited to this. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Obtaining a first video, a classification label of the first video, and title information of the first video; Performing frame sampling on the first video to obtain at least one video frame of the first video; Performing text recognition on at least one video frame of the first video to obtain text information of each of the at least one video frame; Processing the text information of each of the at least one video frame to obtain a first word vector of the first video, and processing the title information of the first video to obtain a second word vector of the first video; training a video classification model based on the first word vector, the second word vector, and the at least one video frame of the first video; The video classification model includes: An image neural network, configured to process the at least one video frame respectively to obtain a first image feature of each of the at least one video frame; an average pooling layer, configured to process the first image feature of each of the at least one video frame to obtain a second image feature of each of the at least one video frame; a first text neural network, configured to process a first word vector of the first video to obtain a first text feature of the first video; a first fully connected layer, configured to process the first text feature of the first video to obtain a second text feature of the first video; a second text neural network, configured to process the second word vector of the first video to obtain a third text feature of the first video; a second fully connected layer, configured to process the third text feature of the first video to obtain a fourth text feature of the first video; The converter is configured to: perform feature concatenation on the second image feature of each of the at least one video frame, the second text feature and the fourth text feature of the first video to obtain (N+2)*c-dimensional features, where N is the number of video frames and c is the feature length input to the converter; perform feature concatenation on the (N+2)*c-dimensional features and the c-dimensional category identifier to obtain (N+3)*c-dimensional features; add the (N+3)*c-dimensional features to the position code of the same dimension to obtain the input of the encoder of the converter, so that the encoder of the converter outputs the fusion feature of the first video; wherein the converter is formed by sequentially connecting L layers of encoders, each layer of the encoder is composed of three modules: layer normalization LN, multi-head attention mechanism MSA and multi-layer perceptron MLP, and L is a positive integer; A classifier is used to process the fusion features of the first video to achieve classification and labeling of the first video.

2. The method according to claim 1, characterized in that Before training the video classification model according to the first word vector, the second word vector and the at least one video frame of the first video, the method further includes: The at least one video frame is processed so that the at least one video frame is adapted to the image neural network.

3. The method according to claim 1 or 2, characterized in that The performing text recognition on at least one video frame of the first video to obtain text information of each of the at least one video frame includes: Performing text detection on the at least one video frame to obtain text positions of the at least one video frame; intercepting a local image at a text position of each of the at least one video frame to obtain a captured image of each of the at least one video frame; Text recognition is performed on the captured images of each of the at least one video frame to obtain text information of each of the at least one video frame.

4. The method according to claim 1 or 2, characterized in that The processing of the text information of each of the at least one video frame to obtain a first word vector for the first video includes: splicing the text information of each of the at least one video frame to obtain spliced text information; The spliced text information is converted into a first word vector of the first video.

5. A video classification method, characterized in that: include: Obtaining a second video and title information of the second video; Performing frame sampling on the second video to obtain at least one video frame of the second video; performing text recognition on at least one video frame of the second video to obtain text information of each of the at least one video frame; Processing the text information of each of the at least one video frame to obtain a third word vector of the second video, and processing the title information of the second video to obtain a fourth word vector of the second video; The third word vector, the fourth word vector and the at least one video frame of the second video are input into the video classification model trained by the method according to any one of claims 1 to 4 to obtain the classification label of the second video.

6. A model training device, characterized in that: include: An acquisition module, configured to acquire a first video, a classification label of the first video, and title information of the first video; a sampling module, configured to perform frame sampling on the first video to obtain at least one video frame of the first video; a text recognition module, configured to perform text recognition on at least one video frame of the first video to obtain text information of each of the at least one video frame; a processing module, configured to process the text information of each of the at least one video frame to obtain a first word vector of the first video, and to process the title information of the first video to obtain a second word vector of the first video; a training module, configured to train a video classification model based on the first word vector, the second word vector, and the at least one video frame of the first video; The video classification model includes: An image neural network, configured to process the at least one video frame respectively to obtain a first image feature of each of the at least one video frame; an average pooling layer, configured to process the first image feature of each of the at least one video frame to obtain a second image feature of each of the at least one video frame; a first text neural network, configured to process a first word vector of the first video to obtain a first text feature of the first video; a first fully connected layer, configured to process the first text feature of the first video to obtain a second text feature of the first video; a second text neural network, configured to process the second word vector of the first video to obtain a third text feature of the first video; a second fully connected layer, configured to process the third text feature of the first video to obtain a fourth text feature of the first video; A converter, configured to: perform feature concatenation on the second image features of each of the at least one video frame, the second text features of the first video, and the fourth text features of the first video to obtain (N+2)*c-dimensional features, where N is the number of video frames and c is the feature length input to the converter; perform feature concatenation on the (N+2)*c-dimensional features and the c-dimensional category identifier to obtain (N+3)*c-dimensional features; add the (N+3)*c-dimensional features to the position code of the same dimension to obtain the input of the encoder of the converter, so that the encoder of the converter outputs the fusion feature of the first video; wherein the converter is composed of L layers of encoders connected in sequence, each layer of encoder is composed of three modules: LN, MSA, and MLP, and L is a positive integer; A classifier is used to process the fusion features of the first video to achieve classification and labeling of the first video.

7. A video classification device, characterized in that: include: An acquisition module, configured to acquire a second video and title information of the second video; a sampling module, configured to perform frame sampling on the second video to obtain at least one video frame of the second video; a text recognition module, configured to perform text recognition on at least one video frame of the second video to obtain text information of each of the at least one video frame; a processing module, configured to process the text information of each of the at least one video frame to obtain a third word vector of the second video, and to process the title information of the second video to obtain a fourth word vector of the second video; An input module is used to input the third word vector, the fourth word vector and the at least one video frame of the second video into the video classification model trained by the method according to any one of claims 1 to 4 to obtain the classification label of the second video.

8. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 5.

Images