Video tag processing method and device, computing equipment, storage medium and product

Abstract

The embodiment of the invention discloses a video tag processing method and device, computing equipment, a storage medium and a product, and the method comprises the steps: constructing an initial video tag generation model through employing a pre-trained convolutional neural network and a pre-trained recurrent neural network; according to a video sample in the training data set and a target video tag corresponding to the video sample, training the initial video tag generation model to obtain a video tag generation model; and processing a to-be-processed video by using the video tag generation model to generate a video tag corresponding to the to-be-processed video. According to the invention, the deep learning technology is introduced into the field of video tag generation, the pre-trained convolutional neural network and the pre-trained recurrent neural network are utilized to construct the model, the trained video tag generation model can accurately and quickly generate the corresponding video tag for the video, automatic and accurate generation of the video tag is realized, and the video tag generation efficiency is improved. The efficiency and accuracy of video tag generation are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to a video tag processing method, apparatus, computing device, storage medium and product. Background Art

[0002] With the rapid growth of video content, it is becoming increasingly difficult for users to search for videos of interest on video platforms. In order to facilitate users to search for videos, it is necessary to set video tags that reflect the video content according to the video content, and classify the videos through video tags. In the prior art, it is usually done manually by watching the video content to set corresponding video tags for the video. However, this video tag processing method not only consumes a lot of manpower and time costs, but also has low processing efficiency, and is affected by subjective factors, resulting in low accuracy and prone to errors. Summary of the invention

[0003] In view of the above problems, the present application proposes a video tag processing method, apparatus, computing device, storage medium and product to solve the following problems: the existing video tag processing method is time-consuming and labor-intensive, has low processing efficiency and low accuracy.

[0004] According to one aspect of an embodiment of the present application, a video tag processing method is provided, including:

[0005] Use pre-trained convolutional neural networks and pre-trained recurrent neural networks to build an initial video label generation model;

[0006] According to the video samples in the training data set and the target video labels corresponding to the video samples, the initial video label generation model is trained to obtain a video label generation model;

[0007] Obtain the video to be processed, use the video label generation model to process the video to be processed, and generate a video label corresponding to the video to be processed.

[0008] Furthermore, the initial video label generation model includes: a video frame extraction module, an image feature extraction module, a sequence processing module and a label generation module; wherein,

[0009] In the video frame extraction module, video key frames are extracted from the video samples;

[0010] In the image feature extraction module, the image feature vector of the video key frame is extracted using a pre-trained convolutional neural network;

[0011] In the sequence processing module, the image feature vectors of the video key frames are processed using a pre-trained recurrent neural network to obtain the video content description information corresponding to the video key frames;

[0012] In the label generation module, the video label corresponding to the video sample is generated according to the video content description information.

[0013] Furthermore, the recurrent neural network includes a bidirectional LSTM; in the sequence processing module, the bidirectional LSTM is also used to capture the front-end dependency relationship in the video frame sequence.

[0014] Furthermore, the video content description information includes at least one or more of the following information:

[0015] Theme information, scene information, object information, action information, emotion information, and video content type information.

[0016] Furthermore, the initial video tag generation model also includes: an attention mechanism module;

[0017] In the attention mechanism module, the attention weights of the video key frames are set according to their importance to enhance the attention of the initial video label generation model to important video key frames.

[0018] Further, according to the video samples in the training data set and the target video labels corresponding to the video samples, the initial video label generation model is trained to obtain the video label generation model further including:

[0019] Extract video samples and target video labels corresponding to the video samples from the training data set;

[0020] Input the video sample into the initial video label generation model for training to obtain the initial video label corresponding to the video sample;

[0021] According to the initial video labels and target video labels corresponding to the video samples, the weight parameters of the initial video label generation model are updated;

[0022] The above steps are iterated in a loop until the iteration end condition is met to obtain a video label generation model.

[0023] According to another aspect of an embodiment of the present application, a video tag processing device is provided, including:

[0024] A construction unit, adapted to construct an initial video label generation model using a pre-trained convolutional neural network and a pre-trained recurrent neural network;

[0025] A training unit, adapted to train an initial video label generation model according to video samples in a training data set and target video labels corresponding to the video samples, to obtain a video label generation model;

[0026] The application unit is adapted to obtain a video to be processed, process the video to be processed using a video tag generation model, and generate a video tag corresponding to the video to be processed.

[0027] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0028] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video tag processing method.

[0029] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned video tag processing method.

[0030] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned video tag processing method.

[0031] According to the technical solution provided in the embodiments of the present application, deep learning technology is introduced into the field of video label generation, and a pre-trained convolutional neural network and a pre-trained recurrent neural network are used to construct an initial video label generation model. The model is trained according to a training data set, and the video label generation model can be easily trained, thereby saving a lot of computing resources and training time, and can effectively improve the generalization ability of the video label generation model; the video label generation model can be used to accurately and quickly generate corresponding video labels for videos without manually determining the video labels, thereby realizing automatic and accurate generation of video labels, greatly reducing the manpower and time costs required in the video label processing process, and effectively improving the efficiency and accuracy of video label generation.

[0032] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0034] Figure 1 A schematic diagram of a process of processing a video tag according to an embodiment of the present application is shown;

[0035] Figure 2a A schematic diagram showing a flow chart of a video tag processing method according to another embodiment of the present application is shown;

[0036] Figure 2b Shows the structure of the initial video tag generation model Figure 1 ;

[0037] Figure 2c The second structural diagram of the initial video tag generation model is shown;

[0038] Figure 3 A structural block diagram of a video tag processing device according to an embodiment of the present application is shown;

[0039] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] First, the terms involved in one or more embodiments of the present application are explained.

[0042] Convolutional Neural Networks (CNN): A type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. It is a deep learning network that is particularly suitable for processing data with a grid structure, such as images, videos, and speech signals. It can automatically extract features and perform classification or regression tasks.

[0043] Recurrent Neural Network (RNN): It is a type of recurrent neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain; it is a deep learning network specifically designed to process sequence data. Unlike traditional feedforward neural networks, RNN has feedback connections that allow information to be passed between time steps, thereby capturing the temporal dependencies in the input data.

[0044] ResNet (Residual Network): is a deep convolutional neural network that introduces residual blocks and solves the gradient vanishing problem in deep networks through jump connections (also called shortcut connections or shortcut connections), making it possible to train very deep networks.

[0045] EfficientNet: is an efficient convolutional neural network that achieves higher accuracy and less computing resource consumption by introducing a compound scaling method to balance the width, depth, and resolution of the network.

[0046] Bidirectional LSTM: Bidirectional Long Short-Term Memory (BiLSTM) is a special recurrent neural network (RNN) that combines two LSTM layers: one for processing forward sequences and the other for processing reverse sequences. This design enables bidirectional LSTM to capture information in both the forward and reverse directions of the input data at the same time, thereby better understanding the context and performing well in tasks such as natural language processing and speech recognition.

[0047] Figure 1 A schematic diagram of a video tag processing method according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0048] Step S101, using a pre-trained convolutional neural network and a pre-trained recurrent neural network to construct an initial video tag generation model.

[0049] Considering that deep learning technology can be well applied to image, video and other processing fields, in an embodiment of the present application, deep learning technology is introduced into the field of video label generation, a video label generation model is constructed and trained, and the video label generation model is used to automatically generate corresponding video labels for the video to be processed.

[0050] In order to improve the training efficiency of the video label generation model, the pre-trained convolutional neural network and the pre-trained recurrent neural network are used to construct the initial video label generation model. Among them, the pre-trained convolutional neural network and the pre-trained recurrent neural network refer to the convolutional neural network and the recurrent neural network that have been trained using large-scale data sets. The pre-trained convolutional neural network and the pre-trained recurrent neural network are usually trained on general tasks such as image classification and target detection, so that the pre-trained convolutional neural network has learned the basic patterns and hierarchical structures of visual features, and the pre-trained recurrent neural network has learned the basic features and patterns of sequence data. By using the pre-trained convolutional neural network and the pre-trained recurrent neural network to construct the initial video label generation model, fine-tuning can be performed on new tasks or smaller data sets, and the video label generation model can be easily trained, thereby saving a lot of computing resources and training time, and can improve the generalization ability of the video label generation model to obtain better performance.

[0051] Step S102: training an initial video label generation model according to the video samples in the training data set and the target video labels corresponding to the video samples to obtain a video label generation model.

[0052] The training data set stores multiple pre-collected video samples and target video tags corresponding to each video sample, wherein the video sample can be obtained from the video library in the video platform, and the target video tag corresponding to the video sample can be determined by analyzing the video content of the video sample. Specifically, the target video tag may include a theme tag, a scene tag, an object tag, an action tag, an emotion tag, and a video content type tag.

[0053] Among them, the training process of the video label generation model is completed through multiple iterations. The video label generation model can accurately and quickly generate corresponding video labels for videos, effectively improving the efficiency and accuracy of video label generation.

[0054] Step S103, obtaining a video to be processed, and using a video tag generation model to process the video to be processed to generate a video tag corresponding to the video to be processed.

[0055] When a user needs to generate a corresponding video tag for a video, the video can be uploaded, and then the video is obtained as a video to be processed in step S103; when a video platform needs to generate a corresponding video tag for a video, the video provided by the video platform can be obtained as a video to be processed. After obtaining the video to be processed, the video to be processed only needs to be input into the video tag generation model for processing, and the video tag corresponding to the video to be processed can be accurately and quickly generated. Video tags may include subject tags, scene tags, object tags, action tags, emotion tags, and video content type tags.

[0056] According to the video label processing method provided in the embodiment of the present application, deep learning technology is introduced into the field of video label generation, and a pre-trained convolutional neural network and a pre-trained recurrent neural network are used to construct an initial video label generation model. The model is trained according to a training data set, and the video label generation model can be easily trained, thereby saving a lot of computing resources and training time, and can effectively improve the generalization ability of the video label generation model; the video label generation model can be used to accurately and quickly generate corresponding video labels for videos without manually determining the video labels, thereby realizing automatic and accurate generation of video labels, greatly reducing the manpower and time costs required in the video label processing process, and effectively improving the efficiency and accuracy of video label generation.

[0057] Figure 2a A schematic diagram of a video tag processing method according to another embodiment of the present application is shown. Figure 2a As shown, the method comprises the following steps:

[0058] Step S201, using a pre-trained convolutional neural network to extract image feature vectors of video key frames, using a pre-trained recurrent neural network to process the image feature vectors of video key frames, and constructing an initial video label generation model.

[0059] Among them, the constructed initial video label generation model may include: a video frame extraction module, an image feature extraction module, a sequence processing module and a label generation module. In the video frame extraction module, video key frames are extracted from video samples; in the image feature extraction module, the image feature vectors of the video key frames are extracted using a pre-trained convolutional neural network; in the sequence processing module, the image feature vectors of the video key frames are processed using a pre-trained recurrent neural network to obtain the video content description information corresponding to the video key frames; in the label generation module, the video label corresponding to the video sample is generated according to the video content description information. Among them, the video content description information includes at least one or more of the following information: theme information, scene information, object information, action information, emotion information and video content type information. Those skilled in the art may also set the video content description information to include other information according to actual needs, which is not specifically limited here.

[0060] Figure 2b Shows the structure of the initial video tag generation model Figure 1 ,like Figure 2b As shown, the video frame extraction module processes the input video samples, extracts and outputs video key frames from the video samples; the pre-trained convolutional neural network is introduced into the image feature extraction module, and the image feature extraction module uses the pre-trained convolutional neural network to extract and output the image feature vectors of the video key frames, wherein a deeper convolutional neural network, such as ResNet or EfficientNet, can be used to extract richer image feature vectors; the pre-trained recurrent neural network is introduced into the sequence processing module, and the sequence processing module uses the pre-trained recurrent neural network to process the image feature sequence formed by the image feature vectors of the video key frames, and generates and outputs the video content description information corresponding to the video key frames; the label generation module generates the video label corresponding to the video sample according to the video content description information, and finally outputs the generated video label.

[0061] Below is a specific code implementation of the construction and training of the initial video label generation model, using Python and TensorFlow framework.

[0062] import tensorflow as tf

[0063] from tensorflow.keras.applications import InceptionV3

[0064] from tensorflow.keras.preprocessing.sequence import pad_sequences

[0065] from tensorflow.keras.models import Model

[0066] from tensorflow.keras.layers import Input,LSTM,Embedding,Dense,TimeDistributed,Bidirectional

[0067] #Load the pre-trained InceptionV3 network

[0068] base_model=InceptionV3(weights='imagenet',include_top=False)

[0069] video_input=Input(shape=(None,299,299,3))

[0070] features=TimeDistributed(base_model)(video_input)

[0071] #Average pooling layer

[0072] avg_pool=TimeDistributed(tf.keras.layers.GlobalAveragePooling2D())(features)

[0073] #LSTM layer processes video features

[0074] lstm=Bidirectional(LSTM(256,return_sequences=True))(avg_pool)

[0075] #Output layer generates labels

[0076] output=TimeDistributed(Dense(num_tags,activation='softmax'))(lstm)

[0077] #Build the model

[0078] model=Model(inputs=video_input,outputs=output)

[0079] model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])

[0080] #Simulate video data and video labels

[0081] import numpy as np

[0082] video_data = np.random.rand(10,20,299,299,3) #10 videos, 20 frames each, 299×299×3 per frame

[0083] labels = np.random.randint(0,num_tags,(10,20,num_tags)) #10 videos, 20 frames per video, one video label per frame

[0084] #Train model

[0085] model.fit(video_data,labels,epochs=10,batch_size=2)

[0086] #Predict video tags

[0087] predicted_labels=model.predict(video_data)

[0088] In order to further improve the performance and accuracy of the video label processing method provided in the embodiment of the present application, the pre-trained recurrent neural network can use a bidirectional LSTM, that is, the recurrent neural network includes a bidirectional LSTM, then in the sequence processing module, the bidirectional LSTM can also be used to capture the front-to-back dependency relationship in the video frame sequence. In addition, the initial video label generation model also includes an attention mechanism module, which can be set between the sequence processing module and the label generation module, such as Figure 2c In the attention mechanism module, the attention weight of the video key frame can be set according to the importance of the video key frame to enhance the initial video label generation model's attention to important video key frames and improve the accuracy of video label generation.

[0089] Specifically, the algorithm formula corresponding to extracting the image feature vector of the key frame of the video using the pre-trained convolutional neural network (CNN) can be expressed as [F_i=CNN(V_i)], where V_i represents the i-th frame and F_i represents the image feature vector of the i-th frame. The algorithm formula corresponding to processing the image feature sequence using the bidirectional LSTM is expressed as [H_t=\text{BiLSTM}(F_t)], where F_t represents the image feature vector of the t-th frame and H_t represents the hidden state of the t-th frame. The algorithm formula corresponding to the attention mechanism can be expressed as [\alpha_t=\frac{\exp(e_t)}{\sum{k=1}^{T}\exp(e_k)}, \quad e_t=\text{tanh}(W_h H_t+b_h)], where \alpha_t represents the attention weight of the t-th frame, and W_h and b_h are trainable parameters. The algorithm formula corresponding to video label generation can be expressed as [y_t=\text{softmax}(W_y H_t+b_y)], where y_t represents the label probability distribution of the tth frame, and W_y and b_y are trainable parameters.

[0090] Below is another concrete code implementation of building and training the initial video label generation model, using Python and TensorFlow framework.

[0091] import tensorflow as tf

[0092] from tensorflow.keras.applications import ResNet50

[0093] from tensorflow.keras.models import Model

[0094] from tensorflow.keras.layers import Input,LSTM,Dense,TimeDistributed,Bidirectional,GlobalAveragePooling2D,Attention

[0095] #Load the pre-trained ResNet50 network

[0096] base_model=ResNet50(weights='imagenet',include_top=False)

[0097] video_input=Input(shape=(None,224,224,3))

[0098] features=TimeDistributed(base_model)(video_input)

[0099] #Average pooling layer

[0100] avg_pool=TimeDistributed(GlobalAveragePooling2D())(features)

[0101] #Bidirectional LSTM layer processes video features

[0102] lstm=Bidirectional(LSTM(256,return_sequences=True))(avg_pool)

[0103] #Attention Mechanism

[0104] attention=Attention()([lstm,lstm])

[0105] #Output layer generates video labels

[0106] output=TimeDistributed(Dense(num_tags,activation='softmax'))(attention)

[0107] #Build the model

[0108] model=Model(inputs=video_input,outputs=output)

[0109] model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])

[0110] #Simulate video data and video labels

[0111] import numpy as np

[0112] video_data = np.random.rand(10,20,224,224,3) #10 videos, 20 frames each, 224×224×3 per frame

[0113] labels = np.random.randint(0,num_tags,(10,20,num_tags)) #10 videos, 20 frames per video, one video label per frame

[0114] #Train model

[0115] model.fit(video_data,labels,epochs=10,batch_size=2)

[0116] #Predict video tags

[0117] predicted_labels=model.predict(video_data)

[0118] By using the deeper ResNet50, bidirectional LSTM and attention mechanism, video tags that reflect the content of the video can be generated more accurately, effectively improving the user experience of the video platform, allowing users to easily determine video tags and accurately and quickly search for videos of interest based on video tags.

[0119] Step S202: training an initial video label generation model according to the video samples in the training data set and the target video labels corresponding to the video samples to obtain a video label generation model.

[0120] The training process of the video tag generation model is completed through multiple iterations. In step S202, video samples and target video tags corresponding to the video samples are extracted from the training data set; the video samples are input into the initial video tag generation model for training to obtain the initial video tags corresponding to the video samples; the weight parameters of the initial video tag generation model are updated according to the initial video tags and target video tags corresponding to the video samples; the above steps are iterated in a loop until the iteration end condition is met to obtain the video tag generation model.

[0121] Specifically, the loss between the initial video label and the target video label corresponding to the video sample can be calculated according to the loss function to obtain the output value of the loss function. According to the loss function, back propagation and other operations are performed, and the weight parameters of the initial video label generation model are updated according to the operation results. In specific applications, the iteration end condition may include: the number of iterations reaches the iteration number threshold; and / or the output value of the loss function is less than the loss threshold. Then, it can be determined whether the iteration end condition is met by judging whether the number of iterations reaches the iteration number threshold, or it can be determined whether the iteration end condition is met according to whether the output value of the loss function is less than the loss threshold. After the iteration end condition is met, the iterative processing is stopped to obtain a trained video label generation model.

[0122] For example, in the above two specific code implementations, the optimization method uses the adam algorithm, which introduces a bias correction mechanism to achieve fast convergence and stable performance. The adam algorithm uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of each parameter. It has the characteristics of adaptive learning rate, insensitivity to hyperparameters, and applicability to sparse gradients. The loss function used is categorical_crossentropy, which is one of the commonly used loss functions in Keras and is widely used in multi-classification problems, especially in the fields of deep learning and machine learning. It measures the difference between the probability distribution predicted by the model and the true label (i.e., the target video label). For a classification task with multiple categories, the model usually outputs the probability value of each category, and the true label is represented in the form of one-hot encoding. categorical_crossentropy uses cross entropy as a difference metric for measuring classification tasks to measure the similarity between two probability distributions.

[0123] Step S203, obtaining the video to be processed, inputting the video to be processed into a video tag generation model, and allowing the video tag generation model to process the video to be processed to generate a video tag corresponding to the video to be processed.

[0124] The video tag generation model obtained through training can automatically analyze and process the video content, and accurately and quickly generate corresponding video tags for the video. In specific applications, the video platform can provide videos for the video tag generation model, use the provided videos as videos to be processed, and use the video tag generation model to generate corresponding video tags for each video to be processed. Users of the video platform can also apply the video tag processing method provided by the embodiment of the present application in the scenarios of searching videos and uploading videos. For example, in the scenario of searching videos, users can search for videos according to the video tags corresponding to each video in the video platform, so as to accurately and quickly search for the videos of interest from the massive videos provided by the video platform; in the scenario of uploading videos, use the uploaded videos as videos to be processed, and use the video tag generation model to generate corresponding video tags for them.

[0125] In step S203, the video to be processed is input into the video label generation model, and the video label generation model processes the video to be processed. Specifically, the video key frames are extracted by the video label generation model, and then the image feature vectors of the video key frames are extracted. Then, the image feature vectors of the video key frames are processed to obtain the video content description information corresponding to the video key frames. Finally, according to the video content description information, a video label corresponding to the video to be processed is generated.

[0126] According to the video label processing method provided in the embodiment of the present application, deep learning technology is introduced into the field of video label generation, a pre-trained convolutional neural network is used to extract image feature vectors of video key frames, and a pre-trained recurrent neural network is used to process the image feature vectors of video key frames to construct an initial video label generation model, and the model is trained according to the training data set, so that the video label generation model can be easily trained, thereby saving a lot of computing resources and training time, and improving the performance and accuracy of the video label generation model; the video label generation model can be used to automatically analyze and process the video content, accurately and quickly generate corresponding video labels for the video, and realize automatic and accurate generation of video labels, which greatly reduces the manpower and time costs required in the video label processing process, and significantly improves the efficiency and accuracy of video label generation, which not only enables users to easily and accurately generate video labels for the videos they upload, but also allows them to search for videos according to video labels, and accurately and quickly search for videos of interest from the massive videos provided by the video platform, significantly improving the user experience of the video platform, and has broad application prospects.

[0127] Figure 3 FIG. 4 shows a structural block diagram of a video tag processing device according to an embodiment of the present application. Figure 3 As shown, the device includes: a construction unit 310, a training unit 320 and an application unit 330.

[0128] The construction unit 310 is adapted to construct an initial video tag generation model using a pre-trained convolutional neural network and a pre-trained recurrent neural network.

[0129] The training unit 320 is adapted to: train the initial video label generation model according to the video samples in the training data set and the target video labels corresponding to the video samples to obtain the video label generation model.

[0130] The application unit 330 is adapted to: obtain a video to be processed, process the video to be processed using a video tag generation model, and generate a video tag corresponding to the video to be processed.

[0131] Optionally, the initial video label generation model includes: a video frame extraction module, an image feature extraction module, a sequence processing module and a label generation module; wherein, in the video frame extraction module, video key frames are extracted from video samples; in the image feature extraction module, a pre-trained convolutional neural network is used to extract image feature vectors of video key frames; in the sequence processing module, a pre-trained recurrent neural network is used to process the image feature vectors of video key frames to obtain video content description information corresponding to the video key frames; in the label generation module, a video label corresponding to the video sample is generated according to the video content description information.

[0132] Optionally, the recurrent neural network includes a bidirectional LSTM; in the sequence processing module, the bidirectional LSTM is also used to capture the front-end dependency relationship in the video frame sequence.

[0133] Optionally, the video content description information includes at least one or more of the following information: theme information, scene information, object information, action information, emotion information and video content type information.

[0134] Optionally, the initial video label generation model also includes: an attention mechanism module; in the attention mechanism module, the attention weights of the video key frames are set according to the importance of the video key frames to enhance the initial video label generation model's attention to important video key frames.

[0135] Optionally, the training unit 320 is further adapted to: extract video samples and target video labels corresponding to the video samples from a training data set; input the video samples into an initial video label generation model for training to obtain initial video labels corresponding to the video samples; update weight parameters of the initial video label generation model according to the initial video labels and target video labels corresponding to the video samples; and iterate the above steps until an iteration end condition is met to obtain a video label generation model.

[0136] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0137] According to the video label processing device provided in the embodiment of the present application, deep learning technology is introduced into the field of video label generation, and a pre-trained convolutional neural network is used to extract image feature vectors of video key frames, and a pre-trained recurrent neural network is used to process the image feature vectors of video key frames to construct an initial video label generation model, and the model is trained according to the training data set, so that the video label generation model can be easily trained, thereby saving a lot of computing resources and training time, and improving the performance and accuracy of the video label generation model; the video label generation model can be used to automatically analyze and process the video content, accurately and quickly generate corresponding video labels for the video, and realize automatic and accurate generation of video labels, which greatly reduces the manpower and time costs required in the video label processing process, and significantly improves the efficiency and accuracy of video label generation, which not only enables users to easily and accurately generate video labels for the videos they upload, but also allows them to search for videos according to video labels, and accurately and quickly search for videos of interest from the massive videos provided by the video platform, significantly improving the user experience of the video platform, and has broad application prospects.

[0138] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the video tag processing method in any of the above method embodiments.

[0139] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the video tag processing method in any of the above method embodiments.

[0140] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0141] like Figure 4 As shown, the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0142] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned video tag processing method embodiment for a computing device.

[0143] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0144] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0145] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0146] The program 410 can be specifically used to enable the processor 402 to execute the video tag processing method in any of the above-mentioned method embodiments. The specific implementation of each step in the program 410 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned video tag processing embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0147] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0148] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0149] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0150] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0151] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0152] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0153] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A video tag processing method, comprising: Use pre-trained convolutional neural networks and pre-trained recurrent neural networks to build an initial video label generation model; According to the video samples in the training data set and the target video labels corresponding to the video samples, the initial video label generation model is trained to obtain a video label generation model; Obtain a video to be processed, and use the video tag generation model to process the video to be processed to generate a video tag corresponding to the video to be processed.

2. According to the method of claim 1, the initial video tag generation model comprises: Video frame extraction module, image feature extraction module, sequence processing module and label generation module; wherein, In the video frame extraction module, extracting video key frames from the video samples; In the image feature extraction module, a pre-trained convolutional neural network is used to extract the image feature vector of the video key frame; In the sequence processing module, the image feature vector of the video key frame is processed by using a pre-trained recurrent neural network to obtain video content description information corresponding to the video key frame; In the label generation module, a video label corresponding to the video sample is generated according to the video content description information.

3. According to the method described in claim 2, the recurrent neural network includes a bidirectional LSTM; in the sequence processing module, the bidirectional LSTM is also used to capture the front-end and back-end dependencies in the video frame sequence.

4. The method according to claim 2, wherein the video content description information includes at least one or more of the following information: Theme information, scene information, object information, action information, emotion information, and video content type information.

5. According to the method according to any one of claims 2 to 4, the initial video tag generation model further comprises: Attention mechanism module; In the attention mechanism module, the attention weight of the video key frame is set according to the importance of the video key frame to enhance the attention of the initial video tag generation model to important video key frames.

6. According to the method according to any one of claims 1 to 5, the training of the initial video label generation model based on the video samples in the training data set and the target video labels corresponding to the video samples to obtain the video label generation model further comprises: Extracting video samples and target video labels corresponding to the video samples from the training data set; Inputting the video sample into the initial video label generation model for training to obtain the initial video label corresponding to the video sample; According to the initial video label and the target video label corresponding to the video sample, updating the weight parameters of the initial video label generation model; The above steps are executed in a loop and iteratively until the iteration end condition is met to obtain the video tag generation model.

7. A video tag processing device, comprising: A construction unit, adapted to construct an initial video label generation model using a pre-trained convolutional neural network and a pre-trained recurrent neural network; A training unit, adapted to train the initial video label generation model according to the video samples in the training data set and the target video labels corresponding to the video samples to obtain a video label generation model; The application unit is adapted to obtain a video to be processed, process the video to be processed using the video tag generation model, and generate a video tag corresponding to the video to be processed.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the video tag processing method according to any one of claims 1 to 6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the video tag processing method according to any one of claims 1 to 6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the video tag processing method according to any one of claims 1 to 6.

Images