Convolutional Neural Network Training Method, Device, and Equipment

Abstract

The present disclosure provides a method, apparatus, and device for training a convolutional neural network, relating to the field of artificial intelligence technologies, specifically to technical fields such as computer vision, image processing, and deep learning. A specific implementation of the method includes: obtaining a first sample video frame sequence; inputting the first sample video frame sequence into a convolutional neural network to obtain a first vector sequence; inputting the first vector sequence into a picture mask self-attention model to obtain a second vector sequence; calculating a first loss based on the first vector sequence and the second vector sequence; and adjusting the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed. This implementation provides a video pre-training method based on contrastive learning, which effectively utilizes a large amount of unlabeled video resources and reduces the model training cost by combining a convolutional neural network and a picture mask self-attention model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, and deep learning. Background Art

[0002] With the rapid development of internet technology, massive amounts of video and image resources have been uploaded online, significantly increasing the complexity of recommendation and search technologies. To effectively meet and enhance the search experience for diverse users, it is necessary to provide them with high-quality resources. Efficiently and accurately understanding this massive amount of video resources directly impacts the accuracy of recommendation and search strategies, ultimately impacting user experience and retention.

[0003] Currently, supervised deep learning approaches are primarily used for small-batch model training to address a specific video problem from a specific dimension or direction. However, this approach fails to fully utilize the knowledge inherent in massive amounts of unlabeled data, and simultaneously addressing multiple video problems requires exponentially more effort. Summary of the Invention

[0004] The embodiments of the present disclosure provide a convolutional neural network training method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a convolutional neural network training method, including: obtaining a first sample video frame sequence; inputting the first sample video frame sequence into a convolutional neural network to obtain a first vector sequence; inputting the first vector sequence into a picture mask self-attention model to obtain a second vector sequence; calculating a first loss based on the first vector sequence and the second vector sequence; adjusting the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network that has completed unsupervised training.

[0006] In a second aspect, an embodiment of the present disclosure proposes a video classification method, comprising: obtaining a video to be classified; extracting a video frame sequence to be classified from the video to be classified; inputting the video frame sequence to be classified into a video classification model to obtain the category of the video to be classified, wherein the video classification model is trained using the method described in the first aspect.

[0007] In a third aspect, an embodiment of the present disclosure provides a convolutional neural network training device, including: a first acquisition module configured to acquire a first sample video frame sequence; a first input module configured to input the first sample video frame sequence into the convolutional neural network to obtain a first vector sequence; a second input module configured to input the first vector sequence into the picture mask self-attention model to obtain a second vector sequence; a first calculation module configured to calculate a first loss based on the first vector sequence and the second vector sequence; and a first adjustment module configured to adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a video classification device, including: an acquisition module configured to acquire a video to be classified; an extraction module configured to extract a sequence of video frames to be classified from the video to be classified; and a classification module configured to input the sequence of video frames to be classified into a video classification model to obtain the category of the video to be classified, where the video classification model is trained by using the device described in the third aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; where the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect or the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure provides a computer program product including a computer program, where the computer program, when executed by a processor, implements the method described in any implementation manner of the first aspect or the second aspect.

[0012] The embodiment of the present disclosure provides a video pre-training method based on contrastive learning. By combining a convolutional neural network and a picture mask self-attention model, it effectively utilizes a large amount of unlabeled video resources and reduces the model training cost. Moreover, it has very good effects in solving various types of video understanding tasks.

[0013] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments read in conjunction with the accompanying drawings. The drawings are used to better understand the solution and do not limit the present disclosure. Among them:

[0015] Figure 1 is a flowchart of an embodiment of the convolutional neural network training method according to the present disclosure;

[0016] Figure 2 is a flowchart of another embodiment of the convolutional neural network training method according to the present disclosure;

[0017] Figure 3 is a flowchart of another embodiment of the convolutional neural network training method according to the present disclosure;

[0018] Figure 4 is a schematic diagram of an unsupervised training process;

[0019] Figure 5 is a schematic diagram of a supervised training process;

[0020] Figure 6 is a flowchart of an embodiment of the video classification method according to the present disclosure;

[0021] Figure 7 is a schematic structural diagram of an embodiment of the convolutional neural network training device according to the present disclosure;

[0022] Figure 8 is a schematic structural diagram of an embodiment of the video classification device according to the present disclosure;

[0023] Figure 9 is a block diagram of an electronic device for implementing the convolutional neural network training method or the video classification method of the embodiments of the present disclosure. Detailed Embodiments

[0024] The following describes exemplary embodiments of the present disclosure in conjunction with the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding and should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0025] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0026] Figure 1Fig. 100 shows the flowchart of an embodiment of a convolutional neural network training method according to the present disclosure. The convolutional neural network training method includes the following steps:

[0027] Step 101, obtaining a first sample video frame sequence.

[0028] In this embodiment, a large number of first sample video frame sequences (V t1 , V t2 , V t3 , …, V tN ) are obtained.

[0029] Generally, a large number of first videos are searched from a search engine according to preset search conditions. N (N is a positive integer) first video frames are extracted from the first videos, and a first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ) can be formed.

[0030] Among them, the preset search conditions can be conditions set according to the classification task. For example, if the classification task is to classify animals, videos related to animals can be searched. Another example is that if the classification task is to classify plants, videos related to plants can be searched.

[0031] Among them, the extraction methods can include but are not limited to: equidistant extraction, key frame extraction, random extraction, etc.

[0032] It should be noted that after searching for the first videos from the search engine, only the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ) needs to be obtained from the first videos, and it is not necessary to label the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ). Therefore, the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ) is unlabeled data.

[0033] Step 102, inputting the first sample video frame sequence into the convolutional neural network to obtain a first vector sequence.

[0034] In this embodiment, the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tNThe first sample video frame V in ti is sequentially input into a Convolutional Neural Networks (CNN) to correspondingly calculate the feature vector C ti , obtaining the first vector sequence (C t1 , C t2 , C t3 , …, C tN ). Wherein, 1 ≤ i ≤ N, and i is a positive integer.

[0035] Among them, the convolutional neural network is a deep learning model that extracts the features of the input image through multiple convolution and pooling operations.

[0036] Step 103: Input the first vector sequence into the picture masked self-attention model to obtain the second vector sequence.

[0037] In this embodiment, the first vector C in the first vector sequence (C t1 , C t2 , C t3 , …, C tN ) is sequentially input into the picture masked self-attention model to correspondingly calculate the feature vector S ti , obtaining the second vector sequence (S ti , S t1 , S t2 , S t3 , …, S tN ).

[0038] Among them, the picture masked self-attention model is a self-attention mechanism model that can calculate weights for the input vector sequence to more effectively capture the relationships between sequences and generate new sequence representations.

[0039] Step 104: Calculate the first loss based on the first vector sequence and the second vector sequence.

[0040] In this embodiment, based on the first vector sequence (C t1 , C t2 , C t3 , …, C tN ) and the second vector sequence (S t1 , S t2 , S t3 , …, S tN ), the first loss S can be calculated.

[0041] Generally, if the first vector C in the first vector sequence (C t1 , C t2 , C t3 , …, C tN ) tiis the same dimension as the second vectors in the second vector sequence (S t1 , S t2 , S t3 , …, S tN ) and in the same vector space, then the first vector C ti can be directly calculated with the second vector S ti to calculate the first loss S; if the first vector C ti is different in dimension from the second vector S ti and / or in a different vector space, then the first vector C ti needs to be projected onto the same dimension and / or the same vector space as the second vector S ti before calculating the first loss S. ti

[0042] Step 105, adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed.

[0043] In this embodiment, based on the first loss S, the parameters of the convolutional neural network and the picture mask self-attention model can be adjusted to obtain a convolutional neural network with unsupervised training completed.

[0044] Generally, using the gradient descent algorithm, adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss S until the first loss S converges to complete unsupervised learning.

[0045] Among them, unsupervised learning is a machine learning method that generates useful representations by discovering patterns and structures in data without labeled data. Unsupervised learning is usually used for tasks such as clustering, dimensionality reduction, or pre-training model weights.

[0046] Among them, the gradient descent algorithm is an optimization algorithm used to adjust the parameters of a model to minimize the loss function. Specifically, the gradient descent algorithm can adjust the value of each parameter according to the gradient direction of the parameter to minimize the loss function.

[0047] In some embodiments, in order to improve the processing effect of the model on video downstream tasks, after unsupervised training is completed, a small amount of labeled data for the target task can also be obtained to continue supervised training. Among them, the target task is the video downstream task, which may include but is not limited to image classification, object detection, semantic segmentation, action recognition, etc.

[0048] The embodiments of the present disclosure provide a video pre-training method based on contrastive learning. By combining a convolutional neural network and a picture mask self-attention model, it effectively utilizes a large amount of unlabeled video resources and reduces the model training cost. Moreover, it has very good effects in solving various types of video understanding tasks.​

[0049] Continue to refer to Figure 2 , which shows the flow 200 of another embodiment of the convolutional neural network training method according to the present disclosure. The convolutional neural network training method includes the following steps:

[0050] Step 201, extract frames from the first video to obtain a first video frame sequence.

[0051] In this embodiment, by extracting frames from the first video, a first video frame sequence can be obtained.

[0052] Generally, a large number of first videos are searched from a search engine according to preset search conditions. Tools such as ffmpeg can be used to extract frames from the first video by the second to obtain a first video frame sequence.

[0053] Among them, the preset search conditions can be conditions set according to the classification task. For example, if the classification task is to classify animals, videos related to animals can be searched. Another example is that if the classification task is to classify plants, videos related to plants can be searched.

[0054] Among them, ffmpeg is an open-source multimedia processing tool that can be used for various tasks such as synthesis, conversion, and streaming media processing.

[0055] Step 202, extract N first video frames from the first video frame sequence according to a preset extraction method to form a first sample video frame sequence.

[0056] In this embodiment, by extracting N (N is a positive integer) first video frames from the first video frame sequence according to a preset extraction method, a first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ) can be formed.

[0057] Among them, the preset extraction method can include but is not limited to: equidistant extraction, key frame extraction, random extraction, etc.

[0058] It should be noted that after searching for the first video from the search engine, only the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ) needs to be extracted from the first video, and it is not necessary to label the first sample video frame sequence (V t1 , V t2 , V t3 , …, V tN ). Therefore, the first sample video frame sequence (V t1 , Vt2 , V t3 , …, V tN ) is unlabeled data.

[0059] It should be understood that by first extracting frames from the video and then extracting according to the preset extraction method, it can be ensured that the length of each first sample video frame sequence obtained is the same, which is convenient for subsequent model training.

[0060] Step 203: Input the first sample video frame sequence into the convolutional neural network to obtain the first vector sequence.

[0061] In this embodiment, the first sample video frames V t1 , V t2 , V t3 , …, V tN ) in the first sample video frame sequence are sequentially input into the convolutional neural network, and the feature vectors C ti are correspondingly calculated to obtain the first vector sequence (C ti , C t1 , C t2 , C t3 , …, C tN ). Where 1 ≤ i ≤ N and i is a positive integer.

[0062] Among them, the convolutional neural network is a deep learning model that extracts the features of the input image through multiple convolution and pooling operations.

[0063] Step 204: Input the first vector sequence into the picture masked self-attention model according to the masking method to obtain the second vector sequence.

[0064] In this embodiment, the first vectors C t1 , C t2 , C t3 , …, C tN ) in the first vector sequence are sequentially input into the picture masked self-attention model according to the masking method, and the feature vectors S ti are correspondingly calculated to obtain the second vector sequence (S ti , S t1 , S t2 , S t3 , …, S tN ).

[0065] Among them, the picture masked self-attention model is a self-attention mechanism model that can calculate weights for the input vector sequence to more effectively capture the relationships between sequences and generate new sequence representations.

[0066] Among them, the masking method can be to first replace the first vector sequence (C t1 , Ct2 , C t3 , …, C tN ) sorted in the current first vector C ti after the first vector (C ti+1 , …, C tN ), and then calculate the second vector S corresponding to the current first vector C by using the self-attention method ti . ti

[0067] It should be understood that by masking the first vector sorted after the current first vector in the mask manner and then calculating the second vector corresponding to the current first vector by using the self-attention method, the relationship between the current first vector and the first vectors before it can be effectively captured.

[0068] Step 205: Input the first vector sequence into the first projection layer to obtain a third vector sequence, and input the second vector sequence into the second projection layer to obtain a fourth vector sequence.

[0069] In this embodiment, inputting the first vector sequence (C t1 , C t2 , C t3 , …, C tN ) into the first projection layer can obtain a third vector sequence (A t1 , A t2 , A t3 , …, A tN ), and inputting the second vector sequence (S t1 , S t2 , S t3 , …, S tN ) into the second projection layer can obtain a fourth vector sequence (B t1 , B t2 , B t3 , …, B tN ).

[0070] Among them, the third vector sequence (A t1 , A t2 , A t3 , …, A tN ) and the fourth vector sequence (B t1 , B t2 , B t3 , …, B tN ) are in the same vector space and have the same dimension.

[0071] Step 206: Calculate the similarity between the third vector in the third vector sequence and the corresponding fourth vector in the fourth vector sequence.

[0072] In this embodiment, calculate the third vector sequence (A​t1 , A t2 , A t3 , …, A tN ) the third vector A ti and the fourth vector sequence (B t1 , B t2 , B t3 , …, B tN ) the corresponding fourth vector B ti of similarity Sim ti , to obtain the similarity sequence (Sim t1 , Sim t2 , Sim t3 , …, Sim tN ).

[0073] Step 207, based on the similarity, calculate the first loss.

[0074] In this embodiment, based on the similarity sequence (Sim t1 , Sim t2 , Sim t3 , …, Sim tN ), the first loss S can be calculated.

[0075] Here, based on the third vector A ti and the corresponding fourth vector B ti of similarity Sim ti , calculate the loss value at time ti as 1 - Sim ti . Further, calculate the mean of the loss values from time t1 to Tn to obtain the first loss S, and its formula is as follows:

[0076] S = 1 / n[(1 - Sim t1 ) + (1 - Sim t2 ) + … + (1 - Sim tN )].[[]END]]

[0077] It should be understood that by projecting the first vector sequence and the second vector sequence into the same vector space and the same dimension through the projection layer, it is convenient to calculate the similarity between vectors.

[0078] Step 208, based on the first loss, adjust the parameters of the convolutional neural network and the picture mask self-attention model to obtain a convolutional neural network with unsupervised training completed.

[0079] In this embodiment, based on the first loss S, the parameters of the convolutional neural network and the picture mask self-attention model can be adjusted to obtain a convolutional neural network with unsupervised training completed.

[0080] Generally, the gradient descent algorithm is used to adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss S until the first loss S converges, completing the unsupervised training.

[0081] Among them, the gradient descent algorithm is an optimization algorithm used to adjust the parameters of the model to minimize the loss function. Specifically, the gradient descent algorithm can adjust the value of each parameter according to the gradient direction of the parameter to minimize the loss function.

[0082] From Figure 2 it can be seen that compared with the corresponding embodiment in Figure 1 the process 200 of the convolutional neural network training method in this embodiment highlights the steps of obtaining the sample video frame sequence, the picture mask self-attention step, and the loss calculation step. Thus, in the solution described in this embodiment, the video is first frame-extracted, and then extracted according to the preset extraction method, which can ensure that the length of each obtained first sample video frame sequence is the same, facilitating subsequent model training. Masking the first vectors sorted after the current first vector according to the mask method, and then calculating the second vector corresponding to the current first vector using the self-attention method can effectively capture the relationship between the current first vector and the first vectors sorted before it. Projecting the first vector sequence and the second vector sequence into the same vector space and the same dimension through the projection layer facilitates calculating the similarity between the vectors.

[0083] Further referring to Figure 3 which shows the process 300 of another embodiment of the convolutional neural network training method according to the present disclosure. The convolutional neural network training method includes the following steps:

[0084] Step 301, obtaining a first sample video frame sequence.

[0085] Step 302, inputting the first sample video frame sequence into the convolutional neural network to obtain a first vector sequence.

[0086] Step 303, inputting the first vector sequence into the picture mask self-attention model to obtain a second vector sequence.

[0087] Step 304, calculating a first loss based on the first vector sequence and the second vector sequence.

[0088] Step 305, adjusting the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed.

[0089] In this embodiment, the specific operations of steps 301 - 305 have been introduced in detail in steps 101 - 105 of the embodiment shown in Figure 1 and will not be elaborated here.

[0090] Step 306: Obtain the second sample video frame sequence.

[0091] In this embodiment, a small number of second sample video frame sequences can be obtained.

[0092] Generally, a small number of second videos are searched from a search engine according to preset search conditions. N (N is a positive integer) second video frames are extracted from the second videos, which can form the second sample video frame sequence. For example, frame extraction is performed on the second video to obtain the second video frame sequence; N second video frames are extracted from the second video frame sequence according to a preset extraction method to form the second sample video frame sequence. Performing frame extraction on the video first and then extracting according to the preset extraction method can ensure that the length of each obtained second sample video frame sequence is the same, which is convenient for subsequent model training.

[0093] Among them, the preset search conditions can be conditions set according to the classification task. For example, if the classification task is to classify animals, videos related to animals can be searched. Another example is that if the classification task is to classify plants, videos related to plants can be searched.

[0094] Among them, the preset extraction method can include but is not limited to: equidistant extraction, key frame extraction, random extraction, etc.

[0095] It should be noted that after searching for the second video from the search engine, not only the second sample video frame sequence needs to be extracted from the second video, but also the second sample video frame sequence needs to be labeled with the sample video category. Therefore, the second sample video frame sequence is labeled data. Among them, the sample video category can be the category of the second video.

[0096] Step 307: Input the second sample video frame sequence into the convolutional neural network that has completed unsupervised training to obtain the fifth vector sequence.

[0097] In this embodiment, the second sample video frames in the second sample video frame sequence are sequentially input into the convolutional neural network that has completed unsupervised training, and the feature vectors are calculated correspondingly to obtain the fifth vector sequence.

[0098] Among them, the convolutional neural network that has completed unsupervised training is obtained through unsupervised training in steps 301 - 305, and its parameters remain unchanged during the supervised training in steps 306 - 312.

[0099] Step 308: Calculate the mean value of the fifth vectors in the fifth vector sequence to obtain the sample video vector.

[0100] In this embodiment, calculating the mean value of the fifth vectors in the fifth vector sequence can obtain the sample video vector.

[0101] Step 309: Input the sample video vector into a multi-layer perceptron to obtain the predicted video category.

[0102] In this embodiment, by inputting the sample video vector into a multi-layer perceptron (MLP), the predicted video category can be obtained.

[0103] Among them, the multi-layer perceptron is a type of artificial neural network layer composed of multiple fully connected layers. It is usually used to process non-linear correlations and construct non-linear function approximations. Here, the last layer of the multi-layer perceptron is the classification output layer, which can be used to predict the video category.

[0104] Step 310: Calculate the second loss based on the sample video category and the predicted video category.

[0105] In this embodiment, the second loss can be calculated based on the sample video category and the predicted video category.

[0106] Generally, by inputting the sample video category and the predicted video category into a loss function, the second loss can be calculated. Among them, the second loss can be used to characterize the difference between the sample video category and the predicted video category.

[0107] Step 311: Adjust the parameters of the multi-layer perceptron based on the second loss to obtain a multi-layer perceptron with supervised training completed.

[0108] In this embodiment, by adjusting the parameters of the multi-layer perceptron based on the second loss, a multi-layer perceptron with supervised training completed can be obtained.

[0109] Generally, use the gradient descent algorithm to fine-tune the parameters of the multi-layer perceptron based on the second loss until the second loss converges, and the supervised learning is completed.

[0110] Among them, supervised learning is a machine learning method. In the case of labeled data, a model is constructed by associating input features with their corresponding target outputs. Supervised learning is usually used for tasks such as classification, regression, or sequence prediction.

[0111] Among them, the gradient descent algorithm is an optimization algorithm used to adjust the parameters of the model to minimize the loss function. Specifically, the gradient descent algorithm can adjust the value of each parameter according to the gradient direction of the parameter to minimize the loss function.

[0112] Step 312: Deploy a video classification model based on the convolutional neural network with unsupervised training completed and the multi-layer perceptron with supervised training completed.

[0113] In this embodiment, a video classification model can be deployed based on the convolutional neural network with unsupervised training completed and the multi-layer perceptron with supervised training completed.

[0114] Generally, deploying a video classification model online can reduce the vector storage cost.

[0115] From Figure 2 it can be seen that compared with the Figure 1 corresponding embodiment, the process 200 of the convolutional neural network training method in this embodiment highlights the supervised training step. Thus, after the unsupervised training is completed, the scheme described in this embodiment uses a small amount of labeled data for supervised training, which can improve the classification effect of the video classification model and reduce the training cost of the video classification model.

[0116] For ease of understanding, Figure 4 a schematic diagram of the unsupervised training process is shown.

[0117] The first step is to obtain a video frame sequence (V t1 , V t2 , V t3 , …, V tN ).

[0118] For example, search for a video resource collection from a search engine, and tools such as ffmpeg can extract frames from the video resource collection by the second and extract N video frames at equal intervals to obtain a video frame sequence (V t1 , V t2 , V t3 , …, V tN ).

[0119] The second step is to input the video frame sequence (V t1 , V t2 , V t3 , …, V tN ) into the CNN model in sequence to calculate feature vectors and obtain a vector sequence (C t1 , C t2 , C t3 , …, C tN ).

[0120] The third step is to input the vector sequence (C t1 , C t2 , C t3 , …, C tN ) into the picture mask self-attention model in a masked manner in sequence to obtain a vector sequence (S t1 , S t2 , S t3 , …, S tN ).

[0121] The fourth step is to input the vector C t2 into the first projection layer to obtain the vector A t2 , and input the vector St2 Input to the second projection layer to obtain vector B t2 .

[0122] In the fifth step, calculate vector A t2 and vector B t2 for similarity Sim t2 . According to similarity Sim t2 , calculate the loss value at time t2 as 1 - Sim t2 ;

[0123] In the sixth step, calculate the loss value 1 - Sim at other times in the manner described in the first to fifth steps ti , where 1 ≤ i ≤ N and i is a positive integer

[0124] In the seventh step, use the gradient descent algorithm to optimize the loss S = 1 / n[(1 - Sim t1 )+(1 - Sim t2 )+…+(1 - Sim tN )], until the loss value converges to complete unsupervised training

[0125] For ease of understanding Figure 5 , a schematic diagram of the supervised training process is shown

[0126] In the first step, obtain the video frame sequence (1, 2, 3, …, N)

[0127] For example, search for a video resource collection from a search engine. Tools such as ffmpeg can extract frames from the video resource collection by the second and extract N video frames at equal intervals to obtain the video frame sequence (1, 2, 3, …, N). And label the sample video categories for the video frame sequence (1, 2, 3, …, N)

[0128] In the second step, sequentially input the video frame sequence (1, 2, 3, …, N) into the CNN model completed by unsupervised training to calculate the feature vectors and obtain the vector sequence (E1, E2, E3, …, E N )

[0129] In the third step, calculate the average value of the vector sequence (E1, E2, E3, …, E N ) to obtain the video vector V E .

[0130] In the fourth step, input the video vector V E into the MLP layer, and the output layer in the MLP layer outputs the predicted video category

[0131] In the fifth step, calculate the loss based on the sample video category and the predicted video category

[0132] In the sixth step, the gradient descent algorithm is used to optimize the loss, and the parameters of the MLP layer are fine-tuned to complete the supervised training.

[0133] In the seventh step, the unsupervised trained CNN model and the supervised trained MLP layer are deployed online for inference tasks.

[0134] Further reference is made to Figure 6 , which shows a flow 600 of an embodiment of the video classification method according to the present disclosure. The video classification method includes the following steps:

[0135] Step 601, obtain the video to be classified.

[0136] In this embodiment, the video to be classified is obtained. Among them, the video to be classified can be a video that needs to be classified according to the video content.

[0137] Step 602, extract the sequence of video frames to be classified from the video to be classified.

[0138] In this embodiment, the sequence of video frames to be classified is extracted from the video to be classified.

[0139] Generally, N (N is a positive integer) video frames are extracted from the video to be classified, and the sequence of video frames to be classified can be formed. For example, tools such as ffmpeg can be used to extract frames from the video to be classified by seconds, and N video frames are extracted according to a preset extraction method to obtain the sequence of video frames to be classified.

[0140] Among them, the preset extraction method can include but is not limited to: equal interval extraction, key frame extraction, random extraction, etc.

[0141] Step 603, input the sequence of video frames to be classified into the video classification model to obtain the category of the video to be classified.

[0142] In this embodiment, when the sequence of video frames to be classified is input into the video classification model, the category of the video to be classified can be obtained. Among them, the video classification model can be trained by using the method shown in Figure 3 , which will not be elaborated here.

[0143] For the video classification method provided in the embodiment of the present disclosure, the video classification model first performs unsupervised training using a large amount of unlabeled data, and then performs supervised training using a small amount of labeled data, thereby improving the classification effect of the video classification model.

[0144] Further reference is made to Figure 7 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a convolutional neural network training device. This device embodiment corresponds to the method embodiment shown in Figure 1 , and this device can be specifically applied to various electronic devices.

[0145] As shown Figure 7 in the figure, the convolutional neural network training device 700 of this embodiment may include: a first acquisition module 701, a first input module 702, a second input module 703, a first calculation module 704, and a first adjustment module 705. Among them, the first acquisition module 701 is configured to acquire a first sample video frame sequence; the first input module 702 is configured to input the first sample video frame sequence into the convolutional neural network to obtain a first vector sequence; the second input module 703 is configured to input the first vector sequence into the picture mask self-attention model to obtain a second vector sequence; the first calculation module 704 is configured to calculate a first loss based on the first vector sequence and the second vector sequence; the first adjustment module 705 is configured to adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed.

[0146] In this embodiment, in the convolutional neural network training device 700: the specific processing of the first acquisition module 701, the first input module 702, the second input module 703, the first calculation module 704, and the first adjustment module 705 and the technical effects brought by them can respectively refer to Figure 1 the relevant descriptions of steps 101-105 in the corresponding embodiment, which will not be elaborated here.

[0147] In some optional implementation manners of this embodiment, the first acquisition module 701 is further configured to: extract frames from the first video to obtain a first video frame sequence; extract N first video frames from the first video frame sequence according to a preset extraction method to form a first sample video frame sequence, where N is a positive integer, and the preset extraction method includes at least one of the following: equal-interval extraction, key frame extraction, and random extraction.

[0148] In some optional implementation manners of this embodiment, the second input module 703 is further configured to: input the first vector sequence into the picture mask self-attention model in a masked manner to obtain a second vector sequence, where the masked manner is to first replace the first vectors sorted after the current first vector in the first vector sequence with a preset value, and then calculate the second vector corresponding to the current first vector by using the self-attention method.

[0149] In some optional implementation manners of this embodiment, the first calculation module 704 is further configured to: input the first vector sequence into a first projection layer to obtain a third vector sequence, and input the second vector sequence into a second projection layer to obtain a fourth vector sequence, where the third vector sequence and the fourth vector sequence are in the same vector space and have the same dimension; calculate the similarity between the third vector in the third vector sequence and the corresponding fourth vector in the fourth vector sequence; calculate the first loss based on the similarity.

[0150] In some alternative implementation manners of this embodiment, the convolutional neural network training device 700 further includes: a second acquisition module configured to acquire a second sample video frame sequence, where the second sample video frame sequence is labeled with a sample video category; a third input module configured to input the second sample video frame sequence into the convolutional neural network that has completed unsupervised training to obtain a fifth vector sequence; a second calculation module configured to calculate the mean value of the fifth vectors in the fifth vector sequence to obtain a sample video vector; a fourth input module configured to input the sample video vector into a multi-layer perceptron to obtain a predicted video category; a third calculation module configured to calculate a second loss based on the sample video category and the predicted video category; a second adjustment module configured to adjust the parameters of the multi-layer perceptron based on the second loss to obtain a multi-layer perceptron that has completed supervised training; and a deployment module configured to deploy a video classification model based on the convolutional neural network that has completed unsupervised training and the multi-layer perceptron that has completed supervised training.

[0151] In some alternative implementation manners of this embodiment, the second acquisition module is further configured to: extract frames from the second video to obtain a second video frame sequence; extract N second video frames from the second video frame sequence according to a preset extraction method to form a second sample video frame sequence, where N is a positive integer, and the preset extraction method includes at least one of the following: equidistant extraction, key frame extraction, and random extraction.

[0152] Further referring to Figure 8 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video classification device. This device embodiment corresponds to the Figure 6 method embodiment shown, and this device can be specifically applied to various electronic devices.

[0153] As shown in Figure 8 , the video classification device 800 in this embodiment may include: an acquisition module 801, an extraction module 802, and a classification module 803. Among them, the acquisition module 801 is configured to acquire a video to be classified; the extraction module 802 is configured to extract a sequence of video frames to be classified from the video to be classified; the classification module 803 is configured to input the sequence of video frames to be classified into a video classification model to obtain the category of the video to be classified, where the video classification model is trained using the Figure 7 device shown.

[0154] In this embodiment, in the video classification device 800: the specific processing of the acquisition module 801, the extraction module 802, and the classification module 803 and the technical effects brought by them can respectively refer to the Figure 6 relevant descriptions of steps 601-603 in the corresponding embodiments, which will not be elaborated here.

[0155] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0157] Figure 9 FIG. shows a schematic block diagram of an exemplary electronic device 900 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0158] As Figure 9 shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0159] A plurality of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disc, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes the various methods and processes described above, such as the convolutional neural network training method or the video classification method. For example, in some embodiments, the convolutional neural network training method or the video classification method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the convolutional neural network training method or the video classification method described above can be executed. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the convolutional neural network training method or the video classification method by any other suitable means (e.g., by means of firmware).

[0161] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0165] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0167] It should be understood that the various forms of processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and no limitations are imposed herein.

[0168] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A method for training a convolutional neural network, comprising: Obtaining a first sample video frame sequence; Inputting the first sample video frame sequence into a convolutional neural network to obtain a first vector sequence; Inputting the first vector sequence into a picture mask self-attention model to obtain a second vector sequence; Calculating a first loss based on the first vector sequence and the second vector sequence; Adjusting the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed; Obtaining a second sample video frame sequence, wherein the second sample video frame sequence is labeled with a sample video category; Inputting the second sample video frame sequence into the convolutional neural network with unsupervised training completed to obtain a fifth vector sequence; Calculating the mean of the fifth vectors in the fifth vector sequence to obtain a sample video vector; Inputting the sample video vector into a multi-layer perceptron to obtain a predicted video category; Calculating a second loss based on the sample video category and the predicted video category; Adjusting the parameters of the multi-layer perceptron based on the second loss to obtain a multi-layer perceptron with supervised training completed; Deploying a video classification model based on the convolutional neural network with unsupervised training completed and the multi-layer perceptron with supervised training completed.

2. The method according to claim 1, wherein The obtaining the first sample video frame sequence includes: Extracting frames from a first video to obtain a first video frame sequence; Extracting N first video frames from the first video frame sequence according to a preset extraction method to form the first sample video frame sequence, where N is a positive integer, and the preset extraction method includes at least one of the following: equidistant extraction, key frame extraction, and random extraction.

3. The method according to claim 1, wherein, The inputting the first vector sequence into a picture mask self-attention model to obtain a second vector sequence includes: Inputting the first vector sequence into the picture mask self-attention model in a masked manner to obtain the second vector sequence, where the masking method is to first replace the first vectors sorted after the current first vector in the first vector sequence with a preset value, and then calculate the second vector corresponding to the current first vector by means of self-attention.

4. The method according to claim 1, wherein The calculating the first loss based on the first vector sequence and the second vector sequence includes: Inputting the first vector sequence into a first projection layer to obtain a third vector sequence, and inputting the second vector sequence into a second projection layer to obtain a fourth vector sequence, where the third vector sequence and the fourth vector sequence are in the same vector space and have the same dimension; Calculating the similarity between the third vectors in the third vector sequence and the corresponding fourth vectors in the fourth vector sequence; Calculating the first loss based on the similarity.

5. The method according to claim 1, wherein The obtaining the second sample video frame sequence includes: Extracting frames from a second video to obtain a second video frame sequence; Extracting N second video frames from the second video frame sequence according to a preset extraction method to form the second sample video frame sequence, where N is a positive integer, and the preset extraction method includes at least one of the following: equidistant extraction, key frame extraction, and random extraction.

6. A video classification method, comprising: Obtaining a video to be classified; Extracting a sequence of video frames to be classified from the video to be classified; Inputting the sequence of video frames to be classified into a video classification model to obtain the category of the video to be classified, wherein the video classification model is trained by using the method according to any one of claims 1-5.

7. A convolutional neural network training device, comprising: A first obtaining module configured to obtain a first sequence of sample video frames; A first input module configured to input the first sequence of sample video frames into a convolutional neural network to obtain a first sequence of vectors; A second input module configured to input the first sequence of vectors into a picture mask self-attention model to obtain a second sequence of vectors; A first calculation module configured to calculate a first loss based on the first sequence of vectors and the second sequence of vectors; A first adjustment module configured to adjust the parameters of the convolutional neural network and the picture mask self-attention model based on the first loss to obtain a convolutional neural network with unsupervised training completed; A second obtaining module configured to obtain a second sequence of sample video frames, wherein the second sequence of sample video frames is labeled with a sample video category; A third input module configured to input the second sequence of sample video frames into the convolutional neural network with unsupervised training completed to obtain a fifth sequence of vectors; A second calculation module configured to calculate the mean value of the fifth vectors in the fifth sequence of vectors to obtain a sample video vector; A fourth input module configured to input the sample video vector into a multi-layer perceptron to obtain a predicted video category; A third calculation module configured to calculate a second loss based on the sample video category and the predicted video category; A second adjustment module configured to adjust the parameters of the multi-layer perceptron based on the second loss to obtain a multi-layer perceptron with supervised training completed; A deployment module configured to deploy a video classification model based on the convolutional neural network with unsupervised training completed and the multi-layer perceptron with supervised training completed.

8. The apparatus according to claim 7, wherein The first obtaining module is further configured to: Extract frames from a first video to obtain a first sequence of video frames; Extract N first video frames from the first sequence of video frames according to a preset extraction method to form the first sequence of sample video frames, wherein N is a positive integer, and the preset extraction method includes at least one of the following: equidistant extraction, key frame extraction, and random extraction.

9. The apparatus according to claim 7, wherein, The second input module is further configured to: Input the first sequence of vectors into the picture mask self-attention model in a masked manner to obtain the second sequence of vectors, wherein the masked manner is to first replace the first vectors sorted after the current first vector in the first sequence of vectors with a preset value, and then calculate the second vector corresponding to the current first vector by using the self-attention method.

10. The device according to claim 7, wherein The first calculation module is further configured to: Input the first vector sequence into a first projection layer to obtain a third vector sequence, and input the second vector sequence into a second projection layer to obtain a fourth vector sequence, where the third vector sequence and the fourth vector sequence are in the same vector space and have the same dimension; Calculate the similarity between a third vector in the third vector sequence and a corresponding fourth vector in the fourth vector sequence; Calculate the first loss based on the similarity; 11. The apparatus according to claim 7, wherein, The second acquisition module is further configured to: Extract frames from a second video to obtain a second video frame sequence; Extract N second video frames from the second video frame sequence according to a preset extraction method to form the second sample video frame sequence, where N is a positive integer, and the preset extraction method includes at least one of the following: equidistant extraction, key frame extraction, and random extraction.

12. A video classification device, comprising: An acquisition module configured to acquire a video to be classified; An extraction module configured to extract a sequence of video frames to be classified from the video to be classified; A classification module configured to input the sequence of video frames to be classified into a video classification model to obtain the category of the video to be classified, where the video classification model is trained using the device according to any one of claims 7-11.

13. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-5 or the method according to claim 6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of claims 1-5 or the method according to claim 6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-5 or the method according to claim 6.

Images