Model training method, video classification method, device and equipment

Abstract

The present disclosure provides a model training method, video classification method, device, and equipment, which relate to the field of artificial intelligence technology, and in particular to technical fields such as deep learning and video processing. A specific implementation scheme is as follows: unsupervised training is performed on a feature extraction model based on an original data set; wherein the feature extraction model includes an image encoder and a text encoder; supervised training is performed on the trained feature extraction model based on a labeled data set; the original data set is input into the feature extraction model obtained through supervised training, and a target data set is selected from the original data set based on the similarity between the image features output by the image encoder and the text features output by the text encoder; and unsupervised training is performed again on the feature extraction model obtained through supervised training based on the target data set to obtain a trained feature extraction model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning and video processing. Background Art

[0002] With the development of artificial intelligence (AI) technology, deep learning models have gained widespread application, for example, in various video processing scenarios. Training deep learning models often requires a large amount of sample data. The larger the sample data volume, the better the learning effect of the deep learning model. However, if the sample data contains excessive noise, it can seriously affect the learning effect of the deep learning model. Summary of the Invention

[0003] The present disclosure provides a method and device for training a feature extraction model, a method and device for training a video classification model, a video classification method and device, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a method for training a feature extraction model is provided, comprising: performing unsupervised training on the feature extraction model based on an original data set; wherein the original data set comprises several first sample video frames and text descriptions corresponding to the first sample video frames, the feature extraction model comprises an image encoder and a text encoder, the image encoder is used to encode the first sample video frames to obtain corresponding image features, and the text encoder is used to encode the text descriptions to obtain corresponding text features; performing supervised training on the trained feature extraction model based on an annotated data set; wherein the annotated data set comprises a second sample video frame, image features annotated for the second sample video frame, text descriptions corresponding to the second sample video frame, and text features annotated for the text descriptions; inputting the original data set into the feature extraction model obtained by supervised training, and screening a target data set from the original data set based on the similarity between the image features output by the image encoder and the text features output by the text encoder; and performing unsupervised training on the feature extraction model obtained by supervised training again based on the target data set to obtain a trained feature extraction model.

[0005] According to another aspect of the present disclosure, a method for training a video classification model is provided, comprising: obtaining a training sample; wherein the training sample comprises a sample video and a classification label of the sample video; inputting at least one video frame in the sample video into a video classification model for classification prediction to obtain a classification result of the sample video; wherein the video classification model comprises an image encoder and a fully connected layer connected in sequence, and the image encoder is an image encoder in a feature extraction model trained using the training method of the above-mentioned feature extraction model; calculating a loss based on the classification label and the classification result, and adjusting parameters of the fully connected layer in the video classification model based on the loss until a convergence condition is met; and outputting the trained video classification model.

[0006] According to another aspect of the present disclosure, a video classification method is provided, comprising: obtaining a video to be classified; inputting at least one video frame in the video to be classified into a video classification model for classification prediction, and obtaining a classification result of the video to be classified; wherein the video classification model is trained according to a training method for a video classification model.

[0007] According to another aspect of the present disclosure, a training device for a feature extraction model is provided, comprising: a first training module for performing unsupervised training on the feature extraction model based on an original data set; wherein the original data set comprises a plurality of first sample video frames and text descriptions corresponding to the first sample video frames, the feature extraction model comprises an image encoder and a text encoder, the image encoder is configured to encode the first sample video frames to obtain corresponding image features, and the text encoder is configured to encode the text descriptions to obtain corresponding text features; a second training module for performing supervised training on the trained feature extraction model based on an annotated data set; wherein the annotated data set comprises a second sample video frame, image features annotated for the second sample video frame, text descriptions corresponding to the second sample video frame, and text features annotated for the text descriptions; a data screening module for inputting the original data set into the feature extraction model obtained through supervised training, and screening a target data set from the original data set based on the similarity between the image features output by the image encoder and the text features output by the text encoder; and a third training module for performing unsupervised training again on the feature extraction model obtained through supervised training based on the target data set to obtain a trained feature extraction model.

[0008] According to another aspect of the present disclosure, a training device for a video classification model is provided, comprising: a first acquisition module for acquiring training samples; wherein the training samples include sample videos and classification labels of the sample videos; a first prediction module for inputting at least one video frame in the sample video into a video classification model for classification prediction to obtain a classification result of the sample video; wherein the video classification model includes an image encoder and a fully connected layer connected in sequence, and the image encoder is an image encoder in a feature extraction model obtained by using the training device of the above-mentioned feature extraction model; a parameter processing module for calculating a loss based on the classification label and the classification result, and adjusting the parameters of the fully connected layer in the video classification model based on the loss until a convergence condition is met; and a model output module for outputting a trained video classification model.

[0009] According to another aspect of the present disclosure, a video classification device is provided, comprising: a second acquisition module for acquiring a video to be classified; a second prediction module for inputting at least one video frame in the video to be classified into a video classification model for classification prediction, to obtain a classification result of the video to be classified; wherein the video classification model is trained according to the above-mentioned video classification model training device.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned feature extraction model training method, the above-mentioned video classification model training method, or the above-mentioned video classification method.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned feature extraction model training method, the above-mentioned video classification model training method or the above-mentioned video classification method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned feature extraction model training method, the above-mentioned video classification model training method, or the above-mentioned video classification method.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flowchart of a method for training a feature extraction model according to an embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of a method for training a feature extraction model according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a method for training a feature extraction model according to another embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a training device for a feature extraction model according to an embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of a method for training a video classification model according to an embodiment of the present disclosure;

[0020] Figure 6 is a structural diagram of a video classification model provided according to an embodiment of the present disclosure;

[0021] Figure 7 2 is a schematic diagram of a training device for a video classification model according to an embodiment of the present disclosure;

[0022] Figure 8 is a flowchart of a video classification method according to an embodiment of the present disclosure;

[0023] Figure 9 is a schematic diagram of a video classification device according to an embodiment of the present disclosure;

[0024] Figure 10 is a block diagram of an electronic device for implementing the method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to an embodiment of the present disclosure, an embodiment of a method for training a feature extraction model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Figure 1 is a flow chart of a method for training a feature extraction model according to an embodiment of the present disclosure, such as Figure 1 As shown, the training method includes the following steps S101 to S104:

[0029] Step S101: Perform unsupervised training on a feature extraction model based on an original dataset. The original dataset includes a plurality of first sample video frames and text descriptions corresponding to the first sample video frames. The feature extraction model includes an image encoder and a text encoder. The image encoder is configured to encode the first sample video frames to obtain corresponding image features, and the text encoder is configured to encode the text descriptions to obtain corresponding text features.

[0030] In a specific implementation, the text description corresponding to the first sample video frame may be a description of the content of the video where the first sample video frame is located, for example, the title of the video.

[0031] The sample video can be intercepted and multiple first sample video frames can be selected from the intercepted video frame sequence. Assume that the original data set includes N first sample video frames and corresponding N text descriptions, where N is an integer greater than 1. In the process of unsupervised training of the feature extraction model, the first sample video frame is input into the image encoder of the feature extraction model to obtain image features, and the text description corresponding to the first sample video frame is input into the text encoder of the feature extraction model to obtain text features, so as to maximize the similarity between the N paired image features and text features, while minimizing N 2 -The similarity between N unpaired image features and text features is used as the target to train the feature extraction model.

[0032] In practical applications, the image features can also be called image vectors, and the text features can also be called text vectors. Figure 2 In the example shown, the original dataset includes 3 first sample video frames and corresponding 3 text descriptions. The 3 first sample video frames are input into the image encoder to obtain image vectors F1, F2 and F3 respectively, and the 3 text descriptions are input into the text encoder to obtain text vectors A1, A2 and A3 respectively, wherein the image vector F1 is paired with the text vector A1, the image vector F2 is paired with the text vector A2, and the image vector F3 is paired with the text vector A3. By calculating the cross entropy loss, the feature extraction model including the image encoder and the text encoder is trained with the goal of maximizing the similarity between F1 and A1, F2 and A2, and F3 and A3, while minimizing the similarity between A2 and F1, A3 and F1, A1 and F2, A3 and F2, A1 and F3, and A2 and F3.

[0033] Step S102: Supervised training is performed on the trained feature extraction model based on the annotated dataset, wherein the annotated dataset includes a second sample video frame, image features annotated for the second sample video frame, a text description corresponding to the second sample video frame, and text features annotated for the text description.

[0034] It should be noted that the data volume of the original data set is usually much larger than that of the annotated data set. In this embodiment, the data volume of the original data set can reach tens of millions.

[0035] In step S102, the annotated dataset is input into the feature extraction model trained in step S101. Specifically, the second sample video frame is input into the image encoder in the feature extraction model to obtain predicted image features, and the text description corresponding to the second sample video frame is input into the text encoder in the feature extraction model to obtain predicted text features. The loss is calculated based on the predicted image features and the annotated image features as well as the predicted text features and the annotated text features, and the parameters of the feature extraction model are adjusted based on the calculated loss until the convergence condition is met.

[0036] Step S103: input the original data set into the feature extraction model obtained through supervised training, and filter the target data set from the original data set according to the similarity between the image features output by the image encoder and the text features output by the text encoder.

[0037] The similarity may be cosine similarity, Euclidean distance, Manhattan distance, or the like.

[0038] The original data set is input into the feature extraction model trained in step S102. The higher the similarity between the output image features and text features, the smaller the noise of the corresponding sample data, and the lower the similarity, the greater the noise of the corresponding sample data. Screening the original data set by similarity can effectively alleviate the sample noise in the large-scale original data set, thereby obtaining a target data set with even smaller sample noise.

[0039] Step S104: performing unsupervised training again on the feature extraction model obtained through supervised training based on the target data set to obtain a trained feature extraction model.

[0040] Retraining the feature extraction model obtained by step S102 based on a target data set with less sample noise can effectively improve the learning effect of the feature extraction model.

[0041] Among them, the conditions for stopping training in the above steps S101, S102 and S104 can be set according to actual conditions, for example, the loss of each iteration does not decrease significantly, or the loss of the iteration reaches a preset loss, etc.

[0042] like Figure 3 As shown, the training method of the feature extraction model provided by the embodiment of the present disclosure first optimizes the feature extraction model using the original data set, and this optimization process is unsupervised training. Secondly, the feature extraction model is optimized using the labeled data set, and this optimization process is supervised training. Then, the feature extraction model obtained by supervised training is used to screen the original data set to obtain the target data set. Finally, the target data set is used to optimize the feature extraction model, and this optimization process is unsupervised training.

[0043] It should be noted that in order to further improve the learning effect of the feature extraction model, the target data set can be further screened, that is, the feature extraction model trained in step S104 is optimized again using the labeled data set, and the target data set is screened using the optimized feature extraction model to obtain a data set with smaller sample noise, until the sample noise of the data set meets the requirements, and finally the optimized feature extraction model is finally trained using the data set that meets the requirements to obtain a trained feature extraction model.

[0044] In an optional embodiment, step S103 specifically includes: if the similarity between the image feature output by the image encoder and the text feature output by the text encoder is greater than a preset value, determining the first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original dataset as the target dataset. The preset value can be set according to actual conditions, for example, 60%.

[0045] In this embodiment, the target data set is filtered from the original data set based on whether the similarity is greater than a preset value. Therefore, the sample noise of the filtered target data set is directly related to the preset value, wherein the larger the preset value is, the smaller the sample noise of the target data set is, and the smaller the preset value is, the greater the sample noise of the target data set is.

[0046] In another optional embodiment, the above step S103 specifically includes the following steps S103a to S103c:

[0047] Step S103a: sort the similarities between the image features output by the image encoder and the text features output by the text encoder from high to low.

[0048] Step S103b: Select a preset number of image features and text features corresponding to similarities that are ranked high, wherein the preset number can be set according to actual conditions.

[0049] Step S103c: Determine the first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original data set as the target data set.

[0050] In this embodiment, the target data set is filtered from the original data set based on the similarities of the preset number of top rankings. Therefore, the sample noise of the filtered target data set is relatively correlated with the preset number, wherein the larger the preset number, the larger the sample noise of the target data set, and the smaller the preset number, the smaller the sample noise of the target data set.

[0051] According to an embodiment of the present disclosure, there is also provided an embodiment of a training device for a feature extraction model, wherein: Figure 4Schematic diagram of a training device for a feature extraction model according to an embodiment of the present disclosure, comprising a first training module 401, a second training module 402, a data screening module 403, and a third training module 404. The first training module 401 is configured to perform unsupervised training on the feature extraction model based on an original dataset, wherein the original dataset comprises a plurality of first sample video frames and text descriptions corresponding to the first sample video frames. The feature extraction model comprises an image encoder and a text encoder, wherein the image encoder is configured to encode the first sample video frames to obtain corresponding image features, and the text encoder is configured to encode the text descriptions to obtain corresponding text features. The second training module 402 is configured to perform supervised training on the trained feature extraction model based on an annotated dataset, wherein the annotated dataset comprises second sample video frames, image features annotated for the second sample video frames, text descriptions corresponding to the second sample video frames, and text features annotated for the text descriptions. The data screening module 403 is configured to input the original dataset into the feature extraction model obtained through supervised training and screen a target dataset from the original dataset based on the similarity between the image features output by the image encoder and the text features output by the text encoder. The third training module 404 is used to perform unsupervised training again on the feature extraction model obtained through supervised training based on the target data set to obtain a trained feature extraction model.

[0052] It should be noted that the above-mentioned first training module 401, second training module 402, data screening module 403 and third training module 404 correspond to steps S101 to S104 in the above-mentioned embodiment. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment.

[0053] In an optional embodiment, the data screening module is specifically used to determine the first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original data set as the target data set when the similarity between the image feature output by the image encoder and the text feature output by the text encoder is greater than a preset value.

[0054] In another optional embodiment, the data screening module is specifically used to sort the similarities between the image features output by the image encoder and the text features output by the text encoder from high to low, and select the image features and text features corresponding to a preset number of similarities that are ranked high, and determine the first sample video frame corresponding to the image features and the text description corresponding to the text features in the original data set as the target data set.

[0055] According to an embodiment of the present disclosure, an embodiment of a method for training a video classification model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] Figure 5 is a flow chart of a method for training a video classification model according to an embodiment of the present disclosure, such as Figure 5 As shown, the training method includes the following steps S501 to S504:

[0057] Step S501: Obtain training samples, wherein the training samples include sample videos and classification labels of the sample videos.

[0058] Step S502: Input at least one video frame from the sample video into a video classification model for classification prediction to obtain a classification result for the sample video. The video classification model includes an image encoder and a fully connected layer connected in sequence, wherein the image encoder is the image encoder in the feature extraction model trained using the training method for the feature extraction model described above.

[0059] Among them, the feature extraction model trained by the training method of the above-mentioned feature extraction model is used to construct a video classification model. Specifically, the video classification model is constructed using the image encoder in the feature extraction model, and a fully connected layer is connected behind the image encoder to obtain a video classification model.

[0060] Step S503: Calculate the loss according to the classification label and the classification result, and adjust the parameters of the fully connected layer in the video classification model according to the loss until the convergence condition is met.

[0061] Step S504: output the trained video classification model.

[0062] In the disclosed embodiments, during the training of the video classification model, only the parameters of the fully connected layer are adjusted, and the parameters of the image encoder are not adjusted. Because the image encoder has already learned the image features of a large number of video frames during the training of the feature extraction model, directly utilizing the image encoder parameters during the training of the video classification model not only provides a good understanding of the image features of the video frames, but also effectively reduces the training cost of the video classification model.

[0063] In order to improve the learning effect of the video classification model, in an optional embodiment, the video classification model further includes a convolutional layer, such as Figure 6As shown, it is provided between the image encoder and the fully connected layer. In this embodiment, the above step S503 specifically includes: adjusting the parameters of the convolutional layer and the fully connected layer in the video classification model according to the loss.

[0064] In a specific implementation, the number of the convolutional layers can be one or more. During the training process of the video classification model, only the parameters of the convolutional layers and the fully connected layers are adjusted, and the parameters of the image encoder are not adjusted.

[0065] According to an embodiment of the present disclosure, there is also provided an embodiment of a training device for a video classification model, wherein: Figure 7 It is a schematic diagram of a training device for a video classification model according to an embodiment of the present disclosure, and the training device includes a first acquisition module 701, a first prediction module 702, a parameter processing module 703, and a model output module 704. The first acquisition module 701 is used to obtain training samples; wherein, the training samples include sample videos and classification labels of the sample videos. The first prediction module 702 is used to input at least one video frame in the sample video into a video classification model for classification prediction to obtain a classification result of the sample video; wherein, the video classification model includes an image encoder and a fully connected layer connected in sequence, and the image encoder is an image encoder in a feature extraction model obtained by the training device of the above-mentioned feature extraction model. The parameter processing module 703 is used to calculate the loss based on the classification label and the classification result, and adjust the parameters of the fully connected layer in the video classification model based on the loss until the convergence condition is met. The model output module 704 is used to output the trained video classification model.

[0066] It should be noted that the first acquisition module 701, the first prediction module 702, the parameter processing module 703 and the model output module 704 correspond to steps S501 to S504 in the above embodiment. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment.

[0067] In an optional embodiment, the video classification model further includes a convolutional layer disposed between the image encoder and the fully connected layer. In this embodiment, the parameter processing module is specifically configured to adjust parameters of the convolutional layer and the fully connected layer in the video classification model based on the loss.

[0068] According to an embodiment of the present disclosure, an embodiment of a video classification method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] Figure 8 is a flow chart of a video classification method according to an embodiment of the present disclosure. Figure 8 As shown, the training method includes the following steps S801 to S802:

[0070] Step S801: Obtain the video to be classified.

[0071] Step S802: Input at least one video frame in the video to be classified into a video classification model for classification prediction to obtain a classification result of the video to be classified. The video classification model is trained according to the above-mentioned video classification model training method.

[0072] In a specific implementation, several video frames can be captured from the video to be classified and input into a video classification model for classification prediction, thereby obtaining a classification result for the video to be classified. The video classification model can classify the video to be classified into different categories, such as content classification, which can include food, film, and travel, etc., and quality classification, which can include high quality, medium quality, and low quality, etc.

[0073] In the disclosed embodiment, since the training effect of the training method of the above-mentioned video classification model is improved, the learning effect of the trained video classification model is also improved, that is, the classification results of videos classified using the video classification model are more accurate.

[0074] According to an embodiment of the present disclosure, a video classification device embodiment is also provided, wherein: Figure 9 1 is a schematic diagram of a video classification device according to an embodiment of the present disclosure, comprising a second acquisition module 901 and a second prediction module 902. The second acquisition module 901 is configured to acquire a video to be classified. The second prediction module 902 is configured to input at least one video frame from the video to be classified into a video classification model for classification prediction, thereby obtaining a classification result for the video to be classified; wherein the video classification model is trained using the aforementioned video classification model training device.

[0075] It should be noted that the second acquisition module 901 and the second prediction module 902 correspond to steps S801 to S802 in the above embodiment. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment.

[0076] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0077] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0078] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0079] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0080] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0081] The computing unit 1001 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method of the feature extraction model, the training method of the video classification model, or the video classification method. For example, in some embodiments, the training method of the feature extraction model, the training method of the video classification model, or the video classification method can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the feature extraction model training method, video classification model training method, or video classification method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the feature extraction model training method, video classification model training method, or video classification method described above in any other appropriate manner (e.g., by means of firmware).

[0082] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0083] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0086] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0087] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0088] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0089] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a feature extraction model, comprising: Performing unsupervised training on a feature extraction model based on an original dataset; wherein the original dataset includes a plurality of first sample video frames and text descriptions corresponding to the first sample video frames, and the feature extraction model includes an image encoder and a text encoder, wherein the image encoder is used to encode the first sample video frames to obtain corresponding image features, and the text encoder is used to encode the text descriptions to obtain corresponding text features; Performing supervised training on the trained feature extraction model based on a labeled dataset; wherein the labeled dataset includes a second sample video frame, image features labeled for the second sample video frame, a text description corresponding to the second sample video frame, and text features labeled for the text description; Inputting the original data set into a feature extraction model obtained through supervised training, and screening a target data set from the original data set based on the similarity between the image features output by the image encoder and the text features output by the text encoder; Performing unsupervised training again on the feature extraction model obtained through supervised training based on the target data set to obtain a trained feature extraction model; The step of filtering the target data set from the original data set according to the similarity between the image features output by the image encoder and the text features output by the text encoder comprises: If the similarity between the image feature output by the image encoder and the text feature output by the text encoder is greater than a preset value, determining the first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original data set as the target data set; or, The similarities between the image features output by the image encoder and the text features output by the text encoder are sorted from high to low; Selecting image features and text features corresponding to a preset number of similarities that are ranked top; The first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original data set are determined as a target data set.

2. The training method according to claim 1, wherein: The text description is the title.

3. A method for training a video classification model, comprising: Obtaining a training sample; wherein the training sample includes a sample video and a classification label of the sample video; Inputting at least one video frame in the sample video into a video classification model for classification prediction to obtain a classification result of the sample video; wherein the video classification model includes an image encoder and a fully connected layer connected in sequence, and the image encoder is an image encoder in a feature extraction model trained using the training method according to claim 1 or 2; Calculating a loss based on the classification label and the classification result, and adjusting parameters of a fully connected layer in the video classification model based on the loss until a convergence condition is met; Output the trained video classification model.

4. The training method according to claim 3, wherein: The video classification model further includes a convolutional layer, which is provided between the image encoder and the fully connected layer; The adjusting the parameters of the fully connected layer in the video classification model according to the loss includes: adjusting the parameters of the convolutional layer and the fully connected layer in the video classification model according to the loss.

5. A video classification method, comprising: Get the video to be classified; Inputting at least one video frame in the video to be classified into a video classification model for classification prediction to obtain a classification result of the video to be classified; The video classification model is trained according to the training method according to claim 3 or 4.

6. A training device for a feature extraction model, comprising: a first training module configured to perform unsupervised training on a feature extraction model based on an original dataset; wherein the original dataset includes a plurality of first sample video frames and text descriptions corresponding to the first sample video frames; and the feature extraction model includes an image encoder and a text encoder, wherein the image encoder is configured to encode the first sample video frames to obtain corresponding image features, and the text encoder is configured to encode the text descriptions to obtain corresponding text features. a second training module, configured to perform supervised training on the trained feature extraction model based on a labeled dataset, wherein the labeled dataset includes a second sample video frame, image features labeled for the second sample video frame, a text description corresponding to the second sample video frame, and text features labeled for the text description; a data screening module, configured to input the original data set into a feature extraction model obtained through supervised training, and screen a target data set from the original data set based on the similarity between the image features output by the image encoder and the text features output by the text encoder; A third training module is used to perform unsupervised training on the feature extraction model obtained by supervised training based on the target data set to obtain a trained feature extraction model; The data screening module is specifically configured to, when the similarity between the image feature output by the image encoder and the text feature output by the text encoder is greater than a preset value, determine the first sample video frame corresponding to the image feature and the text description corresponding to the text feature in the original data set as the target data set; Alternatively, the data screening module is specifically used to sort the similarities between the image features output by the image encoder and the text features output by the text encoder from high to low, and select the image features and text features corresponding to a preset number of similarities that are ranked high, and determine the first sample video frame corresponding to the image features and the text description corresponding to the text features in the original data set as the target data set.

7. The training device according to claim 6, wherein: The text description is the title.

8. A training device for a video classification model, comprising: A first acquisition module is configured to acquire a training sample; wherein the training sample includes a sample video and a classification label of the sample video; a first prediction module, configured to input at least one video frame in the sample video into a video classification model for classification prediction, thereby obtaining a classification result of the sample video; wherein the video classification model comprises an image encoder and a fully connected layer connected in sequence, and the image encoder is an image encoder in a feature extraction model obtained by using the training device according to claim 6 or 7; a parameter processing module, configured to calculate a loss based on the classification label and the classification result, and adjust parameters of a fully connected layer in the video classification model based on the loss until a convergence condition is met; The model output module is used to output the trained video classification model.

9. The training device according to claim 8, wherein: The video classification model further includes a convolutional layer, which is provided between the image encoder and the fully connected layer; The parameter processing module is specifically used to adjust the parameters of the convolutional layer and the fully connected layer in the video classification model according to the loss.

10. A video classification device comprising: The second acquisition module is used to acquire the video to be classified; A second prediction module is used to input at least one video frame in the video to be classified into a video classification model to perform classification prediction and obtain a classification result of the video to be classified; The video classification model is trained using the training device according to claim 8 or 9.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the feature extraction model described in claim 1 or 2, the training method of the video classification model described in claim 3 or 4, or the video classification method described in claim 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the training method of the feature extraction model according to claim 1 or 2, the training method of the video classification model according to claim 3 or 4, or the video classification method according to claim 5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the training method of the feature extraction model according to claim 1 or 2, the training method of the video classification model according to claim 3 or 4, or the video classification method according to claim 5.

Images