Video classification model training method, video classification method, device and electronic equipment
By using video samples and picture samples at the same time to train the video classification model, and compute and update the model weight parameters, the problem of poor generalization ability of the video classification model is solved, the classification accuracy and feature capture ability are improved, and video memory is saved.
Patent Information
- Application Number
- CN202010737337.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-07-28
AI Technical Summary
The existing video classification model has poor generalization ability, resulting in low classification accuracy, mainly due to the lack of diversity in synthetic video samples and insufficient feature capture capabilities.
By inputting video samples and picture samples to the training model at the same time, the difference between the prediction results and the calibration results is calculated, and the model weight parameters are updated to improve the generalization ability and feature fusion ability of the model.
It enhances the generalization ability and accuracy of the video classification model, avoids the problem of the same image occupancy of video memory in synthetic videos, saves video memory, and improves the performance of the convolutional layer and feature fusion layer.
Smart Images

Figure CN111860399B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a video classification model training method, a video classification method, a device and an electronic device. Background Art
[0002] At present, video content is mainly classified based on deep learning technology, but the generalization ability of video classification models trained based on deep learning technology in related technologies is poor, which affects the accuracy of video classification models. Generalization ability refers to the ability of a model to adapt to fresh samples after training. Summary of the invention
[0003] In view of this, the embodiments of the present application hope to provide a video classification model training method, a video classification method, an apparatus and an electronic device to solve the technical problem of poor generalization ability of the video classification model in the related art.
[0004] To achieve the above purpose, the technical solution of this application is implemented as follows:
[0005] The present application provides a video classification model training method, including:
[0006] Inputting at least one video sample into a video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample;
[0007] Inputting a feature vector corresponding to each image sample in at least one image sample into a set image classification model to obtain a second prediction result corresponding to each image sample in the at least one image sample; wherein the feature vector corresponding to the image sample is obtained by inputting the image sample into a convolutional layer in the video classification model;
[0008] Calculating a total loss value of the video classification model based on a difference between each prediction result and a corresponding calibration result in at least two prediction results; the at least two prediction results include a first prediction result corresponding to each video sample in the at least one video sample and a second prediction result corresponding to each picture sample in the at least one picture sample;
[0009] The weight parameters of the video classification model are updated according to the total loss value.
[0010] In the above solution, the total loss value of the video classification model is calculated based on the difference between each prediction result of at least two prediction results and the corresponding calibration result, including:
[0011] Calculating a first loss value based on a difference between a first prediction result corresponding to each video sample in the at least one video sample and a corresponding calibration result;
[0012] Calculating a second loss value based on a difference between a second prediction result corresponding to each picture sample in the at least one picture sample and a corresponding calibration result;
[0013] Based on the first loss value and the second loss value, a total loss value of the video classification model is calculated.
[0014] In the above solution, calculating the total loss value of the video classification model based on the first loss value and the second loss value includes:
[0015] The first loss value and the second loss value are weighted to obtain a total loss value of the video classification model; a first weight value corresponding to the first loss value is greater than a second weight value corresponding to the second loss value.
[0016] In the above scheme, the video classification model training method also includes: randomly obtaining at least one batch of samples from a sample library; each batch of samples in the at least one batch includes at least one video sample and at least one picture sample.
[0017] In the above solution, each batch of samples in the at least one batch includes samples corresponding to at least two different calibration results.
[0018] The embodiment of the present application provides a video classification method, comprising: using a first model to classify a video file to obtain a classification result; wherein:
[0019] The first model is a video classification model trained by using any of the above-mentioned video classification model training methods.
[0020] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.
[0021] Wherein, when the processor is used to run the computer program, it executes the steps of any one of the above-mentioned video classification model training methods, or executes the steps of the above-mentioned video classification method.
[0022] An embodiment of the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned video classification model training methods are implemented, or the steps of the above-mentioned video classification method are executed.
[0023] In an embodiment of the present application, a video classification model is trained by at least one video sample and at least one picture sample. During the training process, the total loss value of the video classification model is calculated based on the first prediction result corresponding to the video sample and the corresponding calibration result, and the second prediction result corresponding to the picture sample and the corresponding calibration result, and the weight parameter of the video classification model is updated according to the total loss value. Since both the video sample and the picture sample are used as training samples during the training process, the diversity of the samples is increased, and the problem of poor generalization ability of the video classification model caused by the lack of diversity of the synthesized video can be overcome, and the generalization ability of the trained video classification model is improved, thereby improving the accuracy of the trained video classification model. In addition, by updating the weight parameters of the video classification model according to the total loss value, the performance of the convolution layer and the feature fusion layer of the video classification model can be improved, and the accuracy of the trained video classification model can be further improved. Since the video sample is a video obtained by actual shooting, compared with a video synthesized by a single picture, it can avoid the situation where the same picture in the synthesized video occupies the video memory, and the video memory can be saved. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A schematic diagram of the implementation flow of the video classification model training method provided in the embodiment of the present application;
[0025] Figure 2 A schematic diagram of the composition structure of the video classification model provided in the embodiment of the present application;
[0026] Figure 3 A schematic diagram of the implementation flow of a video classification model training method provided in another embodiment of the present application;
[0027] Figure 4 A schematic diagram of an implementation flow of calculating the total loss value of a video classification model in the video classification model training method provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of the structure of a video classification model training device provided in an embodiment of the present application;
[0029] Figure 6 Schematic diagram of the hardware structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION
[0030] The prediction accuracy of the video classification model trained based on deep learning technology depends on the size of the video dataset involved in the training and the accuracy of the calibration values of the video samples in the video dataset. The calibration values of the video samples in the dataset usually need to be manually labeled. Since the calibration values of manually labeled video samples are not only costly and inefficient, but also the difficulty of labeling increases with the number of categories of video content, there are currently few video datasets in the industry that contain millions of manually labeled video samples, such as the ImageNet image dataset.
[0031] A video classification model training method is provided in the related art, which uses a single image to synthesize a video sample, and trains the video classification model through the synthesized video sample. However, the lack of diversity in the images in the synthesized video sample will lead to poor generalization ability of the video classification model. In addition, since all the images in the video sample synthesized from a single image contain the same features, the video classification model trained by the synthesized video sample has poor ability to capture features. When the video classification model is used to classify a new video, the accuracy of the classification result obtained is low.
[0032] In order to solve the technical problems of poor generalization ability and low accuracy of video classification models in related technologies, an embodiment of the present application provides a video classification model training method, which uses video samples and image samples to train the video classification model, increases the diversity of training samples, and can enhance the generalization ability of the trained video classification model and improve the accuracy of the video classification model.
[0033] The technical solution of the present application is further elaborated in detail below in conjunction with the accompanying drawings and specific embodiments of the specification.
[0034] Figure 1 The following is a schematic diagram of the implementation process of the video classification model training method provided in the embodiment of the present application. In the embodiment of the present application, the execution subject of the video classification model training method can be an electronic device such as a terminal or a server.
[0035] Reference Figure 1 The video classification model training method provided in the embodiment of the present application includes:
[0036] S101: Input at least one video sample into a video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample.
[0037] The electronic device determines at least one video sample, inputs the determined at least one video sample into a video classification model, classifies the at least one video sample using the video classification model, and obtains a first prediction result corresponding to each video sample in the at least one video sample output by the video classification model.
[0038] Here, the video samples can be obtained from a sample library. The sample library can exist in a local database of the electronic device or in a remote database. The video samples in the sample library are videos actually shot, so the image frames constituting the video samples have different image contents. Compared with videos synthesized from a single picture, in this embodiment, the video samples actually shot are used for training, which can reduce the waste of video memory caused by the same picture occupying video memory.
[0039] The following is a detailed description of the implementation process of the video classification model to classify video samples:
[0040] Please refer to Figure 2 , Figure 2 FIG. 1 is a schematic diagram of the composition structure of the video classification model provided in the embodiment of the present application. Figure 2 As shown, the video classification model includes a convolutional layer, a feature fusion layer, and a video classifier.
[0041] The electronic device inputs the acquired at least one video sample into the video classification model, performs downsampling processing on the at least one video sample according to a set frame interval, and obtains at least one picture corresponding to each video sample. Here, the set frame interval can be 1 or other values.
[0042] The electronic device inputs the image corresponding to each video sample obtained after the downsampling process into the convolution layer of the video classification model for feature extraction, and obtains the feature vector corresponding to each image in at least one image corresponding to each video sample (i.e., the image frame constituting the video sample) output by the convolution layer of the video classification model. Here, the convolution layer can be composed of a convolutional neural network (CNN), and the feature vector output by the convolution layer is used to characterize the local features of the corresponding image. Local features include line features, contour features, etc.
[0043] After the convolution layer outputs the feature vectors corresponding to each of the at least one picture corresponding to each video sample, these feature vectors are respectively input into the feature fusion layer of the video classification model for feature fusion processing to obtain the fused feature vectors corresponding to each video sample output by the feature fusion layer of the video classification model. Here, the video classification model fuses the feature vectors of each of the at least one picture corresponding to each video sample into a fused feature vector in the feature fusion layer. The feature fusion layer is composed of a neural network, which includes a long short-term memory network (LSTM), a recurrent neural network (RNN), and the like.
[0044] The electronic device inputs the fused feature vector corresponding to each video sample into the video classifier of the video classification model for classification processing, and obtains the first prediction result corresponding to each video sample output by the video classifier. Here, the video classifier determines the feature information with category distinction based on the fused feature vector, performs category prediction based on the determined feature information with category distinction, and obtains the first prediction result corresponding to each video sample. The video classifier can be composed of a fully connected layer, and the video classifier can also be a support vector machine (SVM) classifier, a softmax classifier, etc.
[0045] S102: Input the feature vector corresponding to each image sample in at least one image sample into a set image classification model to obtain a second prediction result corresponding to each image sample in the at least one image sample; wherein the feature vector corresponding to the image sample is obtained by inputting the image sample into a convolutional layer in the video classification model.
[0046] The electronic device obtains at least one picture sample, inputs each picture sample in the at least one picture sample into the convolution layer in the video classification model for feature extraction processing, and obtains a feature vector corresponding to each picture sample in the at least one picture sample. Here, the picture samples can be obtained from a sample library. The picture samples in the sample library can be obtained by taking pictures by an electronic device with a camera function, or can be obtained from a set picture database. At least one video sample in S101 and at least one picture sample in S102 belong to the same batch of samples. In practical applications, the convolution layer in the video classification model can process a batch of samples in parallel, that is, process at least one video sample and at least one picture sample belonging to the same batch in parallel.
[0047] The electronic device inputs the feature vector corresponding to each image sample in the at least one image sample into the set image classification model for classification processing to obtain a second prediction result corresponding to each image sample in the at least one image sample. Here, the image classifier can be composed of a fully connected layer, and the image classifier can also be an SVM classifier, a softmax classifier, etc.
[0048] It should be noted that the number of the picture samples in S102 may be the same as or different from the number of the video samples in S101. The calibration results corresponding to the picture samples in S102 may be the same as or different from the calibration results corresponding to the video samples in S101.
[0049] S103: Calculate the total loss value of the video classification model based on the difference between each prediction result in at least two prediction results and the corresponding calibration result; the at least two prediction results include a first prediction result corresponding to each video sample in the at least one video sample and a second prediction result corresponding to each picture sample in the at least one picture sample.
[0050] When the electronic device obtains a first prediction result corresponding to each video sample in at least one video sample and a second prediction result corresponding to each picture sample in at least one picture sample, the electronic device calculates a total loss value of the video classification model based on the difference between the first prediction result corresponding to each video sample and the corresponding calibration result, and the difference between the second prediction result corresponding to each picture sample and the corresponding calibration result.
[0051] S104: Update the weight parameters of the video classification model according to the total loss value.
[0052] The electronic device updates the weight parameters of the video classification model according to the total loss value of the video classification model to improve the accuracy of the prediction results output by the video classification model. The electronic device back-propagates the total loss value of the video classification model in the video classification model, and in the process of back-propagating the total loss value to each layer of the video classification model, calculates the gradient of the loss function according to the total loss value, and updates the weight parameters back-propagated to the current layer along the descending direction of the gradient.
[0053] The electronic device uses the updated weight parameters as the weight parameters used by the trained video classification model.
[0054] Here, an update stop condition can be set. When the update stop condition is met, the weight parameter obtained by the last update is determined as the weight parameter used by the trained video classification model. The update stop condition is such as a set training epoch. A training epoch is a process of training the video classification model once based on at least one video sample and at least one image sample. Of course, the update stop condition is not limited to this, and can also be, for example, a set mean average precision (mAP, mean Average Precision).
[0055] It should be noted that the loss function is used to measure the degree of inconsistency between the predicted value of the model and the true value (calibrated value). In practical applications, model training is achieved by minimizing the loss function.
[0056] Back propagation is relative to forward propagation. Forward propagation refers to the feedforward processing of the model, while the direction of back propagation is opposite to that of forward propagation. Back propagation refers to updating the weight parameters of each layer of the model according to the results of the model output. For example, if the model includes convolutional layer, feature fusion layer and fully connected layer, forward propagation refers to processing in the order of convolutional layer-feature fusion layer-fully connected layer, and back propagation refers to updating the weight parameters of each layer in the order of fully connected layer-feature fusion layer-convolutional layer.
[0057] In the scheme provided by the present embodiment, the video classification model is trained by at least one video sample and at least one picture sample. During the training process, the total loss value of the video classification model is calculated based on the first prediction result corresponding to the video sample and the corresponding calibration result, and the second prediction result corresponding to the picture sample and the corresponding calibration result, and the weight parameter of the video classification model is updated according to the total loss value. Since both the video sample and the picture sample are used as training samples during the training process, the diversity of the samples is increased, and the problem of poor generalization ability of the video classification model caused by the lack of diversity of the synthesized video can be overcome, and the generalization ability of the trained video classification model is improved, thereby improving the accuracy of the trained video classification model. In addition, updating the weight parameters of the video classification model according to the total loss value can improve the performance of the convolution layer and the feature fusion layer of the video classification model, thereby further improving the accuracy of the trained video classification model. Since the video sample is a video obtained by actual shooting, compared with a video synthesized by a single picture, it can avoid the situation where the same picture in the synthesized video occupies the video memory, and can save the video memory.
[0058] In addition, the video samples are videos actually shot, not videos synthesized from a single picture. Since the actual videos have obvious differences in feature distribution ratios compared to the synthesized videos, for example, a feature of the same object only exists in some images of the actual video, but exists in all images of the synthesized video. Training with actual videos can improve the feature capture capability of the feature fusion layer and further improve the accuracy of the trained video classification model.
[0059] As another embodiment of the present application, Figure 3 A schematic diagram of the implementation flow of a video classification model training method provided in another embodiment of the present application is shown. Figure 3 exist Figure 1 S301 was added on the basis of:
[0060] S301: randomly acquiring at least one batch of samples from a sample library; each batch of samples in the at least one batch includes at least one video sample and at least one picture sample.
[0061] The sample library includes multiple video samples and multiple image samples. Each video sample and image sample in the sample library corresponds to a corresponding calibration result, which is used to characterize the true category of the image content. In practical applications, given that it takes a lot of time to annotate the calibration results of video samples, in order to reduce the difficulty of making a sample library, the number of image samples in the sample library is greater than the number of video samples. For example, the sample library can include a large number of image samples and a small number of video samples.
[0062] In one embodiment, each batch of samples in the at least one batch includes samples corresponding to at least two different calibration results.
[0063] The electronic device may randomly obtain at least one video sample from multiple video samples, and randomly obtain at least one picture sample from multiple picture samples, and the obtained at least one video sample and at least one picture sample constitute a batch of samples. Alternatively, the electronic device may divide the video samples and picture samples in the sample library into multiple batches according to set requirements, and when training the video classification model, randomly obtain a batch of samples from the multiple batches of samples in the sample library. The set requirements are used to characterize the relevant requirements for the video samples and image samples in each batch of samples. For example, the number of video samples and the number of picture samples, the number of image frames contained in the video samples, the requirements for the calibration results corresponding to the video samples and the picture samples, etc.
[0064] The number of pictures contained in each video sample in at least one video sample in the same batch may be the same or different; the number of video samples and the number of picture samples in the same batch may be the same or different. The calibration results corresponding to the picture samples in the same batch may be the same or different from the calibration results corresponding to the video samples. Video samples and picture samples in samples from different batches may not be completely the same.
[0065] It should be noted that, in the process of training the video classification model, the electronic device uses the video samples and image samples in the same batch to complete one training. That is, the electronic device inputs the video samples in the same batch into the video classification model, and inputs the image samples in the batch into the convolution layer of the video classification model. Different training rounds can correspond to different batches of samples. Among them, S302 to S305 in this embodiment are the same as Figure 1 S101 to S104 in the corresponding embodiments are the same and will not be described in detail here.
[0066] In one embodiment, the picture samples used in the same training round may be different from the pictures included in the video samples, which can further improve the diversity of image content in the samples participating in the training and further improve the generalization of the training video classification model.
[0067] As another embodiment of the present application, Figure 4 The schematic diagram of the implementation process of calculating the total loss value of the video classification model in the video classification model training method provided in the embodiment of the present application is shown. The total loss value of the video classification model is calculated based on the difference between each prediction result and the corresponding calibration result in at least two prediction results, including:
[0068] S401: Calculate a first loss value based on a difference between a first prediction result corresponding to each video sample in the at least one video sample and a corresponding calibration result.
[0069] The electronic device uses a first loss function to calculate the difference loss1 between a first prediction result corresponding to each video sample in at least one video sample and a corresponding calibration result, and calculates a first loss value based on the difference loss1 between the first prediction result corresponding to each video sample in at least one video sample and a corresponding calibration result.
[0070] Here, the first loss function includes but is not limited to the Softmax loss function, which is a cross entropy loss function based on Softmax. The first loss value may be an average loss value calculated based on the loss1 corresponding to each video sample in at least one video sample; the first loss value may also be a variance value calculated based on the loss1 corresponding to each video sample in at least one video sample.
[0071] S402: Calculate a second loss value based on a difference between a second prediction result corresponding to each picture sample in the at least one picture sample and a corresponding calibration result.
[0072] The electronic device uses a second loss function to calculate the difference loss2 between the second prediction result corresponding to each picture sample in at least one picture sample and the corresponding calibration result, and calculates the second loss value based on the difference loss2 between the second prediction result corresponding to each picture sample in at least one picture sample and the corresponding calibration result.
[0073] Here, the second loss function includes but is not limited to the Softmax loss function. The second loss value may be an average loss value calculated based on the loss2 corresponding to each image sample in at least one image sample; the second loss value may also be a variance value calculated based on the loss2 corresponding to each image sample in at least one image sample.
[0074] It should be noted that when the first loss value is an average loss value, the second loss value is also an average loss value. When the first loss value is a variance value, the second loss value is also a variance value.
[0075] S403: Calculate a total loss value of the video classification model based on the first loss value and the second loss value.
[0076] The electronic device may calculate the average value between the first loss value and the second loss value to obtain the total loss value of the video classification model. The electronic device may also calculate the variance value based on the first loss value and the second loss value to obtain the total loss value of the video classification model.
[0077] As another embodiment of the present application, in order to improve the accuracy of the video classification model, the total loss value of the video classification model is calculated based on the first loss value and the second loss value, including:
[0078] The first loss value and the second loss value are weighted to obtain a total loss value of the video classification model; a first weight value corresponding to the first loss value is greater than a second weight value corresponding to the second loss value.
[0079] Here, the electronic device performs a weighted summation of the first loss value and the second loss value to obtain a total loss value of the video classification model. The first weight value and the second weight value can be set according to the actual application scenario, and the following conditions can be met: the first weight value corresponding to the first loss value is greater than the second weight value corresponding to the second loss value, and the sum of the first weight value corresponding to the first loss value and the second weight value corresponding to the second loss value is equal to 1.
[0080] In practical applications, the first weight corresponding to the first loss value may be twice the second weight corresponding to the second loss value.
[0081] As another embodiment of the present application, after the video classification model is trained, the video classification model can be put into use. For example, in a video classification scenario, an electronic device can classify video files using the video classification model trained by the above embodiment. It should be noted that the electronic device in the embodiment corresponding to the training video classification model may be different from the electronic device that uses the video classification model to classify video files in this embodiment.
[0082] The implementation process of the electronic device using the trained video classification model to classify the video file is as follows:
[0083] The electronic device performs downsampling processing on the acquired video file to obtain at least one picture corresponding to the video file; inputs the at least one picture corresponding to the video file into the convolution layer of the trained video classification model for feature extraction processing to obtain a feature vector corresponding to each picture in the at least one picture corresponding to the video file; inputs the feature vector corresponding to each picture in the at least one picture corresponding to the video file into the feature fusion layer of the video classification model for processing to obtain a fused feature vector corresponding to the video file; inputs the fused feature vector corresponding to the video file into the video classifier of the video classification model for classification processing to obtain multiple video categories and confidences corresponding to the multiple video categories; determines the video category whose confidence is greater than the confidence threshold as the video category corresponding to the video file. Here, the sum of the confidences corresponding to each video category in the multiple video categories is equal to 1. The confidence threshold can be 0.5, and of course it can also be set according to actual conditions.
[0084] The confidence level indicates the probability that the corresponding result is credible. For example, if the confidence level of a certain video category is 80%, it means that the probability that the video category corresponding to the video file is the video category is 80%.
[0085] Here, the video category is used to represent the category of the video content, and the video content category includes categories such as scenery, objects, and animals. Each video content category may correspond to multiple subcategories. For example, when the video content category is animals, the subcategories may include cats, dogs, rabbits, and the like.
[0086] In the solution provided in this embodiment, a video classification model obtained by training with at least one video sample and at least one image sample is used to classify video files. Since the video classification model has good generalization, when the video classification model is used to classify video files, the accuracy of the obtained video category is high.
[0087] To implement the method of the embodiment of the present application, the embodiment of the present application also provides a video classification model training device, which is arranged on an electronic device such as a terminal or a server, such as Figure 5 As shown, the video classification model training device includes:
[0088] A first processing unit 501 is configured to input at least one video sample into a video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample;
[0089] The second processing unit 502 is used to input the feature vector corresponding to each picture sample in the at least one picture sample into the set picture classification model to obtain the second prediction result corresponding to each picture sample in the at least one picture sample; wherein the feature vector corresponding to the picture sample is obtained by inputting the picture sample into the convolution layer in the video classification model;
[0090] A calculation unit 503 is used to calculate a total loss value of the video classification model based on a difference between each prediction result and a corresponding calibration result in at least two prediction results; the at least two prediction results include a first prediction result corresponding to each video sample in the at least one video sample and a second prediction result corresponding to each picture sample in the at least one picture sample;
[0091] The updating unit 504 is used to update the weight parameters of the video classification model according to the total loss value.
[0092] In one embodiment, the calculation unit 503 is specifically used for:
[0093] Calculating a first loss value based on a difference between a first prediction result corresponding to each video sample in the at least one video sample and a corresponding calibration result;
[0094] Calculating a second loss value based on a difference between a second prediction result corresponding to each picture sample in the at least one picture sample and a corresponding calibration result;
[0095] Based on the first loss value and the second loss value, a total loss value of the video classification model is calculated.
[0096] In one embodiment, the calculation unit 503 is also used to: perform weighted processing on the first loss value and the second loss value to obtain a total loss value of the video classification model; a first weight value corresponding to the first loss value is greater than a second weight value corresponding to the second loss value.
[0097] In one embodiment, the video classification model training device further includes:
[0098] The acquisition unit is used to randomly acquire at least one batch of samples from the sample library; each batch of samples in the at least one batch includes at least one video sample and at least one picture sample.
[0099] In one embodiment, each batch of samples in the at least one batch includes samples corresponding to at least two different calibration results.
[0100] In actual application, each unit included in the video classification model training device can be implemented by a processor in the video classification model training device. Of course, the processor needs to run the program stored in the memory to implement the functions of the above-mentioned program modules.
[0101] It should be noted that: the video classification model training device provided in the above embodiment only uses the division of the above program modules as an example when training the video classification model. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the video classification model training device is divided into different program modules to complete all or part of the processing described above. In addition, the video classification model training device provided in the above embodiment and the video classification model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0102] In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides a video classification device, which is arranged on an electronic device such as a terminal or a server, and the video classification device includes:
[0103] A classification unit is used to classify the video file using a first model to obtain a classification result; wherein the first model is a video classification model trained using the video classification model training method in any of the above embodiments.
[0104] In actual application, the classification unit included in the video classification device can be implemented by a processor in the video classification device. Of course, the processor needs to run the program stored in the memory to implement the functions of the above program modules.
[0105] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 6 Schematic diagram of the hardware structure of the electronic device of the present application embodiment. Figure 6 As shown, the electronic equipment includes:
[0106] Communication interface 1, capable of exchanging information with other devices such as network devices;
[0107] The processor 2 is connected to the communication interface 1 to realize information exchange with other devices, and is used to execute the video classification model training method and / or the video classification method provided by one or more technical solutions when running the computer program. The computer program is stored in the memory 3.
[0108] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 4 .
[0109] The memory 3 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.
[0110] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 3 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.
[0111] The method disclosed in the above embodiment of the present application can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 2 or the instruction in the form of software. The above processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 3, and the processor 2 reads the program in the memory 3 and completes the steps of the above method in combination with its hardware.
[0112] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present application are implemented, which will not be repeated here for the sake of brevity.
[0113] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 3 storing a computer program, and the computer program can be executed by a processor 2 to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.
[0114] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0115] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0116] In addition, all functional units in the embodiments of the present application may be integrated into one processing module, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0117] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.
[0118] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.
[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A video classification model training method, It is characterized in that include: Inputting at least one video sample into a video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample; Inputting a feature vector corresponding to each image sample in at least one image sample into a set image classification model to obtain a second prediction result corresponding to each image sample in the at least one image sample; wherein the feature vector corresponding to the image sample is obtained by inputting the image sample into a convolutional layer in the video classification model; Calculating a total loss value of the video classification model based on a difference between each prediction result and a corresponding calibration result in at least two prediction results; the at least two prediction results include a first prediction result corresponding to each video sample in the at least one video sample and a second prediction result corresponding to each picture sample in the at least one picture sample; Updating weight parameters of the video classification model according to the total loss value; The step of calculating the total loss value of the video classification model based on the difference between each prediction result and the corresponding calibration result in at least two prediction results includes: Calculating a first loss value based on a difference between a first prediction result corresponding to each video sample in the at least one video sample and a corresponding calibration result; Calculating a second loss value based on a difference between a second prediction result corresponding to each picture sample in the at least one picture sample and a corresponding calibration result; Based on the first loss value and the second loss value, a total loss value of the video classification model is calculated.
2. The video classification model training method according to claim 1, It is characterized in that The calculating the total loss value of the video classification model based on the first loss value and the second loss value includes: The first loss value and the second loss value are weighted to obtain a total loss value of the video classification model; a first weight value corresponding to the first loss value is greater than a second weight value corresponding to the second loss value.
3. The video classification model training method according to claim 1 or 2, It is characterized in that Before inputting at least one video sample into the video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample, the method includes: At least one batch of samples is randomly acquired from a sample library; each batch of samples in the at least one batch includes the at least one video sample and the at least one picture sample.
4. The video classification model training method according to claim 3, It is characterized in that Each batch of samples in the at least one batch includes samples corresponding to at least two different calibration results.
5. A video classification method, It is characterized in that include: The first model is used to classify the video file to obtain a classification result; wherein, The first model is a video classification model trained using the video classification model training method according to any one of claims 1 to 4.
6. A video classification model training device, It is characterized in that include: A first processing unit, configured to input at least one video sample into a video classification model to obtain a first prediction result corresponding to each video sample in the at least one video sample; A second processing unit is used to input the feature vector corresponding to each picture sample in the at least one picture sample into a set picture classification model to obtain a second prediction result corresponding to each picture sample in the at least one picture sample; wherein the feature vector corresponding to the picture sample is obtained by inputting the picture sample into a convolutional layer in the video classification model; A calculation unit, configured to calculate a first loss value based on a difference between a first prediction result corresponding to each video sample in the at least one video sample and a corresponding calibration result; calculate a second loss value based on a difference between a second prediction result corresponding to each picture sample in the at least one picture sample and a corresponding calibration result; and calculate a total loss value of the video classification model based on the first loss value and the second loss value; An updating unit is used to update the weight parameters of the video classification model according to the total loss value.
7. A video classification device, It is characterized in that include: The classification unit is used to classify the video file using the first model to obtain a classification result; wherein, The first model is a video classification model trained using the video classification model training method according to any one of claims 1 to 4.
8. An electronic device, It is characterized in that include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the video classification model training method described in any one of claims 1 to 4, or executes the steps of the video classification method described in claim 5.
9. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements the steps of the video classification model training method described in any one of claims 1 to 4, or implements the steps of the video classification method described in claim 5.
Citation Information
Patent Citations
Fall detection method based on skeleton points and a fall detection device thereof
CN109492612A
A multimedia data matching information processing method
CN109697236A
System and method for identifying fire in video based on convolutional neural network
CN110633675A
Cognitive indexing of images in digital video content
US20190236371A1
Method of and apparatus for processing video signals
US4725887A
