Video classification method, apparatus, processor, and electronic device

Abstract

The application discloses a video classification method and device, a processor and an electronic device. It relates to the field of artificial intelligence. The method comprises the following steps: obtaining a video to be classified; determining key frames and sample frames of the video to be classified; merging the key frames and the sample frames to obtain an image frame sequence according to the time sequence of the key frames and the sample frames in the video to be classified; and determining a classification result of the video to be classified based on the image frame sequence. The application solves the problem of poor video classification effect in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video classification, and in particular, to a video classification method and device, a processor and an electronic device. BACKGROUND

[0002] For video classification using a deep neural network, the current solution is to first decompose the video frame by frame, and then use a deep neural network to classify and identify frame by frame (or randomly sample frames, such as keeping one frame every 5 frames), and finally weighted average to calculate the final classification result.

[0003] This technology is time-consuming and labor-intensive. In general, a video is 30 frames per second, and a 10-minute video will generate 18000 frames, i.e. 18000 images. Currently, a V100 graphics card can achieve about 60 FPS for real-time ResNet model prediction, which means that it takes 5 minutes to process a 10-minute video completely. Therefore, in order to speed up the prediction, multiple GPUs are often used for parallel computing acceleration. Random sampling can reduce the GPU resource overhead, but due to the randomness of the sampling, it is difficult to ensure that the selected frames contain key information, and often serious interference will lead to poor results. However, even if all frame information is used, too much irrelevant information in the video may affect the final result when weighted calculation is performed.

[0004] In view of the poor video classification effect in the related art, an effective solution has not yet been proposed. SUMMARY

[0005] The main purpose of the present application is to provide a video classification method, device, processor and electronic device to solve the problem of poor video classification effect in the related art.

[0006] In order to achieve the above purpose, according to one aspect of the present application, a video classification method is provided. The method comprises: obtaining a video to be classified; determining key frames and sample frames of the video to be classified; merging the key frames and the sample frames according to their time sequence in the video to be classified to obtain an image frame sequence; and determining a classification result of the video to be classified based on the image frame sequence.

[0007] Optionally, determining the key frames and the sample frames of the video to be classified comprises: determining a plurality of image frames of the video to be classified; sampling in the plurality of image frames to obtain the sample frames; and clustering the plurality of image frames to determine the key frames.

[0008] Optionally, the sampling frames are obtained by extracting image frames from the plurality of image frames according to a predetermined frame interval or a predetermined time interval.

[0009] Optionally, the image frame sequence is obtained by merging the key frame and the sampling frames according to a time sequence of the key frame and the sampling frames in the video to be classified, including: determining a playing time of the key frame in the video to be classified as a first playing time; determining a playing time of the sampling frame in the video to be classified as a second playing time; determining an arrangement order of the key frame and the sampling frames according to a time sequence of the first playing time and the second playing time; and determining the image frame sequence according to the arrangement order of the key frame and the sampling frames.

[0010] Optionally, the classification result of the video to be classified is determined based on the image frame sequence, including: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames include the key frame and the sampling frames, and the preset convolutional neural network model is trained according to sample images with labeled features; and determining the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence.

[0011] Optionally, the classification result of the video to be classified is determined based on the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence, including: determining a product of the feature matrix and the probability of each predetermined image frame to obtain a feature result; determining a feature result sequence according to an order of each predetermined image frame in the image frame sequence, wherein the feature result sequence includes the feature results of the plurality of predetermined image frames in the image frame sequence; and inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and feature result sequences corresponding to the sample videos.

[0012] Optionally, after the plurality of predetermined image frames in the image frame sequence are input into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame, the method further includes: adjusting the probability of the key frame to a preset value; and deleting the sampling frame with a probability lower than a preset threshold from the image frame sequence.

[0013] To achieve the above object, according to another aspect of the present application, a video classification device is provided, comprising: an acquisition unit configured to acquire a video to be classified; a first determination unit configured to determine a key frame and a sample frame of the video to be classified; a merging unit configured to merge the key frame and the sample frame to obtain a sequence of image frames according to a time sequence of the key frame and the sample frame in the video to be classified; and a second determination unit configured to determine a classification result of the video to be classified based on the sequence of image frames.

[0014] To achieve the above object, according to another aspect of the present application, a processor is provided. The processor is configured to run a program, wherein the program performs the above video classification method when running.

[0015] To achieve the above object, according to another aspect of the present application, an electronic device is provided. The electronic device comprises one or more processors and a memory, wherein the memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above video classification method.

[0016] By the present application, the following steps are adopted: acquiring a video to be classified; determining a key frame and a sample frame of the video to be classified; merging the key frame and the sample frame to obtain a sequence of image frames according to a time sequence of the key frame and the sample frame in the video to be classified; and determining a classification result of the video to be classified based on the sequence of image frames, thereby solving the problem of poor video classification effect in the related art. Furthermore, the effect of accurately determining the classification result of the video is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and do not constitute improper limitations to the present application. In the drawings:

[0018] Figure 1 is a flowchart of a video classification method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a predetermined convolutional neural network model according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a predetermined recurrent neural network according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a video classification device according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0024] In order for those skilled in the technical field to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] The present application will be described below in conjunction with preferred implementation steps, Figure 1 is a flowchart of a video classification method provided according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0027] Step S101, obtaining a video to be classified;

[0028] Step S102, determining key frames and sample frames of the video to be classified;

[0029] Step S103, merging the key frames and the sample frames to obtain an image frame sequence according to the time sequence of the key frames and the sample frames in the video to be classified;

[0030] Step S104, determining a classification result of the video to be classified based on the image frame sequence.

[0031] It should be noted that a video is composed of still pictures, which are called frames, so the video to be classified includes a plurality of image frames.

[0032] In step S101, the video to be classified can be in any known non-encrypted video format (MP4, MAV, MOV, etc.), and the video time is not limited.

[0033] In step S102, the sample frame can be randomly extracted from the image frames of the video to be classified, and the key frame can be an image frame with special image features in the video to be classified.

[0034] Optionally, the key frame and the image frame have special differences in image features such as color, contrast, or brightness.

[0035] In step S103, the image frame represents the picture played at a specific playing time of the video to be classified, so the image frame has a time attribute, and the key frame and the sample frame determined according to the image frame also have a time attribute. Therefore, after the key frame and the sample frame are determined, the playing time of each key frame and sample frame in the video to be classified can be determined, and the key frame and the sample frame can be combined according to the time order of the playing time to obtain the image frame sequence.

[0036] In step S103, the image frame sequence includes at least one key frame and at least one sample frame.

[0037] In the above embodiments of the present application, the image frame sequence includes both sample frames and key frames, so that the key information of the video to be classified is greatly retained in the image frame sequence, and the video to be classified can be accurately classified according to the image frame sequence.

[0038] In step S104, the image frame sequence can be analyzed using machine learning to determine the classification result of the video to be classified.

[0039] Optionally, in the video classification method provided in the embodiments of the present application, determining the key frame and the sample frame of the video to be classified includes: determining a plurality of image frames of the video to be classified; extracting the sample frame from the plurality of image frames; and clustering the plurality of image frames to determine the key frame.

[0040] In the above embodiments of the present application, the sample frame is extracted from the plurality of image frames of the video to be classified in a simple frame extraction manner, and the simple frame extraction means that some images are removed according to a certain rule; and the key frame is determined from the plurality of image frames of the video to be classified by clustering.

[0041] Optionally, the video to be classified is processed frame by frame to generate an image, and an original frame picture set (i.e., a plurality of image frames) P={F1, F2, F3,..., Fn} is obtained.

[0042] Optionally, a simple frame extraction process is performed on the original frame image set (i.e., a plurality of image frames) P = {F1, F2, F3... Fn} to obtain a simple frame-extracted image set (i.e., sample frames) F = {F1, F6, F11... Fn}.

[0043] It should be noted that simple frame extraction refers to removing some frames according to a certain rule. Common methods include retaining one frame every five frames or retaining 1 / 3 of the frame number per second (if the video is 30 FPS, then 10 frames are retained).

[0044] Optionally, the key frame calculation is based on the original frame image set (i.e., a plurality of image frames) P = {F1, F2, F3... Fn}. The key frame is different from other frames in that the key frame is special in terms of color, contrast, or brightness. Therefore, the original frame image set (i.e., a plurality of image frames) P = {F1, F2, F3... Fn} is analyzed by K-means clustering algorithm to obtain different clusters, and then the histogram of the images in each cluster is analyzed. Those whose mean value deviates from the average too much are selected as key frames to obtain the key frame set K.

[0045] Optionally, in the video classification method provided in the embodiments of the present application, the extracting in the plurality of image frames to obtain the sample frames includes: extracting image frames corresponding to a specified frame number as sample frames according to a predetermined frame number interval in the plurality of image frames; or extracting image frames corresponding to a specified time as sample frames according to a predetermined time interval in the plurality of image frames.

[0046] In the process of obtaining sample frames in the above embodiments of the present application, extraction can be performed according to a predetermined frame number interval, for example, after the video to be classified is split into a plurality of image frames, one frame can be retained every five frames as a sample frame. Extraction can also be performed according to a predetermined time interval, for example, 1 / 3 of the frame number per second of the plurality of image frames corresponding to the video to be classified is retained as a sample frame.

[0047] Optionally, in the video classification method provided in the embodiments of the present application, the merging of the key frames and the sample frames according to the time sequence of the key frames and the sample frames in the video to be classified to obtain the image frame sequence includes: determining the playing time of the key frames in the video to be classified as a first playing time; determining the playing time of the sample frames in the video to be classified as a second playing time; determining the arrangement order of the key frames and the sample frames according to the time sequence of the first playing time and the second playing time; and determining the image frame sequence according to the arrangement order of the key frames and the sample frames.

[0048] In the above embodiment of the present application, the key frame and the sample frame are image frames in the video to be classified, and each key frame and sample frame has a corresponding playing time in the video to be classified. The arrangement order of the key frame and the sample frame is determined according to the playing time of the key frame and the sample frame in the video to be classified, and then the image frame sequence retaining the key information of the video to be classified is obtained.

[0049] Optionally, in the video classification method provided by the embodiment of the present application, the step of determining the classification result of the video to be classified based on the image frame sequence comprises: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames comprise the key frame and the sample frame, and the preset convolutional neural network model is trained according to sample images with labeled features; and determining the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence.

[0050] In the above embodiment of the present application, the plurality of predetermined image frames in the image frame sequence are analyzed according to the preset convolutional neural network model, so that the feature matrix and the probability of each predetermined image frame can be quickly determined, and then the video to be classified is classified according to the determined feature matrix and probability.

[0051] Optionally, in the video classification method provided by the embodiment of the present application, the step of determining the classification result of the video to be classified based on the image frame sequence comprises: determining a feature result of each predetermined image frame according to the product of the feature matrix and the probability; sorting the feature results of the plurality of predetermined image frames in the image frame sequence according to the order of the predetermined image frames in the image frame sequence to obtain a feature result sequence; and inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and the feature result sequences corresponding to the sample videos.

[0052] In the above embodiment of the present application, the feature result of the predetermined image frame can be determined according to the product of the feature matrix and the probability of the predetermined image frame, and then the feature result of the predetermined image frame is analyzed according to the preset recurrent neural network model, so that the video to be classified can be classified according to the feature results of the plurality of predetermined image frames in the image frame sequence.

[0053] Optionally, in the video classification method provided by the embodiment of the present application, after the plurality of predetermined image frames in the image frame sequence are input into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame, the method further comprises: adjusting the probability of the key frame to a preset value; and deleting the sample frame with a probability lower than a preset threshold from the image frame sequence.

[0054] In the above embodiments of the present invention, adjusting the probability of key frames to a preset value can highlight the key information of the video to be classified. Deleting sampled frames with a probability lower than a preset threshold from the image frame sequence can reduce interference information in the video to be classified. Therefore, the classification result of the video to be classified can be accurately determined based on the adjusted image frame sequence.

[0055] The present invention also provides a preferred embodiment, which provides a video classification method based on keyframes.

[0056] This invention addresses the issue of missing key information caused by random frame sampling by pre-extracting key frames from the video, and dynamically adjusts the final weighting factor while using an RNN model (i.e., a pre-defined convolutional neural network model) to mitigate the impact of irrelevant information in the video on the overall result.

[0057] As an optional example, the present invention provides a video classification method based on keyframes, comprising the following steps:

[0058] S201: Input the video to be categorized.

[0059] Optionally, the videos to be categorized can be any known unencrypted video format (MP4, MAV, MOV, etc.), and the video length is not limited.

[0060] S202: After performing frame extraction on the video to be classified, input it into a preset convolutional neural network model to obtain image features and probabilities.

[0061] Optionally, the video to be classified is processed frame by frame to generate multiple image frames, and then simple frame extraction is performed. In addition, keyframes need to be calculated for all image frames. If keyframes that meet the requirements are filtered out after simple frame extraction, they need to be added back in chronological order.

[0062] Optionally, the pre-defined convolutional neural network model can adopt the structure of ResNet or VGG, and needs to output the feature matrix of the image frame and the classification probability corresponding to the image frame.

[0063] S203: Input the feature matrix and probability of the image frame into a preset recurrent neural network model to obtain the classification result.

[0064] The feature matrix of the image frame output by S202 is multiplied by the probability to generate a new matrix (i.e., the feature result), and then stacked in chronological order and input into a preset recurrent neural network model. After passing through softmax, the final classification result is obtained.

[0065] As an optional example, the detailed process of the keyframe-based video classification method of the present invention includes the following steps:

[0066] S301: Input the video to be categorized.

[0067] S302: The video to be classified is decomposed into multiple image frames frame by frame, and a key frame set (i.e., multiple key frames) K is calculated. The picture is retained to obtain a sample frame set (i.e., multiple sample frames) F according to every N frames. The key frame set (i.e., multiple key frames) K and the sample frame set (i.e., multiple sample frames) F are combined according to time to obtain an image frame sequence FN.

[0068] Optionally, the video to be classified is processed frame by frame to generate pictures to obtain an original frame picture set (i.e., multiple image frames) P = {F1, F2, F3... Fn}, and then simple frame extraction processing is performed. The so-called simple frame extraction refers to removing some pictures according to a certain rule. Common methods include retaining one frame every five frames or retaining 1 / 3 of the frame number per second (if the video is 30 FPS, 10 frames are retained). Thus, a sample frame set (i.e., multiple sample frames) F = {F1, F6, F11... Fn} after simple frame extraction is obtained.

[0069] Optionally, key frame calculation is performed based on the original frame picture set (i.e., multiple image frames) P. The difference between the key frame and other frames lies in that the color, contrast, or brightness of the key frame is relatively special. Therefore, different clusters are obtained by performing clustering analysis on the original frame picture set (i.e., multiple image frames) P through a K-means clustering algorithm, and then histogram analysis is performed on the pictures in each cluster to select those pictures whose mean value deviates from the average too much as key frames, so as to obtain the key frame set (i.e., multiple key frames) K.

[0070] Optionally, the sample frame set (i.e., multiple sample frames) F and the key frame set (i.e., multiple key frames) K are combined according to time sequence, that is, if the frame in the key frame set (i.e., multiple key frames) K is already in the sample frame set (i.e., multiple sample frames) F, no operation is performed, and if the frame in the key frame set (i.e., multiple key frames) K is not in the sample frame set (i.e., multiple sample frames) F, the frame is inserted into the sample frame set (i.e., multiple sample frames) F according to time sequence to obtain an image frame sequence FN with key frames.

[0071] For example, F = {F1, F6, F11}, K = {F1, F3, F9}, and FN = {F1, F3, F6, F9, F11}.

[0072] S303: The image frame sequence FN is input into a preset convolutional neural network model to obtain a feature matrix of each preset image frame and a probability of belonging to a certain category.

[0073] Figure 2 is a schematic diagram of a predetermined convolutional neural network model according to an embodiment of the present application. As shown in FIG. 3, the convolutional neural network model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Figure 2As shown, preset image frames contained in the image frame sequence FN need to be input into the pre-trained predetermined convolutional neural network model one by one, and the output feature matrix is the output before the full connection layer, and the probability is the result output by the softmax.

[0074] Optionally, when training the predetermined convolutional neural network model, the traditional training method is still used, that is, the CNN and the full connection layer are trained together.

[0075] S304: The feature matrix of the preset image frame is multiplied by its probability, and the result is stacked according to the original time sequence and input into the preset recurrent neural network model.

[0076] Optionally, the probability and the feature matrix come from the output of S303, but it needs to be noted that the probability of the key frame is manually adjusted to 1, and the frame with a probability less than 0.5 is discarded because the probability is too low to affect the final recognition result and become an interference term.

[0077] Figure 3 It is a schematic diagram of a preset recurrent neural network according to an embodiment of the application. As shown, Figure 3 The preset recurrent neural network model contains a predetermined convolutional neural network model RNN+full connection layer.

[0078] S305: Output the classification result.

[0079] It needs to be noted that in the past, video recognition is performed by fixed frame extraction, which is easy to miss some key information and interfere with the result, and at the same time, a large amount of computing power is needed for calculation even in the case of frame extraction.

[0080] The present application introduces the pre-extraction of key frames, so that the number of extracted frames can be less and more sparse, thereby reducing the calculation time and improving the result accuracy. After simple image classification, the RNN network is introduced for global analysis of the video, which is more accurate than the traditional weighted average.

[0081] The present application adopts the frame extraction technology based on key frames, which greatly retains the key information of the video in the case of more sparse frame extraction, and the introduced RNN network can better understand the whole streaming video.

[0082] The video classification method provided by the embodiment of the application acquires a video to be classified; determines key frames and sample frames of the video to be classified; merges the key frames and the sample frames according to the time sequence of the key frames and the sample frames in the video to be classified to obtain an image frame sequence; and determines a classification result of the video to be classified based on the image frame sequence, thereby solving the problem of poor video classification effect in the related art. Furthermore, the effect of accurately determining the video classification result is achieved.

[0083] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0084] The embodiment of the present application also provides a video classification device. It should be noted that the video classification device of the embodiment of the present application can be used to execute the method for video classification provided by the embodiment of the present application. The video classification device provided by the embodiment of the present application is introduced as follows.

[0085] Figure 4 FIG. 1 is a schematic diagram of a video classification device according to an embodiment of the present application. As shown in FIG. 1, the device includes an acquisition unit 41 configured to acquire a video to be classified; a first determination unit 42 configured to determine a key frame and a sample frame of the video to be classified; a merging unit 43 configured to merge the key frame and the sample frame to obtain a sequence of image frames according to a time sequence of the key frame and the sample frame in the video to be classified; and a second determination unit 44 configured to determine a classification result of the video to be classified based on the sequence of image frames. Figure 4

[0086] It should be noted that the acquisition unit 41 in this embodiment can be used to execute step S101 in the embodiment of the present application, the first determination unit 42 in this embodiment can be used to execute step S102 in the embodiment of the present application, the merging unit 43 in this embodiment can be used to execute step S103 in the embodiment of the present application, and the second determination unit 44 in this embodiment can be used to execute step S104 in the embodiment of the present application. The above-mentioned units have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above-mentioned embodiments.

[0087] Optionally, in the video classification device provided by the embodiment of the present application, the first determination unit includes a first determination module configured to determine a plurality of image frames of the video to be classified; an extraction module configured to extract a sample frame from the plurality of image frames; and a clustering module configured to cluster the plurality of image frames to determine a key frame.

[0088] Optionally, in the video classification device provided by the embodiment of the present application, the extraction module includes a first extraction module configured to extract a specified number of image frames as the sample frame from the plurality of image frames according to a predetermined frame interval; or a second extraction module configured to extract a specified number of image frames as the sample frame from the plurality of image frames according to a predetermined time interval.

[0089] ​Optionally, in the video classification apparatus provided in the embodiments of the present application, the merging unit comprises: a third determination module, configured to determine that the playing time of the key frame in the video to be classified is a first playing time; a fourth determination module, configured to determine that the playing time of the sample frame in the video to be classified is a second playing time; a fifth determination module, configured to determine the arrangement order of the key frame and the sample frame according to the time order of the first playing time and the second playing time; and a sixth determination module, configured to determine the image frame sequence according to the arrangement order of the key frame and the sample frame.

[0090] Optionally, in the video classification apparatus provided in the embodiments of the present application, the second determination unit comprises: a sixth determination module, configured to input a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determine the feature matrix and the probability of each predetermined image frame, wherein the predetermined image frames comprise the key frame and the sample frame, and the preset convolutional neural network model is trained according to the sample image with labeled features; and a seventh determination module, configured to determine the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence.

[0091] Optionally, in the video classification apparatus provided in the embodiments of the present application, the sixth determination module comprises: a seventh determination module, configured to determine the feature result of each predetermined image frame according to the product of the feature matrix and the probability; an eighth determination module, configured to sort the feature results of the plurality of predetermined image frames in the image frame sequence according to the order of the predetermined image frames in the image frame sequence, to obtain a feature result sequence; and a ninth determination module, configured to input the feature result sequence into a preset recurrent neural network model, and determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to the sample video with labeled classification result and the feature result sequence corresponding to the sample video.

[0092] Optionally, in the video classification apparatus provided in the embodiments of the present application, the apparatus further comprises: an adjusting unit, configured to adjust the probability of the key frame to a preset value after inputting the plurality of predetermined image frames in the image frame sequence into the preset convolutional neural network model and determining the feature matrix and the probability of each predetermined image frame; and a deleting unit, configured to delete the sample frame with a probability lower than a preset threshold from the image frame sequence.

[0093] The video classification apparatus provided in the embodiments of the present application acquires a video to be classified, determines a key frame and a sample frame of the video to be classified, merges the key frame and the sample frame according to the time order of the key frame and the sample frame in the video to be classified to obtain an image frame sequence, and determines the classification result of the video to be classified based on the image frame sequence, thereby solving the problem of poor video classification effect in the related art, and achieving the effect of accurately determining the classification result of the video

[0094] The video classification device comprises a processor and a memory, the units are stored in the memory as program units, and the corresponding functions are realized by executing the program units stored in the memory by the processor.

[0095] The processor comprises a core, and the core calls the corresponding program units in the memory.

[0096] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0097] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the video classification method.

[0098] The embodiment of the present application provides a processor, which is used for running a program, wherein the program is executed to perform the video classification method.

[0099] Figure 5 It is a schematic diagram of an electronic device according to the embodiment of the present application. Figure 5 As shown in the figure, the embodiment of the present application provides an electronic device 50, which comprises a processor 501, a memory 502 and a program stored in the memory and capable of running on the processor, and the processor realizes the following steps when executing the program: obtaining a video to be classified; determining a key frame and a sample frame of the video to be classified; merging the key frame and the sample frame to obtain an image frame sequence according to the time sequence of the key frame and the sample frame in the video to be classified; and determining a classification result of the video to be classified based on the image frame sequence.

[0100] Optionally, the processor realizes the following steps when executing the program: determining a plurality of image frames of the video to be classified; extracting a sample frame from the plurality of image frames; and clustering the plurality of image frames to determine a key frame.

[0101] Optionally, the processor realizes the following steps when executing the program: extracting a specified number of image frames as sample frames from the plurality of image frames according to a predetermined frame interval; or extracting a specified number of image frames as sample frames from the plurality of image frames according to a predetermined time interval.

[0102] Optionally, the processor implements the following steps when executing the program: determining that the playing time of the key frame in the video to be classified is a first playing time; determining that the playing time of the sample frame in the video to be classified is a second playing time; determining the arrangement order of the key frame and the sample frame according to the time order of the first playing time and the second playing time; and determining the image frame sequence according to the arrangement order of the key frame and the sample frame.

[0103] Optionally, the processor implements the following steps when executing the program: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames include the key frame and the sample frame, and the preset convolutional neural network model is trained according to sample images with labeled features; and determining the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence.

[0104] Optionally, the processor implements the following steps when executing the program: determining a feature result of each predetermined image frame according to the product of the feature matrix and the probability; sorting the feature results of the plurality of predetermined image frames in the image frame sequence according to the order of the predetermined image frames in the image frame sequence to obtain a feature result sequence; and inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and the feature result sequences corresponding to the sample videos.

[0105] Optionally, the processor implements the following steps when executing the program: after inputting the plurality of predetermined image frames in the image frame sequence into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame, adjusting the probability of the key frame to a preset value; and deleting the sample frame with a probability lower than a preset threshold in the image frame sequence

[0106] Optionally, the electronic device in the present application can be a server, a PC, a PAD, a mobile phone, etc.

[0107] The present application also provides a computer program product adapted to execute the program initializing the following method steps when executed on a data processing device: obtaining a video to be classified; determining a key frame and a sample frame of the video to be classified; merging the key frame and the sample frame according to the time order of the key frame and the sample frame in the video to be classified to obtain an image frame sequence; and determining the classification result of the video to be classified based on the image frame sequence.

[0108] Optionally, the computer program product is adapted to execute the program initializing the following method steps when executed on a data processing device: determining a plurality of image frames of a video to be classified; extracting a sample frame from the plurality of image frames; and clustering the plurality of image frames to determine a key frame.

[0109] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: extracting, from a plurality of image frames, image frames corresponding to a specified number of frames as sample frames at predetermined frame number intervals; or extracting, from the plurality of image frames, image frames corresponding to a specified time as sample frames at predetermined time intervals.

[0110] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: determining a play time of the key frame in the video to be classified as a first play time; determining a play time of the sample frame in the video to be classified as a second play time; determining an arrangement order of the key frame and the sample frame according to a time order of the first play time and the second play time; and determining the image frame sequence according to the arrangement order of the key frame and the sample frame.

[0111] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames include the key frame and the sample frame, and the preset convolutional neural network model is trained according to sample images with labeled features; and determining a classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence.

[0112] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: determining a feature result of each predetermined image frame according to a product of the feature matrix and the probability; sorting the feature results of the plurality of predetermined image frames in the image frame sequence according to an order of the predetermined image frames in the image frame sequence to obtain a feature result sequence; and inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and feature result sequences corresponding to the sample videos.

[0113] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: after inputting the plurality of predetermined image frames in the image frame sequence into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame, adjusting the probability of the key frame to a preset value; and deleting the sample frame with a probability lower than a preset threshold value in the image frame sequence.

[0114] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0115] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0116] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0117] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0118] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0119] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0120] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0121] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0122] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0123] The above merely provides embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method of video classification, characterized by, The method comprises: acquiring a video to be classified; determining key frames and sample frames of the video to be classified; merging the key frames and the sample frames according to the time sequence of the key frames and the sample frames in the video to be classified to obtain an image frame sequence; determining a classification result of the video to be classified based on the image frame sequence; wherein determining the classification result of the video to be classified based on the image frame sequence comprises: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames include the key frames and the sample frames, and the preset convolutional neural network model is trained according to sample images with labeled features; determining the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence; wherein after inputting the plurality of predetermined image frames in the image frame sequence into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame, the method further comprises: adjusting the probability of the key frames to a preset value; deleting the sample frames with a probability lower than a preset threshold in the image frame sequence; wherein determining the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence comprises: determining a feature result of each predetermined image frame according to the product of the feature matrix and the probability; sorting the feature results of the plurality of predetermined image frames in the image frame sequence according to the order of the predetermined image frames in the image frame sequence to obtain a feature result sequence; inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and the feature result sequences corresponding to the sample videos.

2. The method of claim 1, wherein, determining the key frames and the sample frames of the video to be classified comprises: determining a plurality of image frames of the video to be classified; extracting the sample frames from the plurality of image frames; clustering the plurality of image frames to determine the key frames.

3. The method of claim 2, wherein, extracting the sample frames from the plurality of image frames comprises: extracting image frames corresponding to a specified number of frames as the sample frames from the plurality of image frames according to a predetermined frame number interval; or extracting image frames corresponding to a specified time as the sample frames from the plurality of image frames according to a predetermined time interval.

4. The method of claim 1, wherein, merging the key frames and the sample frames according to the time sequence of the key frames and the sample frames in the video to be classified to obtain an image frame sequence comprises: determining the playing time of the key frames in the video to be classified as a first playing time; determining the playing time of the sample frames in the video to be classified as a second playing time; determining the arrangement order of the key frames and the sample frames according to the time sequence of the first playing time and the second playing time; determining the image frame sequence according to the arrangement order of the key frames and the sample frames.

5. A video classification apparatus characterized by comprising: The method comprises: an acquisition unit configured to acquire a video to be classified; The first determining unit is configured to determine key frames and sample frames of the video to be classified. The merging unit is configured to merge the key frames and the sample frames to obtain an image frame sequence according to time sequences of the key frames and the sample frames in the video to be classified. The second determining unit is configured to determine a classification result of the video to be classified based on the image frame sequence. The second determining unit includes: The sixth determining module is configured to input a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model to determine a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames include the key frames and the sample frames, and the preset convolutional neural network model is trained according to sample images with labeled features. The seventh determining module is configured to determine the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence. The device further includes: The adjusting unit is configured to adjust the probability of the key frames to a preset value after inputting the plurality of predetermined image frames in the image frame sequence into the preset convolutional neural network model to determine the feature matrix and the probability of each predetermined image frame. The deleting unit is configured to delete the sample frames with a probability lower than a preset threshold from the image frame sequence. The determination of the classification result of the video to be classified according to the feature matrix and the probability of the plurality of predetermined image frames in the image frame sequence includes: determining a feature result of each predetermined image frame according to a product of the feature matrix and the probability; sorting the feature results of the plurality of predetermined image frames in the image frame sequence according to an order of the predetermined image frames in the image frame sequence to obtain a feature result sequence; and inputting the feature result sequence into a preset recurrent neural network model to determine the classification result of the video to be classified, wherein the preset recurrent neural network model is trained according to sample videos with labeled classification results and feature result sequences corresponding to the sample videos.

6. A processor, comprising: The processor is configured to run a program, wherein the program performs the video classification method in any one of claims 1 to 4 when the program is run.

7. An electronic device, comprising: The device includes one or more processors and a memory, and the memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the video classification method in any one of claims 1 to 4.

Images