Video classification method and device based on bit stream analysis and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a video classification method and device based on bit stream analysis and electronic equipment. The method comprises the following steps: acquiring a time sequence of a compressed video bit stream; performing feature extraction processing on the time sequence to obtain target features; and predicting the video category of the compressed video according to the target feature. According to the method and the device, the time sequence of the compressed video bit stream is used as the processing data instead of the decompressed pixel data, so that the computing resources and the storage space are greatly reduced, and meanwhile, privacy protection can be provided when the video containing the sensitive information is processed without the operation of decrypting or decompressing the video. Furthermore, feature extraction processing and video category prediction are carried out on the time sequence, various information in the compressed video can be effectively applied and analyzed, the efficient and accurate performance target is achieved in a video classification task, the classification problem of large-scale video data is solved, and practicability and universality are extremely high.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of artificial intelligence, and specifically, relates to a video classification method based on bitstream analysis, a video classification device based on bitstream analysis, and an electronic device. Background Art

[0002] The main purpose of video classification technology is to identify and classify video content, which is crucial for fields such as security monitoring, content filtering, and entertainment. Traditional video classification methods are mainly implemented based on parsing the pixel data of videos. This requires decompressing the compressed video data into the original format and then analyzing the image features therein, such as color, texture, motion, etc. to determine the category of the video. Usually, it involves using some complex algorithms, such as Dynamic Time Warping (DTW), which is used to compare the similarity between different video sequences; and deep learning methods, which can utilize neural networks to learn and identify complex patterns in videos, such as Convolutional Neural Networks (CNN) and Recurrent Neural Network (RNN), which can identify complex patterns and features in videos.

[0003] These networks improve the accuracy of video classification by learning a large amount of video data, but this also means that they require a large amount of labeled data for training. For example, a deep learning model requires thousands of hours of videos to learn different types of video content. In addition, these technologies usually need to decompress the compressed video data into the original format to analyze features, so they require a large amount of computing resources and storage space, especially when processing high-resolution and long-duration videos. In addition, they do not perform well when processing videos of low quality and also involve privacy issues during the analysis process. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a video classification method based on bitstream analysis, a video classification device based on bitstream analysis, and an electronic device.

[0005] According to the first aspect of the embodiments of the present disclosure, a video classification method based on bitstream analysis is provided. The method includes:

[0006] Obtaining a time series of a compressed video bitstream;

[0007] Performing feature extraction processing on the time series to obtain target features;

[0008] Predicting the video category of the compressed video according to the target features.

[0009] Optionally, the obtaining of the target features by performing feature extraction processing on the time series includes:

[0010] Performing feature extraction processing on the time series by using a convolutional neural network to obtain target features.

[0011] Optionally, the performing feature extraction processing on the time series by using a convolutional neural network to obtain target features includes:

[0012] Performing feature extraction processing on the time series by using a convolutional neural network to obtain first features;

[0013] Performing feature extraction processing on the first features by using a convolutional neural network to obtain second features;

[0014] Performing feature extraction processing on the second features by using a convolutional neural network to obtain target features.

[0015] Optionally, the convolutional neural network includes: a residual network; and the residual network includes: three residual blocks.

[0016] Optionally, the predicting of the video category of the compressed video according to the target features includes:

[0017] Constructing a feature map of the target features;

[0018] Performing mapping processing on the feature map to obtain the video category of the compressed video.

[0019] Optionally, the constructing of the feature map of the target features includes:

[0020] Constructing the feature map of the target features by using a global average pooling layer.

[0021] Optionally, the performing mapping processing on the feature map to obtain the video category of the compressed video includes:

[0022] Performing mapping processing on the feature map by using a normalized exponential function to obtain the video category of the compressed video.

[0023] According to a second aspect of the embodiments of the present disclosure, there is provided a video classification device based on bitstream analysis, including:

[0024] A sequence acquisition module configured to acquire a time series of a compressed video bitstream;

[0025] A feature extraction module configured to perform feature extraction processing on the time series to obtain target features;

[0026] A category prediction module configured to predict the video category of the compressed video according to the target features.

[0027] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0028] A processor;

[0029] A memory for storing instructions executable by the processor;

[0030] Wherein, the processor is configured to: execute the executable instructions to implement the steps of the method according to any one of the first aspects of the present disclosure.

[0031] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0032] In the method and device provided by the exemplary embodiments of the present disclosure, the time series of the compressed video bitstream is used as the processing data instead of the decompressed pixel data, which greatly reduces the computing resources and storage space. At the same time, without the need to perform operations such as decrypting or decompressing the video, privacy protection can also be provided when processing videos containing sensitive information. Further, feature extraction processing and video category prediction are performed on the time series, which can effectively apply and analyze various information in the compressed video, achieving efficient and accurate performance goals in the video classification task, solving the classification problem of large-scale video data, and having extremely strong practicability and generality.

[0033] Other features and advantages of the present disclosure will be described in detail in the subsequent specific implementation section.

[0034] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0036] Figure 1 Schematically shows a flowchart of a video classification method based on bitstream analysis in an exemplary embodiment of the present disclosure;

[0037] Figure 2 Schematically shows a structural diagram of a residual network in an exemplary embodiment of the present disclosure;

[0038] Figure 3 Schematically shows a flowchart of a method for feature extraction processing in an exemplary embodiment of the present disclosure;

[0039] Figure 4 Schematically shows a flowchart of a method for predicting video categories in an exemplary embodiment of the present disclosure;

[0040] Figure 5 Schematically show the structural diagram of a video classification device based on bitstream analysis in an exemplary embodiment of the present disclosure;

[0041] Figure 6 Schematically show an electronic device for implementing a video classification method based on bitstream analysis in an exemplary embodiment of the present disclosure;

[0042] Figure 7 Schematically show another electronic device for implementing a video classification method based on bitstream analysis in an exemplary embodiment of the present disclosure. Detailed implementation manners

[0043] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0044] It should be noted that all actions of obtaining signals, information, or data in the present disclosure are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining the authorization given by the owner of the corresponding device.

[0045] The main purpose of video classification technology is to identify and classify video content. This is very crucial for fields such as security monitoring, content filtering, and entertainment. Traditional video classification methods are mainly implemented based on parsing the pixel data of videos, which requires decompressing the compressed video data into the original format and then analyzing the image features therein, such as color, texture, motion, etc. to determine the category of the video.

[0046] It usually involves using some complex algorithms, such as dynamic time warping, which is used to compare the similarity between different video sequences; and deep learning methods, which can use neural networks to learn and identify complex patterns in videos, such as convolutional neural networks and recurrent neural networks, which can identify complex patterns and features in videos.

[0047] These networks improve the accuracy of video classification by learning a large amount of video data, but this also means that they require a large amount of labeled data for training.

[0048] For example, a deep learning model requires thousands of hours of videos to learn different types of video content.

[0049] In addition, these technologies typically require decompressing the compressed video data into its original format to analyze these features, thus requiring a large amount of computing resources and storage space, especially when dealing with high-resolution and long-duration videos.

[0050] Moreover, they perform poorly when dealing with videos of low quality and also involve privacy issues during the analysis process.

[0051] Generally, neural network architectures using deep learning are used to process compressed videos. However, these methods require fully decompressing the video data, which means additional processing time and storage space. These systems first convert the video into a series of images (frames) and then analyze these images for classification.

[0052] Among them, a short video scene classification method and system based on a self-attention model can achieve video classification technology. The system determines the shot boundaries of the video to be processed and performs shot segmentation, thereby extracting the scene features of the key frames, and then denoising these features. Next, frame-level channel attention processing is used to enhance the frame weights, and a self-attention model is used to associate different shots, thereby improving the accuracy of short video classification.

[0053] This technical solution requires precise shot segmentation and key frame extraction. This process has a certain dependence on the coherence and integrity of the video content and is easily affected by the video editing style and quality, especially in user-generated short videos.

[0054] Secondly, the denoising process of this technical solution increases the complexity of the processing flow, and the dependence on the self-attention model leads to a high computational complexity.

[0055] Another method and system for multi-source data normalization processing and fusion can be used for video classification technology. This method mainly relies on detailed partitioning of video frames and complex data processing and recognition based on the proportion of the human body contour area, which increases the complexity of the overall processing flow.

[0056] In addition, when decoding and analyzing frame domain information, the privacy of video data cannot be effectively protected.

[0057] Another video classification technology can be a multi-modal video classification method based on a dynamic fusion gate mechanism. This method requires detailed extraction of unimodal features and dynamically configuring the usage quantity of each feature, which is relatively complex and computationally intensive in implementation.

[0058] Secondly, this method relies on a dynamic fusion gate mechanism to automatically select the location of multi-modal fusion, which increases the complexity of the model and may lead to dependence on specific modal features.

[0059] Moreover, this method requires a large amount of training and adjustment of the model to adapt to different video features and fusion strategies.

[0060] It can be found that these solutions generally adopt deep learning-based technologies, such as convolutional neural networks, to extract features in videos and perform classification.

[0061] To train these models, a large amount of labeled video data is required, and these data sets need to be carefully crafted and maintained.

[0062] Moreover, these methods usually require decoding video files to access and analyze the original pixel data.

[0063] Therefore, due to the dependence on large-scale data sets, the training cost of these models is very high and the time consumption is huge.

[0064] These models may not perform well when dealing with video types different from the training data set because they mainly rely on the information in the data set.

[0065] When dealing with videos containing sensitive information, these methods may involve privacy leakage and ethical issues because they need to fully access and analyze the video content.

[0066] The present disclosure provides a video classification method based on bitstream analysis. Figure 1 It is a schematic flowchart of a video classification method based on bitstream analysis shown according to an exemplary embodiment, as Figure 1 shown, this method may at least include the following steps:

[0067] Step S110. Obtain the time series of the compressed video bitstream.

[0068] Step S120. Perform feature extraction processing on the time series to obtain target features.

[0069] Step S130. Predict the video category of the compressed video according to the target features.

[0070] In the exemplary embodiment of the present disclosure, the time series of the compressed video bitstream is used as the processing data instead of the decompressed pixel data, which greatly reduces the computing resources and storage space. At the same time, without the need to decrypt or decompress the video, it can also provide privacy protection when dealing with videos containing sensitive information. Further, performing feature extraction processing and video category prediction on the time series can effectively apply and analyze various information in the compressed video, achieving efficient and accurate performance goals in the video classification task, solving the classification problem of large-scale video data, and having extremely strong practicality and versatility.

[0071] The following will explain in detail each step of the video classification method based on bitstream analysis.

[0072] In step S110, a time series of the compressed video bitstream is obtained.

[0073] In an exemplary embodiment of the present disclosure, the process of reducing the size of a file is called compression, and a set of well-defined and documented mathematical tools and algorithms, such as video codecs, are used to compress videos. After a video is compressed, it is usually in a format (bitstream) that can only be understood by software that decodes the corresponding bitstream. For example, a video encoded using the H.264 / AVC (a highly compressed digital video codec standard proposed by the Joint Video Team) video codec cannot be decoded by the HEVC (High Efficiency Video Coding) codec, and vice versa.

[0074] A bitstream refers to a digital stream composed of a string of 0s or 1s, and each digit is called a bit. A bitstream can represent any type of data, such as text, audio, video, etc., and converts these data into binary form for transmission and storage through a specific encoding method.

[0075] It should be noted that a bitstream is not a fixed-length data type, nor is there a specified storage method. In practical applications, the length and storage method of a bitstream depend on the application scenario and specific requirements. Techniques such as bit padding and byte alignment are usually used to adjust the length and format of the bitstream.

[0076] The principle of a bitstream is based on data communication technologies of binary encoding and transmission. It converts various types of data into binary form and then transmits and stores them in a certain order through a data stream. During the transmission process, a bitstream can adopt serial transmission or parallel transmission methods to achieve high-speed data transmission and sharing.

[0077] A bitstream refers to the binary data stream output by a video encoder, which contains the encoding information of all video frames. Among them, each bit represents an information unit of a video frame. The information in the bitstream includes pixel information, prediction mode, motion vector, quantization parameter, code table, etc. of the video frame.

[0078] In digital signal processing and communication protocols, bitstreams are often used to represent short control information, status signals, etc. For example, when controlling a hardware device, instructions, status information, etc. can be sent through a bitstream. In addition, in audio and video encoding, bitstreams are also used to represent compressed audio and video data streams for easy transmission and decoding.

[0079] After obtaining the compressed video bitstream, the time series can be further obtained.

[0080] The time series eigenvalue can be a certain time period, such as 0.1s to 0.2s, or a timestamp, which is used to identify the time period when each target appears in the video. The time series refers to a sequence formed by arranging the values of the same statistical indicator in the order of their occurrence time.

[0081] To extract the time series between video frames, the time scale of the video frames can be extracted first, and then each time scale in the video frames is sorted in ascending order to obtain the time series. For example, when a video frame is transmitted every second, the time scale is one second.

[0082] In step S120, feature extraction processing is performed on the time series to obtain target features.

[0083] In an exemplary embodiment of the present disclosure, after obtaining the time series of the compressed video bitstream, the target features of the time series can be further extracted.

[0084] In an alternative embodiment, a convolutional neural network is used to perform feature extraction processing on the time series to obtain target features.

[0085] In an alternative embodiment, the convolutional neural network includes: a residual network; the residual network includes: three residual blocks.

[0086] The Residual Network (ResNet) is a deep convolutional neural network that solves the problem of gradient disappearance in deep networks by introducing residual connections. In ResNet, each convolutional layer does not directly map the input to the output, but adds the input to the output, that is, it is implemented through "shortcut" connections that span several layers.

[0087] This design can make the model easier to optimize and still achieve good performance in deep networks with more than 100 layers.

[0088] Figure 2 Shows a schematic structural diagram of the residual network, as Figure 2 As shown, the residual network is composed of three residual blocks and can be used for the purpose of extracting target features from the input time series. The number of filters of these three convolutional blocks can be set to 256, 512, and 512 respectively.

[0089] When training this residual network, fixed parameter initialization and an automatically adjusted learning rate can be used. In the initial settings, the learning rate adjustment factor is set to 0.5 and the patience level is set to 40. Also, to avoid overfitting, an early stopping strategy can be adopted and the patience level at stopping is set to 80. In addition, an optimizer such as adam (Adaptive Moment Estimation) can be used to automatically adjust parameters such as the learning rate.

[0090] In the task of video classification, a categorical cross - entropy loss function can also be used as the loss for supervised learning. The formula for this loss function is as follows:

[0091]

[0092] where, y i represents the one - hot encoding of the actual video category, represents the one - hot encoding of the predicted video category, and C represents the total number of video categories.

[0093] The optimization objective of this formula is to maximize the likelihood of predicting the true label given the model parameters.

[0094] Cross - entropy can measure the difference between two different probability distributions in the same random variable, which in machine learning represents the difference between the true probability distribution and the predicted probability distribution. The smaller the value of cross - entropy, the better the model prediction effect. Cross - entropy can be used in conjunction with a softmax classifier. The output results are processed to make the predicted values of multiple classifications sum to 1, and then the loss is calculated through cross - entropy.

[0095] In a multi - classification task, the softmax activation function and the cross - entropy loss function are adopted. Since cross - entropy describes the difference between two probability distributions, while the output of a neural network is a vector, not in the form of a probability distribution. Therefore, the softmax activation function is needed to "normalize" a vector into the form of a probability distribution, and then the cross - entropy loss function is used to calculate the loss.

[0096] In an alternative embodiment, Figure 3 a schematic flowchart of the method for feature extraction processing is shown, as Figure 3 shown, this method may at least include the following steps: In step S310, a convolutional neural network is used to perform feature extraction processing on the time series to obtain a first feature.

[0097] When this convolutional neural network is a residual network, the first residual block of the residual network can be used to perform the first - layer feature extraction processing on the input time series to obtain a first feature. This first feature can be a distribution feature used to characterize the peak value and distribution.

[0098] In step S320, the convolutional neural network is used to perform feature extraction processing on the first feature to obtain a second feature.

[0099] When the convolutional neural network is a residual network, the second residual block of the residual network can be used to perform second-layer feature extraction processing on the input first feature to obtain a second feature. This second feature can also represent distribution features, such as peak values and distribution conditions, etc., and can also represent other contents, which are not specifically limited in this exemplary embodiment.

[0100] In step S330, the convolutional neural network is used to perform feature extraction processing on the second feature to obtain a target feature.

[0101] When the convolutional neural network is a residual network, the third residual block of the residual network can be used to perform third-layer feature extraction processing on the input second feature to obtain a target feature. This target feature can also represent distribution features, such as peak values and distribution conditions, etc., and can also represent other contents, which are not specifically limited in this exemplary embodiment.

[0102] In this exemplary embodiment, through the designed structure of the convolutional neural network, compressed video data can be effectively processed, the powerful capabilities of deep learning are exerted, and accurate classification can be further performed without decompressing the video.

[0103] In step S130, the video category of the compressed video is predicted according to the target feature.

[0104] In the exemplary embodiment of the present disclosure, after the target feature is extracted, the video category of the compressed video can be predicted by using this target feature.

[0105] In an alternative embodiment, Figure 4 shows a schematic flowchart of a method for predicting a video category, as Figure 4 shown, the method may at least include the following steps: In step S410, a feature map of the target feature is constructed.

[0106] In an alternative embodiment, a global average pooling layer is used to construct a feature map of the target feature.

[0107] The global average pooling layer can generate a predicted feature map for different video categories in this video classification task after the target feature is extracted.

[0108] Replacing the fully connected layer with global average pooling reduces the number of parameters on the one hand, and on the other hand, global average pooling itself plays a role in regularization and prevents overfitting of the overall structure. In addition, global average pooling summarizes spatial information, so it is more robust to spatial transformations of the input.

[0109] In step S420, the feature map is mapped to obtain the video category of the compressed video.

[0110] In an alternative embodiment, the feature map is mapped using the sigmoid function to obtain the video category of the compressed video.

[0111] Among them, the sigmoid function can be a softmax classifier. The softmax classifier maps the feature vector represented by the generated feature map to the probability distribution of the video category, thereby determining the video category of the compressed video.

[0112] Based on this, in the supervised learning task of this video classification task, when the input time series X is given, the process of the classifier predicting the video category can be represented by formula (2):

[0113]

[0114] Among them, is the predicted output, represents the skip connection in each residual block, θ corresponds to the parameters of each layer, and f corresponds to the superposition result of each one.

[0115] The final output of the residual network is generated by a global average pooling layer and a softmax classifier. The global average pooling layer generates the feature map for different video categories in the video classification task. Then, the softmax classifier maps the feature vector to the probability distribution of the output category. The residual connection between the convolutional layers effectively avoids the problem of gradient disappearance during the training process.

[0116] It should be noted that in addition to using the residual network as the model for processing the compressed video bitstream, different deep learning architectures can also be used, such as VGG (Visual Geometry Group, a deep convolutional neural network architecture), Inception (a deep learning structure), or Transformer (a sequence model based on the attention mechanism) network. These architectures may provide similar performance, especially in terms of feature extraction and classification efficiency.

[0117] In addition, other time series analysis methods can also be used, such as Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU). These methods can be applied to process time series data in the compressed bitstream, such as frame size or interval time, so as to provide effective classification without decoding the video.

[0118] In an exemplary embodiment of the present disclosure, compared with the method of the short video scene classification method and system based on the self-attention model, the present application directly uses the compressed video bitstream as the input without the need for complex shot segmentation and key frame extraction. Since the short videos generated by users vary greatly in shooting and editing, this method is more suitable for processing short videos generated by users. Moreover, the complex video preprocessing steps are omitted, and there is no need to decompress the video data, which is more efficient in terms of calculation and storage.

[0119] Compared with the method of the method for multi-source data normalization processing and fusion, the present application directly uses the compressed video bitstream as the input, avoiding complex video frame processing and partitioning steps. Especially when processing a large amount of video data, the present application has obvious advantages in saving computing resources and improving processing efficiency.

[0120] When compared with the multi-modal video classification method based on the dynamic fusion gate mechanism again, the present application reduces the dependence on specific models, such as ViT (Vision Transformer, the graphical version of Transformer) and Bert (Bidirectional Encoder Representation from Transformers, the encoder of the bidirectional Transformer), by directly using the compressed video bitstream. This enables the present application to have stronger generalization ability for different types and qualities of video content. Regardless of the complexity of the video content, effective classification can be carried out. In addition, the present application reduces the additional training and adjustment costs by directly processing the compressed video bitstream.

[0121] Therefore, in terms of computing requirements and storage requirements, since the present application uses the compressed video bitstream as the input instead of the decompressed pixel data, the computing and storage requirements are greatly reduced. Traditional methods need to completely decode the compressed video to extract pixel data, resulting in a storage increase ratio as high as 75:1, while the present application avoids this process. In addition, since there is no need to store a large amount of decoded video data, the demand for storage space is reduced, especially when processing high-resolution and long-duration videos.

[0122] In terms of protecting privacy and security, since the present application does not need to decrypt or decompress the video, privacy protection is enhanced when processing videos containing sensitive information, which is particularly important for videos that comply with the Digital Rights Management (DRM) policy. Such videos cannot even be decrypted during transmission. The present application can effectively classify without revealing the video content, providing an extensible solution for network operators.

[0123] In terms of classification performance, taking the use of a single Nvidia A100 GPU as an example, the present application can process 1,818 test videos, each containing 3,000 frames, in less than 13 seconds, which is equivalent to a real-time factor of approximately 15,000 for a 30fps video. Compared with the traditional non-deep learning-based dynamic time warping algorithm, the speed of the present application is about 7.5×10^7 times faster, demonstrating the limitations of traditional methods in terms of time complexity.

[0124] In terms of robustness under various coding settings, the present application performs video classification by analyzing information in the bitstream, such as prediction modes, motion vectors, and bitrate allocation. In particular, the deep learning model used in the present application is optimized for compressed video bitstreams and can remain robust under various coding settings, such as the Average Bitrate (ABR) mode, the Constant Bit Rate (CBR) mode, and the presence or absence of B-frames (bidirectional difference frames). Under different combinations of training bitrates, test bitrates, and coding settings, the classification performance is excellent and consistent.

[0125] In addition, in an exemplary embodiment of the present disclosure, a video classification device based on bitstream analysis is also provided. Figure 5 The structural schematic diagram of the video classification device based on bitstream analysis is shown, as Figure 5 As shown, the video classification device 500 based on bitstream analysis may include: a sequence acquisition module 510, a feature extraction module 520, and a class prediction module 530. Among them:

[0126] The sequence acquisition module 510 is configured to acquire the time sequence of the compressed video bitstream;

[0127] The feature extraction module 520 is configured to perform feature extraction processing on the time sequence to obtain target features;

[0128] The class prediction module 530 is configured to predict the video class of the compressed video according to the target features.

[0129] In an exemplary embodiment of the present invention, the feature extraction module 520 includes:

[0130] The network processing sub-module is configured to perform feature extraction processing on the time sequence using a convolutional neural network to obtain target features.

[0131] In an exemplary embodiment of the present invention, the network processing sub-module includes:

[0132] The first extraction unit is configured to perform feature extraction processing on the time sequence using a convolutional neural network to obtain first features;

[0133] A second extraction unit, configured to perform feature extraction processing on the first feature by using a convolutional neural network to obtain a second feature;

[0134] A third extraction unit, configured to perform feature extraction processing on the second feature by using a convolutional neural network to obtain a target feature.

[0135] In an exemplary embodiment of the present invention, the convolutional neural network includes: a residual network; the residual network includes: three residual blocks.

[0136] In an exemplary embodiment of the present invention, the class prediction module 530 includes:

[0137] A feature construction sub-module, configured to construct a feature map of the target feature;

[0138] A mapping processing sub-module, configured to perform mapping processing on the feature map to obtain the video class of the compressed video.

[0139] In an exemplary embodiment of the present invention, the feature construction sub-module includes:

[0140] A global construction unit, configured to construct a feature map of the target feature by using a global average pooling layer.

[0141] In an exemplary embodiment of the present invention, the feature construction sub-module includes:

[0142] A class prediction unit, configured to perform mapping processing on the feature map by using a softmax function to obtain the video class of the compressed video.

[0143] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0144] Figure 6 is a block diagram of an electronic device 600 shown according to an exemplary embodiment. As Figure 6 shown, the electronic device 600 may include: a processor 601, a memory 602. The electronic device 600 may further include one or more of a multimedia component 603, an input / output (I / O) interface 604, and a communication component 605.

[0145] Among them, the processor 601 is used to control the overall operation of the electronic device 600 to complete all or part of the steps in the above video classification method based on bitstream analysis. The memory 602 is used to store various types of data to support the operation of the electronic device 600. These data may include, for example, instructions for any application or method operating on the electronic device 600, as well as application-related data, such as contact data, messages sent and received, pictures, audio, video, and so on. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 603 may include a screen and an audio component. Among them, the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 602 or sent through the communication component 605. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 604 provides an interface between the processor 601 and other interface modules, and the above other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited herein. Therefore, the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module, and so on.

[0146] In one exemplary embodiment, the electronic device 600 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above-mentioned video classification method based on bitstream analysis.

[0147] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-mentioned video classification method based on bitstream analysis are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 602 including program instructions, and the above-mentioned program instructions can be executed by the processor 601 of the electronic device 600 to complete the above-mentioned video classification method based on bitstream analysis.

[0148] Figure 7 is a block diagram of an electronic device 700 shown according to an exemplary embodiment. For example, the electronic device 700 can be provided as a server. Referring to Figure 7 , the electronic device 700 includes a processor 722, the number of which can be one or more, and a memory 732 for storing computer programs executable by the processor 722. The computer programs stored in the memory 732 can include one or more modules each corresponding to a set of instructions. In addition, the processor 722 can be configured to execute the computer program to execute the above-mentioned video classification method based on bitstream analysis.

[0149] In addition, the electronic device 700 can further include a power supply component 726 and a communication component 750. The power supply component 726 can be configured to perform power management of the electronic device 700, and the communication component 750 can be configured to implement communication of the electronic device 700, for example, wired or wireless communication. In addition, the electronic device 700 can further include an input / output (I / O) interface 758. The electronic device 700 can operate based on an operating system stored in the memory 732.

[0150] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-described video classification method based on bitstream analysis are implemented. For example, the non-transitory computer-readable storage medium may be the above-described memory 732 including program instructions, and the above program instructions may be executed by the processor 722 of the electronic device 700 to complete the above-described video classification method based on bitstream analysis.

[0151] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program capable of being executed by a programmable device, and the computer program has a code portion for executing the above-described video classification method based on bitstream analysis when executed by the programmable device.

[0152] The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0153] In addition, it should be noted that, in the above specific embodiments, the various specific technical features described can be combined in any suitable manner without conflict. To avoid unnecessary repetition, the present disclosure does not separately describe various possible combination manners.

[0154] In addition, any combination can be made between various different embodiments of the present disclosure as long as it does not violate the idea of the present disclosure, and it should also be regarded as the content disclosed by the present disclosure.

Claims

1. A video classification method based on bitstream analysis, characterized in that The method includes: Obtaining a time series of a compressed video bitstream; Performing feature extraction processing on the time series to obtain target features; Predicting the video category of the compressed video according to the target features.

2. The video classification method based on bitstream analysis according to claim 1, characterized in that The performing feature extraction processing on the time series to obtain target features includes: Using a convolutional neural network to perform feature extraction processing on the time series to obtain target features.

3. The video classification method based on bitstream analysis according to claim 2, characterized in that The using a convolutional neural network to perform feature extraction processing on the time series to obtain target features includes: Using a convolutional neural network to perform feature extraction processing on the time series to obtain first features; Using a convolutional neural network to perform feature extraction processing on the first features to obtain second features; Using a convolutional neural network to perform feature extraction processing on the second features to obtain target features.

4. The video classification method based on bitstream analysis according to claim 2, characterized in that The convolutional neural network includes: a residual network; the residual network includes: three residual blocks.

5. The video classification method based on bitstream analysis according to claim 1, characterized in that The predicting the video category of the compressed video according to the target features includes: Constructing a feature map of the target features; Performing mapping processing on the feature map to obtain the video category of the compressed video.

6. The video classification method based on bitstream analysis according to claim 5, characterized in that The constructing a feature map of the target features includes: Using a global average pooling layer to construct a feature map of the target features.

7. The video classification method based on bitstream analysis according to claim 5, characterized in that The performing mapping processing on the feature map to obtain the video category of the compressed video includes: Using a softmax function to perform mapping processing on the feature map to obtain the video category of the compressed video.

8. A video classification device based on bitstream analysis, characterized in that It includes: A sequence acquisition module configured to obtain a time series of a compressed video bitstream; A feature extraction module configured to perform feature extraction processing on the time series to obtain target features; A category prediction module configured to predict the video category of the compressed video according to the target features.

9. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that It includes: A memory having a computer program stored thereon; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images