Video classification method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video classification method and device, electronic equipment, a storage medium and a program product. The video classification method comprises the following steps: acquiring a target video frame sequence; performing tree sampling on the target video frame sequence to obtain a video frame sequence and a key frame of a two-layer structure; performing feature extraction processing on the video frame sequence based on a convolutional neural network model to obtain time sequence features; performing feature extraction processing on the key frame based on a pulse neural network model to obtain pulse features; performing fusion processing on the time sequence features and the pulse features to obtain video fusion features; and performing classification processing according to the video fusion features to obtain category information of the target video frame sequence. According to the method, the feature expressive power of the video can be improved, so that the accuracy of video classification is improved, and a video classification task can be better completed.

Description

Technical Field

[0001] The present disclosure generally relates to the field of classification technology, and more specifically, to a video classification method, device, electronic device, storage medium and program product. Background Art

[0002] Deep convolutional networks are used to perform classification tasks, and although they have achieved great success in most tasks, they are often limited when processing videos.

[0003] In related technologies, when using traditional convolutional neural networks to extract video features, the information is relatively single, making it difficult for the network to capture the diverse features of the video, resulting in low model recognition performance. Therefore, how to better complete the video classification task is an urgent problem to be solved in the current classification field. Summary of the invention

[0004] The present disclosure provides a video classification method, apparatus, electronic device, storage medium and program product, which are used to solve at least one of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a video classification method is provided, which includes: obtaining a target video frame sequence; performing tree sampling on the target video frame sequence to obtain a two-layer video frame sequence and key frames; performing feature extraction processing on the video frame sequence based on a convolutional neural network model to obtain a temporal feature; performing feature extraction processing on the key frames based on a pulse neural network model to obtain a pulse feature; performing fusion processing on the temporal feature and the pulse feature to obtain a video fusion feature; performing classification processing according to the video fusion feature to obtain category information of the target video frame sequence.

[0006] Optionally, the convolutional neural network model includes a first convolutional layer and a second convolutional layer, wherein the feature extraction processing on the video frame sequence based on the convolutional neural network model to obtain the timing features includes: performing grayscale processing on the video frame sequence to obtain a grayscale video frame sequence; performing feature extraction processing on the grayscale video frame sequence based on the first convolutional layer to obtain short-term timing features; performing feature extraction processing on the short-term timing features based on the second convolutional layer to obtain long-term timing features as the timing features.

[0007] Optionally, the feature extraction processing of the key frames based on the pulse neural network model to obtain the pulse characteristics includes: pulse coding processing of the key frames to obtain a key frame pulse sequence; and dynamic feature extraction processing of the key frame pulse sequence based on the pulse neural network model to obtain the pulse characteristics.

[0008] Optionally, the convolutional neural network model uses a ReLU activation function to perform data nonlinear operations, and the pulse neural network model uses a LIF neuron as an activation function to perform data nonlinear operations.

[0009] Optionally, the convolutional neural network model and the pulse neural network model are trained through the following steps: obtaining a sample video frame sequence and sample category information; performing tree sampling on the sample video frame sequence to obtain a two-layer structure of a sample video frame sequence and sample key frames; performing feature extraction processing on the sample video frame sequence based on the convolutional neural network model to be trained to obtain sample time series features; performing feature extraction processing on the sample key frames based on the pulse neural network model to be trained to obtain sample pulse features; performing fusion processing on the sample time series features and the sample pulse features to obtain sample video fusion features; performing classification processing according to the sample video fusion features to obtain predicted category information of the sample video frame sequence; determining a loss value between the predicted category information and the sample category information based on a loss function; and performing back-propagation update on the convolutional neural network model to be trained and the pulse neural network model to be trained using the loss value to obtain the convolutional neural network model and the pulse neural network model.

[0010] Optionally, the sample video frame sequence is a preprocessed video frame sequence, and the preprocessing includes at least one of the following: data augmentation processing, size adjustment processing, normalization processing, and outlier removal processing.

[0011] Optionally, the back propagation update of the spiking neural network model to be trained uses a substitute gradient, and the substitute gradient includes: Among them, α is the learning rate and x is the loss value.

[0012] According to a second aspect of an embodiment of the present disclosure, a video classification device is provided, comprising: an acquisition unit, configured to acquire a target video frame sequence; a sampling unit, configured to perform tree-like sampling on the target video frame sequence to obtain a two-layer structured video frame sequence and key frames; a first extraction unit, configured to perform feature extraction processing on the video frame sequence based on a convolutional neural network model to obtain a temporal feature; a second extraction unit, configured to perform feature extraction processing on the key frame based on a pulse neural network model to obtain a pulse feature; a fusion unit, configured to perform fusion processing on the temporal feature and the pulse feature to obtain a video fusion feature; and a classification unit, configured to perform classification processing according to the video fusion feature to obtain category information of the target video frame sequence.

[0013] Optionally, the convolutional neural network model includes a first convolutional layer and a second convolutional layer, and the first extraction unit is further configured to: perform grayscale processing on the video frame sequence to obtain a grayscale video frame sequence; perform feature extraction processing on the grayscale video frame sequence based on the first convolutional layer to obtain short-term time series features; perform feature extraction processing on the short-term time series features based on the second convolutional layer to obtain long-term time series features as the time series features.

[0014] Optionally, the second extraction unit is further configured to: perform pulse coding processing on the key frame to obtain a key frame pulse sequence; and perform dynamic feature extraction processing on the key frame pulse sequence based on the pulse neural network model to obtain the pulse feature.

[0015] Optionally, the convolutional neural network model uses a ReLU activation function to perform data nonlinear operations, and the pulse neural network model uses a LIF neuron as an activation function to perform data nonlinear operations.

[0016] Optionally, the convolutional neural network model and the pulse neural network model are trained through the following steps: obtaining a sample video frame sequence and sample category information; performing tree sampling on the sample video frame sequence to obtain a two-layer structure of a sample video frame sequence and sample key frames; performing feature extraction processing on the sample video frame sequence based on the convolutional neural network model to be trained to obtain sample time series features; performing feature extraction processing on the sample key frames based on the pulse neural network model to be trained to obtain sample pulse features; performing fusion processing on the sample time series features and the sample pulse features to obtain sample video fusion features; performing classification processing according to the sample video fusion features to obtain predicted category information of the sample video frame sequence; determining a loss value between the predicted category information and the sample category information based on a loss function; and performing back-propagation update on the convolutional neural network model to be trained and the pulse neural network model to be trained using the loss value to obtain the convolutional neural network model and the pulse neural network model.

[0017] Optionally, the sample video frame sequence is a preprocessed video frame sequence, and the preprocessing includes at least one of the following: data augmentation processing, size adjustment processing, normalization processing, and outlier removal processing.

[0018] Optionally, the back propagation update of the spiking neural network model to be trained uses a substitute gradient, and the substitute gradient includes: Among them, α is the learning rate and x is the loss value.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a video classification method according to an exemplary embodiment of the present disclosure.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute a video classification method according to an exemplary embodiment of the present disclosure.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute a video classification method according to an exemplary embodiment of the present disclosure.

[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: According to the video classification method and device, electronic device, and storage medium of the present disclosure, by simultaneously utilizing a convolutional neural network model and a pulse neural network model, firstly, sampling is performed from the target video frame sequence to obtain a video frame sequence and key frames respectively suitable for the two models, and then the pulse neural network model is used to extract pulse features from the key frames, the pulse neural network model is used to control power consumption, and the convolutional neural network model is combined to extract timing features from the video frame sequence to construct a video fusion feature with stronger representation capability, which can improve the feature expression of the video, thereby improving the accuracy of video classification, and can better complete the video classification task.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0025] Figure 1 is a flowchart of a video classification method according to an exemplary embodiment of the present disclosure.

[0026] Figure 2 It is a schematic diagram of the training process of a convolutional neural network model and a pulse neural network model according to an exemplary embodiment of the present disclosure.

[0027] Figure 3 It is a schematic diagram of the architecture of a video classification model according to a specific embodiment of the present disclosure.

[0028] Figure 4 is a block diagram of a video classification apparatus according to an exemplary embodiment of the present disclosure.

[0029] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0032] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0033] Hereinafter, a video classification method and apparatus, an electronic device, and a storage medium according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0034] Figure 1 is a flowchart of a video classification method according to an exemplary embodiment of the present disclosure.

[0035] Reference Figure 1 , in step S101, a target video frame sequence is obtained.

[0036] The target video frame sequence is a sequence of video frames in the target video to be classified. As an example, the target video frame sequence is obtained by decoding the target video. The target video can be a color RGB three-channel video captured by an ordinary camera, and the target video can be obtained from a video dataset or database.

[0037] In step S102, tree sampling is performed on the target video frame sequence to obtain a two-layer structured video frame sequence and key frames.

[0038] Tree sampling is to organize the video frame sequence into a tree-like structure, which can help to select and process video frames efficiently. The tree structure can be regarded as a multi-level sampling of the time series, which makes it more systematic to extract key information from it. When extracting the two-layer structure, several video frames are usually selected from the original video frame sequence (here is the target video frame sequence) according to certain rules. The video frame sequence composed of these video frames constitutes a more detailed level. The key frame is a representative frame extracted from the video frame sequence of the first layer, which is used to summarize the video content or capture important moments, forming the second layer structure.

[0039] In step S103, feature extraction processing is performed on the video frame sequence based on the convolutional neural network model to obtain temporal features.

[0040] As the most commonly used neural network model at present, the convolutional neural network model can extract discriminative temporal features, which helps to ensure the basic effect of video classification.

[0041] Optionally, the convolutional neural network model includes a first convolutional layer and a second convolutional layer, and step S103 includes: performing grayscale processing on the video frame sequence to obtain a grayscale video frame sequence; performing feature extraction processing on the grayscale video frame sequence based on the first convolutional layer to obtain short-term time series features; performing feature extraction processing on the short-term time series features based on the second convolutional layer to obtain long-term time series features as time series features.

[0042] By first gray-scaling the video frame sequence, reducing multiple color channels to only a single brightness channel, the amount of data and computing requirements can be greatly reduced, and it helps to reduce the complexity of the model, save computing resources, and improve the speed of inference. In addition, in the video classification task, color information may not be critical information, and may even cause interference (for example, color changes under different lighting conditions may unnecessarily increase the model's sensitivity to color changes, and when performing specific medical image analysis or satellite image analysis, color information may cover or interfere with important structural information). The brightness information (i.e., grayscale information) of the video may be more important. Grayscale processing helps the model to better extract more recognizable features such as brightness changes, thereby improving the video classification effect. In addition, by first extracting short-term temporal features and then further extracting long-term temporal features on this basis, rather than extracting short-term temporal features and long-term temporal features separately, the short-term temporal features and long-term temporal features can be naturally fused to obtain temporal features that simultaneously reflect the characteristics of the target video in the short-term dimension and the long-term dimension, providing a certain reference for video classification.

[0043] As an example, the convolutional neural network model may include multiple parallel network structures, each of which includes a first convolution layer and a second convolution layer, so as to perform parallel feature extraction on the same video frame sequence to obtain multiple time series features, and then average these time series features (perform Mean operation), and use the average value as the final time series feature of the video frame sequence. The present disclosure is not limited to this.

[0044] In step S104, feature extraction processing is performed on the key frame based on the pulse neural network model to obtain pulse features.

[0045] Spiking Neural Network (SNN) has attracted more and more attention from scholars because it is closer to the operating mechanism of the human brain. A popular method for constructing a spiking neural network is transformed from a deep convolutional network. As a highly biologically simulated neural network model, the spiking neural network can simulate the ability of the human brain to process visual information, making it show significant potential in small target recognition tasks. By extracting the pulse features in the key frames based on the spiking neural network model and combining it with the timing features extracted in step S103, richer video features can be extracted, and the feature expression of the video can be improved, thereby improving the accuracy of video classification and being able to better complete video classification tasks.

[0046] Optionally, step S104 includes: performing pulse coding processing on the key frame to obtain a key frame pulse sequence; performing dynamic feature extraction processing on the key frame pulse sequence based on the pulse neural network model to obtain pulse features. By adopting pulse coding processing to obtain the key frame pulse sequence, the obtained data can be made more suitable for pulse neural network model processing. When processing the key frame pulse sequence, not only can the spatial features of the key frame be processed statically, but also the information can be interpreted dynamically from the time dimension. By simulating this dynamic processing capability, the pulse neural network model can extract pulse features that reflect the key frame pulse characteristics while maintaining low energy consumption, which helps to improve the video classification accuracy.

[0047] As an example, after pulse coding processing, multiple key frame pulse sequences can be obtained. These key frame pulse sequences are dynamically extracted using a pulse neural network model to obtain pulse features including multiple time series. These pulse features can be averaged (Mean operation is performed) to obtain the final pulse features.

[0048] As an example, pulse coding includes Poisson coding or other types of coding for key frames. Specifically, the time step T is set to N, N is a natural number greater than 1 (for example, N = 16), and in the interval [t, t + τ], the probability distribution of the number of pulses generated by Poisson coding is as follows:

[0049]

[0050] Among them, λ is the reciprocal of each pixel value after image normalization, k is the number of pulses occurring in the interval [t, t+τ], and τ is the time interval.

[0051] In step S105, the time series feature and the pulse feature are fused to obtain a video fusion feature.

[0052] As an example, when the video fusion feature is obtained by fusion, a Concat operation can be performed on the time series feature and the pulse feature, so as to splice the two together to obtain the video fusion feature. Of course, other fusion processing can also be used, and the present disclosure does not limit this.

[0053] In step S106, classification processing is performed according to the video fusion features to obtain category information of the target video frame sequence.

[0054] This step performs the final video classification based on the video fusion features obtained in the previous steps. As an example, the video fusion features can be input into a multi-level fully connected layer with gradually decreasing dimensions to achieve further dimensionality reduction and extraction of the video fusion features. The last level of fully connected layer is a classifier, such as a softmax classifier, whose dimension is the number of identifiable categories, and the probability that the target video frame sequence belongs to each category can be obtained. For a binary classification problem, the dimension of the classifier is 1, and its output value represents the probability that the target video frame sequence is a positive class. If the probability output by the classifier is greater than or equal to the probability threshold, the target video frame sequence is considered to be a positive class, otherwise the target video frame sequence is considered to be a negative class; for a multi-classification problem, the category with the highest probability can be determined as the category information of the target video frame sequence, or the category with the highest probability and exceeding a certain threshold can be determined as the category information of the target video frame sequence, and when all probability values ​​output by the classifier are less than the threshold, it is determined that the category of the target video frame sequence cannot be identified.

[0055] According to the video classification method of the exemplary embodiment of the present disclosure, considering that the information extracted by the traditional convolutional neural network model is relatively single, and the pulse neural network model relies on its advantages, although it shows significant potential in the small target recognition task, there is information loss when the pulse neural network model is used to extract video features in practice, which leads to poor video recognition effect. By using the convolutional neural network model and the pulse neural network model at the same time, first sampling from the target video frame sequence to obtain video frame sequences and key frames suitable for the two models respectively, and then using the pulse neural network model to extract pulse features from the key frames, relying on the pulse neural network model to control power consumption, and combining the convolutional neural network model to extract temporal features from the video frame sequence to construct a video fusion feature with stronger representation ability, it can improve the feature expression of the video, thereby improving the accuracy of video classification, and can better complete the video classification task.

[0056] It should be understood that this article Figure 1 and about Figure 1 In the text introduction, different steps are marked with serial numbers. This is to facilitate the distinction between different steps, and does not limit the execution order of different steps. The execution order of the steps is naturally limited by the logical sequence of execution. For example, step S101 must be executed first to obtain the target video frame sequence, and then step S102 can be executed to perform tree sampling on the target video frame sequence. As for step S103 and step S104, the two use different models to perform feature extraction on different video data, so there is no logical sequence relationship. Therefore, in actual implementation, they can be executed in parallel or in sequence, and the present disclosure does not limit this.

[0057] In some embodiments, optionally, the convolutional neural network model uses a ReLU (Rectified Linear Unit) activation function to perform data nonlinear operations, and the spiking neural network model uses a LIF (Leaky Integrate-and-Fire) neuron as an activation function to perform data nonlinear operations. By making the activation functions for performing data nonlinear operations in the convolutional neural network model and the spiking neural network model use the ReLU activation function and the LIF neuron, respectively, the calculation needs of the two models can be met.

[0058] Specifically, the current equation of the LIF neuron model is expressed as:

[0059] I(t)=I R +I C .

[0060] Where I(t) is the current passing through the LIF neuron model at time t, I R is the current through the resistor R, IC is the current passing through capacitor C. According to the above formula, we can get:

[0061]

[0062] Among them, U(t) is the voltage across the LIF neuron model at time t, U res is the resting potential, U0 is the membrane potential at t = 0, τ m is the membrane time constant, where τ m =RC.

[0063] In some embodiments, optionally, the convolutional neural network model and the pulse neural network model are trained by the following steps: obtaining a sample video frame sequence and sample category information; performing tree sampling on the sample video frame sequence to obtain a two-layer structure of a sample video frame sequence and sample key frames; performing feature extraction processing on the sample video frame sequence based on the convolutional neural network model to be trained to obtain sample time series features; performing feature extraction processing on the sample key frames based on the pulse neural network model to be trained to obtain sample pulse features; performing fusion processing on the sample time series features and the sample pulse features to obtain sample video fusion features; performing classification processing according to the sample video fusion features to obtain predicted category information of the sample video frame sequence; determining the loss value between the predicted category information and the sample category information based on the loss function; and using the loss value to back-propagate and update the convolutional neural network model to be trained and the pulse neural network model to be trained to obtain a convolutional neural network model and a pulse neural network model.

[0064] In this embodiment, a convolutional neural network model to be trained and a pulse neural network model to be trained are used to perform the same method as in the previous embodiment on a sample video frame sequence to implement video classification processing, and the category information finally obtained is recorded as predicted category information. The loss value between the predicted analog information and the clearly marked sample category information is calculated as a reference for updating the two models to be trained. After back-propagation updating, the training of the two models can be realized, thereby improving the feature representation capability and thus improving the video classification performance.

[0065] As an example, the loss function is a cross entropy loss function. For a binary classification problem, the cross entropy loss function is expressed as follows:

[0066] Loss=-(ylog(p)+(1-y)log(1-p)).

[0067] Among them, y is the label corresponding to the sample category information, for positive samples y = 1, for negative samples y = 0. p is the predicted category information output by the final model, that is, the probability value of the sample video frame sequence being a positive class.

[0068] For multi-classification problems, the cross entropy loss function is expressed as follows:

[0069]

[0070] Among them, c represents the category, C represents the number of categories that the model can recognize, and y c Indicates that the sample category information corresponds to the label of category c. If y c =1, it means the sample video frame sequence belongs to category c, otherwise it does not belong to category c. c It is the part of the predicted category information output by the final model that corresponds to category c, that is, the probability that the sample video frame sequence is category c.

[0071] In practical applications, the values ​​of the loss function can be obtained for multiple sample video frame sequences respectively, and then the average value of the loss function of each sample is calculated as the loss value used to update the model parameters.

[0072] As an example, a loss threshold can be configured, and when the loss value is greater than or equal to the loss threshold, back propagation updates can be performed to update the model parameters, and then the model with updated parameters is used as the model to be trained, and the above training steps are repeated to obtain an updated loss value, which is then compared with the loss threshold until the loss value is less than the loss threshold. The training can be terminated, and the latest model can be used as the trained convolutional neural network model and pulse neural network model. For example, since the loss value obtained for the first time is often greater than or equal to the loss threshold, back propagation updates can be performed directly after the loss value is obtained for the first time, and then the updated model can be used to continue calculating the new loss value, and then the new loss value can be compared with the loss threshold.

[0073] In some embodiments, optionally, the sample video frame sequence is a preprocessed video frame sequence, and the preprocessing includes at least one of the following: data augmentation processing, size adjustment processing, normalization processing, and outlier removal processing.

[0074] In these embodiments, by performing data augmentation processing to expand the number of sample video frames, the number of samples required for model training can be guaranteed, reducing the risk of the model falling into overfitting. By performing size adjustment processing, the size of the sample video frames input to the model can be made to meet the data size standard allowed by the model, ensuring that the sample video frame sequence is effectively processed. Specifically, the video frames in the sample video frame sequence usually meet the size standard, but for the video frames obtained by the data augmentation processing, their size usually needs to be adjusted. At this time, the newly added video frames after the data augmentation processing can be adjusted separately, or the entire video frame sequence obtained after the data augmentation processing can be resized. The present disclosure does not limit this. Normalization processing can reduce the risk of data imbalance between samples, improve sample quality, and help improve the robustness of subsequent model training. The outlier removal process can reduce the interference of outliers on the video classification process.

[0075] As an example, data augmentation processing can be achieved by flipping, rotating, translating, scaling, noise perturbation, brightness contrast transformation, etc., or by using a generative adversarial network to expand the number of samples.

[0076] As an example, the goal of the resizing process is related to the data size standard allowed by the model. Specifically, if the size of the video frame to be processed is often small, the video frame pixel expansion process can be performed to increase its size. If the size of the video frame to be processed is large, its size can be reduced by removing pixels in the edge area.

[0077] As an example, for each pixel point in the sample video frame, the normalized pixel value is calculated by the following formula to implement the normalization of the sample video frame:

[0078]

[0079] Among them, z norm is the pixel value after normalization, z is the pixel value before normalization, min(z) is the minimum pixel value in the current sample video frame, and max(z) is the maximum pixel value in the current sample video frame.

[0080] As an example, the sample video frame sequence is obtained through the following preprocessing steps: performing data amplification processing on the sample video frames and adjusting the size of the amplified video frames to obtain multiple sample video frames; performing normalization processing on the multiple sample video frames to obtain a sample video frame sequence.

[0081] In some embodiments, optionally, the back propagation update of the spiking neural network model to be trained uses a surrogate gradient, the surrogate gradient comprising:

[0082]

[0083] Among them, α is the learning rate and x is the loss value.

[0084] In this embodiment, since the pulse neural network model processes a discontinuous function, it may be difficult to calculate accurate extraction. By using an alternative gradient to perform the back propagation update of the pulse neural network model, the update process can be guaranteed to be reliable.

[0085] Figure 2 It is a schematic diagram of the training process of a convolutional neural network model and a pulse neural network model according to an exemplary embodiment of the present disclosure.

[0086] Reference Figure 2 ,Training convolutional neural network models and pulse neural network models mainly includes the following 5 steps.

[0087] (1) Initialize the network parameters (corresponding to the neural network parameters) of the convolutional neural network model and the pulse neural network model.

[0088] (2) The preprocessed grayscale sampling video frame sequence is input into the convolutional neural network model to extract the sample time series features, the preprocessed sample key frames are Poisson encoded to obtain a pulse sequence, and the pulse sequence is input into the pulse neural network model to extract the sample pulse features. After fusing the sample time series features and the sample pulse features, the predicted category information corresponding to the sample video is output.

[0089] (3) The loss value (corresponding error) between the predicted category information and the sample category information of the sample video frame sequence calculated based on the cross entropy loss function.

[0090] (4) If the loss value is greater than or equal to the loss threshold, execute step (5) to continue training; until the training is completed when the loss value is less than the loss threshold.

[0091] (5) The convolutional neural network model and the spiking neural network model are back-propagated and updated using the loss value, and steps (2) and (3) are repeatedly executed based on the updated convolutional neural network model and the spiking neural network model to iteratively update the loss value, and then step (5) is executed.

[0092] In this embodiment, the ReLU activation function is used in the back-propagation update process to prevent gradient vanishing and gradient exploding during model training.

[0093] Next, a video classification method according to a specific embodiment of the present disclosure is introduced.

[0094] The video classification method of this specific embodiment is based on a video classification model to classify and identify the target video to obtain category information as a classification and identification result. Figure 3As shown, based on the convolutional neural network model above and the pulse neural network model below, the convolutional neural network includes three parallel network structures, each of which includes the first convolution layer (corresponding to Figure 3 The first block of the grayscale video frame sequence, the number at the bottom of the block indicates the number of output neurons of this layer, and the other blocks are similar, so they will not be explained one by one) and the second convolution layer (corresponding to the 9 blocks after the first convolution layer). Accordingly, a video classification method of fusion pulse features is generated, and the steps of obtaining video fusion features in this method mainly include the following 3 steps.

[0095] (1) The grayscale video frame sequence is input into the first convolution layer of each network structure of the convolutional neural network model to obtain short-term temporal features, and then the short-term temporal features are input into the second convolution layer of each network structure of the convolutional neural network model for further processing to obtain long-term temporal features that integrate the short-term temporal features as a temporal feature. The three temporal features output by the three network structures are then averaged (Mean operation is performed) to obtain the final temporal features.

[0096] (2) The key frame pulse sequence is input into the pulse neural network model to obtain pulse features including multiple time series, and then these pulse features are averaged (Mean operation is performed) to obtain the pulse features of the video key frame.

[0097] (3) The temporal features output by the convolutional neural network model and the pulse features output by the pulse neural network model are fused through the Concat operation to obtain the video fusion features.

[0098] The spiking neural network model is used to extract pulse features containing time series information from key frames, fuse the time series features output by the convolutional neural network model and the pulse features output by the spiking neural network model, and then calculate the output prediction value through the fully connected layer. The convolutional neural network model and the spiking neural network model are iteratively trained in turn, and when the network converges, a usable video classification model is obtained for subsequent video classification tasks.

[0099] In this specific embodiment, the spiking neural network model includes at least one first convolution kernel (corresponding to Figure 3The first block of the key frame pulse sequence passed in) and multiple residual blocks (corresponding to the 9 blocks after the first convolution kernel), the multiple residual blocks are determined based on multiple second convolution kernels, that is, the residual block adopts the form of convolution kernel, and is described as the second convolution kernel to distinguish it from the previous first convolution kernel; the activation layer included in each residual block is set as LIF neuron; the convolution kernel in the convolution neural network model is the same as the first convolution kernel in the pulse neural network model, the residual block in the convolution neural network model is the same as the residual block in the pulse neural network model, and the ReLU activation function is used in the convolution neural network model for data nonlinear operation, and the LIF neuron is used as the activation function for data nonlinear operation in the pulse neural network model. In other words, the difference between the convolution neural network model and the pulse neural network model is that the activation functions used in the residual block are different. In addition, the pulse neural network model has a time step dimension, that is, the LIF neuron can emit multiple pulses and at most one pulse per time step.

[0100] In this specific embodiment, the sizes of the first convolution kernel and the second convolution kernel can be selected according to user needs. For example, the first convolution kernel is a 5*5 convolution kernel, and the second convolution kernel is at least one of a 3*3 convolution kernel and a 1*1 convolution kernel.

[0101] In this specific embodiment, the pulse neural network model includes a 5*5 convolution kernel (i.e., the first convolution kernel) and three residual blocks (i.e., the 9 blocks after the first convolution kernel), which can ensure the effective transmission of key features, wherein each residual block is composed of two 3*3 convolution kernels and one 1*1 convolution kernel, and the activation function adopts LIF neurons, which can balance biological rationality and practicality.

[0102] In this specific embodiment, the convolutional neural network model includes three parallel network structures. Each network structure includes a 5*5 convolution kernel and three residual blocks like the pulse neural network model, which can ensure the effective transmission of key features. Each residual block is composed of two 3*3 convolution kernels and one 1*1 convolution kernel, and ReLU is used as the activation function, which effectively solves the problems of gradient disappearance and gradient explosion.

[0103] In this specific embodiment, the video classification model also includes three fully connected layers, the number of output neurons in each fully connected layer is 1024, 256, and the number of categories (classes), respectively, and a ReLU activation function is used after each fully connected layer.

[0104] In this specific embodiment, the video classification model is trained by the training method described above, using sample videos as training samples and video fusion features as training features. The relevant parameters in the training process include: using Stochastic Gradient Descent (SGD) as the optimizer, the learning rate is set to 0.1, the sample batch batch_size is 32, and the maximum number of training epochs is 30.

[0105] The video recognition method for integrating pulse features provided in this specific embodiment constructs a pulse neural network model and a convolutional neural network model through at least one first convolution kernel and multiple residual blocks. The activation function in the pulse neural network model is constructed using LIF neurons, and the convolution kernel in the convolutional neural network model uses a ReLU activation function to perform data nonlinear operations. While ensuring the effective transmission of key features, it alleviates the problems of gradient vanishing and gradient exploding during network training.

[0106] Figure 4 is a block diagram of a video classification apparatus according to an exemplary embodiment of the present disclosure. Figure 4 The device includes an acquisition unit 401, a sampling unit 402, a first extraction unit 403, a second extraction unit 404, a fusion unit 405, and a classification unit 406.

[0107] The acquisition unit 401 may acquire a target video frame sequence.

[0108] The sampling unit 402 may perform tree sampling on the target video frame sequence to obtain a two-layer structured video frame sequence and key frames.

[0109] The first extraction unit 403 can perform feature extraction processing on the video frame sequence based on the convolutional neural network model to obtain temporal features.

[0110] The second extraction unit 404 may perform feature extraction processing on the key frame based on the pulse neural network model to obtain pulse features.

[0111] The fusion unit 405 may fuse the time series feature and the pulse feature to obtain a video fusion feature.

[0112] The classification unit 406 may perform classification processing according to the video fusion features to obtain category information of the target video frame sequence.

[0113] Optionally, the convolutional neural network model includes a first convolutional layer and a second convolutional layer, and the first extraction unit 403 can also: perform grayscale processing on the video frame sequence to obtain a grayscale video frame sequence; perform feature extraction processing on the grayscale video frame sequence based on the first convolutional layer to obtain short-term time series features; perform feature extraction processing on the short-term time series features based on the second convolutional layer to obtain long-term time series features; and fuse the short-term time series features and the long-term time series features to obtain time series features.

[0114] Optionally, the second extraction unit 404 may further: perform pulse coding processing on the key frame to obtain a key frame pulse sequence; and perform dynamic feature extraction processing on the key frame pulse sequence based on a pulse neural network model to obtain pulse features.

[0115] Optionally, the convolutional neural network model uses a ReLU activation function to perform data nonlinear operations, and the spiking neural network model uses a LIF neuron as an activation function to perform data nonlinear operations.

[0116] Optionally, the convolutional neural network model and the pulse neural network model are trained by the following steps: obtaining a sample video frame sequence and sample category information; performing tree sampling on the sample video frame sequence to obtain a two-layer structure of a sample video frame sequence and sample key frames; performing feature extraction processing on the sample video frame sequence based on the convolutional neural network model to be trained to obtain sample time series features; performing feature extraction processing on the sample key frames based on the pulse neural network model to be trained to obtain sample pulse features; performing fusion processing on the sample time series features and the sample pulse features to obtain sample video fusion features; performing classification processing according to the sample video fusion features to obtain predicted category information of the sample video frame sequence; determining the loss value between the predicted category information and the sample category information based on the loss function; and performing back-propagation update on the convolutional neural network model to be trained and the pulse neural network model to be trained using the loss value to obtain a convolutional neural network model and a pulse neural network model.

[0117] Optionally, the sample video frame sequence is a preprocessed video frame sequence, and the preprocessing includes at least one of the following: data augmentation processing, size adjustment processing, normalization processing, and outlier removal processing.

[0118] Optionally, the back-propagation update of the spiking neural network model to be trained uses a surrogate gradient, which includes: Among them, α is the learning rate and x is the loss value.

[0119] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0120] Figure 5A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0121] Reference Figure 5 The electronic device includes: at least one memory 501 and at least one processor 502, wherein the at least one memory 501 stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor 502, the at least one processor is prompted to execute the target corresponding method as described in the above exemplary embodiment.

[0122] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0123] In an electronic device, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. As an example and not limitation, the processor 502 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0124] The processor 502 may execute instructions or codes stored in the memory 501, wherein the memory 501 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0125] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory.

[0126] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0127] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause the at least one processor to execute the target corresponding method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0128] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including computer instructions, which, when executed by at least one processor, execute the target corresponding method as described in the above exemplary embodiment.

[0129] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video classification method, characterized in that: The video classification method comprises: Get the target video frame sequence; Performing tree sampling on the target video frame sequence to obtain a two-layer structured video frame sequence and key frames; Performing feature extraction processing on the video frame sequence based on a convolutional neural network model to obtain temporal features; Performing feature extraction processing on the key frame based on a pulse neural network model to obtain pulse features; Performing fusion processing on the time series feature and the pulse feature to obtain a video fusion feature; Classification processing is performed according to the video fusion features to obtain category information of the target video frame sequence.

2. The video classification method according to claim 1, characterized in that: The convolutional neural network model includes a first convolutional layer and a second convolutional layer, wherein the feature extraction process is performed on the video frame sequence based on the convolutional neural network model to obtain the temporal features, including: Performing grayscale processing on the video frame sequence to obtain a grayscale video frame sequence; Performing feature extraction processing on the grayscale video frame sequence based on the first convolutional layer to obtain short-term temporal features; Based on the second convolutional layer, feature extraction processing is performed on the short-term time series features to obtain long-term time series features as the time series features.

3. The video classification method according to claim 1, characterized in that: The step of performing feature extraction processing on the key frame based on the pulse neural network model to obtain pulse features includes: Performing pulse coding processing on the key frame to obtain a key frame pulse sequence; The key frame pulse sequence is dynamically subjected to feature extraction processing based on the pulse neural network model to obtain the pulse feature.

4. The video classification method according to claim 1, characterized in that: The convolutional neural network model uses the ReLU activation function to perform data nonlinear operations, and the spiking neural network model uses the LIF neuron as an activation function to perform data nonlinear operations.

5. The video classification method according to any one of claims 1 to 4, characterized in that: The convolutional neural network model and the spiking neural network model are trained by the following steps: Obtain sample video frame sequence and sample category information; Performing tree sampling on the sample video frame sequence to obtain a sample video frame sequence and sample key frames of a two-layer structure; Performing feature extraction processing on the sampled video frame sequence based on the convolutional neural network model to be trained to obtain sample time series features; Performing feature extraction processing on the sample key frame based on the pulse neural network model to be trained to obtain sample pulse features; Performing fusion processing on the sample time series feature and the sample pulse feature to obtain a sample video fusion feature; Performing classification processing according to the sample video fusion features to obtain predicted category information of the sample video frame sequence; Determine a loss value between the predicted category information and the sample category information based on a loss function; The loss value is used to perform back-propagation update on the convolutional neural network model to be trained and the pulse neural network model to be trained to obtain the convolutional neural network model and the pulse neural network model.

6. The video classification method according to claim 5, characterized in that: The sample video frame sequence is a preprocessed video frame sequence, wherein the preprocessing includes at least one of the following: data augmentation processing, size adjustment processing, normalization processing, and outlier removal processing; and / or The back propagation update of the spiking neural network model to be trained uses a surrogate gradient, and the surrogate gradient includes: Among them, α is the learning rate and x is the loss value.

7. A video classification device, characterized in that: The video classification device comprises: An acquisition unit, configured to acquire a target video frame sequence; A sampling unit is configured to perform tree sampling on the target video frame sequence to obtain a video frame sequence and a key frame of a two-layer structure; A first extraction unit is configured to perform feature extraction processing on the video frame sequence based on a convolutional neural network model to obtain a temporal feature; A second extraction unit is configured to perform feature extraction processing on the key frame based on a pulse neural network model to obtain a pulse feature; A fusion unit is configured to fuse the time series feature and the pulse feature to obtain a video fusion feature; The classification unit is configured to perform classification processing according to the video fusion features to obtain category information of the target video frame sequence.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the video classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the video classification method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the at least one processor is prompted to perform the video classification method according to any one of claims 1 to 6.

Images