A video classification model training method and related device

Abstract

The present application discloses a training method for a video classification model, which is applied to the field of artificial intelligence. The method includes: obtaining data to be trained; obtaining a first model and a second model; inputting the data to be trained into the first model and the second model respectively to obtain the first temporal features and the first spatial features extracted by the first model and the second temporal features and the second spatial features extracted by the second model; using the first model as a teacher network, and performing knowledge distillation on the second model according to the first loss function and the second loss function to update the second model, wherein the first loss function is obtained based on the first temporal features and the second temporal features, and the second loss function is obtained based on the first spatial features and the second spatial features. Based on this solution, the computational overhead of training a new model can be reduced while ensuring that the trained new model has a high classification accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method and related devices for a video classification model. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] With the rapid development of social media, a large number of new videos are generated on social media platforms every day, which puts forward an urgent demand for video classification methods. At present, the video classification method in the related art is to classify videos using a trained video classification model. After the video classification model is trained based on the old video data, if the new video data is significantly different from the old video data, it is difficult for the video classification model to classify the new video data well.

[0004] Based on this, a related technology is to retrain a new video classification model based on new video data and old video data. However, retraining the model requires a large number of iterative training, which brings huge computational overhead.

[0005] Therefore, under the condition of limited computing resources, how to obtain a video classification model with high classification accuracy has become an urgent problem to be solved. Summary of the invention

[0006] The present application provides a method for training a video classification model, which reduces the computational overhead of training a new model while ensuring that the new model obtained through training has high classification accuracy.

[0007] In a first aspect, the present application provides a method for training a video classification model, which is applied to an electronic device. The method comprises: the electronic device obtains data to be trained, wherein the data to be trained comprises a plurality of video data, wherein the plurality of video data comprises video data of different categories.

[0008] Then, the electronic device obtains a first model and a second model. The first model is a trained model, and the second model is a model to be trained. The first model is used as a teacher network to train the second model as a student network.

[0009] Secondly, the electronic device inputs the data to be trained into the first model and the second model respectively to obtain the first time series feature and the first spatial feature extracted by the first model and the second time series feature and the second spatial feature extracted by the second model. After extracting the features of the input data to be trained, the first model compresses the spatial dimension features of the data to be trained to obtain the first time series feature. The first model compresses the time dimension features of the data to be trained to obtain the first spatial feature. Similarly, the second model compresses the spatial dimension features of the data to be trained to obtain the second time series feature; and the second model compresses the time dimension features of the data to be trained to obtain the second spatial feature.

[0010] Finally, the electronic device uses the first model as a teacher network and performs knowledge distillation on the second model according to a first loss function and a second loss function to update the second model. The first loss function is obtained based on the first temporal feature and the second temporal feature, and the second loss function is obtained based on the first spatial feature and the second spatial feature.

[0011] In this solution, the electronic device decomposes the temporal features and spatial features of the data to be trained based on the characteristics of the video data itself, and performs knowledge distillation on the old video classification model based on the temporal features and spatial features to obtain a new video classification model.

[0012] Compared with training a new video classification model from scratch, obtaining a new video classification model through knowledge distillation can reduce the number of iterations in the training process and reduce the computational overhead of training a new model. In addition, because knowledge distillation is based on the temporal and spatial features in the video data, and the temporal and spatial features unique to the video data are taken into account, it can ensure that the new model learns the key features of the video data, thereby ensuring that the new model obtained by knowledge distillation has a higher classification accuracy.

[0013] In a possible implementation, the data to be trained is obtained based on a first data set and a second data set. The first data set and the second data set both include a plurality of video data. The video data in the first data set is old video data, that is, the electronic device has performed model training based on the video data in the first data set and obtained a corresponding video classification model. The first model is a video classification model trained based on the first data set. The video data in the second data set is video data newly acquired by the electronic device (that is, new video data), and the electronic device needs to train a new video classification model based on the second data set.

[0014] In a possible implementation, the electronic device obtains the data to be trained including: obtaining first video data in a first data set; determining multiple key frames in the first video data to obtain second video data, wherein the multiple key frames are used to characterize the motion information of objects in the first video data. Among the multiple key frames, the positions of the same object in the first video data are different in different key frames, so the multiple key frames can be used to characterize the motion information of the object in the first video data. Based on the second video data and the second data set, the data to be trained is obtained.

[0015] In this solution, the electronic device selects a portion of representative video data from the old video data and trains a new model based on the new video data and a portion of the old video data, thereby reducing the video data that needs to be stored and lowering storage overhead.

[0016] In a possible implementation manner, a difference value between any two adjacent key frames among the multiple key frames determined by the electronic device is greater than or equal to a preset threshold.

[0017] In this solution, the electronic device can remove redundant images describing action changes in the video data by selecting key frames in the video data based on a preset threshold, and the extracted key frame sequence can maintain the difference, continuity and integrity of the action. Therefore, the electronic device performs model training based on the extracted key frame sequence, which does not affect the classification accuracy of the model, and further extracting the key frame sequence in the video data can further reduce the amount of video data that needs to be stored and reduce storage overhead.

[0018] In a possible implementation, the electronic device obtains the first video data in the first data set, including: the electronic device obtains the classification accuracy of each video data in the first data set; the electronic device determines the first video data based on the classification accuracy of each video data in the first data set. For example, the electronic device may also select some video data with high classification accuracy in the first data set as representative video data, and the first video data is the video data with high classification accuracy.

[0019] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained.

[0020] In a possible implementation, the first spatial feature is obtained by performing a pooling operation on the time dimension feature of the third feature by the first model, and the third feature is obtained by aligning the features of each image frame in the data to be trained based on the trajectory information by the first model. The trajectory information is used to indicate the position of the target object in each image frame of the data to be trained. For example, for a video data in the data to be trained, the target object in the video data is a person, and the trajectory information is used to indicate the position of the person in each image frame of the video data.

[0021] In this solution, by introducing the trajectory information of the video data to align the features of each image frame in the video data when extracting the spatial features of the video data, more accurate spatial features can be obtained.

[0022] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained.

[0023] In a possible implementation, the category of the video data in the second data set is different from the category of the video data in the first data set. That is, compared with the first data set, the categories of the video data in the second data set are all new categories. The category to which some or all of the video data in the second data set belongs does not exist in the first data set.

[0024] In a possible implementation, the method also includes: the electronic device inputs the data to be trained into a second model to obtain a classification result output by the second model; the electronic device determines a third loss function based on the classification result and the true category of the data to be trained; the electronic device performs knowledge distillation on the second model according to the first loss function and the second loss function, including: the electronic device performs knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function to update the second model.

[0025] In this solution, the electronic device determines the classification loss function of the second model based on the classification result predicted by the second model, and then constructs the total loss function of the second model based on the distillation loss function and the classification loss function. In this way, when training the second model, the electronic device trains the second model based on the total loss function, ensuring that the second model can simultaneously learn the knowledge transferred by the first model and the classification knowledge, thereby ensuring the classification accuracy of the second model after training.

[0026] The second aspect of the present application provides a video classification method, including: obtaining a video to be classified; inputting the video to be classified into a second model to obtain a classification result; wherein the second model is obtained by using the first model as a teacher network and performing knowledge distillation according to a first loss function and a second loss function, the first loss function is obtained based on a first temporal feature and a second temporal feature, the second loss function is obtained based on a first spatial feature and a second spatial feature, the first temporal feature and the first spatial feature are obtained after inputting the data to be trained into the first model, and the second temporal feature and the second spatial feature are obtained after inputting the data to be trained into the second model.

[0027] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0028] In a possible implementation, the data to be trained is obtained based on second video data and the second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on first video data, the multiple key frames are used to characterize motion information of objects in the first video data, and the first video data belongs to the first data set.

[0029] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0030] In a possible implementation manner, the first video data is determined based on a classification accuracy of each video data in the first data set.

[0031] In a possible implementation, the first time series feature is obtained by compressing the spatial dimension feature of the data to be trained by the first model;

[0032] The first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0033] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0034] or,

[0035] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0036] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0037] or,

[0038] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0039] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0040] or,

[0041] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0042] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0043] In a possible implementation, the third model is obtained by using the first model as a teacher network and the second model as a student network, and performing knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function, wherein the third loss function is determined based on the classification result and the true category of the data to be trained, and the classification result is obtained after the data to be trained is input into the second model.

[0044] In a third aspect, the present application provides a training device, comprising an acquisition unit and a processing unit; the acquisition unit is used to acquire data to be trained; the acquisition unit is also used to acquire a first model and a second model; the processing unit is used to input the data to be trained into the first model and the second model respectively to acquire a first temporal feature and a first spatial feature extracted by the first model and a second temporal feature and a second spatial feature extracted by the second model; the processing unit is also used to use the first model as a teacher network, and perform knowledge distillation on the second model according to a first loss function and a second loss function to update the second model, wherein the first loss function is obtained based on the first temporal feature and the second temporal feature, and the second loss function is obtained based on the first spatial feature and the second spatial feature.

[0045] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0046] In a possible implementation, the acquisition unit is further used to acquire the first video data in the first data set; the processing unit is further used to determine a plurality of key frames in the first video data to obtain the second video data, wherein the plurality of key frames are used to characterize the motion information of the objects in the first video data; the processing unit is further used to obtain the data to be trained based on the second video data and the second data set.

[0047] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0048] In a possible implementation, the acquisition unit is further used to acquire the classification accuracy of each video data in the first data set; and the processing unit is further used to determine the first video data based on the classification accuracy of each video data in the first data set.

[0049] In a possible implementation, the first time series feature is obtained after the first model compresses the spatial dimension feature of the data to be trained; the first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0050] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0051] or,

[0052] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0053] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0054] or,

[0055] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0056] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0057] or,

[0058] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0059] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0060] In a possible implementation, the processing unit is further used to: input the data to be trained into a second model to obtain a classification result output by the second model; determine a third loss function based on the classification result and the true category of the data to be trained; and perform knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function to update the second model.

[0061] In a fourth aspect, the present application provides a video classification device, comprising an acquisition unit and a processing unit; the acquisition unit is used to acquire a video to be classified; the processing unit is used to input the video to be classified into a second model to obtain a classification result; wherein the second model is obtained by using the first model as a teacher network and performing knowledge distillation according to a first loss function and a second loss function, the first loss function is obtained based on a first temporal feature and a second temporal feature, the second loss function is obtained based on a first spatial feature and a second spatial feature, the first temporal feature and the first spatial feature are obtained after inputting the data to be trained into the first model, and the second temporal feature and the second spatial feature are obtained after inputting the data to be trained into the second model.

[0062] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0063] In a possible implementation, the data to be trained is obtained based on second video data and the second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on first video data, the multiple key frames are used to characterize motion information of objects in the first video data, and the first video data belongs to the first data set.

[0064] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0065] In a possible implementation manner, the first video data is determined based on a classification accuracy of each video data in the first data set.

[0066] In a possible implementation, the first time series feature is obtained after the first model compresses the spatial dimension feature of the data to be trained; the first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0067] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0068] or,

[0069] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0070] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0071] or,

[0072] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0073] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0074] or,

[0075] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0076] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0077] In a possible implementation, the third model is obtained by using the first model as a teacher network and the second model as a student network, and performing knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function, wherein the third loss function is determined based on the classification result and the true category of the data to be trained, and the classification result is obtained after the data to be trained is input into the second model.

[0078] In a fifth aspect, the present application provides an electronic device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or the second aspect is implemented. For the processor to execute the steps in each possible implementation of the first aspect or the second aspect, the details can be referred to the first aspect or the second aspect, and will not be repeated here.

[0079] In a sixth aspect, the present application provides a server, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or the second aspect is implemented. For the processor to execute the steps in each possible implementation of the first aspect or the second aspect, the details can be referred to the first aspect or the second aspect, and will not be repeated here.

[0080] In a seventh aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first or second aspect above.

[0081] In an eighth aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or the second aspect.

[0082] In a ninth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first or second aspect above.

[0083] The tenth aspect of the present application provides a chip system, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the first aspect, for example, sending or processing the data and / or information involved in the above method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0085] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0088] Figure 5 A flowchart of a video classification model training method 500 provided in an embodiment of the present application;

[0089] Figure 6a A schematic diagram of extracting features of video data provided by an embodiment of the present application;

[0090] Figure 6b A schematic diagram of extracting timing characteristics of video data provided by an embodiment of the present application;

[0091] Figure 6c A schematic diagram of extracting spatial features of video data provided by an embodiment of the present application;

[0092] Figure 7 A schematic diagram of a pooling operation provided in an embodiment of the present application;

[0093] Figure 8 A schematic diagram of selecting a key frame in video data provided by an embodiment of the present application;

[0094] Fig. 9 A schematic diagram of a process for training a video classification model provided in an embodiment of the present application;

[0095] Fig.10 A schematic diagram of the workflow of the video understanding platform provided in the embodiment of the present application;

[0096] Fig.11 A comparative diagram of the test results provided for this application;

[0097] Fig.12 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0098] Fig.13 A schematic diagram of the structure of a video classification device provided in an embodiment of the present application;

[0099] Fig.14 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0100] Fig.15 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0102] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0103] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0104] (1) Infrastructure.

[0105] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0106] (2)Data.

[0107] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0108] (3) Data processing

[0109] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0110] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0111] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0112] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0113] (4) General ability.

[0114] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0115] (5) Smart products and industry applications.

[0116] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart electronic devices, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0117] The method provided in this application is described below from the model training side and the model application side:

[0118] The model training method provided in the embodiment of the present application can be specifically applied to data training, machine learning, deep learning and other data processing methods to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on the training data, and finally obtain a trained neural network model (such as the target neural network model in the embodiment of the present application); and the target neural network model can be used for model reasoning, specifically, the input data can be input into the target neural network model to obtain output data.

[0119] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0120] (1) Neural network.

[0121] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0122]

[0123] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0124] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal (for example, the first convolution layer and the second convolution layer in this embodiment). In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way of extracting image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0125] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0126] Specifically, Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0127] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in the present application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0128] Convolutional layer / pooling layer 120. Convolutional layer: Figure 2 The convolution layer / pooling layer 120 shown may include layers 121-126 as shown in the examples. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0129] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0130] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0131] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0132] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2 The layers 121-126 illustrated in 120 may be a convolutional layer followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers.

[0133] Neural network layer 130: After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 2 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0134] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 140 to 110 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0135] It should be noted that if Figure 2 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 3 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0136] (3) Deep neural networks.

[0137] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0138] (4) Loss function

[0139] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0140] (5) Back propagation algorithm.

[0141] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0142] (6) Linear operation.

[0143] Linearity refers to the proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first-order derivative is a constant. Linear operations can be, but are not limited to, addition operations, null operations, identity operations, convolution operations, batch normalization (BN) operations, and pooling operations. Linear operations can also be called linear mappings. Linear mappings need to meet two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0144] Among them, homogeneity means f(ax)=af(x); additivity means f(x+y)=f(x)+f(y); for example, f(x)=ax is linear. It should be noted that x, a, and f(x) here are not necessarily scalars, but can be vectors or matrices to form a linear space of any dimension. If x and f(x) are n-dimensional vectors, when a is a constant, they are equivalent to satisfying homogeneity, and when a is a matrix, they are equivalent to satisfying additivity. Relatively speaking, a function graph that is a straight line does not necessarily conform to a linear mapping. For example, f(x)=ax+b neither satisfies homogeneity nor additivity, so it belongs to a nonlinear mapping.

[0145] In the embodiment of the present application, a combination of multiple linear operations may be referred to as a linear operation, and each linear operation included in a linear operation may also be referred to as a sub-linear operation.

[0146] (7)Knowledge distillation.

[0147] Knowledge distillation is a method in transfer learning. Knowledge distillation is to train another network by using the output of a pre-trained model as a supervisory signal. Simply put, knowledge distillation is to use a pre-trained network as a teacher network and the network to be trained as a student network. Under the guidance of the teacher network, the student network is trained so that the student network can have the same or similar data processing capabilities as the teacher network. Usually, the teacher network can be a complex network model, and the student network can be a simple network model; through knowledge distillation, the feature representation knowledge learned by the complex network model can be transferred to the simple network model.

[0148] Figure 4 is a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 4 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user can input data to the I / O interface 112 through a client device 140 .

[0149] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 120 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0150] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0151] Optionally, the client device 140, for example, may be a control unit in an autonomous driving system, or a functional algorithm module in a mobile electronic device. For example, the functional algorithm module may be used to implement related tasks.

[0152] It is worth noting that the training device 120 can generate corresponding target models / rules (such as the target neural network model in this embodiment) based on different training data for different goals or different tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0153] exist Figure 4 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific form can be display, sound, action and other specific methods. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0154] It is worth noting that Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 4 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0155] The training method of the video classification model provided in the embodiment of the present application can be applied to electronic devices, especially electronic devices that need to perform classification on videos. Exemplarily, the electronic device can be, for example, a server, a smart phone, a personal computer (PC), a laptop, a tablet computer, a smart TV, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0156] Taking the training method of the video classification model provided in the embodiment of the present application as an example, a video understanding platform and a social media platform are deployed on the server. The video understanding platform in the server can classify the acquired video data based on the video classification model, thereby generating a classification label corresponding to the video data. Then, the video understanding platform in the server can transmit the video data and the classification label corresponding to the video data to the social media platform to realize the classified display of the video data on the social media platform.

[0157] When the video understanding platform in the server obtains new video data, and the new video data is significantly different from the old video data, the server can execute the video classification model training method provided in the embodiment of the present application to train a new video classification model. After the new video classification model is trained, the server continues to classify the obtained video data based on the new video classification model.

[0158] For ease of understanding, the training ideas of the video classification model training method provided in the embodiment of the present application are first explained below.

[0159] In the embodiment of the present application, the electronic device uses knowledge distillation to implement the training of the video classification model. Among them, knowledge distillation uses a pre-trained network as a teacher network and a network to be trained as a student network. Under the guidance of the teacher network, the student network is trained so that the student network can have the same or similar data processing capabilities as the teacher network.

[0160] Specifically, in the embodiment of the present application, the electronic device uses the video classification model trained based on the old video data as the teacher network, and the video classification model to be trained as the student network. Moreover, based on the old video data and the new video data, the electronic device uses the temporal features and spatial features of the video data extracted by the teacher network as supervisory signals to train the student network, thereby achieving rapid training of the student network.

[0161] See also Figure 5 , Figure 5 FIG. 5 is a flow chart of a video classification model training method 500 provided in an embodiment of the present application. Figure 5 As shown, the method 500 includes the following steps 501 to 506.

[0162] Step 501, obtaining data to be trained.

[0163] In this embodiment, before executing the training of the video classification model, the electronic device first obtains the data to be trained for training the video classification model. The data to be trained can be obtained based on the first data set and the second data set, and the first data set and the second data set both include multiple video data. The video data in the first data set is old video data, that is, the electronic device has performed model training based on the video data in the first data set and obtained the corresponding video classification model. The video data in the second data set is the video data newly acquired by the electronic device (that is, new video data), and the electronic device needs to train a new video classification model based on the second data set to obtain a video classification model with high classification accuracy for both new video data and old video data.

[0164] In a possible embodiment, the categories of the video data in the second data set may be the same as the categories of the video data in the first data set. For example, the categories of the video data in the first data set include food, sports, news, and life; the categories of the video data in the second data set also include food, sports, news, and life, or the categories of the video data in the second data set only include food or sports. In other words, for any video data in the second data set, video data of the same category can be found in the first data set.

[0165] However, there are significant differences between the content of the video data in the second data set and the content of the video data in the first data set. Therefore, it is difficult to accurately classify the video data in the second data set using a video classification model trained based on the first data set.

[0166] For example, for the video data belonging to the food category in the first data set, the content of this part of the video data mainly presents the food processing process before cooking, the food cooking process, and the shape of the food after cooking from the perspective of food. However, for the newly acquired second data set, the video data belonging to the food category in the second data set mainly presents various foods from the perspective of people tasting food. Obviously, although some video data in the first data set and some video data in the second data set both belong to the food category, the content of these two parts of video data is quite different. Therefore, it is difficult to accurately classify the video data in the second data set using the video classification model trained based on the first data set.

[0167] In another possible embodiment, the categories of the video data in the second data set are different from the categories of the video data in the first data set. That is, compared with the first data set, the categories of the video data in the second data set are all new categories. The categories to which some or all of the video data in the second data set belong do not exist in the first data set. For example, the categories to which the video data in the second data set belong include entertainment and digital; while the categories to which the video data in the first data set belong include food, sports, news, and life.

[0168] When the category of the video data in the second data set is different from the category of the video data in the first data set, the video classification model trained based on the first data set can only identify the video category corresponding to the first data set, and obviously cannot classify the video data in the second data set.

[0169] Step 502: Obtain a first model and a second model.

[0170] The first model is a model trained based on the first data set, that is, the first model is an old video classification model. When classifying the video data in the first data, the first model can have a higher classification accuracy. The second model is a model to be trained. The first model is used as a teacher network to train the second model as a student network.

[0171] Step 503: input the data to be trained into the first model and the second model respectively to obtain the first temporal features and the first spatial features extracted by the first model and the second temporal features and the second spatial features extracted by the second model.

[0172] For video data, the video data is composed of multiple frames of continuous images. Therefore, compared with the image features that only have three dimensions, the features of video data usually include four dimensions, namely: time dimension (T), channel dimension (C), height dimension (H) and width dimension (W). Among them, the height dimension and width dimension belong to the spatial dimension of video data. In other words, in addition to the spatial dimension of the image frame, the features of video data also have a time dimension. For example, you can refer to Figure 6a , Figure 6a A schematic diagram of extracting features of video data provided by an embodiment of the present application. Figure 6a As shown, after a video including multiple frames of images is input into the first model, the first model extracts features of the video data. The feature matrix of the video data can be represented as X, X∈[T,C,H,W].

[0173] Therefore, in this embodiment, after extracting the features of the input data to be trained, the first model compresses the spatial dimension features of the training data to obtain the first time series features. That is, the first model compresses the height dimension features and width dimension features belonging to the spatial dimension in the features of the training data to obtain the first time series features. The first time series features include the features of the time dimension and the features of the channel dimension. For example, see Figure 6b , Figure 6b A schematic diagram of extracting the timing characteristics of video data provided by an embodiment of the present application. Figure 6b As shown, after the features of the video data are obtained, the first temporal features are obtained by compressing the spatial features in the features of the video data. The matrix of the first temporal features can be expressed as X1, X1∈[T,C].

[0174] Similarly, the first model compresses the time dimension features of the training data to obtain the first spatial features. The first time series features only include channel dimension features and spatial dimension features, namely, channel dimension features, height dimension features, and width dimension features. For example, see Figure 6c , Figure 6c A schematic diagram of extracting spatial features of video data provided by an embodiment of the present application. Figure 6c As shown, after obtaining the features of the video data, the first spatial features are obtained by compressing the temporal features in the features of the video data. The matrix of the first spatial features can be expressed as X2, X2∈[C,H,W].

[0175] The first model may be a neural network model based on deep learning, such as a convolutional neural network (CNN). This embodiment does not specifically limit the type of the first model.

[0176] In addition, the input data corresponding to the first temporal feature and the first spatial feature are the same, that is, the first temporal feature and the first spatial feature are obtained by the electronic device inputting the same video data into the first model. Similarly, the input data corresponding to the second temporal feature and the second spatial feature are also the same. Furthermore, the input data corresponding to the first temporal feature and the input data corresponding to the second temporal feature are also the same.

[0177] Simply put, the electronic device inputs the same video data in the training data into the first model and the second model, and obtains the first temporal features and the first spatial features output by the middle layer of the first model, and the second temporal features and the second spatial features output by the middle layer of the second model.

[0178] Similarly, the second time series feature is obtained after the second model compresses the spatial dimension feature of the data to be trained; the second spatial feature is obtained after the second model compresses the time dimension feature of the data to be trained.

[0179] Optionally, the second model may be obtained based on the first model. Exemplarily, when the category of the video data in the second data set is the same as the category of the video data in the first data set, the structure of the second model may be the same as the structure of the first model. The electronic device obtains the second model by initializing the weight parameters in the first model. Optionally, the second model may also be a model with a simpler structure, that is, the structure of the second model is simpler than that of the first model.

[0180] When the category of the video data in the second data set is different from the category of the video data in the first data set, the structure of the second model can be similar to the structure of the first model. The electronic device obtains the second model by adjusting the branch structure used to predict the classification result in the first model.

[0181] In addition, the structure of the second model may be different from that of the first model. For example, compared with the first model, the structure of the second model is simpler and is a more lightweight model. The second model may also be a neural network model based on deep learning, such as CNN. This embodiment does not specifically limit the type of the second model.

[0182] Step 504, using the first model as a teacher network, and performing knowledge distillation on the second model according to a first loss function and a second loss function to update the second model, wherein the first loss function is obtained based on the first time series feature and the second time series feature, and the second loss function is obtained based on the first spatial feature and the second spatial feature.

[0183] In this embodiment, the process of the electronic device performing knowledge distillation on the second model using the first model as the teacher network may be: the electronic device determines a final distillation loss function based on the first loss function and the second loss function, and trains the second model based on the distillation loss function until a convergence condition is reached to update the second model. After the update is completed, the second model can be used to perform classification of new video data and old video data.

[0184] Among them, the process of the electronic device training the second model based on the distillation loss function is: the electronic device updates the weight vector of each layer of the neural network in the second model based on the value of the distillation loss function, so that the value of the distillation loss function obtained based on the updated second model continues to decrease. In addition, the convergence condition of the second model may be that the value of the distillation loss function is less than the first threshold or the number of training times of the second model reaches the second threshold. Among them, the first threshold and the second threshold can be set according to the actual training situation, and the embodiment of the present application does not limit the values ​​of the first threshold and the second threshold.

[0185] Specifically, after obtaining the first time series feature and the second time series feature based on the first model and the second model respectively, the electronic device may determine the first loss function based on the two time series features. Exemplarily, the electronic device may determine the first loss function based on the following formula 1.

[0186]

[0187] Among them, L tfKD (θ k ) represents the first loss function; V represents the data to be trained, i.e., video data; y represents the classification label corresponding to the data to be trained; D represents the second data set; M represents the first data set; F tf (f(V;θ k-1 )) represents the first time series feature; F tf (f(V;θ k )) represents the second time series feature; k-1 represents the first model; k represents the second model; f represents the feature extractor; θ represents the parameter of f; || || 2 It represents the 2-norm, which is the square root of the sum of the squares of the absolute values ​​of the vector elements.

[0188] Similarly, after obtaining the first spatial feature and the second spatial feature based on the first model and the second model respectively, the electronic device may determine the second loss function based on the two spatial features. Exemplarily, the electronic device may determine the second loss function based on the following formula 2.

[0189]

[0190] Among them, L sfKD (θk ) represents the second loss function; V represents the data to be trained, i.e., video data; y represents the classification label corresponding to the data to be trained; D represents the second data set; M represents the first data set; F sf (f(V;θ k-1 )) represents the first spatial feature; F sf (f(V;θ k )) represents the second spatial feature; k-1 represents the first model; k represents the second model; f represents the feature extractor; θ represents the parameter of f; || || 2 It represents the 2-norm, which is the square root of the sum of the squares of the absolute values ​​of the vector elements.

[0191] Exemplarily, after determining the first loss function and the second loss function, the electronic device may determine the distillation loss function based on the following Formula 3.

[0192] L fKD (θ k )=L sfKD (θ k )+αL tfKD (θ k ) Formula 3

[0193] Among them, L fKD (θ k ) represents the distillation loss function; L tfKD (θ k ) represents the first loss function; L sfKD (θ k ) represents the second loss function; α is a hyperparameter used to balance the importance of the first loss function and the second loss function.

[0194] In this embodiment, by using old video data and new video data as training data, based on the characteristics of the video data itself, the temporal features and spatial features of the training data are decomposed, and the old video classification model is subjected to knowledge distillation based on the temporal features and spatial features to obtain a new video classification model.

[0195] Compared with training a new video classification model from scratch, obtaining a new video classification model through knowledge distillation can reduce the number of iterations in the training process and reduce the computational overhead of training a new model. In addition, because knowledge distillation is based on the temporal and spatial features in the video data, and the temporal and spatial features unique to the video data are taken into account, it can ensure that the new model learns the key features of the video data, thereby ensuring that the new model obtained by knowledge distillation has a higher classification accuracy.

[0196] In a possible embodiment, in order to ensure the classification accuracy of the trained third model, the electronic device may also determine the classification loss function of the second model based on the classification result predicted by the second model, and then construct the total loss function of the second model based on the distillation loss function and the classification loss function. In this way, during the process of training the second model, the electronic device trains the second model based on the total loss function, ensuring that the second model can simultaneously learn the knowledge transmitted by the first model and the classification knowledge.

[0197] Exemplarily, the method 500 of the above embodiment may further include the following steps: the electronic device inputs the data to be trained into a second model to obtain a classification result output by the second model, and the classification result is the result predicted by the second model. The electronic device determines a third loss function based on the classification result and the true category of the data to be trained. Among them, the third loss function is a classification loss function, which can be determined based on the classification result predicted by the second model and the true category. The third loss function can be, for example, a cross entropy function. The method for obtaining the third loss function can be to refer to the method for obtaining the existing classification loss function, which will not be repeated in this embodiment.

[0198] In the above-mentioned step 506, after obtaining the third loss function, the electronic device can determine the total loss function of the second model according to the first loss function, the second loss function and the third loss function, and train the second model based on the total loss function to update the second model.

[0199] Exemplarily, the total loss function of the second model can be obtained based on the following formula 4.

[0200] Loss = L fkD (θk)+loss truth Formula 4

[0201] Among them, Loss represents the total loss function of the second model; L fKD (θ k ) represents the distillation loss function, which is obtained based on the first loss function and the second loss function; loss truth Represents the third loss function.

[0202] For ease of understanding, the process of obtaining temporal features and spatial features based on the first model and the second model mentioned above will be described in detail below.

[0203] For the first model, after the electronic device inputs the data to be trained into the first model, the first model extracts the features of the data to be trained. The features of the data to be trained include features of four dimensions, namely, time dimension features, channel dimension features, height dimension features, and width dimension features. Then, the first model performs a pooling operation on the width dimension features of the training data to obtain the first feature; the first model performs a pooling operation on the height dimension features of the training data to obtain the second feature. Finally, the first model concatenates the obtained first feature and the second feature to obtain the first time series feature.

[0204] Exemplarily, the process of obtaining the first time series feature by the first model can be expressed by the following formula 5.

[0205] F tf (f(V;θ k ))=pool W (f(V;θ k ))+pool H (f(V;θ k )) Formula 5

[0206] Among them, F tf (f(V;θ k )) represents the first time series feature; pool W (f(V;θ k )) indicates the first feature; pool H (f(V;θ k )) represents the second feature; V represents the data to be trained, that is, video data; y represents the classification label corresponding to the data to be trained; D represents the second data set; M represents the first data set; k-1 represents the first model; k represents the second model; f represents the feature extractor; θ represents the parameter of f.

[0207] That is to say, the first time series feature is obtained by concatenating the first feature and the second feature, the first feature is obtained by the first model performing a pooling operation on the width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on the height dimension feature of the data to be trained.

[0208] Exemplarily, assuming that the first feature is an N*N matrix and the second feature is an M*N matrix, the first feature and the second feature are concatenated to obtain a (N+M)*N matrix, and the (N+M)*N matrix is ​​the first time series feature.

[0209] The pooling operation refers to treating the feature as a matrix and dividing the matrix into several non-overlapping regions, then calculating the mean or maximum value of the feature in each region to obtain the pooling result. The pooling operation actually integrates the feature points in a small neighborhood to obtain new features.

[0210] For example, see Figure 7 , Figure 7 A schematic diagram of a pooling operation provided in an embodiment of the present application. Figure 7 As shown in , pooling operations usually include two types, namely maximum pooling operations and mean pooling operations. Figure 7 In the maximum pooling operation shown, the 4*4 feature is transformed into a 2*2 feature. Specifically, the 4*4 feature is first divided into 4 non-overlapping regions, each of which is 2*2 in size, and then the maximum pooling operation determines the maximum value in each region as the value in the transformed feature. For example, in the region in the upper left corner, the maximum value is 100; in the region in the upper right corner, the maximum value is 184; in the region in the lower left corner, the maximum value is 12; and in the region in the lower right corner, the maximum value is 45. Therefore, the feature obtained after performing the maximum pooling operation is [100, 184, 12, 45].

[0211] Similarly, the mean pooling operation is to determine the mean of each region obtained by the division, and use the mean of each region as the value in the transformed feature. For example, in the region of the upper left corner, the mean is 36; in the region of the upper right corner, the mean is 80; in the region of the lower left corner, the mean is 12; in the region of the lower right corner, the mean is 15. Therefore, the feature obtained after performing the mean pooling operation is [36,80,12,15].

[0212] Optionally, after the electronic device inputs the data to be trained into the first model, the first model extracts the features of the data to be trained. Then, the first model performs a pooling operation on the time dimension features of the data to be trained to obtain a first spatial feature. In other words, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension features of the data to be trained.

[0213] Exemplarily, the process of obtaining the first spatial feature by the first model can be expressed by the following formula 6.

[0214] F sf (f(V;θ k ))=pool T (f(V;θ k )) Formula 6

[0215] Among them, F sf (f(V;θk )) represents the first spatial feature; pool T (f(V;θ k )) represents the pooling operation performed on the time dimension features of the training data; V represents the training data, that is, the video data; y represents the classification label corresponding to the training data; D represents the second data set; M represents the first data set; k-1 represents the first model; k represents the second model; f represents the feature extractor; θ represents the parameter of f.

[0216] Optionally, for video data, objects in the video data are constantly moving, that is, the same object included in each frame of the video data may be located at different positions. Therefore, the trajectory information of the object itself in the video data can also be regarded as a kind of timing information of the video data. Among them, the trajectory information is used to indicate the position of the target object in each image frame of the data to be trained. For example, for a video data in the data to be trained, the target object in the video data is a person, and the trajectory information is used to indicate the position of the person in each image frame of the video data, such as the trajectory information indicates the specific coordinates of the person in each image frame of the video data.

[0217] Therefore, when extracting the spatial features of video data, the trajectory information of the video data is introduced to align the features of each image frame in the video data, so that more accurate spatial features can be obtained.

[0218] Exemplarily, after the electronic device inputs the data to be trained into the first model, the first model extracts the features of the data to be trained. Then, the first model aligns the features of each image frame in the data to be trained based on the trajectory information of the data to be trained to obtain the fifth feature, which is the aligned feature, and the fifth feature also includes the features of the aforementioned four dimensions. Finally, the first model performs a pooling operation on the time dimension feature of the fifth feature to obtain the first spatial feature. Specifically, in the process of the first model performing a pooling operation on the time dimension feature of the third feature, the first model needs to select a certain area or specific pixel in each image frame to perform the pooling operation. After the first model aligns the features of each image frame in the data to be trained based on the trajectory information of the data to be trained, the features representing the same object in each image frame can be obtained, so that the first model selects the aligned area or pixel in each image frame when performing the pooling operation.

[0219] The trajectory information of the data to be trained may be extracted based on an existing trajectory extraction method, which is not described in detail in this embodiment.

[0220] Specifically, the process of obtaining the first spatial feature based on the trajectory information by the first model can be expressed by the following formula 7.

[0221] Fsf (f(V;θ k )) = poolT(TrajAlign(f(V;θ k ))) Formula 7

[0222] Among them, F sf (f(V;θ k )) represents the first spatial feature; pool T (TrajAlign(f(V;θ k ))) represents the pooling operation on the time dimension feature of the third feature; TrajAlign() represents the feature alignment function based on trajectory information; TrajAlign(f(V;θ k )) represents the third feature; V represents the data to be trained, that is, video data; y represents the classification label corresponding to the data to be trained; D represents the second data set; M represents the first data set; k-1 represents the first model; k represents the second model; f represents the feature extractor; θ represents the parameter of f.

[0223] It is understandable that the above description is about the process of extracting the first temporal feature and the first spatial feature by the first model. Similarly, the second model can also extract the second temporal feature and the second spatial feature based on the same method as the first model. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0224] In general, the second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0225] Optionally, the second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0226] Optionally, the second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0227] The above describes the process of extracting temporal features and spatial features through the model. The following describes the process of an electronic device acquiring data to be trained.

[0228] In this embodiment, in order to ensure that the newly trained third model has a high classification accuracy for both the old video data and the new video data, the electronic device needs to perform training of the third model based on the old video data (i.e., the first data set) and the new video data (i.e., the second data set) at the same time. However, video data is composed of a continuous sequence of images, and each complete video data usually takes up a large storage space. Therefore, when it is necessary to store the old video data for a long time in order to implement the new model training, the old video data often occupies a large amount of storage space, which brings huge storage overhead to the electronic device.

[0229] In view of this, in a possible embodiment, the electronic device may select a portion of representative video data from the old video data, and train a new model based on the new video data and part of the video data from the old video data, thereby reducing the video data that needs to be stored and reducing storage overhead.

[0230] Specifically, after acquiring the first data set, the electronic device may select part of the video data in the first data set as representative video data. Then, the electronic device obtains the data to be trained based on the part of the video data in the first data set and the second data set, that is, the data to be trained includes the part of the video data in the first data set and the second data set.

[0231] Furthermore, in addition to selecting some representative video data in the first data set, the electronic device can further select some key frames in the selected video data, and construct video data with a smaller data volume based on the selected key frames. Finally, the electronic device obtains the data to be trained based on the second data set and the newly constructed video data with a smaller data volume.

[0232] It is understandable that video data is composed of a continuous sequence of images, and there are many redundant images in the image sequence that describe the changes in action. The electronic device can obtain a key frame sequence by extracting key frames from the video data. The key frame sequence can well maintain the difference, continuity and integrity of the action, thereby achieving the characterization of the content characteristics of the video data. Therefore, performing model training based on the extracted key frame sequence does not affect the classification accuracy of the model.

[0233] Exemplarily, in the above step 501, the electronic device obtaining the data to be trained may specifically include the following steps 5011-5013.

[0234] Step 5011: The electronic device obtains first video data in a first data set.

[0235] The first video data may be representative video data selected by the electronic device from the first data set.

[0236] Optionally, the electronic device may randomly select a certain number of video data from the first data set as representative video data, and the first video data may be one of the video data selected by the electronic device. For example, assuming that the categories of the video data in the first data set include food, sports, news, and life, the electronic device may select 100 video data from each category of video data in the first data set as representative video data. In this way, the electronic device may select 400 video data from the first data set, namely, 100 video data of food, 100 video data of sports, 100 video data of news, and 100 video data of life.

[0237] Optionally, the electronic device may also select some video data with high classification accuracy in the first data set as representative video data. Exemplarily, the electronic device may obtain the classification accuracy of each video data in the first data set. For example, the electronic device may input each video data in the first data set into the first model to obtain the probability that the video data predicted by the first model belongs to each category. Based on the probability that the video data belongs to each category and the real category to which the video data belongs, the classification accuracy corresponding to each video data may be determined. Among them, in the prediction result, the greater the probability that the video data belongs to the real category, the higher the classification accuracy corresponding to the video data. In this way, the electronic device may determine the first video data based on the classification accuracy of each video data in the first data set.

[0238] Specifically, after determining the classification accuracy of each video data in the first data set, the electronic device can select multiple video data with higher classification accuracy as representative video data, and the first video data is one of the multiple video data with higher classification accuracy. For example, the electronic device can select 400 video data with the highest classification accuracy in the first data set as representative video data. It is worth noting that in the process of selecting video data, the electronic device needs to ensure that a certain amount of video data is selected in each category to avoid the selected video data only including some categories in the first data set.

[0239] Step 5012: The electronic device determines a plurality of key frames in the first video data to obtain second video data, where the second video data includes the plurality of key frames.

[0240] The multiple key frames are used to represent the motion information of the object in the first video data. In the multiple key frames, the positions of the same object in the first video data are different in different key frames, so the multiple key frames can be used to represent the motion information of the object in the first video data.

[0241] Optionally, the electronic device may select multiple key frames in the first video data by determining the difference value between the image in the first video data and the latest key frame. In simple terms, after determining that the first image frame in the first video data is a key frame, if the movement of the object in the second image frame in the first video data does not change from that of the object in the first image frame, the second image frame is not selected as a key frame. In the image frames after the second image frame, if the object in a certain image frame undergoes a larger movement change compared to the object in the first image frame, the image frame can be determined as a key frame, thereby representing the motion information of the object.

[0242] Exemplarily, the electronic device may first determine the first image in the first video data as a key frame, and then the electronic device sequentially calculates the difference value between the image in the first video data and the key frame. If the difference value between the image in the first video data and the current key frame is less than a preset threshold, the electronic device continues to calculate the difference value between the next image in the first video data and the current key frame; if the difference value between the image in the first video data and the current key frame is greater than or equal to the preset threshold, the electronic device adds the image to the key frame set and determines the image as the latest key frame. In this way, the electronic device continues to sequentially calculate the difference value between the image in the first video data and the latest key frame to determine the key frame in the first video data.

[0243] Specifically, the electronic device can determine the difference value between the image and the key frame by calculating the vector distance between the image in the first video data and the latest key frame. Exemplarily, the electronic device can calculate the vector distance between the image in the first video data and the latest key frame based on Formula 8.

[0244] Dis(v i , v j )=||v i -v j || 2 Formula 8

[0245] Among them, v i , v j Respectively represent the image in the first video data and the latest key frame; Dis(v i , v j) represents the vector distance between the image in the first video data and the latest key frame, || || 2 It represents the 2-norm, which is the square root of the sum of the squares of the absolute values ​​of the vector elements.

[0246] That is, after the electronic device determines to obtain a plurality of key frames in the first video, a difference value between any two adjacent key frames in the plurality of key frames is greater than or equal to a preset threshold.

[0247] For example, see Figure 8 , Figure 8 A schematic diagram of selecting a key frame in video data provided by an embodiment of the present application. Figure 8 As shown, Figure 8 The video data in includes 12 continuous images, and the 12 images in the video data are numbered 1-12. First, the electronic device adds the first image (i.e., image 1) in the video data to the key frame set, and then the electronic device calculates the vector distance between image 2 and image 1. After calculation, it is found that the vector distance between image 2 and image 1 is less than the threshold, and the electronic device continues to calculate the vector distance between image 3 and image 1. When the vector distance between image 3 and image 1 is still less than the threshold, the electronic device continues to calculate the vector distance between image 4 and image 1.

[0248] Since the vector distance between image 4 and image 1 is greater than the threshold, the electronic device adds image 4 to the key frame set and calculates the vector distance between the subsequent images and image 4. Since the vector distances between image 5, image 6, image 7 and image 8 located after image 4 and image 4 are all less than the threshold, image 5, image 6, image 7 and image 8 are not added to the key frame set. Since the vector distance between image 9 and image 4 is greater than the threshold, the electronic device adds image 9 to the key frame set and continues to calculate the vector distance between the images after image 9 and image 9. Since the vector distances between image 10, image 11 and image 12 and image 9 are all less than the threshold, image 10, image 11 and image 12 are not added to the key frame set. Finally, the key frames selected by the electronic device in the video data are: image 1, image 4 and image 9.

[0249] Step 5013: Obtain the data to be trained based on the second video data and the second data set.

[0250] After selecting key frames in the video data and recombining to obtain new video data, the electronic device can obtain the data to be trained based on the video data recombined in the second data set and the first data set. That is, the old video data included in the data to be trained is obtained by the electronic device first selecting part of the video data in the first data set and then selecting key frames in the part of the video data.

[0251] In this embodiment, a small number of representative video samples are first selected at the video sample granularity, and then a small number of key frames are selected for each representative video sample at the video sample internal granularity, thereby obtaining video data composed of key frames. By selecting part of the video data in the first data set as the training data at two granularities, the amount of video data can be greatly reduced, saving the storage overhead of the electronic device.

[0252] For example, see Fig. 9 , Fig. 9 A flow chart of a training video classification model provided in an embodiment of the present application. Fig. 9 As shown, after the electronic device obtains the first data set (i.e., the existing video to be classified), the first model is obtained by training based on the first data set. Moreover, based on the above-mentioned dual-granularity sample selection method, the electronic device selects part of the video data in the first data set as representative video data, and stores the selected representative video data in the video database. Then, after the electronic device obtains the second data set (i.e., the newly added video to be classified), the old video data (i.e., the representative video data in the first data set) is read from the video database, and based on the old video data and the second data set, the knowledge distillation of the second model is performed through the first model, thereby realizing the training of the second model. After the training of the second model is completed, the electronic device also selects part of the video data in the second data set as representative video data through the dual-granularity sample selection method, and stores the selected representative video data in the video database, so as to facilitate the subsequent training of new models.

[0253] For ease of understanding, the training method of the video classification model provided in the embodiment of the present application will be introduced below with reference to specific examples.

[0254] For example, a video understanding platform is deployed on an electronic device, and the video understanding platform can classify the input video, thereby generating a category label for the input video. The category label can be, for example, a news label, a sports label, an entertainment label, or a food label.

[0255] For example, see Fig.10 , Fig.10The workflow diagram of the video understanding platform provided in the embodiment of the present application is as follows. The workflow of the video understanding platform includes the following steps 1 to 3.

[0256] Step 1: Train the initial model based on the first batch of video data.

[0257] The video understanding platform obtains the first batch of video data D 1 Afterwards, the video understanding platform is based on the first batch of video data D 1 A video classification model is trained from scratch to obtain a video classification model F1. Then, the video understanding platform selects a representative sample from the first batch of video data D based on the dual-granularity representative sample selection method described in steps 5011-5013. 1 Select representative video data (i.e., representative samples) from the video data, and continue to select key frames from the representative video data to finally form a video data set M 1 After obtaining the video data set M 1 Afterwards, the video understanding platform collects the video data set M 1 Store in the corresponding storage space.

[0258] Step 2: When the video understanding platform obtains the second batch of video data D 2 Afterwards, the video understanding platform reads the video data set M from the storage space 1 And, based on the second batch of video data D 2 and video data set M 1 , through the training method described in 501-506 and the video classification model F1, the video classification model F2 is trained to obtain the trained video classification model F2.

[0259] After the video classification model F2 is trained, the video understanding platform selects the representative samples from the second batch of video data D based on the dual-granularity representative sample selection method described in steps 5011-5013. 2 Select representative video data (ie, representative samples) from the video data, and continue to select key frames from the representative video data to obtain a video data set N 2 Finally, the video understanding platform collects video data sets N 2 Stored in the corresponding storage space, the original video data set M in the storage space 1 And the newly stored video data set N 2 Together they constitute the video data set M 2 .

[0260] Step 3. Whenever the video understanding platform obtains a new batch of video data, the video understanding platform repeats step 2 to obtain a new video classification model.

[0261] For example, when the video understanding platform obtains the kth batch of video data D k Afterwards, the video understanding platform reads the video data set M from the storage space k-1 And, based on the kth batch of video data D k and video data set M k-1 , through the training method described in 501-506 and the video classification model Fk-1, the video classification model Fk is trained to obtain the trained video classification model Fk.

[0262] After the video classification model Fk is trained, the video understanding platform selects the kth batch of video data D based on the dual-granularity representative sample selection method described in steps 5011-5013. k Select representative video data (ie, representative samples) from the video data, and continue to select key frames from the representative video data to obtain a video data set N k Finally, the video understanding platform collects video data sets N k Stored in the corresponding storage space, the original video data set M in the storage space k-1 And the newly stored video data set N k Together they constitute the video data set M k .

[0263] In order to facilitate verification of the classification accuracy of the model trained by the training method provided in the embodiment of the present application, this embodiment also provides corresponding test results.

[0264] Specifically, in this embodiment, a corresponding model is trained based on each method of the prior art and the training method provided in the embodiment of the present application, and is tested on an incremental video learning task.

[0265] For example, see Fig.11 , Fig.11 This is a comparative diagram of the test results provided in this application. Fig.11 As shown in the figure, Acc.(%) represents the classification performance of the model obtained after the last incremental learning task on all previously appeared category test data; Forget.(%) represents the degradation value of the classification performance of the model on the first batch of category test data (that is, the accuracy of the model obtained in the first incremental learning task minus the accuracy of the model obtained in the last incremental learning task), and Mem.(G) represents the storage overhead for storing representative video key frames in the incremental video learning task.

[0266] Depend on Fig.11It can be seen that on the Something-Something V2 and Kinetics datasets, the classification accuracy of the model trained by the training method provided in the embodiment of the present application on the newly added video categories is significantly better than the existing method, and the forgetting rate is also the lowest, achieving a significant improvement in performance.

[0267] The above introduces the training method of the video classification model provided by the embodiment of the present application. The following will introduce a video classification method provided by the embodiment of the present application.

[0268] The embodiment of the present application also provides a video classification method, which is applied to an electronic device, on which the third model described in the above embodiment is deployed. The electronic device can perform classification on the video based on the third model. Among them, the electronic device used to perform video classification in this embodiment and the electronic device used to train the model in the above embodiment can be different electronic devices. For example, the electronic device used to train the model can be a server, and the electronic device used to perform video classification can be a smart phone. After the server trains the video classification model, it sends the video classification model to the smart phone, and the smart phone performs video classification based on the video classification model. The electronic device used to perform video classification in this embodiment and the electronic device used to train the model in the above embodiment can also be the same electronic device. For example, the electronic device used to train the model and the electronic device used to perform video classification are the same server. After the server trains the video classification model, it obtains the video to be classified and performs video classification based on the video classification model.

[0269] Specifically, the video classification method includes: the electronic device obtains a video to be classified, and inputs the video to be classified into a second model to obtain a classification result. For example, a video understanding platform can be deployed on the electronic device, and when a user uploads a video to the video understanding platform, the electronic device can obtain the video to be classified. Based on the second model deployed on the electronic device, the electronic device can classify the obtained video to be classified to obtain a classification result.

[0270] Among them, the second model is obtained by using the first model as the teacher network and performing knowledge distillation according to the first loss function and the second loss function, the first loss function is obtained based on the first time series feature and the second time series feature, the second loss function is obtained based on the first space feature and the second space feature, the first time series feature and the first space feature are obtained after the data to be trained is input into the first model, and the second time series feature and the second space feature are obtained after the data to be trained is input into the second model.

[0271] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0272] In a possible implementation, the data to be trained is obtained based on second video data and the second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on first video data, the multiple key frames are used to characterize motion information of objects in the first video data, and the first video data belongs to the first data set.

[0273] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0274] In a possible implementation manner, the first video data is determined based on a classification accuracy of each video data in the first data set.

[0275] In a possible implementation, the first time series feature is obtained by compressing the spatial dimension feature of the data to be trained by the first model;

[0276] The first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0277] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0278] or,

[0279] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0280] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0281] or,

[0282] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0283] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0284] or,

[0285] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0286] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0287] In a possible implementation, the third model is obtained by using the first model as a teacher network and the second model as a student network, and performing knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function, wherein the third loss function is determined based on the classification result and the true category of the data to be trained, and the classification result is obtained after the data to be trained is input into the second model.

[0288] See also Fig.12 , Fig.12 This is a schematic diagram of the structure of a training device provided in an embodiment of the present application. Fig.12 As shown, a training device provided by an embodiment of the present application includes: an acquisition unit 1201 and a processing unit 1202; the acquisition unit 1201 is used to acquire data to be trained; the acquisition unit 1201 is also used to acquire a first model and a second model; the processing unit 1202 is used to input the data to be trained into the first model and the second model respectively to obtain the first time series features and the first spatial features extracted by the first model and the second time series features and the second spatial features extracted by the second model; the processing unit 1202 is also used to use the first model as a teacher network, and perform knowledge distillation on the second model according to a first loss function and a second loss function to update the second model, wherein the first loss function is obtained based on the first time series features and the second time series features, and the second loss function is obtained based on the first spatial features and the second spatial features.

[0289] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0290] In a possible implementation, the acquisition unit 1201 is also used to acquire the first video data in the first data set; the processing unit 1202 is also used to determine multiple key frames in the first video data to obtain second video data, and the multiple key frames are used to characterize the motion information of the objects in the first video data; the processing unit 1202 is also used to obtain the data to be trained based on the second video data and the second data set.

[0291] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0292] In a possible implementation, the acquisition unit 1201 is further used to acquire the classification accuracy of each video data in the first data set; the processing unit 1202 is further used to determine the first video data based on the classification accuracy of each video data in the first data set.

[0293] In a possible implementation, the first time series feature is obtained after the first model compresses the spatial dimension feature of the data to be trained; the first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0294] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0295] or,

[0296] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0297] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0298] or,

[0299] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0300] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0301] or,

[0302] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0303] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0304] In one possible implementation, the processing unit 1202 is further used to: input the data to be trained into a second model to obtain a classification result output by the second model; determine a third loss function based on the classification result and the true category of the data to be trained; and perform knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function to update the second model.

[0305] See also Fig.13 , Fig.13 The structure diagram of a video classification device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, a video classification device provided by an embodiment of the present application includes: an acquisition unit 1301 and a processing unit 1302; the acquisition unit 1301 is used to acquire a video to be classified; the processing unit 1302 is used to input the video to be classified into a second model to obtain a classification result; wherein, the second model is obtained by using the first model as a teacher network and performing knowledge distillation according to a first loss function and a second loss function, the first loss function is obtained based on a first time series feature and a second time series feature, the second loss function is obtained based on a first spatial feature and a second spatial feature, the first time series feature and the first spatial feature are obtained after inputting the data to be trained into the first model, and the second time series feature and the second spatial feature are obtained after inputting the data to be trained into the second model.

[0306] In a possible implementation, the data to be trained includes a first data set and a second data set, and the first model is a video classification model trained based on the first data set.

[0307] In a possible implementation, the data to be trained is obtained based on second video data and the second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on first video data, the multiple key frames are used to characterize motion information of objects in the first video data, and the first video data belongs to the first data set.

[0308] In a possible implementation manner, a difference value between any two adjacent key frames in the multiple key frames is greater than or equal to a preset threshold.

[0309] In a possible implementation manner, the first video data is determined based on a classification accuracy of each video data in the first data set.

[0310] In a possible implementation, the first time series feature is obtained after the first model compresses the spatial dimension feature of the data to be trained; the first spatial feature is obtained after the first model compresses the time dimension feature of the data to be trained.

[0311] In a possible implementation, the first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by the first model performing a pooling operation on a width dimension feature of the data to be trained, and the second feature is obtained by the first model performing a pooling operation on a height dimension feature of the data to be trained;

[0312] or,

[0313] The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model performing a pooling operation on the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model performing a pooling operation on the height dimension feature of the data to be trained.

[0314] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information;

[0315] or,

[0316] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

[0317] In a possible implementation, the first spatial feature is obtained after the first model performs a pooling operation on the time dimension feature of the data to be trained;

[0318] or,

[0319] The second spatial feature is obtained after the second model performs a pooling operation on the time dimension feature of the data to be trained.

[0320] In a possible implementation manner, a category of the video data in the second data set is different from a category of the video data in the first data set.

[0321] In a possible implementation, the third model is obtained by using the first model as a teacher network and the second model as a student network, and performing knowledge distillation on the second model according to the first loss function, the second loss function and the third loss function, wherein the third loss function is determined based on the classification result and the true category of the data to be trained, and the classification result is obtained after the data to be trained is input into the second model.

[0322] Next, an execution device provided by an embodiment of the present application is introduced. Fig.14 , Fig.14 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1400 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Among them, the execution device 1400 can be deployed with Fig.14 The data processing device described in the corresponding embodiment is used to implement Fig.14 The data processing function in the corresponding embodiment. Specifically, the execution device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403 and a memory 1404 (wherein the number of the processor 1403 in the execution device 1400 can be one or more, Fig.14 In the example of FIG. 1403 , the processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403 and the memory 1404 may be connected via a bus or other means.

[0323] The memory 1404 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1403. A portion of the memory 1404 may also include a non-volatile random access memory (NVRAM). The memory 1404 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0324] The processor 1403 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0325] The method disclosed in the above embodiment of the present application can be applied to the processor 1403, or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1403. The above processor 1403 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1403 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1404, and the processor 1403 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0326] The receiver 1401 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1402 can be used to output digital or character information through the first interface; the transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1402 can also include a display device such as a display screen.

[0327] In one embodiment of the present application, the processor 1403 is configured to execute Figure 5 The method in the corresponding embodiment.

[0328] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0329] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0330] The execution device, training device or electronic device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the image processing method described in the above embodiment, or so that the chip in the training device executes the image processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0331] For details, please refer to Fig.15 , Fig.15 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1500, NPU 1500 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1503, which is controlled by the controller 1504 to extract matrix data from the memory and perform multiplication operations.

[0332] In some implementations, the operation circuit 1503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1503 is a two-dimensional systolic array. The operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1503 is a general-purpose matrix processor.

[0333] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1508.

[0334] The unified memory 1506 is used to store input data and output data. The weight data is directly transferred to the weight memory 1502 through the direct memory access controller (DMAC) 1505. The input data is also transferred to the unified memory 1506 through the DMAC.

[0335] BIU stands for Bus Interface Unit 1515 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1509 .

[0336] The bus interface unit 1515 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0337] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or to transfer weight data to the weight memory 1502 or to transfer input data to the input memory 1501.

[0338] The vector calculation unit 1507 includes multiple operation processing units, and further processes the output of the operation circuit 1503 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0339] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function; or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in a subsequent layer in a neural network.

[0340] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0341] Unified memory 1506, input memory 1501, weight memory 1502 and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0342] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0343] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0344] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0345] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0346] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A video classification model training method, characterized in that: include: Get the data to be trained; Obtain a first model and a second model; Inputting the to-be-trained data into the first model and the second model respectively to obtain the first temporal features and the first spatial features extracted by the first model and the second temporal features and the second spatial features extracted by the second model; Taking the first model as a teacher network, performing knowledge distillation on the second model according to a first loss function and a second loss function to update the second model, wherein the first loss function is obtained based on the first temporal feature and the second temporal feature, and the second loss function is obtained based on the first spatial feature and the second spatial feature; The first model is a video classification model trained based on a first data set, and obtaining the data to be trained includes: Acquire first video data in the first data set; Determine a plurality of key frames in the first video data to obtain second video data, wherein the plurality of key frames are used to represent motion information of an object in the first video data; The data to be trained is obtained based on the second video data and the second data set.

2. The method according to claim 1, characterized in that A difference value between any two adjacent key frames in the plurality of key frames is greater than or equal to a preset threshold.

3. The method according to any one of claims 1 to 2, characterized in that: The first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by operating the first model on a width dimension feature of the data to be trained, and the second feature is obtained by operating the first model on a height dimension feature of the data to be trained; or, The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model operating the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model operating the height dimension feature of the data to be trained.

4. The method according to any one of claims 1 to 2, characterized in that: The first spatial feature is obtained after the first model operates the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information; or, The second spatial feature is obtained after the second model operates the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

5. A video classification method, characterized in that: include: Get the video to be classified; Inputting the video to be classified into a second model to obtain a classification result; The second model is obtained by performing knowledge distillation based on the first loss function and the second loss function using the first model as the teacher network, the first loss function is obtained based on the first time series feature and the second time series feature, the second loss function is obtained based on the first space feature and the second space feature, the first time series feature and the first space feature are obtained after the data to be trained is input into the first model, and the second time series feature and the second space feature are obtained after the data to be trained is input into the second model; The first model is a video classification model trained based on a first data set, the data to be trained is obtained based on second video data and a second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on the first video data, the multiple key frames are used to characterize the motion information of objects in the first video data, and the first video data belongs to the first data set.

6. The method according to claim 5, characterized in that A difference value between any two adjacent key frames in the plurality of key frames is greater than or equal to a preset threshold.

7. The method according to any one of claims 5 to 6, characterized in that: The first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by operating the first model on a width dimension feature of the data to be trained, and the second feature is obtained by operating the first model on a height dimension feature of the data to be trained; or, The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model operating the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model operating the height dimension feature of the data to be trained.

8. The method according to any one of claims 5 to 6, characterized in that: The first spatial feature is obtained after the first model is executed on the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information; or, The second spatial feature is obtained after the second model operates the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

9. A training device, characterized in that: include: Acquisition unit and processing unit; The acquisition unit is used to acquire the data to be trained; The acquisition unit is further used to acquire the first model and the second model; The processing unit is used to input the to-be-trained data into the first model and the second model respectively, so as to obtain the first temporal features and the first spatial features extracted by the first model and the second temporal features and the second spatial features extracted by the second model; The processing unit is further configured to use the first model as a teacher network, and perform knowledge distillation on the second model according to a first loss function and a second loss function to update the second model, wherein the first loss function is obtained based on the first time series feature and the second time series feature, and the second loss function is obtained based on the first spatial feature and the second spatial feature; The first model is a video classification model trained based on a first data set; The acquisition unit is further used to acquire the first video data in the first data set; The processing unit is further used to determine a plurality of key frames in the first video data to obtain second video data, wherein the plurality of key frames are used to represent motion information of an object in the first video data; The processing unit is further configured to obtain the data to be trained based on the second video data and the second data set.

10. The device according to claim 9, characterized in that A difference value between any two adjacent key frames in the plurality of key frames is greater than or equal to a preset threshold.

11. The device according to any one of claims 9 to 10, characterized in that: The first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by operating the first model on a width dimension feature of the data to be trained, and the second feature is obtained by operating the first model on a height dimension feature of the data to be trained; or, The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model operating the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model operating the height dimension feature of the data to be trained.

12. The device according to any one of claims 9 to 10, characterized in that The first spatial feature is obtained after the first model operates the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information; or, The second spatial feature is obtained after the second model operates the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

13. A video classification device, characterized in that: include: Acquisition unit and processing unit; The acquisition unit is used to acquire the video to be classified; The processing unit is used to input the video to be classified into a second model to obtain a classification result; wherein the second model is obtained by using the first model as a teacher network and performing knowledge distillation according to a first loss function and a second loss function, the first loss function is obtained based on a first temporal feature and a second temporal feature, the second loss function is obtained based on a first spatial feature and a second spatial feature, the first temporal feature and the first spatial feature are obtained after inputting the data to be trained into the first model, and the second temporal feature and the second spatial feature are obtained after inputting the data to be trained into the second model; The first model is a video classification model trained based on a first data set, the data to be trained is obtained based on second video data and a second data set, the second video data includes multiple key frames, the multiple key frames are obtained based on the first video data, the multiple key frames are used to characterize the motion information of objects in the first video data, and the first video data belongs to the first data set.

14. The device according to claim 13, characterized in that A difference value between any two adjacent key frames in the plurality of key frames is greater than or equal to a preset threshold.

15. The device according to claim 13 or 14, characterized in that The first time series feature is obtained by concatenating a first feature and a second feature, the first feature is obtained by operating the first model on a width dimension feature of the data to be trained, and the second feature is obtained by operating the first model on a height dimension feature of the data to be trained; or, The second time series feature is obtained by concatenating the third feature and the fourth feature, the third feature is obtained by the second model operating the width dimension feature of the data to be trained, and the fourth feature is obtained by the second model operating the height dimension feature of the data to be trained.

16. The device according to claim 13 or 14, characterized in that The first spatial feature is obtained after the first model operates the time dimension feature of the fifth feature, and the fifth feature is obtained after the first model aligns the features of each image frame in the data to be trained based on trajectory information; or, The second spatial feature is obtained after the second model operates the time dimension feature of the sixth feature, and the sixth feature is obtained after the second model aligns the features of each image frame in the data to be trained based on trajectory information.

17. An electronic device, characterized in that: The electronic device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the electronic device executes the method according to any one of claims 1 to 8.

18. A server, characterized in that: The server comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the server executes the method according to any one of claims 1 to 8.

19. A computer storage medium, characterized in that: The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 8.

20. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 8.

Images