Video data processing method and device

Abstract

One or more embodiments of the invention disclose a video data processing method and device. The method comprises the following steps: firstly, acquiring a target video and a plurality of video category labels related to the video category of the target video, and secondly, performing feature extraction on the target video through an image-text cross-modal pre-training model to obtain a text feature of the target video and a second video feature comprising a plurality of video frames, inputting a second video feature of the target video into the trained video modulation model, and modulating the target video in a preset time period through a time sequence modulation matrix in the video modulation model to obtain a modulated second video feature corresponding to the target video; and finally calculating a first similarity between the modulated second video feature and the plurality of video category labels, calculating a second similarity between the modulated second video feature and the text feature of the target video, and determining the video category of the target video based on the first similarity and the second similarity.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a video data processing method and device. Background Art

[0002] Video data processing is also called video understanding. In the video data processing business, videos with less data are also called small sample videos. Small sample video understanding is the task of processing video data with less data. Specifically, it refers to the task of understanding and analyzing videos using deep learning or machine learning methods when the data volume is very limited.

[0003] In the related art, when processing small sample video data, it is usually necessary to train the model on a large amount of annotated data. However, due to the complexity of video data, such as the diversity of character actions in the video, new categories of video content that are different from the trained video categories may appear in the application scenario. The new categories of video content have less data and corresponding annotations in the initial stage, so it is difficult to obtain large-scale video annotation data in practical applications. The existing cross-modal model of image and text can provide a good representation space for image and text modality matching, but due to the lack of temporal characteristics of the corresponding video of the image modality, it is not feasible to use it directly for video action understanding. Therefore, it is necessary to provide a video data processing method for a small amount of data that can take into account the temporal characteristics of video data. Summary of the invention

[0004] On the one hand, one or more embodiments of the present specification provide a video data processing method, including: obtaining a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is lower than a preset data amount; performing feature extraction on the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data; calculating a first similarity between the modulated second video features and the multiple video category labels, calculating a second similarity between the modulated second video features and the text features of the target video, and determining the video category of the target video based on the first similarity and the second similarity.

[0005] On the other hand, one or more embodiments of the present specification provide a video data processing device, including: a data acquisition module, acquiring a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is lower than a preset data amount; a modulation processing module, extracting features of the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data; a video category determination module, calculating a first similarity between the modulated second video features and the multiple video category labels, calculating a second similarity between the modulated second video features and the text features of the target video, and determining the video category of the target video based on the first similarity and the second similarity.

[0006] On the other hand, one or more embodiments of the present specification provide an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, when the executable instructions are executed, the processor can: a processor; and a memory arranged to store computer executable instructions, when the executable instructions are executed, the processor can: obtain a target video and multiple video category labels related to the video category of the target video, the amount of data in the target video is lower than a preset amount of data; extract features of the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, The second video feature of the target video is input into the trained video modulation model, and the target video is modulated within a preset time period by the timing modulation matrix in the video modulation model to obtain the modulated second video feature corresponding to the target video, wherein the timing modulation matrix is ​​a model parameter constructed based on the video category for enhancing the timing feature of the video data; a first similarity between the modulated second video feature and the multiple video category labels is calculated, a second similarity between the modulated second video feature and the text feature of the target video is calculated, and the video category of the target video is determined based on the first similarity and the second similarity.

[0007] On the other hand, one or more embodiments of the present specification provide a storage medium for storing a computer program, which can be executed by a processor to implement the following process: a processor; and a memory arranged to store computer executable instructions, which, when the executable instructions are executed, can enable the processor to: obtain a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is less than a preset amount of data; perform feature extraction on the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and input the second video features of the target video into a trained video modulation model, modulate the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data; calculate a first similarity between the modulated second video features and the multiple video category labels, calculate a second similarity between the modulated second video features and the text features of the target video, and determine the video category of the target video based on the first similarity and the second similarity.

[0008] On the other hand, one or more embodiments of the present specification also provide a computer program product, including a computer program, which implements the following process when executed by a processor: a processor; and a memory arranged to store computer executable instructions, which, when the executable instructions are executed, can enable the processor to: obtain a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is less than a preset amount of data; perform feature extraction on the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and input the second video features of the target video into a trained video modulation model, modulate the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data; calculate a first similarity between the modulated second video features and the multiple video category labels, calculate a second similarity between the modulated second video features and the text features of the target video, and determine the video category of the target video based on the first similarity and the second similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 is a schematic flow chart of a video data processing method according to an embodiment of the present specification; Figure 2 is a schematic flow chart of a video data processing method according to another embodiment of the present specification; Figure 3 is a schematic diagram of the implementation principle of a video data processing method according to an embodiment of this specification; Figure 4 is a schematic block diagram of a video data processing device according to an embodiment of the present specification; Figure 5 It is a schematic block diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0011] One or more embodiments of the present specification provide a method and apparatus for processing video data.

[0012] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0013] like Figure 1As shown, an embodiment of this specification provides a video data processing method, and the execution subject of the method can be a terminal device or a server, wherein the terminal device can be a certain terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car device, etc.). The server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server in the financial field or the online shopping field, or a background server of an application, etc. In this embodiment, the server is used as an example for detailed description. For the execution process of the terminal device, please refer to the following related content, which will not be repeated here. The method can specifically include the following steps: In step S102, a target video and a plurality of video category labels related to the video category of the target video are obtained, and the data amount in the target video is lower than a preset data amount.

[0014] The target video is a video to be processed, and there are multiple video category labels related to the video category of the target video, each video category label corresponds to a video category, and the video category of the current target video is determined by performing data processing on the target video. Taking the video data processing service of action recognition as an example, the video category of the target video may be one of the actions of raising the head, turning the head, looking far away, or lowering the head. Accordingly, the video category labels related to the video category of the target video include: raising the head, turning the head, looking far away, and lowering the head. By performing video data processing on the target video, it is determined that the action corresponding to the current target video is one of the actions of raising the head, turning the head, looking far away, or lowering the head.

[0015] The target video in the embodiments of the present specification is a video whose data volume is lower than a preset data volume, that is, a small sample video with a limited video data volume.

[0016] In step S104, feature extraction is performed on the target video through the text-image cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and the second video features of the target video are input into the trained video modulation model. The target video is modulated within a preset time period through the timing modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video. The timing modulation matrix is ​​a model parameter constructed based on the video category for enhancing the timing features of video data.

[0017] The cross-modal pre-training model of images and texts in the embodiments of this specification belongs to a multimodal pre-training model, and an existing cross-modal pre-training model of images and texts can be used. This model is a deep learning model for processing data of multiple modalities. By fusing information of different modalities, it provides a more comprehensive understanding and analysis capability for the data to be processed. The cross-modal pre-training model of images and texts learns general cross-modal representations by pre-training on large-scale multimodal data. The pre-training stage usually includes two steps: modal feature extraction and modal fusion. In the modal feature extraction stage, a modality-specific network structure (such as a convolutional neural network, a recurrent neural network, etc.) is used to extract feature representations from the data of each modality, and in the modal fusion stage, the features of different modalities are fused to establish a cross-modal aligned representation space.

[0018] The image-text cross-modal model in the embodiments of this specification may adopt a CLIP (Contrastive Language–Image Pre-training) model, a conceptual subtitle model, and the like.

[0019] The second video feature is an image feature of the target video. The second video feature including multiple video frames can be understood as extracting multiple video frames from all video frames of the target video to represent the image feature of the target video. For example, 16 frames are extracted from a target image including 200 frames, and the image features of the 16 frames are used as the second video feature of the target video.

[0020] In implementation, a method for extracting features from a target video through a cross-modal pre-trained model of images and text to obtain text features of the target video and second video features including multiple video frames can be a method of directly obtaining video features and text features by using a model trained in an end-to-end learning manner, or can be a method of separately processing the image data and text data of the video to obtain the text features and second video features of the target video.

[0021] The video modulation model is a model obtained by training based on a preset loss function, a first video feature library corresponding to multiple videos under multiple video category labels, and each first video feature, wherein the first video feature library includes first video features corresponding to multiple video category labels. As an upstream model, the image-text cross-modal pre-training model can obtain the text features of the target video and the second video features including multiple video frames. As a downstream model, the trained video modulation model can perform modulation processing based on the obtained second video features to obtain the modulated second video features.

[0022] The temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data. In implementation, the temporal modulation matrix can be constructed based on the number of multiple video category labels (or the number of multiple video categories) and the dimension of the feature vector contained in a single video frame in the target video, wherein each video category label corresponds to a video category related to the video category of the target video, the number of video category labels is the same as the number of video categories, and the dimension of the feature vector contained in a single video frame is the dimension of the single-frame feature channel. For example: each row of the temporal modulation matrix represents the dimension of the single-frame feature channel under a video category label (or a video category), and each column of the temporal modulation matrix represents a single-frame feature channel. If the number of video category labels of a temporal modulation matrix is ​​N, and the dimension of the feature vector contained in a single video frame in the target video is D (D-dimensional single-frame feature channel), then the temporal modulation matrix can be expressed as a D*K matrix.

[0023] In the embodiment of the present specification, the number of video frames included in the second video feature after modulation processing is much lower than the second video feature including multiple video frames before modulation. The second video feature after modulation processing can be a second video feature including one video frame.

[0024] In implementation, the second video feature of the target video is input into the trained video modulation model, and the method of modulating the target video within a preset time period through the timing modulation matrix in the video modulation model can be to perform timing analysis processing on the second video feature including multiple video frames within the preset time period, and use the second video feature after the timing analysis processing as the second video feature after modulation processing, or to first perform timing analysis processing on the second video feature including multiple video frames within the preset time period, and then perform temporal average pooling processing on the second video feature after the timing analysis processing to obtain the second video feature after modulation processing.

[0025] In step S106, a first similarity between the modulated second video feature and multiple video category labels is calculated, a second similarity between the modulated second video feature and the text feature of the target video is calculated, and the video category of the target video is determined based on the first similarity and the second similarity.

[0026] In the embodiment of this specification, the similarity used to determine the video category of the target video includes at least a first similarity and a second similarity, wherein the first similarity is the similarity between the modulated second video feature and the video category label, and the second similarity is the visual-text similarity.

[0027] In implementation, the method for calculating the first similarity can be to directly calculate the first similarity based on the modulated second video feature and multiple video category labels respectively, or to calculate the similarity based on the modulated second video feature and a first video feature library (including the first video feature corresponding to multiple video category labels), and determine the first similarity based on the similarity calculation result and multiple video category labels.

[0028] The second similarity may be calculated by using cosine similarity, Euclidean distance, or the like.

[0029] An embodiment of the present specification provides a video data processing method, which first obtains a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is lower than a preset amount of data; then, a feature extraction is performed on the target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and the second video features of the target video are input into a trained video modulation model, and the target video is modulated within a preset time period through a timing modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the timing modulation matrix is ​​a model parameter constructed based on the video category for enhancing the timing features of video data, and finally, a first similarity between the modulated second video features and multiple video category labels is calculated, a second similarity between the modulated second video features and the text features of the target video is calculated, and the video category of the target video is determined based on the first similarity and the second similarity. In this way, through the time-series modulation matrix in the trained video modulation model, multiple video features on the time dimension of the target video can be modulated, so that a single video feature in the target video has a time-series meaning, and a single video feature with a time-series meaning in the target video can be mapped to the same representation space with the video category label and the text feature of the target video, thereby determining the video category of the target video based on the first similarity between the second video feature and the multiple video category labels, the second similarity between the second video feature after the modulation process and the text feature of the target video, and the above-mentioned method of determining the video category can effectively enhance the time-series characteristics of the video features corresponding to the target video, which helps to improve the accuracy of video understanding and the processing efficiency of video data. In addition, since there is only one learnable time-series modulation matrix in the video modulation model, the model parameters are relatively small, and for small sample video scenes with a small amount of data, the degree of overfitting in the video modulation model training process can be effectively reduced. Moreover, in the embodiment of this specification, when determining the video category of the target video, the similarity is calculated based on the first similarity and the second similarity at the same time. The method of calculating the similarity can realize a network structure similar to the residual, and can also further reduce the degree of overfitting in the video modulation model training process.

[0030] like Figure 2 In an embodiment of the present specification, the image-text cross-modal pre-training model in the above step S104 includes a visual encoder and / or a text encoder. The image-text cross-modal pre-training model is used to extract features of the target video, and the processing of obtaining text features of the target video and second video features including multiple video frames can be varied. An optional processing method is provided below, and specific details can be referred to the processing of the following steps S1042-S1044.

[0031] In step S1042, multiple video frames are extracted from the target video, and the multiple video frames are input into a visual encoder to obtain a second video feature including the multiple video frames.

[0032] In implementation, a method of extracting multiple video frames from a target video may adopt a uniform sampling method. For example, if the target video includes 200 video frames, and a uniform sampling method is adopted to extract video frames, then sampling is performed every 10 frames, and a total of 20 times are extracted to obtain 20 video frames. The uniform sampling method can cover more comprehensive video features in the target video. A method of extracting multiple video frames from a target video may also adopt a random sampling method. For example, if the target video includes 200 video frames, and a random sampling method is adopted to extract video frames, and 20 times are randomly extracted to obtain 20 video frames.

[0033] The embodiment of the present application uses a visual encoder to extract video features of the target video. The visual encoder is usually composed of a deep learning model and can use a convolutional neural network or a recurrent neural network. Using a visual encoder for video feature extraction can effectively improve the feature extraction efficiency and the reliability of the extracted video features.

[0034] In step S1044, text description information is extracted from the target video, and the text description information is input into a text encoder to obtain text features of the target video.

[0035] The text description information in the target video includes the text description information in the video data and / or the text description information in the video category label. Continuing with the video data processing service of action recognition as an example, the multiple video category labels related to the video category in the target video include: looking up, turning the head, looking far away, and looking down. Then, the text description information extracted from the target video can be looking up, turning the head, looking far away, and looking down. Then, the above text description information is input into the text encoder to obtain the text features of the target video.

[0036] In implementation, if there is no text description information in the video, step S1044 may not be executed.

[0037] Continue to see Figure 2In the embodiment of the present specification, in the above step S104, the second video feature of the target video is input into the trained video modulation model, and the target video is modulated within a preset time period through the timing modulation matrix in the video modulation model to obtain the second video feature after modulation corresponding to the target video. The processing can be varied, and an optional processing method is provided below. For details, please refer to the processing of the following steps S1046-S1048.

[0038] In step S1046, based on the trained video modulation model, the second video feature including multiple video frames is subjected to timing analysis processing using the trained timing modulation matrix within a preset time period to obtain the second video feature including multiple video frames after the timing analysis processing.

[0039] In implementation, the second video feature including multiple video frames and the time modulation matrix may be subjected to element-by-element multiplication calculation according to the category vector to obtain the second video feature including multiple video frames after time analysis processing based on a preset time period.

[0040] In step S1048, average pooling is performed on the second video features including multiple video frames after the time series analysis process over a preset time period to obtain the second video features after modulation process corresponding to the target video.

[0041] By performing average pooling processing on the second video features including multiple video frames after timing analysis in the time dimension, the second video features can be reduced in dimensionality. For example, the second video features including multiple video frames can be pooled into second video features including one video frame, and the second video features including one video frame are used to represent the video features of the target video. In this way, based on the timing features of the target video, the key features of the target video can be extracted, and the subsequent similarity calculation related to the video features can be facilitated, which is beneficial to improving the efficiency and accuracy of video data processing.

[0042] In the embodiment of this specification, during the execution of steps S1046-S1048, the processing of the above step S104 also includes the processing of the following step S1040.

[0043] In step S1040, based on the trained video modulation model, each first video feature in the first video feature library is subjected to timing analysis processing using the trained timing modulation matrix within a preset time period, and the first video features after the timing analysis processing are subjected to average pooling processing within the preset time period to obtain the first video feature library after modulation processing, wherein the first video feature library includes first video features corresponding to multiple video category labels.

[0044] Multiple video category labels correspond to multiple videos, each video corresponds to a video category, each video category corresponds to a first video feature (i.e., each video category label corresponds to a first video feature), and multiple first video features form a first video feature library. In the embodiment of this specification, the time series analysis processing method for each first video feature in the first video feature library is the same as the above step S1046, and the method of performing average pooling processing on the first video features after time series analysis processing in a preset time period is the same as the above step S1048, which will not be repeated here.

[0045] Corresponding to step S1040, in the embodiment of the present specification, the processing of calculating the first similarity between the second video feature after modulation processing and multiple video category labels in the above step S106 can be varied. An optional processing method is provided below, and specific details can be referred to the processing of the following steps S1060-S1064.

[0046] In step S1060, based on the second video feature after modulation processing and the first video feature library after modulation processing, a visual category similarity vector corresponding to the second video feature after modulation processing is determined, and the visual category similarity vector is used to represent the similarity between the second video feature after modulation processing and each first video feature in the first video feature library after modulation processing.

[0047] In implementation, based on the second video feature after modulation and the first video feature library after modulation, a method for determining the visual category similarity vector corresponding to the second video feature after modulation can be to perform a vector multiplication calculation by category on the second video feature after modulation and the first video feature library after modulation to obtain the visual category similarity vector corresponding to the second video feature after modulation. By performing a vector multiplication calculation by category, the similarity between the second video feature after modulation and each first video feature in the first video feature library after modulation can be obtained.

[0048] In step S1062, one-hot encoding is performed on the multiple video category labels to obtain multiple video category labels after encoding.

[0049] Performing one-hot encoding on video category labels so that each video category label (or video category) corresponds to an independent binary feature can effectively improve the accuracy and reliability of data processing, thereby improving data processing efficiency.

[0050] In step S1064, based on the visual category similarity vector and the multiple video category labels after the encoding process, a first similarity between the modulated second video feature and the multiple video category labels is determined.

[0051] The schematic diagram of the implementation principle of the video data processing method in the embodiment of this specification can be found in Figure 3 , Figure 3 In the formula, N is the number of current video categories (i.e., the number of multiple video category labels), K is the number of training samples in each video category, T is the number of single video sampling frames, and D is the dimension of single frame feature channel. is the element-wise multiplication of class vectors, For the average pooling process according to the time dimension T, is matrix multiplication, is vector multiplication by category, For the summation symbol, is the visual features extracted frame by frame. One-hot encoding for small sample video categories, is the temporal modulation matrix (i.e., the learnable feature modulation vector of each video category, D*N matrix), The test video features after time series analysis processing, are the features of each video after modulation, is the modulated test video feature, is the text feature of the category prompt, is the visual category similarity of the small sample (i.e., the visual category similarity corresponding to the second video feature after modulation processing), .Depend on Figure 3 It can be seen that multiple video frames are extracted from each video set corresponding to multiple video category labels to form a small sample video frame, and the small sample video frame is input into the visual encoder to obtain the frame-by-frame extracted visual features in the form of a matrix of D*T*N*K (i.e., the first video feature library), using the timing modulation matrix After performing time series analysis on the time dimension of the T first video features in the first video feature library, the modulated video feature matrix of D*N*K is obtained. At the same time, the target video is input into the visual encoder of the cross-modal pre-trained model of images and texts to obtain the second video feature containing multiple video frames in the form of a D*T matrix. , using the timing modulation matrix A second video feature comprising a plurality of video frames After the timing analysis is performed, the second video feature after modulation processing is obtained in the form of a D*N matrix Then the second video feature after modulation processing And the first video feature library after modulation processing Perform vector multiplication calculation according to category to obtain the visual category similarity vector corresponding to the second video feature after modulation processing Based on the visual category similarity vector and multiple video category labels after encoding , determine the first similarity between the second video feature after modulation processing and multiple video category labels The text description information extracted from the target video (taking action recognition business as an example, it can be the text description information such as head raising and head turning related to the action recognition label) is input into the text encoder of the image-text cross-modal pre-training model to obtain the text features of the target video. , calculate the second similarity between the second video feature after modulation processing and the text feature of the target video , and finally the first similarity with weight coefficient α is assigned and the second similarity The sum is calculated to obtain the similarity between the video category of the target video and multiple video category labels, thereby determining the video category of the target video.

[0052] In the embodiment of the present specification, the process of determining the video category of the target video based on the first similarity and the second similarity in the above step S106 can be varied. An optional processing method is provided below, and the details can be referred to the processes of the following steps S1062-S1064.

[0053] In step S1062, according to a preset similarity weight rule, a first weight coefficient is configured for the first similarity to obtain the first similarity configured with the first weight coefficient.

[0054] In implementation, the first weight coefficient can be set according to the specific video data processing business, and can take a value of 1.0. By setting the first weight coefficient, the size of the category logits can be effectively adjusted to balance the parameters and improve the accuracy of the visual category of the target video.

[0055] In step S1064 , a video category of the target video is determined based on the first similarity and the second similarity configured with the first weight coefficient.

[0056] The first similarity and the second similarity configured with the first weight coefficient can be summed and calculated, and then the similarities obtained by the summation calculation can be sorted in order from high to low, to determine the video category label with the first ranking in similarity between the target video and multiple video category labels, and to determine the video category of the target video based on the video category label ranked first.

[0057] In the embodiments of this specification, there may be various methods for training the video modulation model. An optional processing method is provided below. For details, please refer to the following processing steps A02-A10.

[0058] In step A02, a plurality of videos including different video categories are obtained, and a temporal modulation matrix to be trained is constructed based on the different video categories of the plurality of videos.

[0059] When constructing the timing modulation matrix to be trained, each video category corresponds to a timing adjustment vector, and multiple timing modulation vectors corresponding to multiple video categories constitute the timing modulation matrix.

[0060] In step A04, feature extraction is performed on each of the multiple videos through a graphic-text cross-modal pre-training model to obtain a first video feature sample including multiple video frames corresponding to each of the multiple videos, and a first video feature sample library is constructed based on the first video feature samples corresponding to each video.

[0061] In step A06, the first video feature sample library is input into the video modulation model, and each first video feature sample in the first video feature sample library is modulated using the constructed timing modulation matrix within a preset time period to obtain the modulated first video feature sample library.

[0062] In implementation, each first video feature sample in the first video feature sample library is modulated using the constructed timing modulation matrix within a preset time period. It can be that each first video feature sample in the first video feature sample library is only modulated using the constructed timing modulation matrix, or it can be that each first video feature sample in the first video feature sample library is first subjected to timing analysis processing using the constructed timing modulation matrix, and then average pooling processing is performed over a preset time period based on each first video feature sample after the timing analysis processing to obtain a first video feature sample library consisting of the modulated first video feature samples. In the model training process, the timing modulation matrix is ​​the only model parameter to be trained, and therefore, it can effectively reduce the degree of overfitting of the model training.

[0063] In step A08, each first video feature sample in the first video feature sample library is traversed, and a single first video feature sample in the first video feature sample library is input into the video modulation model. The single first video feature sample is modulated using the constructed timing modulation matrix within a preset time period to obtain the modulated first video feature sample corresponding to the single video.

[0064] In step A10, the video modulation model is trained based on the first video feature sample library, the single first video feature sample, the first video feature sample library after modulation processing, the first video feature sample after modulation processing and a preset loss function to obtain a trained video modulation model, and the trained video model includes a trained timing modulation matrix.

[0065] Among them, the loss function can adopt the cross entropy loss function.

[0066] The video data processing method in the embodiments of this specification can be used for different video understanding services, such as action recognition service, scene recognition service, video annotation service, event detection service, face recognition service, video classification service, etc. Accordingly, the target video can be any video in the action recognition service, the scene recognition service, the video annotation service, the event detection service, the face recognition service, or the video classification service.

[0067] An embodiment of the present specification provides a video data processing method, which first obtains a target video and multiple video category labels related to the video category of the target video, wherein the amount of data in the target video is lower than a preset amount of data; then, a feature extraction is performed on the target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and the second video features of the target video are input into a trained video modulation model, and the target video is modulated within a preset time period through a timing modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the timing modulation matrix is ​​a model parameter constructed based on the video category for enhancing the timing features of video data, and finally, a first similarity between the modulated second video features and multiple video category labels is calculated, a second similarity between the modulated second video features and the text features of the target video is calculated, and the video category of the target video is determined based on the first similarity and the second similarity. In this way, through the time-series modulation matrix in the trained video modulation model, multiple video features on the time dimension of the target video can be modulated, so that a single video feature in the target video has a time-series meaning, and a single video feature with a time-series meaning in the target video can be mapped to the same representation space with the video category label and the text feature of the target video, thereby determining the video category of the target video based on the first similarity between the second video feature and the multiple video category labels, the second similarity between the second video feature after the modulation process and the text feature of the target video, and the above-mentioned method of determining the video category can effectively enhance the time-series characteristics of the video features corresponding to the target video, which helps to improve the accuracy of video understanding and the processing efficiency of video data. In addition, since there is only one learnable time-series modulation matrix in the video modulation model, the model parameters are relatively small, and for small sample video scenes with a small amount of data, the degree of overfitting in the video modulation model training process can be effectively reduced. Moreover, in the embodiment of this specification, when determining the video category of the target video, the similarity is calculated based on the first similarity and the second similarity at the same time. The method of calculating the similarity can realize a network structure similar to the residual, and can also further reduce the degree of overfitting in the video modulation model training process.

[0068] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0069] The above is a video data processing method provided by one or more embodiments of this specification. Based on the same idea, one or more embodiments of this specification also provide a video data processing device, such as Figure 4 shown.

[0070] The video data processing device includes: a data acquisition module 210, a modulation processing module 220 and a video category determination module 230. A data acquisition module 210 acquires a target video and a plurality of video category labels related to a video category of the target video, wherein the amount of data in the target video is less than a preset amount of data; The modulation processing module 220 extracts features of the target video through the image-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and inputs the second video features of the target video into the trained video modulation model, and modulates the target video within a preset time period through the timing modulation matrix in the video modulation model to obtain the second video features after modulation corresponding to the target video, wherein the timing modulation matrix is ​​a model parameter constructed based on the video category for enhancing the timing features of the video data; The video category determination module 230 calculates a first similarity between the second video feature after modulation processing and multiple video category labels, calculates a second similarity between the second video feature after modulation processing and the text feature of the target video, and determines the video category of the target video based on the first similarity and the second similarity.

[0071] In the embodiment of this specification, the modulation processing module 220 includes: A timing analysis processing unit, based on the trained video modulation model, performs timing analysis processing on the second video features including multiple video frames using the trained timing modulation matrix within a preset time period to obtain the second video features including multiple video frames after the timing analysis processing; The average pooling processing unit performs average pooling processing on the second video feature including multiple video frames after the time series analysis processing in a preset time period to obtain the second video feature after modulation processing corresponding to the target video.

[0072] In the embodiment of this specification, the modulation processing module 220 further includes: The first video feature library modulation processing unit, based on the trained video modulation model, performs timing analysis processing on each first video feature in the first video feature library using the trained timing modulation matrix within a preset time period, and performs average pooling processing on the first video features after the timing analysis processing in the preset time period to obtain the first video feature library after modulation processing, wherein the first video feature library includes first video features corresponding to multiple video category labels.

[0073] In the embodiment of this specification, the video category determination module 230 includes: A first similarity calculation unit is used to calculate a first similarity between the modulated second video feature and a plurality of video category labels; A second similarity calculation unit, which calculates a second similarity between the second video feature after the modulation process and the text feature of the target video; The video category determination unit determines the video category of the target video based on the first similarity and the second similarity.

[0074] In the embodiment of this specification, the first similarity calculation unit further includes: A visual category similarity vector determination subunit determines a visual category similarity vector corresponding to the second video feature after modulation based on the second video feature after modulation and the first video feature library after modulation, wherein the visual category similarity vector is used to represent the similarity between the second video feature after modulation and each first video feature in the first video feature library after modulation; A one-hot encoding processing subunit performs one-hot encoding processing on multiple video category labels to obtain multiple video category labels after encoding processing; The first similarity determination subunit determines a first similarity between the modulated second video feature and the multiple video category labels based on the visual category similarity vector and the multiple video category labels after encoding.

[0075] In the embodiment of the present specification, the temporal modulation matrix is ​​a matrix constructed based on the number of multiple video category labels and the dimension of the feature vector contained in a single video frame in the target video.

[0076] In the embodiment of this specification, the video category determination unit further includes: The weight coefficient setting subunit configures a first weight coefficient for the first similarity according to a preset similarity weight rule to obtain the first similarity configured with the first weight coefficient; The video category determination subunit determines the video category of the target video based on the first similarity and the second similarity configured with a first weight coefficient.

[0077] In the embodiment of this specification, the modulation processing module 220 also includes a video modulation model training unit, and the video modulation model training unit further includes: A time-series modulation matrix construction subunit acquires a plurality of videos including different video categories, and constructs a time-series modulation matrix to be trained based on different video categories of the plurality of videos; A first video feature sample library construction subunit extracts features from each of the multiple videos through a graphic-text cross-modal pre-training model to obtain a first video feature sample including multiple video frames corresponding to each of the multiple videos, and constructs a first video feature sample library based on the first video feature samples corresponding to each video; The first modulation processing subunit inputs the first video feature sample library into the video modulation model, and performs modulation processing on each first video feature sample in the first video feature sample library using the constructed timing modulation matrix within a preset time period to obtain the first video feature sample library after modulation processing; The second modulation processing subunit, for each first video feature sample in the first video feature sample library, inputs a single first video feature sample in the first video feature sample library into the video modulation model, performs modulation processing on the single first video feature sample using the constructed timing modulation matrix within a preset time period, and obtains a modulated first video feature sample corresponding to the single video; The model training subunit performs model training on the video modulation model based on the first video feature sample library, the single first video feature sample, the first video feature sample library after modulation processing, the first video feature sample after modulation processing and a preset loss function to obtain a trained video modulation model, wherein the trained video model includes a trained timing modulation matrix.

[0078] In the embodiment of this specification, the image-text cross-modal pre-training model in the modulation processing module 220 includes a visual encoder and / or a text encoder, and the modulation processing module 220 also includes: A video feature acquisition unit extracts a plurality of video frames from a target video and inputs the plurality of video frames into a visual encoder to obtain a second video feature including the plurality of video frames; and / or, The text feature acquisition unit extracts text description information from the target video and inputs the text description information into a text encoder to obtain text features of the target video.

[0079] The embodiment of the present specification provides a video data processing device, which first acquires a target video and multiple video category labels related to the video category of the target video through a data acquisition module, and the amount of data in the target video is lower than a preset data amount; then, a modulation processing module is used to extract features of the target video through a graphic-text cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and the second video features of the target video are input into a trained video modulation model, and the target video is modulated within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, the temporal modulation matrix is ​​a model parameter constructed based on the video category for enhancing the temporal features of video data, and finally, a video category determination module is used to calculate a first similarity between the modulated second video features and multiple video category labels, and a second similarity between the modulated second video features and the text features of the target video, and the video category of the target video is determined based on the first similarity and the second similarity. In this way, through the time-series modulation matrix in the trained video modulation model, multiple video features on the time dimension of the target video can be modulated, so that a single video feature in the target video has a time-series meaning, and a single video feature with a time-series meaning in the target video can be mapped to the same representation space with the video category label and the text feature of the target video, thereby determining the video category of the target video based on the first similarity between the second video feature and the multiple video category labels, the second similarity between the second video feature after the modulation process and the text feature of the target video, and the above-mentioned method of determining the video category can effectively enhance the time-series characteristics of the video features corresponding to the target video, which helps to improve the accuracy of video understanding and the processing efficiency of video data. In addition, since there is only one learnable time-series modulation matrix in the video modulation model, the model parameters are relatively small, and for small sample video scenes with a small amount of data, the degree of overfitting in the video modulation model training process can be effectively reduced. Moreover, in the embodiment of this specification, when determining the video category of the target video, the similarity is calculated based on the first similarity and the second similarity at the same time. The method of calculating the similarity can realize a network structure similar to the residual, and can also further reduce the degree of overfitting in the video modulation model training process.

[0080] Those skilled in the art should understand that the above-mentioned video data processing device can be used to implement the video data processing method described above, and the detailed description thereof should be similar to the description of the method part above, and will not be further described here to avoid redundancy.

[0081] Based on the same idea, one or more embodiments of this specification also provide an electronic device, such as Figure 5As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and the memory 302 may store one or more storage applications or data. Among them, the memory 302 may be a temporary storage or a permanent storage. The application stored in the memory 302 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the electronic device. The electronic device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input and output interfaces 305, and one or more keyboards 306.

[0082] Specifically in this embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions: Obtaining a target video and a plurality of video category labels related to a video category of the target video, wherein an amount of data in the target video is less than a preset amount of data; Extracting features of a target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including a plurality of video frames, inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; A first similarity between the modulated second video feature and a plurality of video category labels is calculated, a second similarity between the modulated second video feature and a text feature of a target video is calculated, and a video category of the target video is determined based on the first similarity and the second similarity.

[0083] One or more embodiments of the present specification provide a storage medium, the storage medium is used to store computer executable instructions, and the executable instructions implement the following process when executed by a processor: Obtaining a target video and a plurality of video category labels related to a video category of the target video, wherein an amount of data in the target video is less than a preset amount of data; Extracting features of a target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including a plurality of video frames, inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; A first similarity between the modulated second video feature and a plurality of video category labels is calculated, a second similarity between the modulated second video feature and a text feature of a target video is calculated, and a video category of the target video is determined based on the first similarity and the second similarity.

[0084] One or more embodiments of the present specification further provide a computer program product, including a computer program, which implements the following process when executed by a processor: Obtaining a target video and a plurality of video category labels related to a video category of the target video, wherein an amount of data in the target video is less than a preset amount of data; Extracting features of a target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including a plurality of video frames, inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; A first similarity between the modulated second video feature and a plurality of video category labels is calculated, a second similarity between the modulated second video feature and a text feature of a target video is calculated, and a video category of the target video is determined based on the first similarity and the second similarity.

[0085] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0086] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0087] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0088] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0089] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0090] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] One or more embodiments of this specification are described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0098] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0099] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0100] The above description is only one or more embodiments of this specification and is not intended to limit this application. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification should be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A video data processing method, comprising: Acquire a target video and a plurality of video category labels related to the video category of the target video, wherein the data amount in the target video is less than a preset data amount; Extracting features of the target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including a plurality of video frames, inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; Calculate a first similarity between the modulated second video feature and the multiple video category labels, calculate a second similarity between the modulated second video feature and the text feature of the target video, and determine the video category of the target video based on the first similarity and the second similarity.

2. According to the method of claim 1, the step of inputting the second video feature of the target video into a trained video modulation model, modulating the target video within a preset time period using a timing modulation matrix in the video modulation model, and obtaining the modulated second video feature corresponding to the target video, comprises: Based on the trained video modulation model, performing time series analysis processing on a second video feature including a plurality of video frames using a trained time series modulation matrix within a preset time period, and obtaining a second video feature including a plurality of video frames after the time series analysis processing; The second video feature including multiple video frames after the time series analysis is averaged and pooled over a preset time period to obtain the second video feature after modulation corresponding to the target video.

3. The method according to claim 2, further comprising: Based on the trained video modulation model, performing time series analysis processing on each first video feature in the first video feature library using the trained time series modulation matrix within a preset time period, and performing average pooling processing on the first video features after the time series analysis processing in the preset time period to obtain a first video feature library after modulation processing, wherein the first video feature library includes first video features corresponding to a plurality of video category labels; The calculating the first similarity between the modulated second video feature and the plurality of video category labels comprises: Based on the second video feature after the modulation process and the first video feature library after the modulation process, determining a visual category similarity vector corresponding to the second video feature after the modulation process, wherein the visual category similarity vector is used to represent the similarity between the second video feature after the modulation process and each first video feature in the first video feature library after the modulation process; Performing one-hot encoding processing on the multiple video category labels to obtain multiple video category labels after encoding processing; Based on the visual category similarity vector and the plurality of video category labels after encoding, a first similarity between the second video feature after modulation and the plurality of video category labels is determined.

4. According to the method of claim 1, the temporal modulation matrix is ​​a matrix constructed based on the number of the multiple video category labels and the dimension of the feature vector contained in a single video frame in the target video.

5. The method according to claim 1, wherein determining the video category of the target video based on the first similarity and the second similarity comprises: According to a preset similarity weight rule, configuring a first weight coefficient for the first similarity to obtain a first similarity configured with the first weight coefficient; Based on the first similarity and the second similarity configured with a first weight coefficient, a video category of the target video is determined.

6. According to the method of claim 1, the training method of the video modulation model comprises: Acquire a plurality of videos including different video categories, and construct a temporal modulation matrix to be trained based on the different video categories of the plurality of videos; Extract features from each of the multiple videos using a graphic-text cross-modal pre-training model to obtain a first video feature sample including multiple video frames corresponding to each of the multiple videos, and construct a first video feature sample library based on the first video feature samples corresponding to each video; Inputting the first video feature sample library into the video modulation model, modulating each first video feature sample in the first video feature sample library using the constructed timing modulation matrix within a preset time period, and obtaining the modulated first video feature sample library; Traversing each first video feature sample in the first video feature sample library, inputting a single first video feature sample in the first video feature sample library into the video modulation model, and performing modulation processing on the single first video feature sample using the constructed timing modulation matrix within a preset time period to obtain a modulated first video feature sample corresponding to the single video; The video modulation model is trained based on the first video feature sample library, a single first video feature sample, the first video feature sample library after modulation processing, the first video feature sample after modulation processing and a preset loss function to obtain a trained video modulation model, wherein the trained video model includes a trained timing modulation matrix.

7. According to the method of claim 1, the image-text cross-modal pre-training model comprises a visual encoder and / or a text encoder, and the feature extraction of the target video by the image-text cross-modal pre-training model to obtain the text features of the target video and the second video features including a plurality of video frames comprises: Extracting a plurality of video frames from the target video, and inputting the plurality of video frames into the visual encoder to obtain a second video feature including the plurality of video frames; and / or, Extract text description information from the target video, and input the text description information into the text encoder to obtain text features of the target video.

8. According to the method of claim 1, the target video is a video in an action recognition service, a video in a scene recognition service, a video in a video tagging service, a video in an event detection service, a video in a face recognition service, or a video in a video classification service.

9. A video data processing device, comprising: A data acquisition module, which acquires a target video and a plurality of video category labels related to the video category of the target video, wherein the amount of data in the target video is lower than a preset amount of data; A modulation processing module, which extracts features of the target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including multiple video frames, and inputs the second video features of the target video into a trained video modulation model, and modulates the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; A video category determination module calculates a first similarity between the second video feature after the modulation process and the multiple video category labels, calculates a second similarity between the second video feature after the modulation process and the text feature of the target video, and determines the video category of the target video based on the first similarity and the second similarity.

10. An electronic device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Acquire a target video and a plurality of video category labels related to the video category of the target video, wherein the data amount in the target video is less than a preset data amount; Extracting features of the target video through a text-image cross-modal pre-training model to obtain text features of the target video and second video features including a plurality of video frames, inputting the second video features of the target video into a trained video modulation model, modulating the target video within a preset time period through a temporal modulation matrix in the video modulation model to obtain the modulated second video features corresponding to the target video, wherein the temporal modulation matrix is ​​a model parameter constructed based on video categories for enhancing temporal features of video data; Calculate a first similarity between the modulated second video feature and the multiple video category labels, calculate a second similarity between the modulated second video feature and the text feature of the target video, and determine the video category of the target video based on the first similarity and the second similarity.

Images