Video classification method and device, video classification model training method and device, equipment and medium

Abstract

The embodiment of the invention discloses a video classification and model training method and device, equipment and a medium. The method comprises the following steps: acquiring a video sample and text description information corresponding to the video sample; extracting video features of the video sample and text features of the text description information; determining target loss of a to-be-trained video classification model according to the video features and the text features; wherein the target loss is used for updating model parameters of the video classification model.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video classification, and more specifically, to a video classification and model training method, device, equipment and medium thereof. Background Art

[0002] At present, there are usually a large number of cameras deployed in public places. Artificial Intelligence (AI) algorithms can be used to analyze videos, for example but not limited to identifying weather in videos, detecting abnormal user behavior, and estimating crowd density, which has important application value.

[0003] However, in related technologies, video classification models are mostly trained in a supervised manner, which requires detailed manual labeling of video samples, which is time-consuming and labor-intensive. Summary of the invention

[0004] The disclosed embodiments provide a new technical solution for training a video classification model.

[0005] According to a first aspect of the present disclosure, a method for training a video classification model is provided, the method comprising:

[0006] Obtain video samples and their corresponding text description information;

[0007] Extracting video features of the video sample and text features of the text description information;

[0008] According to the video features and the text features, a target loss of the video classification model to be trained is determined; wherein the target loss is used to update the model parameters of the video classification model.

[0009] Optionally, obtaining the category to be classified and the synonyms of the category to be classified as keywords to be searched;

[0010] Searching based on the keywords to be searched, obtaining original video samples and their corresponding original text description information;

[0011] Sampling the original video sample to obtain the video sample;

[0012] The original text description information and synonymous text description information of the original text description information are used as the text description information.

[0013] Optionally, extracting video features of the video sample includes:

[0014] The video features of the video sample are extracted through a video feature extraction network in a video classification model to be trained.

[0015] Optionally, extracting the video features of the video sample by using a video feature extraction network in a video classification model to be trained includes:

[0016] Inputting the video sample into the video feature extraction network to obtain the original video features of the video sample;

[0017] The original video feature is processed into blocks to obtain a first video feature;

[0018] Performing mask processing on the first video feature at a set scale to obtain a second video feature; wherein the set scale includes at least one of the following: video frame, color channel, pixel;

[0019] The second video feature is encoded to obtain the video feature of the video sample.

[0020] Optionally, extracting text features of the text description sample includes:

[0021] The text features of the text description samples are extracted through a video classification network in a video classification model to be trained.

[0022] Optionally, the target loss includes a first loss and a second loss.

[0023] The step of determining a target loss of a video classification model to be trained according to the video features and the text features includes:

[0024] Determine the first loss of the video feature extraction network in the video classification model to be trained according to the video feature; wherein the first loss is used to update the network parameters of the video feature extraction network;

[0025] The second loss of the video classification network in the video classification model to be trained is determined according to the video features and the text features; wherein the second loss is used to update the network parameters of the video classification network.

[0026] Optionally, determining the second loss of the video classification network in the video classification model to be trained according to the video feature and the text feature includes:

[0027] Obtaining an average value of the video feature to obtain a third video feature, and obtaining an average value of the text feature to obtain a first text feature;

[0028] Mapping the first text feature and the third video feature to target dimensions respectively to obtain a second text feature and a fourth video feature;

[0029] Normalizing the second text feature and the fourth video feature respectively to obtain a normalized text feature and a normalized video feature;

[0030] Obtaining a similarity matrix between the normalized text features and the normalized video features;

[0031] The second loss is determined according to the similarity matrix.

[0032] According to a second aspect of the present disclosure, a video classification method is provided, the method comprising:

[0033] Obtaining the video to be classified and the text description information corresponding to the video to be classified in different categories;

[0034] The trained video classification model is used to process the video to be classified and the different text description information to obtain the category to which the video to be classified belongs.

[0035] Optionally, the using a trained video classification model to process the video to be classified and the different text description information to obtain the category to which the video to be classified belongs includes:

[0036] Using the video feature extraction network in the trained video classification model to extract the video features of the video to be classified, and using the video classification network in the trained video classification model to extract text features corresponding to different text description information; and,

[0037] Determine a similarity matrix between the video features and the text features using a video classification network in a trained video classification model;

[0038] The category corresponding to the maximum similarity in the similarity matrix is ​​obtained as the category to which the video to be classified belongs.

[0039] According to a third aspect of the present disclosure, a video classification device is provided, the device comprising:

[0040] The second acquisition module is used to acquire the video to be classified and the text description information corresponding to the video to be classified in different categories;

[0041] The classification module is used to process the video to be classified and the different text description information using the trained video classification model to obtain the category to which the video to be classified belongs.

[0042] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to the first aspect or the second aspect above.

[0043] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect or the second aspect above is implemented.

[0044] One beneficial effect of the disclosed embodiment is that after obtaining the video sample and its corresponding text description information, the video features of the video sample and the text features of the text description information are extracted, and then the target loss of the video classification model to be trained is determined based on the video features and the text features, so as to update the model parameters of the video classification model based on the target loss. That is, the text description information can be understood as the fuzzy text label corresponding to the video sample. By comparing and learning the video sample and its corresponding text description information, there is no need to manually annotate the video sample, which reduces the manpower and training time costs, and has a good training effect in the training of video classification models in different scenarios.

[0045] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0047] Figure 1 is a schematic diagram of the hardware configuration of an electronic device according to an embodiment of the present disclosure;

[0048] Figure 2 is a flowchart of a method for training a video classification model according to an embodiment of the present disclosure;

[0049] Figure 3 is a schematic diagram of a video sample and its corresponding text description information according to an embodiment of the present disclosure;

[0050] Figure 4 is a schematic diagram of the principle of a multi-scale mask according to an embodiment of the present disclosure;

[0051] Figure 5 is a flowchart of a video classification method according to an embodiment of the present disclosure;

[0052] Figure 6 is a flowchart of a method for training a video classification model based on an example;

[0053] Figure 7 is a flowchart of a method for training a video classification model according to another example;

[0054] Figure 8is a principle block diagram of a training device for a video classification model according to an embodiment of the present disclosure;

[0055] Fig. 9 is a principle block diagram of a video classification device according to an embodiment of the present disclosure;

[0056] Fig.10 is a schematic diagram of the hardware configuration of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0060] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0061] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0062] <Hardware Configuration>

[0063] Figure 1 A block diagram showing a hardware configuration of an electronic device 1000 that can implement an embodiment of the present disclosure is shown.

[0064] The electronic device 1000 may be a portable computer, a desktop computer, a mobile phone, a tablet computer, etc. Figure 1As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. Among them, the processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is, for example, capable of wired or wireless communication, and may specifically include Wifi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, and the like. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0065] Figure 1 The electronic device shown is merely illustrative and does not in any way imply any limitation on the present disclosure, its application or use. In the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to perform any one of the video classification model training methods or video classification methods provided in the embodiments of the present disclosure. It should be understood by those skilled in the art that although in Figure 1 In the electronic device 1000, multiple devices are shown, but the present disclosure may only involve some of the devices, for example, the electronic device 1000 only involves the processor 1100 and the storage device 1200. A technician can design instructions according to the scheme disclosed in the present disclosure. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0066] <Method Example 1>

[0067] The present disclosure discloses a method for training a video classification model. Figure 2 As shown, the training method of the video classification model includes the following steps S2100 to S2300.

[0068] Step S2100, obtaining video samples and their corresponding text description information.

[0069] The video samples are samples obtained by sampling the original video samples.

[0070] The text description information may be the subject information of the video sample or the key content information of the video sample.

[0071] In an optional embodiment, the step S2100 of obtaining the video sample and its corresponding text description information may further include the following steps S2110 to S2140:

[0072] Step S2110, obtaining the category to be classified and the synonyms of the category to be classified as keywords to be searched.

[0073] Specifically, first determine the category to be classified and the synonyms corresponding to the category to be classified, then use the category to be classified and its synonyms as the keywords to be searched on the video website. For example, if a video classification task of three categories, namely, quiet, fighting, and smoking, is required, quiet (synonyms include, but are not limited to, orderly, neat, and harmonious), fighting (synonyms include, but are not limited to, brawl, fight, and conflict), and smoking (synonyms include, but are not limited to, puffing, lighting a cigarette, and smoking) are used as the keywords to be searched.

[0074] Step S2120: Search based on the keyword to be searched to obtain the original video sample and its corresponding original text description information.

[0075] Specifically, the category to be classified and its synonyms are used as search keywords to obtain the original video sample and its corresponding original text description information. That is, the original video sample and its corresponding original text description information are data directly searched out based on the category to be classified and its synonyms.

[0076] Continuing with the above example, quiet, orderly, neat, harmonious, fighting, brawl, conflict, smoking, puffing, lighting a cigarette, and smoking are searched as keywords, and the searched original video samples and their corresponding original text description information are batch downloaded by crawling, and the original video samples and their corresponding original text description information are stored in the form of video-text pairs.

[0077] Step S2130: sampling the original video sample to obtain the video sample.

[0078] Specifically, to facilitate subsequent processing, all original video samples are uniformly sampled to a fixed frame number L to obtain video samples.

[0079] Step S2140: taking the original text description information and synonymous text description information of the original text description information as the text description information.

[0080] Specifically, in order to better hit the accurate category to be classified, it calls the application programming interface (API) of the large language model, such as ChatGPT (Chat Generative Pre-trained Transformer), to perform synonymous transformation on the original text description information, and obtain synonymous text description information of the original text description information for text data enhancement, so that the trained video classification model can better respond to precise category labels.

[0081] Reference Figure 3 , if the original text description information corresponding to the original video sample is "On a rainy day, two people got into a fight after drinking", the following three synonymous text description information are generated through ChatGPT: Synonymous text description information 1 "On a rainy day, the two people started a fierce fight after drinking", synonymous text description information 2 "When raindrops hit the window, the two people were involved in a fierce fight because of the effect of alcohol", synonymous text description information 3 "On a rainy night, a fierce physical conflict broke out between the two drunks". In this way, the same video sample corresponds to four text description information, and these four text description information also hit the category to be classified "fighting".

[0082] According to the above steps S2110 to S2140, training samples of the video classification model can be obtained, so as to train the video classification model based on the training samples.

[0083] After obtaining the video sample and its corresponding text description information, enter:

[0084] Step S2200: extracting video features of the video sample and text features of the text description information.

[0085] Among them, the video features of the video samples can be understood as the key features of the video samples, which can accurately reflect the category to which the video samples belong.

[0086] In this embodiment, the video features of the video sample can be extracted by the video feature extraction network in the video classification model to be trained. In an optional embodiment, extracting the video features of the video sample by the video feature extraction network in the video classification model to be trained in this step S2200 can further include the following steps S2210 to S2240:

[0087] Step S2210: input the video sample into the video feature extraction network to obtain the original video features of the video sample.

[0088] In this step S2210, the video sample is input into the video feature extraction network in the video classification model to be trained, so that the original video features of the video sample can be extracted first.

[0089] Specifically, for a batch size of training, the original video feature dimension of the extracted video sample is [N, L, H, W, 3], where N means that there are N videos participating in the training in the batch, L means the number of video frames, H means the height of the video image, W means the width of the video image, and 3 means the color channel.

[0090] Step S2220, performing block processing on the original video feature to obtain a first video feature.

[0091] In this step S2220, the video samples are divided into p*p blocks, so that the entire video becomes [N, L, H / p, W / p, p*p*3]. The video dimension is expanded to obtain the first video feature dimension of [N, L*(H / p)*(W / p), p*p*3].

[0092] Step S2230: performing mask processing on the first video feature at a set scale to obtain a second video feature.

[0093] The set scale includes at least one of the following: video frame, color channel, pixel.

[0094] In step S2230, the first video feature is masked at the three scales of video frame, color channel, and pixel, and the second video feature dimension is obtained as [N, K, p*p*3], where K is the number of remaining pixel blocks that are not masked. Figure 4 , a cube represents a frame of image, each frame of image has 3 channels, the length and width are represented by 5x5 grids, and the gray color of the image represents the mask of the position. This not only helps the video classification model learn multi-scale video features, but also can add interference such as frame loss and color channel loss in the training of the video classification model, making the video classification model more robust when processing such videos.

[0095] Specifically, if the data is stored sequentially in memory, consecutive (H / p)*(W / p)*(p*p*3) elements form a frame, and consecutive p*p*3 elements form a pixel block unit. The pixel-level mask operation is to fill the randomly adjacent p*p*3 elements with 0, and discard these elements and do not participate in the encoding of the video feature extraction network, and only use them in subsequent decoding. The channel-level mask operation is to randomly fill all values ​​of one color channel with 0 within the range of adjacent (H / p)*(W / p)*(p*p*3) elements. For the convenience of subsequent calculations, these 0 values ​​are still retained to participate in the encoding and decoding of the video feature extraction network. The frame-level mask operation is to randomly fill the consecutive (H / p)*(W / p)*(p*p*3) elements with 0, and discard these elements and do not participate in the encoding of the video feature extraction network, and only use them in decoding.

[0096] Step S2240: Encode the second video feature to obtain the video feature of the video sample.

[0097] Specifically, the second video feature is sent to the encoder of the ViT subnetwork in the video feature extraction network for encoding, and the video feature dimension of the obtained video sample is [N, K, D], where D is the learned feature dimension.

[0098] According to the above steps S2210 to S2240, on the one hand, the key features of the video sample can be extracted as the video features of the video sample, so as to perform comparative learning based on the video features and text features and train the video classification model.

[0099] On the other hand, it introduces multi-scale masked autoencoders during the training process and performs masking at three scales: video frame, color channel, and pixel. This enables the video classification model to learn better video feature representations, making the video classification model more robust when dealing with videos with lost frames, lost color channels, etc. At the same time, the mask method reduces the data that needs to be processed during training, speeds up the training process, and reduces training time.

[0100] In this embodiment, the text features of the text description information can be extracted through the video classification network in the video classification model to be trained. In the specific implementation, the text description information is input into the video classification network in the video classification model to be trained, and the original text features of the text description information are first extracted; then, the word vectors of the original text features are extracted through the BERT sub-network in the video classification network to obtain the text features of the text description information. It should be noted that the BERT sub-network here is an open source network, and the parameters of the BERT sub-network are usually pre-set, and the word embedding vector used to extract the original text features usually does not need to be updated during the model training process.

[0101] Specifically, for a training batch, the dimension of the original text feature information of the extracted text description information is [N, T], where N is the number of samples in the batch and T is the uniform sentence length (less padded and more truncated). In addition, the text feature dimension of the text description information obtained by extracting word vectors through the BERT sub-network in the video classification network is [N, T, D1], where D1 is the extracted feature dimension.

[0102] After extracting the video features of the video sample and the text features of the text description information, proceed to:

[0103] Step S2300: determining a target loss of a video classification model to be trained according to the video features and the text features.

[0104] The target loss is used to update the model parameters of the video classification model.

[0105] The target loss includes a first loss and a second loss, the first loss is used to update the network parameters of the video feature extraction network, and the second loss is used to update the network parameters of the video classification network.

[0106] In an optional embodiment, the step S2300 of determining the target loss of the video classification model to be trained according to the video features and the text features may further include the following steps S2310 to S2320:

[0107] Step S2310: Determine the first loss of the video feature extraction network in the video classification model to be trained according to the video feature.

[0108] Specifically, the video features of the video sample and the masked elements in step S2230 are arranged in the original order, and the masked elements are uniformly assigned to D-dimensional column vectors with an initial value of 0. In this way, the input video feature dimension of the decoder of the video feature extraction network is obtained as [N, L*(H / p)*(W / p), D]. After further passing through the fully connected layer of the video feature extraction network, the final video feature dimension is [N, L*(H / p)*(W / p), p*p*3], which has the same dimension as the original video feature of the video sample.

[0109] Next, the final video features and original video features of dimensions [N, L*(H / p)*(W / p), p*p*3] are normalized respectively, and then the mean square error is calculated to get the loss. The model is trained by gradient backpropagation. The calculation formula of the mean square error is:

[0110]

[0111] Among them, M represents the number of pixels in a frame of image, x i Represents the mean square error between the final video features and the original video features of the corresponding pixels in the frame image.

[0112] Step S2320: Determine the second loss of the video classification network in the video classification model to be trained based on the video features and the text features.

[0113] Specifically, in step S2320, determining the second loss of the video classification network in the video classification model to be trained according to the video features and the text features may further include the following steps S2321 to S2325:

[0114] Step S2321, obtaining an average value of the video feature to obtain a third video feature, and obtaining an average value of the text feature to obtain a first text feature.

[0115] In this step S2321, the video features [N, K, D] of the video samples are averaged to obtain the third video feature [N, D], so that the video features of all samples are obtained. For easy distinction, the third video feature is defined as [N, D_video].

[0116] In this step S2321, the text features [N, T, D1] of the text description information are encoded and decoded through the transformer subnetwork in the video classification network to obtain features [N, T, D_text], and the average is taken to obtain the first text feature [N, D_text], so that the text features of all samples are obtained.

[0117] Step S2322: Map the first text feature and the third video feature to target dimensions respectively to obtain a second text feature and a fourth video feature.

[0118] In this step S2322, the first text feature [N, D_text] and the third video feature [N, D_video] are projected to the same dimension D_feature through the fully connected layer of the video classification network to obtain the second text feature and the fourth video feature.

[0119] Step S2323: normalize the second text feature and the fourth video feature respectively to obtain a normalized text feature and a normalized video feature.

[0120] In this step S2323, the second text feature and the fourth video feature are respectively normalized to obtain normalized text features and normalized video features, both of which can be marked as [N, D_feature], that is, two feature matrices of [N, D_feature] are obtained.

[0121] Step S2324, obtaining a similarity matrix between the normalized text features and the normalized video features.

[0122] In this step S2324, the cosine similarity is calculated for each element in the two feature matrices of normalized text features and normalized video features to obtain a similarity matrix of [N, N]. The (i, j)th element in the similarity matrix represents the similarity between the i-th video sample and the j-th text description information.

[0123] Step S2325: determine the second loss according to the similarity matrix.

[0124] In this step S2325, a standard label = [0, 1, ... N-1] is generated, indicating that the ith video corresponds to the ith text. The similarity matrix and the standard label are used to calculate the loss using symmetric cross entropy, and are optimized by gradient backpropagation. The symmetric cross entropy calculation formula is:

[0125] loss_v=cross_entropy(logits,label,axis=0)

[0126] loss_t=cross_entropy(logits,label,axis=1)

[0127] loss=(loss_v+loss_t) / 2,

[0128]

[0129] Among them, logits is the similarity matrix, label is the standard label, axis=0 means that the cross entropy of the similarity matrix is ​​calculated by row, axis=1 means that the cross entropy of the similarity matrix is ​​calculated by column, and N is the total number of samples.

[0130] According to the above steps S2310 to S2320, two tasks are trained in parallel. On the one hand, text features are extracted from the text description information, and video features are extracted from the video samples. Since the two have different dimensions but actually describe the same thing, they are projected into the same feature space and trained by comparative learning.

[0131] On the other hand, it performs multi-scale masked autoencoders learning on video features, and the learning goal is to learn features from masked videos to restore the original unmasked videos.

[0132] According to the method of the embodiment of the present disclosure, after obtaining the video sample and its corresponding text description information, the video features of the video sample and the text features of the text description information are extracted, and then the target loss of the video classification model to be trained is determined based on the video features and the text features, so as to update the model parameters of the video classification model according to the target loss. That is, the text description information can be understood as the fuzzy text label corresponding to the video sample. By comparing and learning the video sample and its corresponding text description information, there is no need to manually annotate the video sample, which reduces the manpower and training time costs, and has a good training effect in the training of video classification models in different scenarios.

[0133] <Method Example 2>

[0134] The present disclosure embodiment discloses a video classification method, such as Figure 5 As shown, the video classification method includes the following steps S5100 to S5200.

[0135] Step S5100, obtaining the video to be classified and the text description information corresponding to the video to be classified in different categories.

[0136] Step S5200: Process the video to be classified and the different text description information using the trained video classification model to obtain the category to which the video to be classified belongs.

[0137] The video classification model may be trained according to the video classification model training method described in the first embodiment of the method.

[0138] In an optional embodiment, this step S5200 uses a trained video classification model to process the video to be classified and the different text description information, and obtaining the category to which the video to be classified belongs may further include: using a video feature extraction network in the trained video classification model to extract video features of the video to be classified, and using a video classification network in the trained video classification model to extract text features corresponding to different text description information respectively; and, using the video classification network in the trained video classification model to determine a similarity matrix between the video features and the text features; obtaining the category corresponding to the maximum similarity in the similarity matrix as the category to which the video to be classified belongs.

[0139] Exemplarily, firstly extract frames of the video to be classified, and extract video features based on the video feature extraction network in the trained video classification model, and then calculate the average of the video features based on the video classification network in the trained video classification model to obtain the video feature [1, D_feature].

[0140] Next, a text description is generated for each category of the video to be classified, and the text features are extracted and averaged based on the video classification network in the trained video classification model. For example, if the categories to be classified are: quiet, fighting, and smoking, three text descriptions are generated: "This is a quiet video", "This is a video containing fighting", and "This is a video containing smoking". The text features are extracted and averaged to obtain the text feature [3, D_feature].

[0141] Furthermore, based on the fully connected layer of the video classification network in the trained video classification model, the video features [1, D_feature] and the text features [3, D_feature] are mapped to the same dimension and then normalized to obtain normalized video features and normalized text features, and the cosine similarity between the normalized video features and the normalized text features is calculated to obtain a similarity matrix of [1, 3] dimensions, and the category corresponding to the maximum similarity in the similarity matrix is ​​selected as the category to which the video to be classified belongs.

[0142] According to the embodiments of the present disclosure, by classifying the video to be classified based on the trained video classification model, the classification accuracy of the video to be classified can be improved.

[0143] <Example>

[0144] Next, we will show an example of a video classification model training method. In this example, refer to Figure 6 and Figure 7 , the training method of the video classification model includes:

[0145] S01, obtaining original video text and original text description information.

[0146] In this step S01, the categories to be classified are first determined, and the categories to be classified and their synonyms are used as keywords to be searched on the video website. For example, if a video classification task of three categories, namely, quiet, fighting, and smoking, is required, quiet (synonyms such as but not limited to: orderly, neat, harmonious), fighting (synonyms such as but not limited to: brawl, fight, conflict), and smoking (synonyms such as but not limited to: puffing, lighting a cigarette, smoking) are used as keywords to be searched for searching; and the original video samples and their corresponding original text description information searched out are downloaded in batches by crawling, and the original video samples and their corresponding original text description information are stored in the form of video-text pairs.

[0147] S02, text enhancement and video frame sampling.

[0148] In this step S02, in order to better hit the accurate category to be classified, it calls the application programming interface of the large language model, such as ChatGPT, to perform synonymous conversion on the original text description information corresponding to the original video sample, and obtain synonymous text description information of the original text description information. Here, the text description information corresponding to the original video sample may include original text description information, synonymous text description information 1, synonymous text description information 2 and synonymous text description information 3.

[0149] In this step S02, all original video samples are uniformly sampled to a fixed frame number L to obtain video samples.

[0150] S03: Model training. The video-text pairs generated by S02 are trained in parallel through two tasks: 1. Extract video features and text features respectively, project the two features into the same feature space, and train them through contrastive learning. The loss function uses symmetric cross entropy. 2. Perform multi-scale masked autoencoders learning on video features. The learning goal is to learn features from masked videos to restore the original unmasked videos. The loss function uses mean square error.

[0151] 03_1: Multi-scale masked autoencoders task. The multi-scale masked autoencoders proposed in this example can perform random masking at three scales: video frame, color channel, and pixel. This not only helps to learn multi-scale key features, but also allows interference such as frame loss and color channel loss to be added during training, making the model more robust when processing such videos. The specific implementation is as follows:

[0152] S03_1_1: For a training batch, the original video feature dimension of the video sample is defined as [N, L, H, W, 3], where N represents a total of N videos participating in the training in the batch, L represents the number of video frames, H represents the video image height, W represents the video image width, and 3 represents the color channel.

[0153] S03_1_2: Divide the image into p*p blocks, so that the entire video becomes [N, L, H / p, W / p, p*p*3], and expand the video dimension to become [N, L*(H / p)*(W / p), p*p*3].

[0154] S03_1_3: Assuming that the data is stored sequentially in memory, the continuous adjacent (H / p)*(W / p)*(p*p*3) elements form a frame, and the continuous adjacent p*p*3 form a pixel block unit. The pixel-level mask operation is to fill the random adjacent p*p*3 elements with 0, and discard these elements and do not participate in the encoder part of the network, and only use them in the decoder; the channel-level mask operation is to randomly fill all the values ​​of one color channel with 0 within the range of adjacent (H / p)*(W / p)*(p*p*3) elements. For the convenience of subsequent calculations, these 0 values ​​are still retained to participate in the encoder and decoder; the frame-level mask operation is to randomly fill the continuous (H / p)*(W / p)*(p*p*3) elements with 0, and discard these elements and do not participate in the encoder part of the network, and only use them in the decoder.

[0155] S03_1_4: After processing in S03_1_3, the feature dimension of the encoder becomes [N, K, p*p*3], where K is the number of remaining pixel blocks that are not masked. It is sent to the encoder part of the ViT network, and the output feature dimension is [N, K, D], where D is the learned feature dimension.

[0156] S03_1_5: Arrange the features learned in S03_1_4 and the masked elements in S03_1_3 in their original order. The masked elements are uniformly assigned to D-dimensional column vectors with an initial value of 0. In this way, the input features of the decoder part are obtained with a dimension of [N, L*(H / p)*(W / p), D]. After the above features pass through the decoder and the fully connected layer, the dimension becomes [N, L*(H / p)*(W / p), p*p*3], which has the same dimension as the original input video.

[0157] S03_1_6: Normalize the features obtained in S03_1_5 and the original video features respectively, then calculate the mean square error to get the loss, and perform model training through gradient backpropagation.

[0158] S03_2: Comparative learning of video features and text features. This part mainly extracts text features from text description information and video features from video samples. Because they have different dimensions but actually describe the same thing, they are projected into the same feature space through a fully connected layer. After projecting into the same feature space, the cosine similarity of video features and text features is calculated pairwise, and the loss is obtained by calculating the symmetric cross entropy with the self-supervised label for training.

[0159] S03_2_1: For a batch in training, define the original text feature dimension of its text description information as [N, T], where N is the number of samples in the batch and T is the uniform sentence length (less padding and more truncation).

[0160] S03_2_2: For the sentences in S03_2_1, word vectors are extracted through the BERT network, and the feature dimension is [N, T, D1], where D1 is the extracted feature dimension.

[0161] S03_2_3_1: For the features obtained in S03_2_2, the encoder and decoder are performed through the transformer network to obtain the feature dimension [N, T, D_text], and the average value is [N, D_text], so that the text features of all samples are obtained.

[0162] S03_2_3_2: For the features [N, K, D] obtained in S03_1_4, take the average to get [N, D], so that the video features of all samples are obtained. For easy distinction, the video features are defined as [N, D_video].

[0163] S03_2_4: The video features obtained in S03_2_3_2 and the text features obtained in S03_2_3_1 are projected to the same feature dimension D_feature through the fully connected layer, and the features are normalized to obtain two feature matrices of [N, D_feature].

[0164] S03_2_5: For the elements in the two feature matrices obtained in S03_2_4, the cosine similarity is calculated pairwise to obtain a similarity matrix of [N, N]. The (i, j)th element in the similarity matrix represents the similarity between the i-th video and the j-th text.

[0165] S03_2_6: Generate a standard label = [0, 1, ... N-1], indicating that the ith video corresponds to the ith text, and use the symmetric cross entropy to calculate the loss for the similarity matrix and label generated in S03_2_5, and optimize it through gradient backpropagation.

[0166] <Device Example 1>

[0167] In this embodiment, a video classification model training device 800 is provided. Figure 8 As shown, the video classification model training device 800 may include a first acquisition module 810 , an extraction module 820 and a training module 830 .

[0168] A first acquisition module 810 is used to acquire a video sample and its corresponding text description information;

[0169] An extraction module 820, configured to extract video features of the video sample and text features of the text description information;

[0170] The training module 830 is used to determine the target loss of the video classification model to be trained based on the video features and the text features; wherein the target loss is used to update the model parameters of the video classification model.

[0171] In one embodiment, the first acquisition module 810 is specifically used to: obtain the category to be classified and the synonyms of the category to be classified as the keywords to be searched; search based on the keywords to be searched to obtain the original video sample and its corresponding original text description information; sample the original video sample to obtain the video sample; and use the original text description information and the synonymous text description information of the original text description information as the text description information.

[0172] In one embodiment, the extraction module 820 is specifically used to extract the video features of the video sample through a video feature extraction network in a video classification model to be trained.

[0173] In one embodiment, the extraction module 820 is specifically used to: input the video sample into the video feature extraction network to obtain the original video feature of the video sample; perform block processing on the original video feature to obtain a first video feature; perform mask processing on the first video feature at a set scale to obtain a second video feature; wherein the set scale includes at least one of the following: video frame, color channel, pixel; encode the second video feature to obtain the video feature of the video sample.

[0174] In one embodiment, the extraction module 820 is specifically configured to extract text features of the text description sample through a video classification network in a video classification model to be trained.

[0175] In one embodiment, the target loss includes a first loss and a second loss, and the training module 830 is specifically used to: determine the first loss of the video feature extraction network in the video classification model to be trained according to the video features; wherein the first loss is used to update the network parameters of the video feature extraction network; determine the second loss of the video classification network in the video classification model to be trained according to the video features and the text features; wherein the second loss is used to update the network parameters of the video classification network.

[0176] In one embodiment, the training module 830 is specifically used to: obtain the average value of the video feature to obtain the third video feature, and obtain the average value of the text feature to obtain the first text feature; map the first text feature and the third video feature to the target dimension respectively to obtain the second text feature and the fourth video feature; normalize the second text feature and the fourth video feature respectively to obtain the normalized text feature and the normalized video feature; obtain the similarity matrix between the normalized text feature and the normalized video feature; and determine the second loss according to the similarity matrix.

[0177] According to this embodiment, after obtaining the video sample and its corresponding text description information, the video features of the video sample and the text features of the text description information are extracted, and then the target loss of the video classification model to be trained is determined according to the video features and the text features, so as to update the model parameters of the video classification model according to the target loss. That is, the text description information can be understood as the fuzzy text label corresponding to the video sample. By comparing and learning the video sample and its corresponding text description information, there is no need to manually annotate the video sample, which reduces the manpower and training time costs, and has a good training effect in the training of video classification models in different scenarios.

[0178] <Device Example 2>

[0179] In this embodiment, a video classification device 900 is provided. Fig. 9 As shown, the video classification device 900 may include a second acquisition module 910 and a classification module 920 .

[0180] The second acquisition module 910 is used to acquire the video to be classified and the text description information corresponding to the video to be classified in different categories;

[0181] The classification module 920 is used to process the video to be classified and the different text description information using the trained video classification model to obtain the category to which the video to be classified belongs.

[0182] In one embodiment, the classification module 920 is specifically used to: use the video feature extraction network in the trained video classification model to extract the video features of the video to be classified, and use the video classification network in the trained video classification model to extract the text features corresponding to the different text description information; and use the video classification network in the trained video classification model to determine the similarity matrix between the video features and the text features; obtain the category corresponding to the maximum similarity in the similarity matrix as the category to which the video to be classified belongs.

[0183] According to this embodiment, the video to be classified is classified based on the trained video classification model, which can improve the classification accuracy of the video to be classified.

[0184] <Equipment Embodiment>

[0185] In this embodiment, an electronic device 100 is also provided. Fig.10 As shown, it includes a memory 120 and a processor 110 .

[0186] The memory 120 is used to store an executable computer program; the computer program is used to control the processor 110 to execute any one of the video classification model training methods provided in the first embodiment or to execute any one of the video classification methods provided in the second embodiment.

[0187] In this embodiment, the electronic device 100 may be Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which is not limited here.

[0188] In another embodiment, the electronic device 100 may include the above video classification model training device 800 or the above video classification device 900.

[0189] <Computer Readable Storage Medium>

[0190] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any embodiment of the present disclosure is implemented.

[0191] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0192] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0193] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0194] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0195] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0196] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0197] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0198] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0199] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A video classification model training method, characterized in that: The method comprises: Obtain video samples and their corresponding text description information; Extracting video features of the video sample and text features of the text description information; According to the video features and the text features, a target loss of the video classification model to be trained is determined; wherein the target loss is used to update the model parameters of the video classification model.

2. The method according to claim 1, characterized in that Get video samples and their corresponding text description information, including: Obtaining the category to be classified and the synonyms of the category to be classified as keywords to be searched; Searching based on the keywords to be searched, obtaining original video samples and their corresponding original text description information; Sampling the original video sample to obtain the video sample; The original text description information and synonymous text description information of the original text description information are used as the text description information.

3. The method according to claim 1, characterized in that: Extracting video features of the video sample includes: The video features of the video sample are extracted through a video feature extraction network in a video classification model to be trained.

4. The method according to claim 3, characterized in that: The extracting the video features of the video sample by using a video feature extraction network in a video classification model to be trained includes: Inputting the video sample into the video feature extraction network to obtain the original video features of the video sample; The original video feature is processed into blocks to obtain a first video feature; Performing mask processing on the first video feature at a set scale to obtain a second video feature; wherein the set scale includes at least one of the following: video frame, color channel, pixel; The second video feature is encoded to obtain the video feature of the video sample.

5. The method according to claim 1, characterized in that Extracting text features of the text description sample includes: The text features of the text description samples are extracted through a video classification network in a video classification model to be trained.

6. The method according to claim 1, characterized in that The target loss includes a first loss and a second loss, The step of determining a target loss of a video classification model to be trained according to the video features and the text features includes: Determine the first loss of the video feature extraction network in the video classification model to be trained according to the video feature; wherein the first loss is used to update the network parameters of the video feature extraction network; The second loss of the video classification network in the video classification model to be trained is determined according to the video features and the text features; wherein the second loss is used to update the network parameters of the video classification network.

7. The method according to claim 6, characterized in that The step of determining the second loss of the video classification network in the video classification model to be trained according to the video feature and the text feature includes: Obtaining an average value of the video feature to obtain a third video feature, and obtaining an average value of the text feature to obtain a first text feature; Mapping the first text feature and the third video feature to target dimensions respectively to obtain a second text feature and a fourth video feature; Normalizing the second text feature and the fourth video feature respectively to obtain a normalized text feature and a normalized video feature; Obtaining a similarity matrix between the normalized text features and the normalized video features; The second loss is determined according to the similarity matrix.

8. A video classification method, characterized in that: The method comprises: Obtaining the video to be classified and the text description information corresponding to the video to be classified in different categories; The trained video classification model is used to process the video to be classified and the different text description information to obtain the category to which the video to be classified belongs.

9. The method according to claim 8, characterized in that The step of processing the video to be classified and the different text description information using the trained video classification model to obtain the category to which the video to be classified belongs includes: Using the video feature extraction network in the trained video classification model to extract the video features of the video to be classified, and using the video classification network in the trained video classification model to extract text features corresponding to different text description information; and, Determine a similarity matrix between the video features and the text features using a video classification network in a trained video classification model; The category corresponding to the maximum similarity in the similarity matrix is ​​obtained as the category to which the video to be classified belongs.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images