A video model training, video classification method and device

Abstract

The present invention discloses a video model training, video classification method and device, which relates to the field of video action recognition. The invention is to solve the problem of high cost and low efficiency of video classification in the prior art. It includes: respectively obtaining multiple training key frames of each training video in the training video set; respectively obtaining the visible image block and the shielded image block of each training key frame; respectively obtaining the coding embedding of each training video; respectively embedding the coding of each training video into the coding module in the input video self-supervision model, and obtaining the training classification flag bit of each training video embedded in the corresponding first output, the training embedding set and the corresponding position embedding corresponding second output; respectively inputting the decoding flag bit corresponding to each training key frame and the corresponding decoding embedding into the decoding module, and obtaining the prediction result of the shielded image block of each training key frame; training the video self-supervision model to obtain a pre-trained video self-supervision model.

Description

Technical Field

[0001] The present invention relates to the field of video action recognition, and in particular to a video model training, video classification method and device. Background Art

[0002] With the development of communication technology, the application of video is becoming more and more extensive. At the same time, the demand for video classification is also becoming more and more widespread. Whether it is sharing video retrieval and recommendation on social networks or monitoring and security, video classification is required.

[0003] In the prior art, classification models such as support vector machines (SVM) are generally used to achieve video classification. The specific process includes: firstly training the classification model through a video-category set, and then inputting the video to be classified into the trained model to obtain the classification result.

[0004] In the prior art, training a classification model generally requires a large amount of manually labeled data, but the cost of manual labeling is very high and the efficiency is low, which in turn leads to high cost and low efficiency of video classification. Summary of the invention

[0005] In view of this, the main purpose of the present invention is to solve the problem of high cost and low efficiency of existing video classification.

[0006] On the one hand, an embodiment of the present invention provides a video model training method, including: S10, respectively obtaining multiple training key frames of each training video in the training video set; S20, respectively obtaining the visible image block and the shielded image block of each training key frame according to preset rules; S30, respectively inserting the training classification flag bit embedding at the first position of each training embedding set, and adding it to the corresponding training position embedding to obtain the coded embedding of each training video; the training embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all training key frames of the corresponding training video; S40, respectively embedding the coded embedding of each training video into the encoding module of the video self-supervisory model, and obtaining the first output corresponding to the training classification flag bit embedding of each training video, the training embedding set and the second output corresponding to the corresponding position embedding; S50, respectively embedding each training key The decoding flag bit input corresponding to the frame and the corresponding decoding embedding are input into the decoding module in the video self-supervisory model to obtain the prediction result of the shielded image block of each training key frame; S60, the video self-supervisory model is trained according to the original image and the prediction result of the shielded image block of each training key frame respectively to obtain a pre-trained video self-supervisory model; the decoding embedding corresponding to each training key frame is composed of the image embedding and the decoding position embedding corresponding to the training key frame; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all the shielded image blocks of the training key frame according to the image position order in the training key frame; the decoding flag bit input corresponding to any training key frame is the first output corresponding to the training classification flag bit embedding of the training video corresponding to the training key frame.

[0007] On the other hand, an embodiment of the present invention provides a video model training method, including: P10, obtaining an initial classification model based on a classification head and the above-mentioned pre-trained video self-supervised model; P20, training the initial classification model through a video-category training set to obtain a pre-trained classification model.

[0008] On the other hand, an embodiment of the present invention provides a video classification method, including: Q10, obtaining multiple classification key frames of the video to be classified; Q20, obtaining the visible image blocks of each classification key frame respectively according to preset rules; Q30, inserting the classification flag bit embedding at the first position of the classification embedding set, and adding it with the corresponding classification position embedding to obtain the target model embedding; the classification embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all classification key frames; Q40, inputting the target model embedding into the above-mentioned pre-trained classification model to obtain the category of the video to be classified.

[0009] In another aspect, an embodiment of the present invention provides a video model training device, comprising:

[0010] A training frame acquisition module, used to respectively acquire a plurality of training key frames of each training video in a training video set;

[0011] An image block acquisition module, connected to the training frame acquisition module, for respectively acquiring visible image blocks and shielded image blocks of each training key frame according to preset rules;

[0012] A training embedding acquisition module, connected to the image block acquisition module, is used to insert the training classification flag bit embedding at the first position of each training embedding set, and then add it to the corresponding training position embedding to obtain the coding embedding of each training video; the training embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all training key frames of the corresponding training video;

[0013] An encoding input module, connected to the training embedding acquisition module, for respectively embedding the encoding of each training video into the encoding module of the input video self-supervised model, to obtain a first output corresponding to the embedding of the training classification flag of each training video, a training embedding set and a second output corresponding to the corresponding position embedding;

[0014] A decoding input module, connected to the encoding input, for respectively inputting the decoding flag bit corresponding to each training key frame and the corresponding decoding module embedded in the input video self-supervisory model to obtain the prediction result of the masked image block of each training key frame;

[0015] A self-supervised training module, connected to the image block acquisition module and the decoding input module, respectively, for training the video self-supervised model according to the original image and the prediction result of the masked image block of each training key frame, to obtain a pre-trained video self-supervised model;

[0016] The decoding embedding corresponding to each training key frame is composed of the image embedding and decoding position embedding corresponding to the training key frame; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all masked image blocks of the training key frame, combined in the order of image positions in the training key frame; the decoding flag input corresponding to any training key frame is the first output corresponding to the training classification flag embedding of the training video corresponding to the training key frame.

[0017] In another aspect, an embodiment of the present invention provides a video model training device, comprising:

[0018] A classification model acquisition module, used to acquire an initial classification model based on the classification head and the pre-trained video self-supervised model acquired by the above-mentioned video model training device;

[0019] The classification model training module is connected to the classification model acquisition module and is used to train the initial classification model through a video-category training set to obtain a pre-trained classification model.

[0020] In another aspect, an embodiment of the present invention provides a video classification device, comprising:

[0021] A classification frame acquisition module is used to obtain multiple classification key frames of the video to be classified;

[0022] A visible image block acquisition module, connected to the classification frame acquisition module, for acquiring the visible image blocks of each classification key frame according to a preset rule;

[0023] A classification embedding acquisition module, connected to the visible image block acquisition module, is used to insert a classification flag bit embedding at the first position of the classification embedding set, and then add it to the corresponding classification position embedding to obtain a target model embedding; the classification embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all classification key frames;

[0024] The video classification module is connected to the classification embedding acquisition module and is used to embed the target model into the pre-trained classification model obtained by the above-mentioned video model training device to obtain the category of the video to be classified.

[0025] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A flowchart of the video model training method provided in Example 1 of the present invention;

[0028] Figure 2A flowchart of a video model training method provided in Example 2 of the present invention;

[0029] Figure 3 A flowchart of a video classification method provided in Embodiment 3 of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a video model training device provided in Example 4 of the present invention;

[0031] Figure 5 A schematic diagram of the structure of a video model training device provided in Example 5 of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of a video classification device provided in Example 6 of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] like Figure 1 As shown, the present invention provides a video model training method, comprising:

[0036] Step 101, respectively obtain multiple training key frames of each training video in the training video set.

[0037] Specifically, in the mainstream video encoding method, each video mainly includes three types of picture frames, which are respectively called: Intra-coded frame (I frame), Predictive frame (P frame) and Bi-Predictive frame (B frame), and the key frame in step 101 of the embodiment of the present application refers to the Intra-coded frame (I frame) among the three picture frames, which is a frame that completely retains the image data in the compressed video. When decoding the key frame, only the image data of this frame is needed to complete the decoding. Since the similarity between each key frame in the video is small, the video can be comprehensively represented by multiple key frames; by extracting multiple key frames and processing the key frames, the processing speed and accuracy can be improved.

[0038] In this embodiment, a method for obtaining multiple training key frames is to extract multiple training key frames from a training video according to preset rules; the preset rules include: one of duration, interval, weight, and click volume.

[0039] Step 102: Obtain the visible image patches and masked image patches of each training key frame according to preset rules.

[0040] In this embodiment, the preset rules in step 102 include a preset partitioning rule and a preset ratio. At this time, step 102 may specifically include: First, divide each training key frame into corresponding non-overlapping image patches according to the preset partitioning rule; then obtain the visible image patches and masked image patches of the corresponding training key frame from the non-overlapping image patches of each training key frame according to the preset ratio.

[0041] Among them, the specific process of obtaining the visible image patches and masked image patches of the corresponding training key frame from the non-overlapping image patches of each training key frame according to the preset ratio may be to randomly obtain the visible image patches and masked image patches of the corresponding training key frame from the non-overlapping image patches of each training key frame according to the preset ratio; or this process may also be that for the i-th training key frame, randomly obtain the visible image patches and masked image patches of the i-th training key frame from the non-overlapping image patches of the i-th training key frame according to the preset ratio; i = 1 + xn, where x is a natural number, and n is the floor of the ratio of the number of non-overlapping image patches to the visible number of visible image patches; 1 ≤ i ≤ the total number of training key frames, and i is a positive integer; for the j-th training key frame, randomly obtain the visible image patches of the j-th training key frame from the optional area of the j-th training key frame according to the visible number, and obtain the masked image patches of the j-th training key frame according to the visible image patches of the j-th training key frame; the j-th training key frame is the key frame other than the i-th training key frame among all training key frames; the optional area of the j-th training key frame is the positions of all visible image patches among all image positions except the positions of all visible image patches from the first key frame to the previous key frame of the j-th training key frame; the first key frame is the nearest i-th training key frame before the j-th training key frame; 1 < j ≤ the total number of training key frames, and j is a positive integer.

[0042] The preset partitioning rule may be a preset partitioning quantity, may also be a preset partitioning ratio, or may be other partitioning rules, which are not limited here; the preset ratio may be any ratio, and the specific ratio of the masked image patches may be 75%, and at this time, the model training effect is better.

[0043] Step 103: After inserting the training classification flag bit embedding at the beginning of each training embedding set respectively, add it to the corresponding training position embedding to obtain the encoded embedding of each training video.

[0044] In this embodiment, the training embedding set in step 103 is obtained by combining the visible embeddings corresponding to all visible image patches of all training key frames of the corresponding training video. Among them, the visible image patches can be generated into visible embeddings through linear mapping and other methods, which will not be elaborated here one by one.

[0045] Step 104, respectively embed the encoding of each training video into the encoding module in the input video self-supervisory model to obtain the first output corresponding to the embedding of the training classification flag of each training video, the training embedding set and the second output corresponding to the corresponding position embedding.

[0046] In this embodiment, the encoding module in the video self-supervision model only processes data corresponding to a portion of the image blocks, namely, the visible image blocks, which can greatly reduce information redundancy and enable the encoding module to learn more advanced features, greatly reducing the amount of calculation.

[0047] Step 105 , respectively input the decoding flag bit corresponding to each training key frame and the corresponding decoding module embedded in the input video self-supervisory model to obtain the prediction result of the masked image block of each training key frame.

[0048] In this embodiment, the decoding embedding corresponding to each training key frame is composed of the image embedding and decoding position embedding corresponding to the training key frame; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all the shielded image blocks of the training key frame in the order of the image position in the training key frame; the decoding flag input corresponding to any training key frame is the first output corresponding to the training classification flag embedding of the training video corresponding to the training key frame. Since the decoding module in step 105 only processes the data corresponding to each key frame, the amount of calculation is greatly reduced, and the decoding flag input corresponding to each training key frame is used as the input of the decoding module, which can retain the semantic information of the entire video.

[0049] In this embodiment, the encoding module and decoding module in the video self-supervision model can be a transformer encoder-decoder architecture. In particular, the structure of the decoder can be similar to or even the same as that of the encoder. The difference is that the encoder has more stacked layers and the decoder has fewer.

[0050] Step 106 , training the video self-supervision model according to the original image and the prediction result of the masked image block of each training key frame, to obtain a pre-trained video self-supervision model.

[0051] In this embodiment, step 106 may specifically include: 1. Obtaining the mean square error of the original image and the current prediction result of the masked image block of all training key frames respectively; 2. Determining whether the mean square error is greater than a preset threshold; 3. If it is greater than, adjusting the trainable parameters in the video self-supervision model, and re-executing steps 104 and 105 using the adjusted model; 4. If it is not greater than, using the video self-supervision model used when it is not greater than the preset threshold as the pre-trained video self-supervision model.

[0052] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0053] Example 2

[0054] like Figure 2 As shown, an embodiment of the present invention provides a video model training method, comprising:

[0055] Step 201, obtaining an initial classification model based on a classification head and a pre-trained video self-supervised model.

[0056] In this embodiment, the method of obtaining the pre-trained video self-supervisory model is similar to that provided in Embodiment 1 of the present invention, and will not be described in detail here. Specifically, the initial classification model in step 201 can be composed of a classification head and an encoding module of the pre-trained video self-supervisory model.

[0057] Step 202: Train the initial classification model using the video-category training set to obtain a pre-trained classification model.

[0058] In this embodiment, the method of training the model through step 102 can be: simultaneously training the classification head in the initial classification model and the pre-trained video self-supervised model through the video-category training set to obtain the pre-trained classification model; or, training the classification head in the initial classification model through the video-category training set to obtain the pre-trained classification model.

[0059] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0060] Example 3

[0061] like Figure 3 As shown, an embodiment of the present invention provides a video classification method, including:

[0062] Step 301: Acquire multiple classification key frames of the video to be classified.

[0063] In this embodiment, the method of obtaining the key frame in step 301 is the same as Figure 1 The steps 101 shown are similar and will not be described in detail here.

[0064] Step 302: Obtain visible image blocks of each classification key frame according to preset rules.

[0065] In this embodiment, the method of obtaining the visible image block in step 302 is the same as that in Figure 1 The step 102 shown is similar and will not be described in detail here.

[0066] Step 303, after inserting the classification flag bit embedding at the first position of the classification embedding set, it is added to the corresponding classification position embedding to obtain the target model embedding.

[0067] In this embodiment, the classification embedding set in step 303 is obtained by combining the visible embeddings corresponding to all visible image blocks of all classification key frames; the visible embeddings can be obtained by linear mapping the visible image blocks, etc., which will not be repeated here.

[0068] Step 304: embed the target model into the pre-trained classification model to obtain the category of the video to be classified.

[0069] In this embodiment, the method of obtaining the pre-trained classification model is similar to that provided in Example 2 of the present invention, and will not be described in detail here.

[0070] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0071] Example 4

[0072] like Figure 4 As shown, an embodiment of the present invention provides a video model training device, comprising:

[0073] The training frame acquisition module 401 is used to respectively acquire a plurality of training key frames of each training video in the training video set;

[0074] The image block acquisition module 402 is connected to the training frame acquisition module and is used to respectively acquire the visible image blocks and the shielded image blocks of each training key frame according to a preset rule;

[0075] The training embedding acquisition module 403 is connected to the image block acquisition module and is used to insert the training classification flag bit embedding at the first position of each training embedding set, and then add it to the corresponding training position embedding to obtain the coding embedding of each training video; the training embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all training key frames of the corresponding training video;

[0076] The encoding input module 404 is connected to the training embedding acquisition module and is used to respectively embed the encoding of each training video into the encoding module of the video self-supervisory model to obtain a first output corresponding to the embedding of the training classification flag of each training video, a training embedding set and a second output corresponding to the corresponding position embedding;

[0077] A decoding input module 405, connected to the encoding input, is used to input the decoding flag bit corresponding to each training key frame and the corresponding decoding module embedded in the input video self-supervisory model to obtain the prediction result of the masked image block of each training key frame;

[0078] A self-supervised training module 406, connected to the image block acquisition module and the decoding input module, respectively, for training the video self-supervised model according to the original image and the prediction result of the masked image block of each training key frame, to obtain a pre-trained video self-supervised model;

[0079] The decoding embedding corresponding to each training key frame is composed of the image embedding and decoding position embedding corresponding to the training key frame; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all masked image blocks of the training key frame, combined in the order of image positions in the training key frame; the decoding flag input corresponding to any training key frame is the first output corresponding to the training classification flag embedding of the training video corresponding to the training key frame.

[0080] In this embodiment, the process of obtaining a pre-trained video self-supervisory model through the above-mentioned module is similar to that provided in Example 1 of the present invention, and will not be described in detail here.

[0081] Among them, the image block acquisition module in the video model training device provided in this embodiment includes:

[0082] A non-overlapping acquisition submodule is used to divide each training key frame into corresponding non-overlapping image blocks according to a preset division rule;

[0083] The image block acquisition submodule is connected to the non-overlapping acquisition submodule and is used to respectively acquire the visible image blocks and the shielded image blocks of the corresponding training key frame from the non-overlapping image blocks corresponding to each training key frame according to a preset ratio.

[0084] The self-supervised training module in the video model training device provided in this embodiment includes:

[0085] The error acquisition submodule is used to respectively obtain the original image of the masked image block of all training key frames and the mean square error of the current prediction result;

[0086] The threshold judgment submodule is connected to the error acquisition submodule and is used to judge whether the mean square error is greater than a preset threshold; if it is, the parameter adjustment submodule is executed; if it is not, the model acquisition submodule is executed;

[0087] A parameter adjustment submodule, connected to the threshold judgment submodule, is used to adjust the trainable parameters in the video self-supervised model and then re-execute the steps of the encoding input module and the decoding input module using the adjusted model;

[0088] The model acquisition submodule is connected to the threshold judgment submodule and is used to use the video self-supervision model used when the value is not greater than the preset threshold as the pre-trained video self-supervision model.

[0089] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0090] Example 5

[0091] like Figure 5 As shown, an embodiment of the present invention provides a video model training device, comprising:

[0092] A classification model acquisition module 501 is used to acquire an initial classification model according to a classification head and a pre-trained video self-supervisory model acquired by a model training device provided by Embodiment 4 of the present invention;

[0093] The classification model training module 502 is connected to the classification model acquisition module and is used to train the initial classification model through the video-category training set to obtain a pre-trained classification model.

[0094] In this embodiment, the process of obtaining a pre-trained classification model through the above-mentioned module is similar to that provided in Example 2 of the present invention, and will not be described in detail here.

[0095] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0096] Example 6

[0097] like Figure 6 As shown, an embodiment of the present invention provides a video classification device, including:

[0098] The classification frame acquisition module 601 is used to acquire multiple classification key frames of the video to be classified;

[0099] A visible image block acquisition module 602, connected to the classification frame acquisition module, is used to acquire the visible image blocks of each classification key frame according to a preset rule;

[0100] A classification embedding acquisition module 603 is connected to the visible image block acquisition module and is used to insert a classification flag bit embedding at the first position of the classification embedding set and then add it to the corresponding classification position embedding to obtain a target model embedding; the classification embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all classification key frames;

[0101] The video classification module 604 is connected to the classification embedding acquisition module, and is used to embed the target model into the pre-trained classification model obtained by the video model training device provided by Example 5 of the present invention to obtain the category of the video to be classified.

[0102] In this embodiment, the process of implementing video classification through the above modules is similar to that provided in Embodiment 3 of the present invention, and will not be described in detail here.

[0103] In summary, the technical solution provided by the embodiment of the present invention can train the video self-supervisory model by only processing the training key frames of the training video to obtain a pre-trained video self-supervisory model. The video classification model is generated by the pre-trained video self-supervisory model and the classification head. Since only a small number of videos with labeled categories are required for the video classification model training, and there is no need to manually label the training videos with categories when training the video self-supervisory model, this solution solves the problem that a large amount of manually labeled category data is generally required when training the classification model in the prior art, and the cost of manual labeling is very high and the efficiency is low, which leads to the high cost and low efficiency of video classification.

[0104] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video model training method, characterized in that: include: S10, respectively obtaining a plurality of training key frames of each training video in the training video set; S20, respectively obtaining visible image blocks and shielded image blocks of each training key frame according to preset rules; S30, inserting the training classification flag bit embedding into the first position of each training embedding set, and adding it to the corresponding training position embedding to obtain the coding embedding of each training video; The training embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all training key frames of the corresponding training video; S40, respectively embedding the encoding of each training video into the encoding module of the input video self-supervisory model, and obtaining a first output corresponding to the embedding of the training classification flag of each training video, a training embedding set, and a second output corresponding to the embedding of the corresponding training position; S50, respectively inputting the decoding flag corresponding to each training key frame and the corresponding decoding into the decoding module in the input video self-supervision model to obtain the prediction result of the masked image block of each training key frame; S60, training the video self-supervision model according to the original image and prediction result of the masked image block of each training key frame to obtain a pre-trained video self-supervision model; The decoding embedding corresponding to each training key frame is composed of the image embedding corresponding to the training key frame and the decoding position embedding; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all the shielded image blocks of the training key frame in the order of the image position in the training key frame; the decoding flag input corresponding to any training key frame is the first output corresponding to the training classification flag embedding of the training video corresponding to the training key frame; The preset rule includes a preset division rule and a preset ratio, and S20 includes: S201, dividing each training key frame into corresponding non-overlapping image blocks according to the preset division rule; S202: Obtain visible image blocks and shielded image blocks corresponding to each training key frame from non-overlapping image blocks corresponding to each training key frame according to the preset ratio.

2. The video model training method according to claim 1, characterized in that: The S202 includes: Randomly acquiring the visible image blocks and the shielded image blocks of the corresponding training key frames from the non-overlapping image blocks corresponding to each training key frame according to the preset ratio; or, For the i-th training key frame, randomly obtain the visible image patches and masked image patches of the i-th training key frame from the non-overlapping image patches corresponding to the i-th training key frame according to a preset ratio; i = 1 + xn, where x is a natural number, and n is the integer obtained by rounding down the ratio of the number of non-overlapping image patches to the visible number of visible image patches; 1 ≤ i ≤ the total number of training key frames, and i is a positive integer; for the j-th training key frame, randomly obtain the visible image patches of the j-th training key frame from the optional region of the j-th training key frame according to the visible number, and obtain the masked image patches of the j-th training key frame according to the visible image patches of the j-th training key frame; the j-th training key frame is the key frame other than the i-th training key frame among all training key frames; the optional region of the j-th training key frame is the positions of all visible image patches among all image positions except for the positions of all visible image patches from the first key frame to the previous key frame of the j-th training key frame; the first key frame is the nearest i-th training key frame before the j-th training key frame; 1 < j ≤ the total number of training key frames, and j is a positive integer.

3. The video model training method according to claim 1 or 2, characterized in that: The S60 includes: S601. Respectively obtain the mean square error between the original image and the current prediction result of the masked image patches of all training key frames; S602. Determine whether the mean square error is greater than a preset threshold; if it is greater, execute step S603; if it is not greater, execute step S604; S603. After adjusting the trainable parameters in the video self-supervised model, use the adjusted model to re-execute steps S40 and S50; S604. Use the video self-supervised model when the mean square error is not greater than the preset threshold as the pre-trained video self-supervised model.

4. A video model training method, characterized in that: It includes: P10. Obtain an initial classification model according to the classification head and the pre-trained video self-supervised model as described in any one of claims 1-3; P20. Train the initial classification model through a video-category training set to obtain a pre-trained classification model.

5. The video model training method according to claim 4, characterized in that: The P20 includes: Simultaneously train the classification head and the pre-trained video self-supervised model in the initial classification model through a video-category training set to obtain a pre-trained classification model; or, Train the classification head in the initial classification model through a video-category training set to obtain a pre-trained classification model.

6. A video classification method, characterized in that: It includes: Q10. Obtain multiple classification key frames of the video to be classified; Q20. Respectively obtain the visible image patches of each classification key frame according to a preset rule; Q30. After inserting a classification flag bit embedding at the first position of the classification embedding set and adding it to the corresponding classification position embedding, obtain a target model embedding; the classification embedding set is obtained by combining all visible embeddings corresponding to all visible image patches of all classification key frames; Q40. Input the target model embedding into the pre-trained classification model as described in claim 4 or 5 to obtain the category of the video to be classified.

7. A video model training device, characterized in that: It includes: A training frame acquisition module, configured to respectively obtain multiple training key frames of each training video in a training video set; An image patch acquisition module, connected to the training frame acquisition module, configured to respectively obtain the visible image patches and masked image patches of each training key frame according to a preset rule; A training embedding acquisition module, connected to the image block acquisition module, is used to insert a training classification flag bit embedding at the first position of each training embedding set, and then add it to the corresponding training position embedding to obtain a coded embedding of each training video; The training embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all training key frames of the corresponding training video; An encoding input module, connected to the training embedding acquisition module, for respectively embedding the encoding of each training video into the encoding module of the input video self-supervisory model, and obtaining a first output corresponding to the embedding of the training classification flag of each training video, a training embedding set, and a second output corresponding to the embedding of the corresponding training position; A decoding input module, connected to the encoding input, for respectively inputting the decoding flag bit corresponding to each training key frame and the corresponding decoding module embedded in the input video self-supervisory model to obtain the prediction result of the masked image block of each training key frame; A self-supervised training module, connected to the image block acquisition module and the decoding input module, respectively, for training the video self-supervised model according to the original image and the prediction result of the masked image block of each training key frame, to obtain a pre-trained video self-supervised model; The decoding embedding corresponding to each training key frame is composed of the image embedding corresponding to the training key frame and the decoding position embedding; the image embedding corresponding to the training key frame is composed of the output of the visible embedding corresponding to the training key frame in the encoding module and the preset shared learnable embedding corresponding to all the shielded image blocks of the training key frame in the order of the image position in the training key frame; the decoding flag input corresponding to any training key frame is the first output corresponding to the training classification flag embedding of the training video corresponding to the training key frame; The preset rule includes a preset division rule and a preset ratio, and the image block acquisition module includes: A non-overlapping acquisition submodule, used to divide each training key frame into corresponding non-overlapping image blocks according to the preset division rule; The image block acquisition submodule is connected to the non-overlapping acquisition submodule and is used to respectively acquire the visible image blocks and the shielded image blocks of the corresponding training key frame from the non-overlapping image blocks corresponding to each training key frame according to the preset ratio.

8. The video model training device according to claim 7, characterized in that: The self-supervised training module includes: The error acquisition submodule is used to respectively obtain the original image of the masked image block of all training key frames and the mean square error of the current prediction result; A threshold judgment submodule, connected to the error acquisition submodule, is used to judge whether the mean square error is greater than a preset threshold; if so, execute the parameter adjustment submodule; if not, execute the model acquisition submodule; The parameter adjustment submodule is connected to the threshold judgment submodule and is used to adjust the trainable parameters in the video self-supervisory model and then re-execute the steps of the encoding input module and the decoding input module using the adjusted model; The model acquisition submodule is connected to the threshold judgment submodule, and is used to use the video self-supervision model used when the value is not greater than a preset threshold as the pre-trained video self-supervision model.

9. A video model training device, characterized in that: include: A classification model acquisition module, used to acquire an initial classification model based on a classification head and a pre-trained video self-supervised model acquired by a model training device as claimed in any one of claims 7 to 8; The classification model training module is connected to the classification model acquisition module and is used to train the initial classification model through a video-category training set to obtain a pre-trained classification model.

10. A video classification device, characterized in that: include: A classification frame acquisition module is used to obtain multiple classification key frames of the video to be classified; A visible image block acquisition module, connected to the classification frame acquisition module, for acquiring the visible image blocks of each classification key frame according to a preset rule; A classification embedding acquisition module, connected to the visible image block acquisition module, is used to insert a classification flag bit embedding at the first position of the classification embedding set, and then add it to the corresponding classification position embedding to obtain a target model embedding; the classification embedding set is obtained by combining the visible embeddings corresponding to all visible image blocks of all classification key frames; The video classification module is connected to the classification embedding acquisition module and is used to embed the target model into the pre-trained classification model obtained by the model training device as described in claim 9 to obtain the category of the video to be classified.

Images