A small sample action recognition method based on Transformer and offset alignment strategy

Abstract

The present invention discloses a small sample action recognition method based on Transformer and misalignment alignment strategy, including: obtaining a query set video and a support set video; obtaining a corresponding first image set and a second image set; sampling and cropping the image set, the cropped first image set forms a first image sequence, each cropped second image set forms a second image sequence, and the first image sequence and the second image sequence are regarded as a group; establishing an action recognition model, the action recognition model includes an improved ResNet network model, a spatiotemporal feature encoder and a multi-granularity feature alignment module connected in sequence, and obtaining multi-granularity action probabilities corresponding to each group of first image sequences and second image sequences; using a classifier to obtain the action category with the highest probability among all multi-granularity action probabilities as the action recognition result of the corresponding action category video to be identified. This method can improve the accuracy and efficiency of small sample action video classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning video understanding, and specifically relates to a small sample action recognition method based on Transformer and misalignment alignment strategy. Background Art

[0002] With the advent of the information age, a large amount of visual information has entered our daily lives. The rapid spread of short videos on the Internet and the comprehensive coverage of monitoring systems in life have enabled various imaging devices to replace human vision, and computer vision technology has also been born. Compared with static image data, dynamic videos contain richer audio-visual information features. Action recognition is mainly used to identify the body movements of people in piles of videos. It uses the powerful computing power of computers to parse data and obtain the information needed by humans in videos or multi-dimensional data. With the increase in the amount of video data close to daily life, video action recognition algorithms have developed rapidly, and at the same time, many new services and application models have been spawned. Action recognition algorithms can be applied to fields such as human-computer interaction, robots, entertainment interaction, and face recognition, and can also provide strong technical support for short videos, live broadcast websites and other businesses. However, the existing action recognition methods based on standard supervised learning environments require massive data input and a lot of computing power support. Therefore, a small sample action recognition algorithm is proposed, which aims to learn to distinguish action categories with very few data samples.

[0003] The current mainstream method in the field of small sample action recognition is to measure the similarity between videos. The key idea is to learn a generalizable metric from the action classes of videos with a large number of labels. Such a metric can be used to measure the similarity between videos. It can effectively calculate the similarity between the query set video and the support set video, thereby determining the action category of the query set video.

[0004] However, existing small-sample action recognition algorithms generally have some shortcomings: 1) It is difficult to match and align feature granularity. For accurate query set video and support set video matching, the ideal metric should be learned from multiple levels, such as video frame, video clip, or even the entire video level. However, the matching granularity from coarse to fine will increase the complexity of the model. In addition, coordinating and combining the matching of each granularity is also a very challenging problem. Most of the existing metric-based small-sample action recognition works compare two videos from a single granularity, mainly including two matching methods, global matching and temporal matching. Global matching will match the similarity of a bunch of features, which encodes the video into a vector of fixed size to calculate the similarity. This has a problem that it cannot distinguish the dynamics of different time sequences of the video. The temporal matching method uses alignment between frames or video clips in time order. This method has made great progress in this field, but the spatial information obtained by this type of model is very limited. For example, the ability to distinguish between categories such as "playing guitar" and "playing bass" is limited. Therefore, researchers should explore a simple and relatively reliable matching mechanism. 2) The model lacks diversity. Early small-sample action recognition methods used cross entropy loss to train global features. In this method, the lack of sample data can easily lead to overfitting of model parameters and difficulty in generalizing action metrics. This usually requires a large number of negative examples to enhance robustness. This method requires human supervision, and manually annotating video training data is very time-consuming and labor-intensive. Therefore, it is very necessary to design an effective deep learning method for these two parts to improve the accuracy of small sample action recognition. Summary of the invention

[0005] The purpose of the present invention is to address the above problems and propose a small sample action recognition method based on Transformer and misalignment alignment strategy, which can improve the accuracy and efficiency of small sample action video classification.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The present invention proposes a small sample action recognition method based on Transformer and misalignment alignment strategy, comprising the following steps:

[0008] S1. Obtain a query set video and a support set video, where the query set video includes a video of an action category to be identified, and the support set video includes multiple videos of known action categories;

[0009] S2, extracting image frames from the video of the action category to be identified and each known action category video respectively, recording the image frames extracted from the video of the action category to be identified as a first image set, and recording the image frames extracted from each known action category video as a second image set;

[0010] S3, sampling the first image set and each second image set according to a preset sampling frame number, and cropping the sampled image frames into the same size, the cropped first image set forms a first image sequence, each cropped second image set forms a second image sequence, and the first image sequence and the second image sequence are regarded as a group;

[0011] S4. Establish an action recognition model, which includes an improved ResNet network model, a spatiotemporal feature encoder, and a multi-granularity feature alignment module connected in sequence, and perform the following operations on each group of the first image sequence and the second image sequence:

[0012] S41, using an improved ResNet network model to perform feature extraction on the first image sequence and the second image sequence respectively, and correspondingly obtain a first basic spatiotemporal feature and a second basic spatiotemporal feature, wherein the improved ResNet network model is to replace at least one convolutional layer in the bottleneck layer of the ResNet network with an expanded convolutional layer;

[0013] S42, respectively inputting the first basic spatiotemporal feature and the second basic spatiotemporal feature into the spatiotemporal feature encoder, and correspondingly obtaining the first multi-granularity video feature and the second multi-granularity video feature, wherein the first multi-granularity video feature includes the query segment granularity feature q s and query frame granularity feature q f The second multi-granularity video feature includes supporting segment granularity features v s and support frame granularity features v f ;

[0014] S43: Input the first multi-granularity video feature and the second multi-granularity video feature into a multi-granularity feature alignment module to obtain the corresponding multi-granularity action probability, as follows:

[0015] According to the support fragment granularity feature v s Get the corresponding support offset feature The formula is as follows:

[0016]

[0017] In the formula, Conv represents the three-dimensional convolution operation, Shfit represents the shift operation;

[0018] Calculate the first action prototype feature of the action category to which the current known action category video belongs and the second action prototype feature The formula is as follows:

[0019]

[0020]

[0021] In the formula, Sc represents the set of known action category videos with action label c in the support set video, c is the action label of the current known action category video, c∈C, C is the total action label of the support set video, c t For S c The t-th known action category video in represents the support offset feature of the t-th known action category video, t = 1~N, N is S c The total number of videos with known action categories in Represents the supporting frame-granular features of the t-th known action category video;

[0022] Calculate the segment granularity likelihood probability P of the current action category video to be identified belonging to the action label c s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ), the formula is as follows:

[0023]

[0024]

[0025] In the formula, ||·|| represents the distance similarity, c p Indicates the predicted label of the current action category video to be identified;

[0026] The fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) to aggregate and obtain the corresponding multi-granularity action probability;

[0027] S5. Use a classifier to obtain the action category with the highest probability among all multi-granularity action probabilities as the action recognition result corresponding to the video of the action category to be recognized.

[0028] Preferably, the spatiotemporal feature encoder performs the following operations:

[0029] S421: input the first basic spatiotemporal feature and the second basic spatiotemporal feature into the Transformer module respectively while maintaining the spatiotemporal order, and obtain the query fragment granularity feature accordingly. and support fragment granularity features Among them, the granularity of the i-th query fragment is and the ith supporting fragment granularity The calculation is as follows:

[0030]

[0031]

[0032] In the formula, Q k K represents the query weight of the kth attention head of the Transformer module. k represents the key weight of the kth attention head of the Transformer module, V k represents the value weight of the kth attention head of the Transformer module, k = 1 to 8, v i represents the i-th segment of the currently known action category video, v j represents the jth segment of the currently known action category video, q i represents the i-th segment of the current action category video to be identified, q j represents the jth segment of the current action category video to be identified, i, j = 1 to n, n is the total number of segments of the known action category video or the action category video to be identified;

[0033] S422, respectively, the query fragment granularity features q s and support fragment granularity features v s Perform average pooling to obtain the query frame granularity features and support frame granularity features Among them, the granularity of the i-th query frame and the i-th supported frame granularity The calculation is as follows:

[0034]

[0035]

[0036] Where Avg() is the average pooling function.

[0037] Preferably, the fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) are aggregated in an additive manner with a weight of 1:1.

[0038] Preferably, the multi-granularity action probability is given by the following formula:

[0039]

[0040] In the formula, q is the current action category video to be identified.

[0041] Preferably, the image frames are extracted according to a preset minimum interval and the resolution of the extracted image frames is determined. If it is the preset resolution, it remains unchanged; otherwise, the resolution of the extracted image frames is adjusted to the preset resolution.

[0042] Preferably, the sampled image frames are cropped into the same size, specifically: a random cropping strategy is adopted in the training phase, and a center cropping strategy is adopted in the testing phase.

[0043] Preferably, the size of the cropped image frame is [224, 224] pixels.

[0044] Preferably, sparse sampling is used for sampling the first image set and each second image set, and the preset sampling frame number M=8.

[0045] Preferably, the dilation rate of the dilated convolutional layer is 2 or 4.

[0046] Preferably, the shift operation implements a quarter shift.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This method obtains support set videos and query set videos, extracts frames from the videos to form corresponding image sets, and then obtains multiple image sequences with the same number of sampling frames as the input of the action recognition model. The action recognition model extracts basic spatiotemporal features of the image sequence by improving the ResNet network model, and uses the ResNet model with dilated convolution to obtain long-sequence spatiotemporal features. It can obtain a larger convolution kernel receptive field while reducing the number of model parameters. The basic spatiotemporal features are then block-encoded by the spatiotemporal feature encoder, and the features are globally associated using the self-attention mechanism of the Transformer module. The multi-granularity feature alignment module adopts a full alignment strategy for fine-grained features (frame granularity features) and a staggered alignment strategy for coarse-grained features (fragment granularity features). The overfitting problem of the small sample model is effectively alleviated through scene deviation. Finally, the classifier is used to obtain the action category with the highest probability as the action recognition result, which can greatly improve work efficiency and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the small sample action recognition method based on Transformer and misalignment alignment strategy of the present invention;

[0050] Figure 2 It is a structural schematic diagram of the action recognition model of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of the improved ResNet network model of the present invention;

[0052] Figure 4 It is a schematic diagram of the structure of the spatiotemporal feature encoder of the present invention;

[0053] Figure 5 It is a structural schematic diagram of the multi-granularity feature alignment module of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the technical field of this application. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0056] like Figure 1-5 As shown in FIG. 1 , a small sample action recognition method based on Transformer and misalignment alignment strategy includes the following steps:

[0057] S1. Obtain query set videos and support set videos. The query set videos include a video of an action category to be identified, and the support set videos include multiple videos of known action categories.

[0058] S2. Extract image frames from the action category video to be identified and each known action category video respectively. The image frames extracted from the action category video to be identified are recorded as a first image set, and the image frames extracted from each known action category video are recorded as a second image set.

[0059] In one embodiment, the image frame is extracted according to the preset minimum interval and the resolution of the extracted image frame is determined. If it is the preset resolution, it remains unchanged. Otherwise, the resolution of the extracted image frame is adjusted to the preset resolution. In this embodiment, the preset resolution is [256, 256] pixels. It is easy to understand that the preset resolution can also be designed according to actual needs.

[0060] S3. Sample the first image set and each second image set according to a preset sampling frame number, and crop the sampled image frames to the same size. The cropped first image set forms a first image sequence, and each cropped second image set forms a second image sequence. The first image sequence and one second image sequence are regarded as a group.

[0061] In one embodiment, the sampled image frames are cropped to the same size, specifically: a random cropping strategy is used in the training phase, and a center cropping strategy is used in the test phase. It is easy to understand that before using the action recognition model for actual testing, the data set can also be used to train the action recognition model. A random cropping strategy can be used in the training phase, and a center cropping strategy can be used in the test phase, and the specific adjustment can be made according to actual needs.

[0062] In one embodiment, the size of the image frame after cropping is [224, 224] pixels. In one embodiment, sparse sampling is used for sampling the first image set and each second image set, and the preset sampling frame number M=8. It is easy to understand that the cropping size of the image frame and the preset sampling frame number can also be designed according to actual needs.

[0063] S4. Establish an action recognition model, which includes an improved ResNet network model, a spatiotemporal feature encoder, and a multi-granularity feature alignment module connected in sequence, and perform the following operations on each group of the first image sequence and the second image sequence:

[0064] S41. Use an improved ResNet network model to extract features from the first image sequence and the second image sequence respectively, and obtain a first basic spatiotemporal feature and a second basic spatiotemporal feature correspondingly. The improved ResNet network model is to replace at least one convolutional layer in the bottleneck layer of the ResNet network with an expanded convolutional layer.

[0065] like Figure 3As shown, the improved ResNet network model in this embodiment is to selectively replace the convolution layer in the bottleneck layer (Bottle Neck) of the ResNet network model of the prior art with an expanded convolution layer (D-Conv2D). In the figure, Conv2D represents a two-dimensional convolution layer, BN represents a normalization layer, ReLU represents a ReLU activation function layer, and Maxpooling represents a maximum pooling layer. The ResNet network model of the prior art includes 5 stages (Stage 0, Stage 1, Stage 2, Stage 3, and Stage 4). Part of the convolution layers in the two bottleneck layers (Bottle Neck1 and Bottle Neck2) in Stage 1, Stage 2, Stage 3, and Stage 4 are replaced with an expanded convolution layer (D-Conv2D). For example, each bottleneck layer replaces two convolution layers, and the expansion rate of the expanded convolution layer (D-Conv2D) in Bottle Neck1 is 2, and the expansion rate of Bottle Neck2 is 2. The expansion rate of the dilated convolution layer (D-Conv2D) in Neck2 is 4. Except for the replacement layer, the structure is the original structure of the ResNet network model in the prior art and will not be repeated here.

[0066] S42, respectively inputting the first basic spatiotemporal feature and the second basic spatiotemporal feature into the spatiotemporal feature encoder, and correspondingly obtaining the first multi-granularity video feature and the second multi-granularity video feature, wherein the first multi-granularity video feature includes the query segment granularity feature q s and query frame granularity feature q f The second multi-granularity video feature includes supporting segment granularity features v s and support frame granularity features v f .

[0067] In one embodiment, the spatiotemporal feature encoder performs the following operations:

[0068] S421: input the first basic spatiotemporal feature and the second basic spatiotemporal feature into the Transformer module respectively while maintaining the spatiotemporal order, and obtain the query fragment granularity feature accordingly. and support fragment granularity features Among them, the granularity of the i-th query fragment is and the ith supporting fragment granularity The calculation is as follows:

[0069]

[0070]

[0071] In the formula, Q k K represents the query weight of the kth attention head of the Transformer module. krepresents the key weight of the kth attention head of the Transformer module, V k represents the value weight of the kth attention head of the Transformer module, k = 1 to 8, v i represents the i-th segment of the currently known action category video, v j represents the jth segment of the currently known action category video, q i represents the i-th segment of the current action category video to be identified, q j represents the jth segment of the current action category video to be identified, i, j = 1 to n, n is the total number of segments of the known action category video or the action category video to be identified;

[0072] S422, respectively, the query fragment granularity features q s and support fragment granularity features v s Perform average pooling to obtain the query frame granularity features and support frame granularity features Among them, the granularity of the i-th query frame and the i-th supported frame granularity The calculation is as follows:

[0073]

[0074]

[0075] Where Avg() is the average pooling function.

[0076] like Figure 4 As shown in Figure 1, the self-attention mechanism of the Transformer module is used to encode the features of different positions in the video. Among them, k ranges from 1 to 8, representing an eight-head attention structure, Q k (Query), K k (Key) and V k (Value) is the learnable weight in the attention head, v i represents the i-th segment of the currently known action category video, v j represents the jth segment of the currently known action category video, q i represents the i-th segment of the current action category video to be identified, q j Represents the jth segment of the current action category video to be identified, associates it with segments at other positions in the same video, and then obtains the global action field of view through the encoder and decoder of the Transformer module.

[0077] In this application, two types of visual clue features with different granularities are defined, namely frame granularity features and fragment granularity features. Frame-level features are obtained through the Transformer module. Although frame-level features (fine-grained features) contain many spatial details, they lack the temporal clues in coarse-grained features (fragment granularity features). Subsequently, fragment-level features are generated through the average pooling layer to display dynamic information in more detail while ensuring spatial order.

[0078] S43: Input the first multi-granularity video feature and the second multi-granularity video feature into a multi-granularity feature alignment module to obtain the corresponding multi-granularity action probability, as follows:

[0079] According to the support fragment granularity feature v s Get the corresponding support offset feature The formula is as follows:

[0080]

[0081] In the formula, Conv represents the three-dimensional convolution operation, Shfit represents the shift operation;

[0082] Calculate the first action prototype feature of the action category to which the current known action category video belongs and the second action prototype feature The formula is as follows:

[0083]

[0084]

[0085] In the formula, S c represents the set of known action category videos with action label c in the support set video, c is the action label of the current known action category video, c∈C, C is the total action label of the support set video, c t For S c The t-th known action category video in represents the support offset feature of the t-th known action category video, t = 1~N, N is S c The total number of videos with known action categories in Represents the supporting frame-granular features of the t-th known action category video;

[0086] Calculate the segment granularity likelihood probability P of the current action category video to be identified belonging to the action label c s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ), the formula is as follows:

[0087]

[0088]

[0089] In the formula, ||·|| represents the distance similarity, c p Indicates the predicted label of the current action category video to be identified.

[0090] The fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) are aggregated to obtain the corresponding multi-granularity action probabilities.

[0091] In one embodiment, the segment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) are aggregated in an additive manner with a weight of 1:1.

[0092] In one embodiment, the multi-granularity action probability is formulated as follows:

[0093]

[0094] In the formula, q is the current action category video to be identified.

[0095] In one embodiment, the dilation rate of the dilated convolutional layer is 2 or 4.

[0096] In one embodiment, the shift operation implements a quarter shift.

[0097] Specifically, the multi-granularity feature alignment module can match and align features of multiple granularities, such as Figure 5 As shown, including:

[0098] The staggered alignment strategy is used to process the fragment granularity features and fully align the frame granularity features. First, the support fragment granularity features v s Along the time dimension, the whole is moved by one quarter. The moving distance can also be adjusted according to actual needs. Subsequently, a three-dimensional convolution operation is used to fuse the rich time domain features to obtain the corresponding support offset features. Complete the feature offset alignment operation. For this support offset feature Since the video action label is known to be c, the corresponding first action prototype feature is obtained by averaging the offset features of other supporting videos of the same action category. and the second action prototype feature And the query fragment granularity feature q s and query frame granularity feature q f With the first action prototype characteristics and the second action prototype feature The distance similarity is calculated by metric method to obtain the segment granularity likelihood probability P of the current action category video to be identified belonging to the action label c s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ), the fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P c |c p =c|q f ) are aggregated to obtain the multi-granularity action probabilities of the corresponding action categories.

[0099] S5. Use a classifier to obtain the action category with the highest probability among all multi-granular action probabilities as the action recognition result of the video of the action category to be recognized. After traversing each group of the first image sequence and the second image sequence, output the multi-granular action probabilities of the query set video belonging to each action category, and use a classifier (Few-shot Classifier) ​​to take the action category with the highest multi-granular action probability as the action recognition result.

[0100] This method obtains support set videos and query set videos, extracts frames from the videos to form corresponding image sets, and then obtains multiple image sequences with the same number of sampling frames as the input of the action recognition model. The action recognition model extracts basic spatiotemporal features of the image sequence by improving the ResNet network model, and uses the ResNet model with dilated convolution to obtain long-sequence spatiotemporal features. It can obtain a larger convolution kernel receptive field while reducing the number of model parameters. The basic spatiotemporal features are then block-encoded by the spatiotemporal feature encoder, and the features are globally associated using the self-attention mechanism of the Transformer module. The multi-granularity feature alignment module adopts a full alignment strategy for fine-grained features (frame granularity features) and a staggered alignment strategy for coarse-grained features (fragment granularity features). The overfitting problem of the small sample model is effectively alleviated through scene deviation. Finally, the classifier is used to obtain the action category with the highest probability as the action recognition result, which can greatly improve work efficiency and classification accuracy.

[0101] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.

Claims

1. A small sample action recognition method based on Transformer and misalignment alignment strategy, characterized by: The small sample action recognition method based on Transformer and misalignment alignment strategy includes the following steps: S1, obtaining a query set video and a support set video, wherein the query set video includes a video of an action category to be identified, and the support set video includes multiple videos of known action categories; S2, extracting image frames from the video of the action category to be identified and each known action category video respectively, recording the image frames extracted from the video of the action category to be identified as a first image set, and recording the image frames extracted from each known action category video as a second image set; S3, sampling the first image set and each second image set according to a preset sampling frame number, and cropping the sampled image frames into the same size, the cropped first image set forms a first image sequence, each cropped second image set forms a second image sequence, and the first image sequence and the second image sequence are regarded as a group; S4. Establish an action recognition model, wherein the action recognition model includes an improved ResNet network model, a spatiotemporal feature encoder, and a multi-granularity feature alignment module connected in sequence, and perform the following operations on each group of the first image sequence and the second image sequence: S41, using an improved ResNet network model to perform feature extraction on the first image sequence and the second image sequence respectively, and correspondingly obtain a first basic spatiotemporal feature and a second basic spatiotemporal feature, wherein the improved ResNet network model is to replace at least one convolutional layer in the bottleneck layer of the ResNet network with an expanded convolutional layer; S42, respectively inputting the first basic spatiotemporal feature and the second basic spatiotemporal feature into the spatiotemporal feature encoder, and correspondingly obtaining a first multi-granularity video feature and a second multi-granularity video feature, wherein the first multi-granularity video feature includes a query segment granularity feature q s and query frame granularity feature q f The second multi-granularity video feature includes a support segment granularity feature v s and support frame granularity features v f ; S43: Input the first multi-granularity video feature and the second multi-granularity video feature into a multi-granularity feature alignment module to obtain the corresponding multi-granularity action probability, as follows: According to the support fragment granularity feature v s Get the corresponding support offset feature The formula is as follows: In the formula, Conv represents the three-dimensional convolution operation, Shfit represents the shift operation; Calculate the first action prototype feature of the action category to which the current known action category video belongs and the second action prototype feature The formula is as follows: In the formula, S c represents the set of known action category videos with action label c in the support set video, c is the action label of the current known action category video, c∈C, C is the total action label of the support set video, c t For S c The t-th known action category video in represents the support offset feature of the t-th known action category video, t = 1~N, N is S c The total number of videos with known action categories in Represents the supporting frame-granular features of the t-th known action category video; Calculate the segment granularity likelihood probability P of the current action category video to be identified belonging to the action label c s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ), the formula is as follows: In the formula, ||·|| represents the distance similarity, c p Indicates the predicted label of the current action category video to be identified; The fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) to aggregate and obtain the corresponding multi-granularity action probability; The spatiotemporal feature encoder performs the following operations: S421: input the first basic spatiotemporal feature and the second basic spatiotemporal feature into the Transformer module respectively while maintaining the spatiotemporal order, and obtain the query fragment granularity feature accordingly. and support fragment granularity features Among them, the granularity of the i-th query fragment is and the ith supporting fragment granularity The calculation is as follows: In the formula, Q k K represents the query weight of the kth attention head of the Transformer module. k represents the key weight of the kth attention head of the Transformer module, V k represents the value weight of the kth attention head of the Transformer module, k = 1 to 8, v i represents the i-th segment of the currently known action category video, v j represents the jth segment of the currently known action category video, q i represents the i-th segment of the current action category video to be identified, q j represents the jth segment of the current action category video to be identified, i, j = 1 to n, n is the total number of segments of the known action category video or the action category video to be identified; S422, respectively, the query fragment granularity features q s and support fragment granularity features v s Perform average pooling to obtain the query frame granularity features and support frame granularity features Among them, the granularity of the i-th query frame and the i-th supported frame granularity The calculation is as follows: Where Avg() is the average pooling function; S5. Use a classifier to obtain the action category with the highest probability among all multi-granularity action probabilities as the action recognition result corresponding to the video of the action category to be recognized.

2. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The fragment granularity likelihood probability P s (c p =c|q s ) and the frame granularity likelihood probability P f (c p =c|q f ) are aggregated in an additive manner with a weight of 1:

1.

3. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 2, characterized in that: The multi-granularity action probability is formulated as follows: In the formula, q is the current action category video to be identified.

4. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The image frames are extracted according to a preset minimum interval and the resolution of the extracted image frames is determined. If the resolution is the preset resolution, it remains unchanged; otherwise, the resolution of the extracted image frames is adjusted to the preset resolution.

5. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The sampled image frames are cropped into the same size, specifically: a random cropping strategy is adopted in the training phase, and a center cropping strategy is adopted in the testing phase.

6. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 5, characterized in that: The size of the cropped image frame is [224, 224] pixels.

7. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The sampling of the first image set and each second image set adopts sparse sampling, and the preset sampling frame number M=8.

8. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The dilation rate of the dilated convolutional layer is 2 or 4.

9. The small sample action recognition method based on Transformer and misalignment alignment strategy as claimed in claim 1, characterized in that: The shift operation achieves a quarter shift.

Images