First person behavior identification method and device based on small sample learning

Abstract

The invention relates to a first person behavior recognition method and device based on small sample learning, and the method comprises the steps: building a meta-training task and a meta-testing task through a first person video sample according to the small sample learning setting; the meta-training task is utilized to perform meta-training on a first person behavior recognition model, and the first person behavior recognition model comprises a feature extraction module and a set matching measurement module; the feature extraction module is used for extracting multi-scale video features of a first person video sample; the set matching measurement module is used for calculating a matching score of the first person video sample in the query set and the support set according to the multi-scale video features of the first person video sample, and identifying a behavior category of the user according to the matching score; and performing user behavior recognition on the meta-test task by using the trained first person behavior recognition model. The method can improve the generalization ability and recognition accuracy of the model, and reduce the calculation amount of model training.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a first-person behavior recognition method and device based on small sample learning. Background Art

[0002] In daily life, people frequently use wearable video acquisition devices, such as smart helmets and head-mounted cameras, to record their various activities, which results in a large number of videos shot from a first-person perspective. First-person videos record the interaction between human hands and interactive objects. As one of the basic tasks of first-person video analysis, first-person action recognition aims to identify the actions generated by the interaction between human hands and interactive objects in first-person videos.

[0003] For the first-person action recognition task, current deep learning methods can be roughly divided into three categories. The first category is based on class activation feature map methods. This type of method uses an image classification network pre-trained on a large-scale image dataset to extract the class activation map of the video frame, and then extracts spatiotemporal fusion information by applying spatial attention, temporal modeling, or long-short time attention modules. However, these methods rely more on prior knowledge from large-scale datasets, and the model reasoning speed is slow and the generalization performance is poor. The second category is based on the joint supervision of action labels and human eye sight data. This type of method uses eye sight data as auxiliary supervision data to achieve attention to the main area. Although it can efficiently focus on key spatial features, the additional eye sight data increases redundancy. The third category is based on key object detection methods. Considering the key role of human hands and their interactive objects in first-person action recognition, this type of method uses a target detection model to detect hands and interactive objects in first-person videos and extract features of the detection area, but it requires the assistance of target detection algorithms, which increases the computational complexity of the network and requires additional bounding box annotations. Summary of the invention

[0004] In order to solve the problems existing in the background technology, to improve the reasoning speed of the model, to reduce manual data labeling, to reduce the amount of network calculation, to improve the generalization performance of the network, and to reduce data redundancy, one aspect of the present application proposes a first-person behavior recognition method based on small sample learning, including:

[0005] S1: Following the few-shot learning setting, we construct meta-training tasks and meta-testing tasks using first-person video samples;

[0006] S2: meta-training a first-person action recognition model using a meta-training task, wherein the first-person action recognition model includes: a feature extraction module and a set matching metric module;

[0007] The feature extraction module is used to extract multi-scale video features of first-person video samples;

[0008] The set matching metric module is used to calculate the matching scores of the first-person video samples in the query set and the support set according to the multi-scale video features of the first-person video samples, and identify the user's behavior category according to the matching scores;

[0009] S3: Use the trained first-person action recognition model to perform user behavior recognition on the meta-test task.

[0010] Preferably, the feature extraction module includes: ResNet50, and a feature fusion branch corresponding to each stage of ResNet50;

[0011] The multi-scale video feature extraction of the first-person video sample includes:

[0012] Inputting the video frames of the first-person video sample into ResNet50 in sequence, and outputting the video frame features of the corresponding scale in each stage of ResNet50; the ResNet50 includes 5 stages, each stage is composed of a different number of convolutional layers and / or residual blocks;

[0013] All video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain video features of the corresponding scale;

[0014] The video features output by all feature fusion branches are combined into multi-scale video features of the first-person video sample.

[0015] Preferably, the feature fusion branch flattens all input video frame features in the spatial dimension respectively, inputs all flattened video frame features into the mapper for spatiotemporal information fusion to generate spatiotemporal video features; and performs average pooling on the spatiotemporal video features to obtain output features of the feature extraction branch.

[0016] Preferably, the mapper takes all flattened video frame features as input sequences and performs single-head self-attention operations to obtain spatial video frame feature sequences; performs average pooling operations on the spatial video frame feature sequences to obtain spatial pooling video frame feature sequences; concatenates all feature elements in the spatial pooling video frame feature sequences to obtain initial video features; inputs the initial video features into MLP-Mixer for processing to obtain temporal video features; and sequentially performs expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features.

[0017] Preferably, the MLP-Mixer includes: a token-mixing MLP and a channel-mixing MLP; the input feature of the token-mixing MLP is the transposition of the initial video feature; the output feature of the token-mixing MLP is transposed and added to the input initial video feature as the input feature of the channel-mixing MLP; the output feature of the channel-mixing MLP is added to the input feature of the channel-mixing MLP to obtain the temporal video feature.

[0018] Preferably, the step of sequentially performing expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features includes:

[0019] The temporal video features of all samples in the support set are combined into the temporal video features of the support set The temporal video features of all samples in the query set constitute the temporal video features of the query set K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames of the sample, and C is the number of channels;

[0020] The temporal video feature of the support set is expanded by one dimension in the 0th dimension and replicated M times along this dimension, where M represents the number of samples in the query set, to obtain the first intermediate video feature of the support set

[0021] Add a new dimension after the first dimension to the temporal video feature of the query set to obtain the first intermediate video feature of the query set

[0022] The first intermediate video feature of the support set and the first intermediate video feature of the query set are averaged in the frame number dimension to obtain the second intermediate video feature of the query set. and the second intermediate video features of the support set

[0023] The second intermediate video feature of the support set and the second intermediate video feature of the query set are concatenated along the second dimension to obtain the first fusion feature.

[0024] Multi-head self-attention is used to process the first fusion feature to obtain the second fusion feature;

[0025] Add a dimension to the second fusion feature after the second dimension and copy it F times along this dimension to get the third fusion feature F sup ∈R M×(K+1)×F×C ;

[0026] The first K video features of the third fusion feature in the second dimension The first intermediate video features of the support set samples Concatenate along the channel dimension to get the first fusion feature F of the support set s ∈R M×K×F×2C , and then get the first fusion feature of each support set sample i represents the i-th first-person video sample in the support set;

[0027] The last video feature of the third fusion feature on the second dimension The first intermediate video feature of the query set The first fusion feature F of the query set is obtained by splicing along the channel dimension p ∈R M×1×F×2C , and then get the first fusion feature of each query set sample j represents the jth first-person video sample in the query set;

[0028] One-dimensional convolution is used to further aggregate the first fused features of the support set samples or the first fused features of the query set samples along the channel dimension so that their channel dimension is the same as the channel dimension of the mapper input feature, thereby obtaining the spatiotemporal video features of the support set samples or the query set samples.

[0029] Preferably, calculating the matching scores of the first-person video samples in the query set and the support set comprises:

[0030]

[0031] D(x q ,S n )=max(d(x q ,S n ),d(S n ,x q ))

[0032] Among them, d M (·) represents the Mahalanobis distance, x q represents a query set sample, represents the support set, x i n Denotes the support set S n The i-th first-person video sample in y i Representation sample The user behavior category of n represents the user behavior category of all samples in the support set is n, f(x q ) represents the query sample x q Multi-scale video features, Represents the support set sample Multi-scale video features of Score(x q ,Sn ) represents the query sample x q and support set S n The matching score.

[0033] Preferably, identifying the user's behavior category according to the matching score includes: selecting a support set S with the largest matching score n The user behavior category n is used as the query sample x q categories of user behavior.

[0034] Another aspect of the present application provides a first-person behavior recognition device based on small sample learning, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor is configured to implement the first-person behavior recognition method based on small sample learning when executing the computer program.

[0035] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the first-person behavior recognition method based on small sample learning can be implemented.

[0036] The present invention has at least the following beneficial effects

[0037] The present invention constructs a meta-training task and a meta-testing task, and trains a first-person behavior recognition model by meta-learning, so that the model can quickly adapt to new scenes and behavior categories. Training on the meta-training task can be transferred to the meta-testing task through a small amount of labeled data, so as to improve the generalization ability of the model and reduce manual data labeling. At the same time, the setting of small sample learning in meta-learning can reduce the training data of the model, reduce the calculation amount of the network, and improve the reasoning speed of the model. The present invention uses a feature extraction module to extract multi-scale video features including edges, details, contours and high-level semantics from the same video, fully realizes the utilization of first-person video data, reduces data redundancy, and measures the matching score between the first-person video sample and the support set in the query set of the meta-training task or the meta-testing task through the set matching metric module. The matching score model can explicitly and accurately measure the matching score between the query set sample and the support set, and realize accurate recognition of user behavior for the query set sample according to the matching score. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 A schematic diagram of the network structure of the first-person behavior recognition model of the present invention;

[0040] Figure 3It is a schematic diagram of the structure of the mapper of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 , Figure 2 and Figure 3 One aspect of the present invention provides a first-person behavior recognition method based on small sample learning, comprising:

[0043] S1: Following the few-shot learning setting, we construct meta-training tasks and meta-testing tasks using first-person video samples;

[0044] S2: meta-training a first-person action recognition model using a meta-training task, wherein the first-person action recognition model includes: a feature extraction module and a set matching metric module;

[0045] The feature extraction module is used to extract multi-scale video features of first-person video samples;

[0046] The set matching metric module is used to calculate the matching scores of the first-person video samples in the query set and the support set according to the multi-scale video features of the first-person video samples, and identify the user's behavior category according to the matching scores;

[0047] S3: Use the trained first-person action recognition model to perform user behavior recognition on the meta-test task.

[0048] In this embodiment, firstly, first-person video samples are obtained to construct a first-person video sample set, wherein the first-person video sample set includes: first-person video samples and user behavior category labels corresponding to the first-person video samples;

[0049] Following the small sample learning setting, the first-person video sample set with the total user behavior category set C is divided into a meta-training set according to the user behavior category and meta-test set Meta-training set The set of user behavior categories is Meta test set The set of user behavior categories is and

[0050] In this embodiment, the first-person video sample set includes first-person video samples of various user behavior categories. The user behavior category set of all first-person video samples in the first-person video sample set is represented as C, and the meta-training set The set of user behavior categories is Meta test set The set of user behavior categories is and

[0051] In this embodiment, constructing the meta-training task and the meta-testing task includes the following process: Randomly select N1 user behavior categories from the meta-test set, and randomly select K+M first-person video samples for each user behavior category; take the K first-person video samples of each user behavior category as the support set, and the M first-person video samples as the query set to construct the meta-training task. Similarly, select the user behavior category set from the meta-test set. Randomly select N2 user behavior categories, and randomly select K+M first-person video samples for each user behavior category; take the K first-person video samples of each user behavior category as the support set, and the M first-person video samples as the query set to construct a meta-test task, where the value of K should be much smaller than M and can be determined based on the experience of technicians in this field.

[0052] In this embodiment, each meta-training task or meta-testing task includes: a support set and a query set. The support set of the meta-training task has a small number of first-person video samples and a label corresponding to each sample. The first-person video samples in the query set of the meta-training task have real labels. The first-person video samples in the support set of the meta-testing task have real labels. The first-person video samples in the query set of the meta-testing task do not have real labels. User behavior recognition is performed on the first-person video samples in the query set without labels through the support set with labels in the meta-testing task.

[0053] First-person video, also known as egocentric video or first person video, refers to video content recorded from a personal perspective by wearing a camera device (such as a head-mounted camera, smart glasses, etc.). The characteristics of this form of video are that the shooting angle and the change of the perspective are completely driven by the head and body movement of the person wearing the device (i.e. the cameraman or the actor), so the captured content can be roughly understood as the scene captured by the human eye.

[0054] In this embodiment, by constructing meta-training tasks and meta-testing tasks, the model can be quickly adapted to new scenarios and behavior categories through meta-learning. After training on the meta-training task, a small amount of labeled data can be migrated to the meta-testing task to improve the generalization ability of the model and reduce manual labeling of data. At the same time, in meta-learning, the setting of small sample learning can reduce the training data of the model, reduce the computational workload of the network, and improve the reasoning speed of the model.

[0055] In this embodiment, the feature extraction module includes: ResNet50, and a feature fusion branch corresponding to each stage of ResNet50;

[0056] The multi-scale video feature extraction of the first-person video sample includes:

[0057] Inputting the video frames of the first-person video sample into ResNet50 in sequence, and outputting the video frame features of the corresponding scale in each stage of ResNet50; the ResNet50 includes 5 stages, each stage is composed of a different number of convolutional layers and / or residual blocks;

[0058] All video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain video features of the corresponding scale;

[0059] The video features output by all feature fusion branches are combined into multi-scale video features of the first-person video sample.

[0060] In this embodiment, each first-person video sample is composed of multiple video frames. In order to reduce the amount of computation and training time of the network, the present invention provides a frame extraction preprocessing process for the video, extracting X video frames from each video, and forming a video frame sequence in time sequence; in order to comprehensively consider the relationship between the amount of computation and the accuracy of reasoning, this embodiment obtains a value of X of 8 based on a large number of experiments, and the effect is better. Of course, those skilled in the art can further conduct a large number of limited experiments according to factors such as the size of the video and the number of frames to obtain a more optimal value. The frame extraction method includes: manual frame extraction, speed time lapse, export page frame extraction, time plug-in or expression frame extraction, card point frame extraction, automatic segmentation and frame extraction and other existing frame extraction methods.

[0061] In this embodiment, ResNet50 includes 5 stages, each of which is composed of a different number of convolutional layers and / or residual blocks. The following is an explanation of the existing network such as ResNet50;

[0062] The first stage of ResNet50, denoted as Stage0, consists of two layers: the first layer includes: CONV layer, BN layer and RELU; the second layer includes: maxpool layer.

[0063] The second stage of ResNet50 is denoted as Stage 1, which consists of 3 residual blocks. The specific composition can be found in existing literature and will not be described in detail in this application.

[0064] The third stage of ResNet50 is denoted as Stage2, which consists of 4 residual blocks. The specific composition can be found in existing literature and will not be described in detail in this application.

[0065] The fourth stage of ResNet50 is denoted as Stage3, which consists of 6 residual blocks. The specific composition can be found in existing literature and will not be described in detail in this application;

[0066] The fifth stage of ResNet50 is denoted as Stage4, which consists of 3 residual blocks. The specific composition can be found in existing literature and will not be described in detail in this application.

[0067] In this embodiment, the feature fusion branch flattens all input video frame features in the spatial dimension respectively, inputs all flattened video frame features into the mapper for spatiotemporal information fusion to generate spatiotemporal video features; and performs average pooling on the spatiotemporal video features to obtain output features of the feature extraction branch.

[0068] For example, for a first-person video sample T1={t1,t2,…,t8} with 8 video frames, all the video frame features output in the first stage are expressed as: X1={x1,x2,…,x8}, for each xi,i∈{1,2,…,8}, it is flattened in the spatial dimension to obtain the flattened video frame feature yi, all the flattened video frame features Y1={y1,y2,…,y8} are input into the mapper to fuse the spatiotemporal information to generate spatiotemporal video features, and the spatiotemporal video features are average pooled to obtain the output feature Z1 of the feature extraction branch; the video features output by all feature fusion branches constitute the multi-scale video features {Z1,Z2,Z3,Z4,Z5} of the first-person video sample, and Zj,j∈{1,2,…,5} represents the output feature of the jth feature fusion branch.

[0069] In this embodiment, the mapper takes all flattened video frame features as input sequences and performs single-head self-attention operations to obtain spatial video frame feature sequences; performs average pooling operations on the spatial video frame feature sequences to obtain spatial pooling video frame feature sequences; concatenates all feature elements in the spatial pooling video frame feature sequences to obtain initial video features; inputs the initial video features into MLP-Mixer for processing to obtain temporal video features; and performs expansion-concatenation-convolution operations on the temporal video features in sequence to generate spatiotemporal video features.

[0070] In this embodiment, the MLP-Mixer includes: a token-mixing MLP and a channel-mixing MLP; the input feature of the token-mixing MLP is the transposition of the initial video feature; the output feature of the token-mixing MLP is transposed and added to the input initial video feature as the input feature of the channel-mixing MLP; the output feature of the channel-mixing MLP and the input feature of the channel-mixing MLP are added to obtain the temporal video feature.

[0071] In this embodiment, the temporal video features are sequentially expanded, concatenated, and convolved to generate spatiotemporal video features, including:

[0072] The temporal video features of all samples in the support set are combined into the temporal video features of the support set The temporal video features of all samples in the query set constitute the temporal video features of the query set K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames of the sample, and C is the number of channels;

[0073] The temporal video feature definition of the support set is expressed as Among them, K is the number of support set samples, F is the number of sampling frames of support set samples, and C is the number of channels;

[0074] The temporal video feature definition of the query set is expressed as Where M is the number of query set samples; F is the number of sampling frames of the query set samples, and C is the number of channels;

[0075] The temporal video feature of the support set is expanded by one dimension in the 0th dimension and replicated M times along this dimension, where M represents the number of samples in the query set, to obtain the first intermediate video feature of the support set

[0076] Add a new dimension after the first dimension to the temporal video feature of the query set to obtain the first intermediate video feature of the query set

[0077] First, in Add a new dimension to the 0th dimension of Then, according to the number of query set samples M, copy along this newly added dimension Multiple times, get the first intermediate video feature of the support set

[0078] Secondly Add a new dimension after the first dimension to get the first intermediate video feature of the query set

[0079] The first intermediate video feature of the support set and the first intermediate video feature of the query set are averaged in the frame number dimension to obtain the second intermediate video feature of the query set. and the second intermediate video features of the support set

[0080] The second intermediate video feature of the support set and the second intermediate video feature of the query set are concatenated along the second dimension to obtain the first fusion feature.

[0081] Multi-head self-attention is used to process the first fusion feature to obtain the second fusion feature. The attention mechanism is used to interact and enhance the features of the support set and the query set, so as to capture better matching features.

[0082] Add a dimension to the second fusion feature after the second dimension and copy it F times along this dimension to get the third fusion feature F sup ∈R M×(K+1)×F×C ;

[0083] In order to make the learned global interaction relationship information and the original local features jointly enhance the discriminative ability of the model:

[0084] The first K video features of the third fusion feature in the second dimension The first intermediate video features of the support set samples Concatenate along the channel dimension to get the first fusion feature F of the support set s ∈R M×K×F×2C , and then get the first fusion feature of each support set sample i represents the i-th first-person video sample in the support set;

[0085] The last video feature of the third fusion feature on the second dimension The first intermediate video feature of the query set The first fusion feature F of the query set is obtained by splicing along the channel dimension p ∈R M×1×F×2C , and then get the first fusion feature of each query set sample j represents the hth first-person video sample in the query set;

[0086] One-dimensional convolution is used to further aggregate the first fused features of the support set samples or the first fused features of the query set samples along the channel dimension so that their channel dimension is the same as the channel dimension of the mapper input feature, thereby obtaining the spatiotemporal video features of the support set samples or the query set samples.

[0087] In this embodiment, by combining the self-attention mechanism and the convolution operation, the process can simultaneously capture the spatial features (such as the shape and texture of the object) and the temporal features (such as the movement trajectory of the object, the change of action, etc.) in the video. The self-attention mechanism is good at processing long-distance dependencies in sequence data, which helps to capture the features in the time dimension; while the convolution operation has a strong extraction capability for local spatial features. The average pooling operation reduces the feature dimension, reduces the computational complexity, and removes redundant information, making subsequent processing more efficient. When necessary, more contextual information or diversity can be introduced to the features through expansion operations (such as increasing the feature dimension), which is conducive to the subsequent operations. The MLP-Mixer network alternately applies nonlinear transformations in the channel and spatial dimensions through a multi-layer perceptron (MLP), which enhances the expressive power of the features. This nonlinear transformation can capture the complex relationship between features, making the generated video features richer and more distinctive. Through the sequence of expansion-splicing-convolution operations, the process further mines the spatiotemporal information in the video. The expansion operation may increase the richness of information by introducing new feature dimensions or duplicating existing features; the splicing operation combines features from different sources or different levels to form a more comprehensive feature representation; the convolution operation effectively captures the spatiotemporal patterns in the video using the characteristics of local connections and weight sharing. The spatiotemporal video features generated by this process provide strong support for subsequent video processing tasks. These features not only have stronger expressive power, but can also better adapt to the needs of different tasks, thereby improving the performance of the overall system and making the generated video features have stronger generalization capabilities. Even in the face of unseen video data, the model can extract useful feature information from it and make accurate predictions and judgments. This is of great significance for robustness and stability in practical applications.

[0088] In this embodiment, calculating the matching scores of the first-person video samples in the query set and the first-person video samples in the support set includes:

[0089]

[0090] D(x q ,S n )=max(d(x q ,S n ),d(S n ,x q ))

[0091] Among them, d M (·) represents the Mahalanobis distance, x q represents a query set sample, represents the support set, Denotes the support set S n The i-th first-person video sample in yi Representation sample The user behavior category of n represents the user behavior category of all samples in the support set is n, f(x q ) represents the query sample x q Multi-scale video features, Represents the support set sample Multi-scale video features of Score(x q ,S n ) represents the query sample x q and support set S n This matching metric method uses the Mahalanobis distance to calculate the maximum distance from one set to the nearest point in another set, which can effectively capture the maximum difference between sets and takes into account the covariance structure of the data, making it more robust in the face of noise and outliers.

[0092] In this embodiment, identifying the user's behavior category according to the matching score includes: selecting the support set S with the largest matching score n The user behavior category n is used as the query sample x q categories of user behavior.

[0093] The present invention constructs a meta-training task and a meta-testing task, and trains a first-person behavior recognition model by meta-learning, so that the model can quickly adapt to new scenes and behavior categories. Training on the meta-training task can be transferred to the meta-testing task through a small amount of labeled data, so as to improve the generalization ability of the model and reduce manual data labeling. At the same time, the setting of small sample learning in meta-learning can reduce the training data of the model, reduce the calculation amount of the network, and improve the reasoning speed of the model. The present invention uses a feature extraction module to extract multi-scale video features including edges, details, contours and high-level semantics from the same video, fully realizes the utilization of first-person video data, reduces data redundancy, and measures the matching score between the first-person video sample and the support set in the query set of the meta-training task or the meta-testing task through the set matching metric module. The matching score model can explicitly and accurately measure the matching score between the query set sample and the support set, and realize accurate recognition of user behavior for the query set sample according to the matching score.

[0094] Another aspect of the present application provides a first-person behavior recognition device based on small sample learning, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor is configured to implement the first-person behavior recognition method based on small sample learning when executing the computer program.

[0095] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the first-person behavior recognition method based on small sample learning can be implemented.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0097] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A first-person action recognition method based on small sample learning, characterized in that: The following steps are involved: S1: Following the few-shot learning setting, we construct meta-training tasks and meta-testing tasks using first-person video samples; S2: meta-training a first-person action recognition model using a meta-training task, wherein the first-person action recognition model includes: a feature extraction module and a set matching metric module; The feature extraction module is used to extract multi-scale video features of first-person video samples; The set matching metric module is used to calculate the matching scores of the first-person video samples in the query set and the support set according to the multi-scale video features of the first-person video samples, and identify the user's behavior category according to the matching scores; S3: Use the trained first-person action recognition model to perform user behavior recognition on the meta-test task.

2. According to the first-person behavior recognition method based on small sample learning according to claim 1, it is characterized in that: The feature extraction module includes: ResNet50, and a feature fusion branch corresponding to each stage of ResNet50; The multi-scale video feature extraction of the first-person video sample includes: Inputting the video frames of the first-person video sample into ResNet50 in sequence, and outputting the video frame features of the corresponding scale in each stage of ResNet50; the ResNet50 includes 5 stages, each stage is composed of a different number of convolutional layers and / or residual blocks; All video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain video features of the corresponding scale; The video features output by all feature fusion branches are combined into multi-scale video features of the first-person video sample.

3. The first-person action recognition method based on small sample learning according to claim 2 is characterized in that: The feature fusion branch flattens all input video frame features in the spatial dimension respectively, inputs all flattened video frame features into the mapper for spatiotemporal information fusion to generate spatiotemporal video features; and performs average pooling on the spatiotemporal video features to obtain output features of the feature extraction branch.

4. The first-person behavior recognition method based on small sample learning according to claim 3 is characterized in that: The mapper takes all flattened video frame features as input sequences and performs single-head self-attention operations to obtain spatial video frame feature sequences; performs average pooling operations on the spatial video frame feature sequences to obtain spatial pooling video frame feature sequences; splices all feature elements in the spatial pooling video frame feature sequences to obtain initial video features; inputs the initial video features into MLP-Mixer for processing to obtain temporal video features; and sequentially performs expansion-splicing-convolution operations on the temporal video features to generate spatiotemporal video features.

5. The first-person action recognition method based on small sample learning according to claim 4 is characterized in that: The MLP-Mixer includes: a token-mixing MLP and a channel-mixing MLP; the input feature of the token-mixing MLP is the transposition of the initial video feature; the output feature of the token-mixing MLP is transposed and added to the input initial video feature as the input feature of the channel-mixing MLP; the output feature of the channel-mixing MLP is added to the input feature of the channel-mixing MLP to obtain the temporal video feature.

6. The first-person behavior recognition method based on small sample learning according to claim 4 is characterized in that: The step of sequentially performing expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features includes: The temporal video features of all samples in the support set are combined into the temporal video features of the support set The temporal video features of all samples in the query set constitute the temporal video features of the query set K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames of the sample, and C is the number of channels; The temporal video feature of the support set is expanded by one dimension in the 0th dimension and replicated M times along this dimension, where M represents the number of samples in the query set, to obtain the first intermediate video feature of the support set Add a new dimension after the first dimension to the temporal video feature of the query set to obtain the first intermediate video feature of the query set The first intermediate video feature of the support set and the first intermediate video feature of the query set are averaged in the frame number dimension to obtain the second intermediate video feature of the query set. and the second intermediate video features of the support set The second intermediate video feature of the support set and the second intermediate video feature of the query set are concatenated along the second dimension to obtain the first fusion feature. Multi-head self-attention is used to process the first fusion feature to obtain the second fusion feature; Add a dimension to the second fusion feature after the second dimension and copy it F times along this dimension to get the third fusion feature F sup ∈R M×(K+1)×F×C ; The first K video features of the third fusion feature in the second dimension The first intermediate video features of the support set samples Concatenate along the channel dimension to get the first fusion feature F of the support set s ∈R M×K×F×2C , and then get the first fusion feature of each support set sample i represents the i-th first-person video sample in the support set; The last video feature of the third fusion feature on the second dimension The first intermediate video feature of the query set The first fusion feature F of the query set is obtained by splicing along the channel dimension p ∈R M×1×F×2V , and then get the first fusion feature of each query set sample j represents the jth first-person video sample in the query set; One-dimensional convolution is used to further aggregate the first fused features of the support set samples or the first fused features of the query set samples along the channel dimension so that their channel dimension is the same as the channel dimension of the mapper input feature, thereby obtaining the spatiotemporal video features of the support set samples or the query set samples.

7. The first-person behavior recognition method based on small sample learning according to claim 1 is characterized in that: The calculation of the matching scores between the first-person video samples in the query set and the support set includes: D(x q ,S n )=max(d(x q ,S n ),d(S n ,x q )) Among them, d M (·) represents the Mahalanobis distance, x q represents a query set sample, represents the support set, Denotes the support set S n The i-th first-person video sample in y i Representation sample The user behavior category of n represents the user behavior category of all samples in the support set is n, f(x q ) represents the query sample x q Multi-scale video features, Represents the support set sample Multi-scale video features of Score(x q ,S n ) represents the query sample x q and support set S n The matching score.

8. The first-person behavior recognition method based on small sample learning according to claim 7 is characterized in that: The method of identifying the user's behavior category according to the matching score includes: selecting a support set S with the largest matching score n The user behavior category n is used as the query sample x q categories of user behavior.

9. A first-person behavior recognition device based on small sample learning, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The processor is configured to implement a first-person behavior recognition method based on small sample learning as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement a first-person behavior recognition method based on small sample learning as described in any one of claims 1 to 8.

Images