A first-person behavior recognition method and device based on small sample learning

Abstract

This invention relates to a method and apparatus for first-person behavior recognition based on few-shot learning. The method includes following a few-shot learning setup, constructing a meta-training task and a meta-testing task using first-person video samples; meta-training a first-person behavior recognition model using the meta-training task, wherein the first-person behavior recognition model includes a feature extraction module and a set matching metric module; the feature extraction module is used to extract multi-scale video features from the first-person video samples; the set matching metric module is used to calculate the matching score between the first-person video samples in the query set and the support set based on the multi-scale video features of the first-person video samples, and to identify the user's behavior category based on the matching score; and the trained first-person behavior recognition model is used to perform user behavior recognition on the meta-testing task. This invention can improve the model's generalization ability and recognition accuracy, while reducing the computational load of model training.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and relates to a method and apparatus for first-person behavior recognition based on few-shot learning. Background Technology

[0002] People frequently use wearable video capture devices in their daily lives, such as smart helmets and head-mounted cameras, to record their various activities, resulting in a large amount of video footage shot from a first-person perspective. First-person video records the interaction between human hands and interacting objects. As one of the fundamental tasks of first-person video analysis, first-person behavior recognition aims to identify behaviors generated by the interaction between human hands and interactive objects in first-person videos.

[0003] For first-person action recognition tasks, current deep learning methods can be broadly categorized into three types. The first type is based on class activation maps. These methods use image classification networks pre-trained on large-scale image datasets to extract class activation maps from video frames. They then extract spatiotemporal fusion information by applying spatial attention, temporal modeling, or long and short-term attention modules. However, these methods heavily rely on prior knowledge from large datasets, and their inference speed is slow, resulting in poor generalization performance. The second type is based on joint supervision using action labels and human eye gaze data. This type uses eye gaze data as auxiliary supervision to focus on key regions. While it efficiently focuses on key spatial features, the additional eye gaze data adds redundancy. The third type is based on key object detection. Considering the crucial role of human hands and interacting objects in first-person action recognition, this type of method uses object detection models to detect hands and interacting objects in first-person videos and extract features from the detected regions. However, it requires object detection algorithms, increasing the computational load of the network, and also necessitates additional bounding box annotations. Summary of the Invention

[0004] To address the problems existing in the background technology, improve the inference speed of the model, reduce manual data annotation, reduce the computational load of the network, improve the generalization performance of the network, and reduce data redundancy, one aspect of this application proposes a first-person behavior recognition method based on few-shot learning, including:

[0005] S1: Following the few-shot learning setup, construct meta-training and meta-testing tasks using first-person video samples;

[0006] S2: Meta-training is performed on the first-person behavior recognition model using a meta-training task, wherein the first-person behavior recognition model includes: a feature extraction module and a set matching metric module;

[0007] The feature extraction module is used to extract multi-scale video features from first-person video samples;

[0008] The set matching metric module is used to calculate the matching score between the first-person video samples in the query set and the support set based on the multi-scale video features of the first-person video samples, and to identify the user's behavior category based on the matching score.

[0009] S3: Use the trained first-person behavior recognition model to perform user behavior recognition on the meta-test task.

[0010] Preferably, the feature extraction module includes: ResNet50, and feature fusion branches corresponding to each stage of ResNet50;

[0011] The multi-scale video features extracted from the first-person video samples include:

[0012] The video frames of the first-person video sample are sequentially input into ResNet50, and the video frame features of the corresponding scale are output at each stage of ResNet50; the ResNet50 includes 5 stages, each stage consisting of a different number of convolutional layers and / or residual blocks;

[0013] The video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain the video features at the corresponding scale.

[0014] The video features output from all feature fusion branches are combined to form multi-scale video features for first-person video samples.

[0015] Preferably, the feature fusion branch flattens all the input video frame features in the spatial dimension, inputs all the flattened video frame features into a mapper for spatiotemporal information fusion to generate spatiotemporal video features, and performs average pooling on the spatiotemporal video features to obtain the output features of the feature extraction branch.

[0016] Preferably, the mapper performs a single-head self-attention operation on all flattened video frame features as input sequences to obtain a spatial video frame feature sequence; performs average pooling on the spatial video frame feature sequence to obtain a spatial pooled video frame feature sequence; concatenates all feature elements in the spatial pooled video frame feature sequence to obtain initial video features; inputs the initial video features into an MLP-Mixer for processing to obtain temporal video features; and sequentially performs expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features.

[0017] Preferably, the MLP-Mixer includes: a token-mixing MLP and a channel-mixing MLP; the input feature of the token-mixing MLP is the transpose of the initial video feature; the output feature of the token-mixing MLP is transposed and added to the input initial video feature as the input feature of the channel-mixing MLP; the output feature of the channel-mixing MLP and the input feature of the channel-mixing MLP are added together to obtain the temporal video feature.

[0018] Preferably, the step of generating spatiotemporal video features by sequentially performing expansion-stitching-convolution operations on the temporal video features includes:

[0019] The time video features of all samples in the support set are used to form the time video features of the support set. The time video features of the query set consist of the time video features of all samples in the query set. K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames for the samples, and C is the number of channels;

[0020] Expand the temporal video features of the support set by one dimension at dimension 0, and replicate this dimension M times, where M represents the number of samples in the query set, to obtain the first intermediate video features of the support set.

[0021] Add a new dimension after the first dimension to the temporal video features of the query set to obtain the first intermediate video features of the query set.

[0022] The second intermediate video feature of the query set is obtained by averaging the first intermediate video features of the support set and the first intermediate video features of the query set along the frame number dimension. and the second intermediate video features of the support set

[0023] The first fused feature is obtained by concatenating the second intermediate video features of the support set and the second intermediate video features of the query set along the second dimension.

[0024] The second fusion feature is obtained by performing attention processing on the first fusion feature using multi-head self-attention;

[0025] Add a dimension after the second dimension to the second fusion feature, and copy it F times along this dimension to obtain the third fusion feature F. sup ∈R M×(K+1)×F×C ;

[0026] The third fusion feature is the top K video features in the second dimension. First intermediate video features of the support set samples The first fusion feature F of the support set is obtained by concatenating along the channel dimension. s ∈R M×K×F×2C This leads to the first fusion feature of each support set sample. i represents the i-th first-person video sample in the support set;

[0027] The third fusion feature is the last video feature in the second dimension. Features of the first intermediate video in the query set The first fusion feature F of the query set is obtained by concatenating along the channel dimension. p ∈R M×1×F×2C This leads to the first fusion feature of each query set sample. j represents the j-th first-person video sample in the query set;

[0028] One-dimensional convolution is used to further aggregate the first fusion features of the support set samples or the first fusion features of the query set samples along the channel dimension, so that their channel dimension is the same as the channel dimension of the input features of the mapper, thus obtaining the spatiotemporal video features of the support set samples or the query set samples.

[0029] Preferably, the calculation of the matching score between the first-person video samples in the query set and the support set includes:

[0030]

[0031] D(x q ,S n )=max(d(x q ,S n ),d(S n ,x q ))

[0032] Where, d M (·) represents Mahalanobis distance, x q This represents a sample of the query set. Indicates the support set, x i n Indicates support for set S n The i-th first-person video sample in the data, y i Indicates sample User behavior categories, where n represents the total number of user behavior categories for all samples in the support set, f(x) q ) represents the query sample x q Multi-scale video features. Indicates support set samples Multi-scale video features; Score(x) q ,Sn ) represents the query sample x q and support set S n The matching score.

[0033] Preferably, identifying the user's behavior category based on the matching score includes: selecting the support set S with the largest matching score. n User behavior category n as query sample x q User behavior categories.

[0034] Another aspect of this application provides a first-person behavior recognition device based on few-shot learning, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, the processor being configured to implement the first-person behavior recognition method based on few-shot learning when executing the computer program.

[0035] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the aforementioned first-person behavior recognition method based on few-shot learning.

[0036] The present invention has at least the following beneficial effects

[0037] This invention constructs meta-training and meta-testing tasks, and trains the first-person behavior recognition model through meta-learning. This enables the model to quickly adapt to new scenes and behavior categories. Training on the meta-training task allows for transfer to the meta-testing task with a small amount of labeled data, improving the model's generalization ability and reducing manual data labeling. Furthermore, the few-shot learning setting in meta-learning reduces the training data required, lowers the computational load of the network, and improves the model's inference speed. This invention utilizes a feature extraction module to extract multi-scale video features, including edges, details, contours, and high-level semantics, from the same video, fully utilizing first-person video data and reducing data redundancy. A set matching metric module measures the matching score between first-person video samples in the query set and the support set in the meta-training or meta-testing task. This matching score model explicitly and accurately measures the matching score between the query set samples and the support set, and uses the matching score to achieve accurate user behavior recognition for the query set samples. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the process of the present invention;

[0039] Figure 2 This is a schematic diagram of the network structure of the first-person behavior recognition model of the present invention;

[0040] Figure 3This is a schematic diagram of the mapper of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1 , Figure 2 and Figure 3 One aspect of the present invention provides a first-person behavior recognition method based on few-shot learning, comprising:

[0043] S1: Following the few-shot learning setup, construct meta-training and meta-testing tasks using first-person video samples;

[0044] S2: Meta-training is performed on the first-person behavior recognition model using a meta-training task, wherein the first-person behavior recognition model includes: a feature extraction module and a set matching metric module;

[0045] The feature extraction module is used to extract multi-scale video features from first-person video samples;

[0046] The set matching metric module is used to calculate the matching score between the first-person video samples in the query set and the support set based on the multi-scale video features of the first-person video samples, and to identify the user's behavior category based on the matching score.

[0047] S3: Use the trained first-person behavior recognition model to perform user behavior recognition on the meta-test task.

[0048] In this embodiment, firstly, first-person video samples are obtained, and a first-person video sample set is constructed. The first-person video sample set includes: first-person video samples and user behavior category tags corresponding to the first-person video samples.

[0049] Following the few-shot learning setup, the first-person video sample set, with a total user behavior category set of C, is divided into a meta-training set according to user behavior category. Heyuan Test Set Meta-training set The set of user behavior categories is Metatest set The set of user behavior categories is and

[0050] In this embodiment, the first-person video sample set includes first-person video samples of various user behavior categories. The set of user behavior categories for all first-person video samples in the first-person video sample set is denoted as C, and the meta-training set is... The set of user behavior categories is Metatest set The set of user behavior categories is and

[0051] In this embodiment, constructing the meta-training task and the meta-testing task includes the following process: From the user behavior category set of the meta-training set... We randomly select N1 user behavior categories, and for each user behavior category, we randomly select K+M first-person video samples. We use the K first-person video samples from each user behavior category as the support set and the M first-person video samples as the query set to construct the meta-training task. Similarly, we use the user behavior category set from the meta-test set... N2 user behavior categories are randomly selected, and K+M first-person video samples are randomly selected for each user behavior category. The K first-person video samples of each user behavior category are used as the support set, and the M first-person video samples are used as the query set to construct a meta-test task. The value of K should be much smaller than M, and can be determined based on the experience of those skilled in the art.

[0052] In this embodiment, each meta-training task or meta-testing task includes a support set and a query set. The support set of the meta-training task has a small number of first-person video samples and a label corresponding to each sample. The first-person video samples in the query set of the meta-training task have real labels. The first-person video samples in the support set of the meta-testing task have real labels. The first-person video samples in the query set of the meta-testing task do not have real labels. User behavior recognition is performed on the first-person video samples in the query set that do not have labels using the labeled support set in the meta-testing task.

[0053] First-person video, also known as egocentric video or first-person perspective video, refers to video content recorded from a personal perspective using camera equipment (such as head-mounted cameras, smart glasses, etc.). The characteristic of this video format is that changes in the shooting angle and perspective are entirely driven by the head and body movements of the person wearing the device (i.e., the cameraman or actor), so the captured content can be approximated as a scene captured by the human eye.

[0054] In this embodiment, by constructing meta-training and meta-testing tasks, the model can quickly adapt to new scenarios and behavior categories through meta-learning. Training on the meta-training task allows for transfer to the meta-testing task with a small amount of labeled data, thereby improving the model's generalization ability and reducing the need for manual data labeling. At the same time, the few-shot learning setting in meta-learning can reduce the model's training data, reduce the computational load of the network, and improve the model's inference speed.

[0055] In this embodiment, the feature extraction module includes: ResNet50, and feature fusion branches corresponding to each stage of ResNet50;

[0056] The multi-scale video features extracted from the first-person video samples include:

[0057] The video frames of the first-person video sample are sequentially input into ResNet50, and the video frame features of the corresponding scale are output at each stage of ResNet50; the ResNet50 includes 5 stages, each stage consisting of a different number of convolutional layers and / or residual blocks;

[0058] The video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain the video features at the corresponding scale.

[0059] The video features output from all feature fusion branches are combined to form multi-scale video features for first-person video samples.

[0060] In this embodiment, each first-person video sample consists of multiple video frames. To reduce the computational load and training time of the network, this invention provides a method for preprocessing the video by extracting frames. X video frames are extracted from each video and formed into a video frame sequence in chronological order. To balance the computational load and inference accuracy, this embodiment uses a value of 8 for X based on extensive experiments, which yields better results. Of course, those skilled in the art can conduct a large number of experiments based on factors such as the size of the video and the number of frames to obtain a better value. The frame extraction methods include existing methods such as manual frame extraction, speed time elapsed, exported page frame extraction, time plugin or expression frame extraction, beat-based frame extraction, and automatic segmentation and frame extraction.

[0061] In this embodiment, ResNet50 includes 5 stages, each consisting of a different number of convolutional layers and / or residual blocks. The following describes this existing network, ResNet50.

[0062] The first stage of ResNet50 is designated Stage0, which consists of two layers: the first layer includes the CONV layer, the BN layer, and the RELU layer; the second layer includes the maxpool layer.

[0063] The second stage of ResNet50, denoted as Stage1, consists of three residual blocks. For details of the specific composition, please refer to existing literature. This application will not elaborate further.

[0064] The third stage of ResNet50, denoted as Stage2, consists of four residual blocks. For details of the specific composition, please refer to existing literature. This application will not elaborate further.

[0065] The fourth stage of ResNet50 is denoted as Stage3, which consists of 6 residual blocks. The specific composition can be found in existing literature and will not be described in detail in this application.

[0066] The fifth stage of ResNet50, denoted as Stage4, consists of three residual blocks. For details of the specific composition, please refer to existing literature. This application will not elaborate further.

[0067] In this embodiment, the feature fusion branch flattens all the input video frame features in the spatial dimension, inputs all the flattened video frame features into the mapper for spatiotemporal information fusion to generate spatiotemporal video features, and performs average pooling on the spatiotemporal video features to obtain the output features of the feature extraction branch.

[0068] For example, for a first-person video sample T1 = {t1, t2, ..., t8} with 8 video frames, the video frame features output in the first stage are represented as: X1 = {x1, x2, ..., x8}. For each xi, i ∈ {1, 2, ..., 8}, it is flattened in the spatial dimension to obtain the flattened video frame feature yi. All flattened video frame features Y1 = {y1, y2, ..., y8} are input into the mapper to fuse spatiotemporal information and generate spatiotemporal video features. The spatiotemporal video features are then subjected to average pooling to obtain the output feature Z1 of the feature extraction branch. The video features output by all feature fusion branches constitute the multi-scale video features {Z1, Z2, Z3, Z4, Z5} of the first-person video sample, where Zj, j ∈ {1, 2, ..., 5} represents the output feature of the j-th feature fusion branch.

[0069] In this embodiment, the mapper takes all flattened video frame features as input sequences and performs a single-head self-attention operation to obtain a spatial video frame feature sequence; it then performs an average pooling operation on the spatial video frame feature sequence to obtain a spatial pooled video frame feature sequence; it concatenates all feature elements in the spatial pooled video frame feature sequence to obtain initial video features; it inputs the initial video features into an MLP-Mixer for processing to obtain temporal video features; and it sequentially performs expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features.

[0070] In this embodiment, the MLP-Mixer includes a token-mixing MLP and a channel-mixing MLP. The input feature of the token-mixing MLP is the transpose of the initial video feature. The output feature of the token-mixing MLP is transposed and added to the input initial video feature to obtain the input feature of the channel-mixing MLP. The output feature of the channel-mixing MLP and the input feature of the channel-mixing MLP are added to obtain the temporal video feature.

[0071] In this embodiment, the process of generating spatiotemporal video features by sequentially expanding, stitching, and convolving the temporal video features includes:

[0072] The time video features of all samples in the support set are used to form the time video features of the support set. The time video features of the query set consist of the time video features of all samples in the query set. K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames for the samples, and C is the number of channels;

[0073] The temporal video features of the support set are defined as follows: Where K is the number of support set samples, F is the number of sampling frames for the support set samples, and C is the number of channels;

[0074] Define the temporal video features of the query set as follows: Where M is the number of query set samples; F is the number of sampling frames for the query set samples; and C is the number of channels.

[0075] Expand the temporal video features of the support set by one dimension at dimension 0, and replicate this dimension M times, where M represents the number of samples in the query set, to obtain the first intermediate video features of the support set.

[0076] Add a new dimension after the first dimension to the temporal video features of the query set to obtain the first intermediate video features of the query set.

[0077] First of all, Add a new dimension to the 0th dimension, making it become Then, based on the number M of the query set samples, copy along this newly added dimension. Multiple times, the first intermediate video features of the support set were obtained.

[0078] Secondly Adding a new dimension after the first dimension yields the first intermediate video feature of the query set.

[0079] The second intermediate video feature of the query set is obtained by averaging the first intermediate video features of the support set and the first intermediate video features of the query set along the frame number dimension. and the second intermediate video features of the support set

[0080] The first fused feature is obtained by concatenating the second intermediate video features of the support set and the second intermediate video features of the query set along the second dimension.

[0081] Multi-head self-attention is used to process the first fusion feature to obtain the second fusion feature. The features of the support set and the query set are interacted and enhanced through the attention mechanism, thereby capturing better matching features.

[0082] Add a dimension after the second dimension to the second fusion feature, and copy it F times along this dimension to obtain the third fusion feature F. sup ∈R M×(K+1)×F×C ;

[0083] To enhance the model's discriminative ability by combining the learned global interaction information with the original local features:

[0084] The third fusion feature is the top K video features in the second dimension. First intermediate video features of the support set samples The first fusion feature F of the support set is obtained by concatenating along the channel dimension. s ∈R M×K×F×2C This leads to the first fusion feature of each support set sample. i represents the i-th first-person video sample in the support set;

[0085] The third fusion feature is the last video feature in the second dimension. Features of the first intermediate video in the query set The first fusion feature F of the query set is obtained by concatenating along the channel dimension. p ∈R M×1×F×2C This leads to the first fusion feature of each query set sample. j represents the h-th first-person video sample in the query set;

[0086] One-dimensional convolution is used to further aggregate the first fusion features of the support set samples or the first fusion features of the query set samples along the channel dimension, so that their channel dimension is the same as the channel dimension of the input features of the mapper, thus obtaining the spatiotemporal video features of the support set samples or the query set samples.

[0087] In this embodiment, by combining self-attention mechanisms and convolutional operations, the process can simultaneously capture spatial features (such as the shape and texture of objects) and temporal features (such as the motion trajectory and action changes of objects) in the video. Self-attention mechanisms excel at handling long-range dependencies in sequential data, helping to capture features in the temporal dimension; while convolutional operations have a powerful ability to extract local spatial features. Average pooling reduces feature dimensionality, lowers computational complexity, and removes redundant information, making subsequent processing more efficient. When needed, expansion operations (such as increasing feature dimensionality) can introduce more contextual information or diversity into the features, which is beneficial for subsequent operations. The MLP-Mixer network enhances the expressive power of features by alternately applying nonlinear transformations in the channel and spatial dimensions using a multilayer perceptron (MLP). This nonlinear transformation can capture complex relationships between features, making the generated video features richer and more discriminative. Through the sequence of expansion-concatenation-convolution operations, this process further mines the spatiotemporal information in the video. Expansion operations can enrich information by introducing new feature dimensions or replicating existing features; concatenation operations merge features from different sources or at different levels to form a more comprehensive feature representation; convolution operations effectively capture spatiotemporal patterns in videos by leveraging local connectivity and weight sharing. The spatiotemporal video features generated by this process provide strong support for subsequent video processing tasks. These features not only have stronger expressive power but also better adapt to the needs of different tasks, thereby improving the overall system performance and giving the generated video features stronger generalization ability. Even when faced with unseen video data, the model can extract useful feature information and make accurate predictions and judgments. This is of great significance for robustness and stability in practical applications.

[0088] In this embodiment, calculating the matching score between first-person video samples in the query set and first-person video samples in the support set includes:

[0089]

[0090] D(x q ,S n )=max(d(x q ,S n ),d(S n ,x q ))

[0091] Where, d M (·) represents Mahalanobis distance, x q This represents a sample of the query set. Indicates support set, Indicates support for set S n The i-th first-person video sample in the data, yi Indicates sample User behavior categories, where n represents the total number of user behavior categories for all samples in the support set, f(x) q ) represents the query sample x q Multi-scale video features. Indicates support set samples Multi-scale video features; Score(x) q ,S n ) represents the query sample x q and support set S n The matching score is calculated using Mahalanobis distance to determine the maximum distance between the nearest points in one set and another. This method effectively captures the maximum differences between sets and takes into account the covariance structure of the data, making it more robust to noise and outliers.

[0092] In this embodiment, identifying the user's behavior category based on the matching score includes: selecting the support set S with the largest matching score. n User behavior category n as query sample x q User behavior categories.

[0093] This invention constructs meta-training and meta-testing tasks, and trains the first-person behavior recognition model through meta-learning. This enables the model to quickly adapt to new scenes and behavior categories. Training on the meta-training task allows for transfer to the meta-testing task with a small amount of labeled data, improving the model's generalization ability and reducing manual data labeling. Furthermore, the few-shot learning setting in meta-learning reduces the training data required, lowers the computational load of the network, and improves the model's inference speed. This invention utilizes a feature extraction module to extract multi-scale video features, including edges, details, contours, and high-level semantics, from the same video, fully utilizing first-person video data and reducing data redundancy. A set matching metric module measures the matching score between first-person video samples in the query set and the support set in the meta-training or meta-testing task. This matching score model explicitly and accurately measures the matching score between the query set samples and the support set, and uses the matching score to achieve accurate user behavior recognition for the query set samples.

[0094] Another aspect of this application provides a first-person behavior recognition device based on few-shot learning, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, the processor being configured to implement the first-person behavior recognition method based on few-shot learning when executing the computer program.

[0095] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the aforementioned first-person behavior recognition method based on few-shot learning.

[0096] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0097] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A first-person behavior recognition method based on few-shot learning, characterized in that, Includes the following steps: S1: Following the few-shot learning setup, construct meta-training and meta-testing tasks using first-person video samples; S2: Meta-training is performed on the first-person behavior recognition model using a meta-training task, wherein the first-person behavior recognition model includes: a feature extraction module and a set matching metric module; The feature extraction module is used to extract multi-scale video features from first-person video samples; The feature extraction module includes: ResNet50, and feature fusion branches corresponding to each stage of ResNet50; The multi-scale video features extracted from the first-person video samples include: The video frames of the first-person video sample are sequentially input into ResNet50, and the video frame features of the corresponding scale are output at each stage of ResNet50; the ResNet50 includes 5 stages, each stage consisting of a different number of convolutional layers and / or residual blocks; The video frame features output by ResNet50 at each stage are input into the feature fusion branch corresponding to that stage to obtain the video features at the corresponding scale. The video features output from all feature fusion branches are combined to form multi-scale video features of the first-person video sample. The feature fusion branch flattens all the input video frame features in the spatial dimension, inputs all the flattened video frame features into the mapper for spatiotemporal information fusion to generate spatiotemporal video features, and performs average pooling on the spatiotemporal video features to obtain the output features of the feature extraction branch. The mapper takes all flattened video frame features as input sequences and performs a single-head self-attention operation to obtain a spatial video frame feature sequence; it then performs an average pooling operation on the spatial video frame feature sequence to obtain a spatial pooled video frame feature sequence; it concatenates all feature elements in the spatial pooled video frame feature sequence to obtain initial video features; it inputs the initial video features into an MLP-Mixer for processing to obtain temporal video features; and it sequentially performs expansion-concatenation-convolution operations on the temporal video features to generate spatiotemporal video features. The set matching metric module is used to calculate the matching score between the first-person video samples in the query set and the support set based on the multi-scale video features of the first-person video samples, and to identify the user's behavior category based on the matching score. The matching scores between the first-person video samples and the support set in the computation query set include: in, Represents Mahalanobis distance, This represents a sample of the query set. Indicates support set, Indicates support set The i-th first-person video sample in the data. Indicates sample The user behavior categories, where n represents the total number of user behavior categories for all samples in the support set. Indicates query sample Multi-scale video features. Indicates support set samples Multi-scale video features; Indicates query sample and support set The matching score; S3: Use the trained first-person behavior recognition model to perform user behavior recognition on the meta-test task.

2. The first-person behavior recognition method based on few-shot learning according to claim 1, characterized in that, The MLP-Mixer includes a token-mixing MLP and a channel-mixing MLP. The input features of the token-mixing MLP are the transpose of the initial video features. The output features of the token-mixing MLP are transposed and added to the initial input video features to obtain the input features of the channel-mixing MLP. The output features of the channel-mixing MLP and the input features of the channel-mixing MLP are added to obtain the temporal video features.

3. The first-person behavior recognition method based on few-shot learning according to claim 1, characterized in that, The step of generating spatiotemporal video features by sequentially expanding, concatenating, and convolving temporal video features includes: The time video features of all samples in the support set are used to form the time video features of the support set. The time video features of all samples in the query set constitute the time video features of the query set. K is the number of support set samples, M is the number of query set samples, F is the number of sampling frames for the samples, and C is the number of channels; Expand the temporal video features of the support set by one dimension at dimension 0, and replicate this dimension M times, where M represents the number of samples in the query set, to obtain the first intermediate video features of the support set. ; Add a new dimension after the first dimension to the temporal video features of the query set to obtain the first intermediate video features of the query set. ; The second intermediate video feature of the query set is obtained by averaging the first intermediate video features of the support set and the first intermediate video features of the query set along the frame number dimension. and the second intermediate video features of the support set ; The first fused feature is obtained by concatenating the second intermediate video features of the support set and the second intermediate video features of the query set along the second dimension. ; The second fusion feature is obtained by performing attention processing on the first fusion feature using multi-head self-attention; Add a dimension after the second dimension to the second fusion feature, and replicate it F times along this dimension to obtain the third fusion feature. ; The third fusion feature is the top K video features in the second dimension. First intermediate video features of the support set samples The first fusion feature of the support set is obtained by splicing along the channel dimension. This leads to the first fusion feature of each support set sample. , The first support set A first-person video sample; The third fusion feature is the last video feature in the second dimension. Features of the first intermediate video in the query set The first fusion feature of the query set is obtained by concatenating along the channel dimension. This leads to the first fusion feature of each query set sample. , Represents the query set's first... A first-person video sample; One-dimensional convolution is used to further aggregate the first fusion features of the support set samples or the first fusion features of the query set samples along the channel dimension, so that their channel dimension is the same as the channel dimension of the input features of the mapper, thus obtaining the spatiotemporal video features of the support set samples or the query set samples.

4. The first-person behavior recognition method based on few-shot learning according to claim 1, characterized in that, The step of identifying user behavior categories based on matching scores includes: selecting the support set with the highest matching scores. User behavior categories As a query sample User behavior categories.

5. A first-person behavior recognition device based on few-shot learning, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor is configured to implement, when executing the computer program, a first-person behavior recognition method based on few-shot learning as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it can implement the first-person behavior recognition method based on few-shot learning as described in any one of claims 1-4.

Images