Event camera-based few-sample fine-grained action recognition method

By extracting and learning the event camera data, the problem of poor action recognition effect in high dynamic or low light environments is solved, and action recognition with high accuracy and environmental adaptability is achieved, suitable for real-world scenarios .

CN118968612APending Publication Date: 2024-11-15BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410955188.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing action recognition systems perform poorly in high dynamic or low lighting environments, and lack effective algorithm support to fail to make full use of event camera data, resulting in limited application effects in real-world scenarios.

Method used

Develop a small sample fine-grained action recognition method based on event cameras, extract features using pre-trained CLIP image encoder and VideoFlow encoder, and perform loss calculations through motion and semantic learning modules , Optimize event feature extractor to adapt to the uniqueness of event data.

Benefits of technology

It significantly improves the accuracy and environmental adaptability of action recognition, can accurately identify subtle actions under high dynamic and low light conditions, reduces data demand and processing time, and is suitable for real-time human-computer interaction systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968612A_ABST
    Figure CN118968612A_ABST
Patent Text Reader

Abstract

The invention relates to a few-sample fine-grained action recognition method based on an event camera, and belongs to the technical field of computer vision and artificial intelligence. The method comprises the following steps: step 1, aligning an RGB frame with an asynchronous event stream; 2, performing motion and semantic feature learning on the data aligned in the step 1 at the same time; and step 3, replacing the event feature extractor trained in the step 2 with an existing RGB-based few-sample algorithm to perform fine-grained action recognition. The method achieves the effects of improving the recognition accuracy, optimizing the complex environment adaptability, improving the learning efficiency and effect of few samples, and reducing the data demand and the processing time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a few-sample fine-grained action recognition method based on an event camera, and belongs to the technical field of computer vision and artificial intelligence. Background Art

[0002] Currently, most action recognition systems rely on features extracted from RGB video frames, which perform poorly in complex environments such as high dynamics or low light. These environmental conditions often lead to information loss or degradation of capture quality, limiting the effectiveness of existing methods in real-world scenarios.

[0003] Event cameras, as a new type of sensor, provide a solution. Such cameras use their microsecond temporal resolution and high dynamic range to independently and sparsely capture scene brightness changes, thereby effectively recording subtle high-speed human movements, especially in low-light and high-contrast environments. Although event cameras have potential advantages in handling these challenging scenes, existing technologies have failed to fully utilize event camera data due to the lack of effective algorithmic support.

[0004] In addition, most of the popular image encoders are designed to process RGB images, and they cannot effectively process event data because the data format and information characteristics generated by event cameras are completely different from traditional RGB images. Therefore, in order to maximize the potential value of event camera data, it is urgent to develop a specialized encoder that can adapt to the uniqueness of event data, thereby improving the accuracy and efficiency of action recognition.

[0005] To this end, this patent proposes a few-sample fine-grained action recognition method based on event camera data. A new feature extraction technology is developed that can robustly extract action features by processing event data with a specially designed algorithm. This method not only improves the accuracy of action recognition, but also enhances the adaptability and stability of the system under various environmental conditions.

[0006] The technology of this patent brings innovation to the field of fine-grained human action recognition and human-computer interaction, especially in the application scenarios of high dynamic and low light conditions and limited sample data, which significantly improves the performance and reliability of action recognition. The application of this technology will promote the development of human-computer interaction technology and provide users with a more accurate and reliable interactive experience. Summary of the invention

[0007] The purpose of the present invention is to solve the problem of application limitations of the existing few-sample fine-grained human action recognition (FGH action recognition) technology in human-computer interaction environments, and propose a few-sample fine-grained action recognition method based on event camera.

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] A method for fine-grained action recognition with few samples based on event cameras comprises the following steps:

[0010] Step 1: Align the RGB frame with the asynchronous event stream.

[0011] Convert the asynchronous event stream into T synchronous event frames by aligning the asynchronous event stream with the corresponding time periods of the RGB frames

[0012] According to RGB frame The exposure time Δs converts the asynchronous event stream ε into a synchronous event frame. It should be noted that the time span of the asynchronous event stream ε is ΔsT.

[0013] First, use event representation (event representation is a method of converting raw event data into a common data form) to convert ε into the event volume of the B channel The specific conversion method is as follows:

[0014]

[0015] Where R(s) normalizes the input timestamp s, and K b (a) is a bilinear interpolation function with variable a, is the value at row x, column y, and channel b.

[0016] Then, the event volume Evenly divided into T consecutive blocks, get the synchronization event frame

[0017] Step 2: The data aligned in step 1 is used to learn motion and semantic features simultaneously.

[0018] Use the pre-trained CLIP image encoder E clip (·) and the pre-trained VideoFlow encoder E vif (·) Extract RGB frame features and supervise the event feature extractor E evt (·), which includes semantic and motion feature learning losses and

[0019] Step 2.1: Get the temporal motion loss through the motion learning module

[0020] For fine-grained human action recognition, temporal motion should be extracted to accurately predict action categories. This method requires the event feature extractor E evt(·) with the pre-trained VideoFlow encoder E vif (·) The extracted motion features remain consistent.

[0021] This method converts the RGB frame Passed to VideoFlow encoder E vif (·),get The VideoFlow encoder output is then average pooled to collapse the spatial dimensions and obtain the motion features of the RGB frame, where Yes D vif dimensional vector.

[0022] At the same time, this method uses the event feature extractor E evt (·)Embedded event frame And use the task header H vif (·) Predict motion features from the extracted embeddings while collapsing the spatial dimensions and unifying the feature dimensions to d vif Similarly, this method expresses the predicted motion features as in

[0023] With motion features predicted from RGB frames and motion features predicted from synchronized event frames Prepare the event feature extractor for motion feature learning. This method uses the Softmax function to transform the motion features and Normalized to and

[0024]

[0025] in Can be or is the jth eigenvalue.

[0026] Then, the Kullback-Leibler (KL) divergence is used to calculate and The deviation between them can be used to obtain the temporal motion loss of this method.

[0027]

[0028] Step 2.2: Obtain temporal semantic loss through semantic learning module

[0029] Learning Synchronous Event Frames Semantics of RGB and synchronized event frames to help fine-grained human action recognition. and The extraction method is:

[0030]

[0031] in D clip is the semantic feature dimension, H clip (·) is a task header.

[0032] Direct Enforcement and The feature consistency between is suboptimal. Considering that fine-grained human action recognition uses semantic changes in the temporal dimension to infer fine-grained categories, this method is and Calculate the normalized temporal semantic feature correlation and

[0033]

[0034] in for or The value of the i-th row and j-th column is ⊙ is a scalar multiplication operation. Then, using the temporal semantic loss To enforce semantic consistency,

[0035]

[0036] Step 2.3: Through temporal motion loss and temporal semantic loss Get the overall loss function.

[0037] The network passes the overall loss Optimizations are performed, including temporal motion loss and temporal semantic loss Balanced by the hyperparameter λ, the loss Defined as:

[0038]

[0039] Step 3: Replace the event feature extractor trained in step 2 into the existing RGB-based few-shot algorithm for fine-grained action recognition.

[0040] Following the existing few-shot fine-grained human action recognition method (Shi et al.), its default image encoder is replaced with the event feature extractor trained in the above steps.

[0041] Specifically, we first use the CLIP text encoder to extract action semantics from action text, then use a feedforward network to fine-tune the extracted event features and calculate the semantic similarity between the extracted text features and the refined event features to classify fine-grained human actions. In the experimental part, we also combine the event feature extractor with other few-shot fine-grained human action recognition methods to demonstrate the effectiveness of the event feature extractor.

[0042] After completing the above steps, the accuracy of action recognition is significantly improved, the environmental adaptability is better, and the efficient feature extraction capability of event data is achieved. It also realizes high-accuracy recognition of fine-grained actions with a small number of samples and improved performance under high dynamic and low-light conditions. At the same time, it optimizes the efficiency and accuracy of the event encoder in extracting event features.

[0043] Beneficial Effects

[0044] 1. The present invention processes event data through a specially designed algorithm, which significantly improves the accuracy of small-sample fine-grained action recognition and can accurately identify subtle and rapid action changes.

[0045] 2. The present invention is particularly suitable for high dynamic and low light conditions, solving the problem that traditional action recognition systems based on RGB video frames perform poorly under these conditions. The high temporal resolution and high dynamic range of the event camera enable it to perform well in capturing scene brightness changes, greatly improving the system's environmental adaptability. Optimize adaptability to complex environments.

[0046] 3. Improve the efficiency and effect of few-sample learning: The new learning strategy enables the model to quickly learn from limited samples and accurately identify new action categories, which is particularly suitable for applications in open sets and real-world environments.

[0047] 4. Reduce data requirements and processing time: Through effective feature extraction technology, the dependence on large amounts of labeled data is reduced, while the processing speed and response time are improved, which is especially important for real-time human-computer interaction systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention.

[0049] Figure 2 FIG. 1 is a diagram of embodiment step 1.

[0050] Figure 3 Diagram of the temporal motion loss training process.

[0051] Figure 4 Diagram of the temporal semantic loss training process.

[0052] Figure 5This is a diagram of the training process of step 2.3 of the embodiment. DETAILED DESCRIPTION

[0053] In order to better illustrate the purpose and advantages of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] This embodiment takes extracting 8 frames from each video and selecting 8 synchronization event frames closest to each frame for alignment as an example.

[0055] Example

[0056] like Figure 1 As shown, a few-sample fine-grained action recognition method based on event camera includes the following steps:

[0057] Step 1: Align the RGB frames with the asynchronous event stream. Extract 8 frames from each video and select the 8 synchronous event frames closest to each frame for alignment.

[0058] Convert the asynchronous event stream into T synchronous event frames by aligning the asynchronous event stream with the corresponding time periods of the RGB frames

[0059] According to RGB frame The exposure time Δs converts the asynchronous event stream ε into a synchronous event frame. It should be noted that the time span of the asynchronous event stream ε is ΔsT.

[0060] First, use the event representation to convert ε into the event volume of the B channel The specific conversion method is as follows:

[0061]

[0062]

[0063] Where R(s) normalizes the input timestamp s, and K b (a) is a bilinear interpolation function with variable a, is the value at row x, column y, and channel b.

[0064] Then, the event volume Evenly divided into T consecutive blocks, get the synchronization event frame like Figure 2 shown.

[0065] Step 2: The data aligned in step 1 is used to learn motion and semantic features simultaneously.

[0066] We train and evaluate our proposed framework on four datasets: N-FineGym, E-FAction, DailyAction, and For event-based few-shot action recognition. The N-FineGym dataset is an event-based extension of FineGym. FineGym videos are converted into asynchronous event streams using ESIM and SuperSloMo. We follow the method of working with few shots of RGB frames and use splits of 72, 13, and 14 categories in the training, validation, and test sets, respectively. E-FAction was made by the authors of this patent, using the DAVIS346 event camera, with 128 classes and more than a thousand sets of data. We divide the coarse actions into 3, 1, and 1 categories for training, validation, and test sets, and use the fine-grained categories in the coarse categories for few-shot fine-grained human action recognition.

[0067] Since there are not enough event-based fine-grained action recognition datasets, we further compare our method on published common action recognition datasets. The DailyAction dataset is captured using the DAVIS128 event camera. We randomly use 7 action categories for training and 5 action categories for testing. There are only 12 action categories in the dataset and we do not use a validation set. The dataset was captured using a CeleX-V camera and is split into 35 training, 5 validation, and 10 testing categories. We use events and paired RGB frames from the training set when training our event feature extractor.

[0068] This example uses the pre-trained ViT-B / 16 from CLIP as E clip (·), and use the layers before FlowHead of the pre-trained Videoflow as E vif (·).H clip (·) and H vif (·) are implemented by a feed-forward network and an average pooling layer. evt (·), we used an 18-layer Resnet3D. For few-shot fine-grained human action recognition, we follow the architecture of the state-of-the-art methods and replace the default CLIP image encoder with E evt (·).

[0069] In order to train E evt (·), the present invention uses an input resolution of 224×224, T=8, λ=0.1 for 200 epochs training, and uses the SGD optimizer. The learning rate, momentum, and weight decay are 0.01, 0.9, and 0.01, respectively.

[0070] Use the pre-trained CLIP image encoder E clip (·) and the pre-trained VideoFlow encoder E vif (·) Extract RGB frame features and supervise the event feature extractor Eevt (·), which includes semantic and motion feature learning losses and

[0071] Step 2.1: Get the temporal motion loss through the motion learning module The training process is as follows Figure 3 shown.

[0072] For fine-grained human action recognition, temporal motion is extracted to accurately predict the action category. The present invention requires the event feature extractor E evt (·) with the pre-trained VideoFlow encoder E vif (·) The extracted motion features remain consistent.

[0073] The present invention converts the RGB frame Passed to VideoFlow encoder E vif (·),get The VideoFlow encoder output is then average pooled to collapse the spatial dimensions and obtain the motion features of the RGB frame, where Yes D vif dimensional vector.

[0074] At the same time, the present invention uses the event feature extractor E evt (·)Embedded synchronization event frame And use the task header H vif (·) Predict motion features from the extracted embeddings, while collapsing the spatial dimensions and unifying the feature dimensions to D vif Similarly, the present invention expresses the predicted motion features as in

[0075] With motion features predicted from RGB frames and motion features predicted from synchronized event frames The present invention uses the Softmax function to learn motion features. and Normalized to and

[0076]

[0077] in Can be or is the jth eigenvalue.

[0078] Then, the Kullback-Leibler (KL) divergence is used to calculate and The deviation between them, and then the temporal motion loss of the present invention is obtained

[0079]

[0080] Step 2.2: Obtain temporal semantic loss through semantic learning module The training process is as follows Figure 4 shown.

[0081] Learning Synchronous Event Frames Semantics of RGB and synchronized event frames to help fine-grained human action recognition. and The extraction method is:

[0082]

[0083] in D clip is the semantic feature dimension, H clip (·) is a task header.

[0084] Direct Enforcement and Considering that fine-grained human action recognition uses semantic changes in the time dimension to infer fine-grained categories, the present invention is respectively and Calculate the normalized temporal semantic feature correlation and

[0085]

[0086] in for or The value of the i-th row and j-th column is ⊙ is a scalar multiplication operation. Then, using the temporal semantic loss To enforce semantic consistency,

[0087]

[0088] Step 2.3: Through temporal motion loss and temporal semantic loss The overall loss function is obtained. The training process is as follows Figure 5 shown.

[0089] The network passes the overall loss Optimizations are performed, including temporal motion loss and temporal semantic loss Balanced by the hyperparameter λ, the loss Defined as:

[0090]

[0091] Step 3: Replace the event feature extractor trained in step 2 into the existing RGB-based few-shot algorithm for fine-grained action recognition.

[0092] The present invention is compared with the state-of-the-art few-shot action recognition method MASTAF and Shi et al.

[0093] In the comparative experiment, the present invention uses the image encoder from CLIP and logistic regression to perform few-sample

[0094]

[0095] Action Recognition. We call them CLIP-L. Replace the default CLIP image encoder of Shi et al. with a pre-trained ResNet3D (using the N-ImageNet and Kinetic-400 datasets). We call them ResNet3D-N and ResNet3D-K.

[0096] The figure above compares the performance of the most advanced and baseline methods on 5-way 5-shot human action recognition. Our method performs best on all four datasets.

[0097] i) Compared with Shi et al., we share the few-shot recognition method but use a different feature extractor. Our event feature extractor is 2.7%, 1.5%, 4.1%, and 5.6% more accurate than Shi et al.'s default feature extractor, respectively. Our performance shows that there is a significant domain difference between the event domain and the RGB domain, indicating that the event feature extractor is crucial for few-shot human action recognition. Our fine-grained few-shot action recognition method still has a significant improvement over general action recognition.

[0098] ii) For ResNet3D-N and ResNet3D-K, the semantic categories of the asynchronous event stream are used to train the event feature extractor. When using the same few-shot method and feature extractor architecture, ResNet3D-N and ResNet3D-K have lower accuracy than ours because our method uses pre-trained CLIP and Videoflow for supervision, which is more robust and effective than using class labels for supervision to learn feature extractors for event streams. iii) Our method performs better than CLIP-L and Shi et al. Our event feature extractor is learned from the image encoder of CLIP, while CLIP-L and Shi et al. directly use the image encoder of CLIP. Using RGB frames as an intermediate step to learn the feature extractor of event streams can help transfer knowledge for fine-grained human action recognition.

[0099]

[0100] In the above table, the present invention is compared with the default feature extractor in the state-of-the-art few-shot fine-grained human action recognition methods. Using our event feature extractor, the few-shot recognition performance is consistently improved, indicating that our extracted features can be widely used in fine-grained few-shot human action recognition methods.

Claims

1. A method for fine-grained action recognition with few samples based on event cameras, characterized in that: The following steps are involved: Step 1, align the RGB frame with the asynchronous event stream; Convert the asynchronous event stream into T synchronous event frames by aligning the asynchronous event stream with the corresponding time periods of the RGB frames According to RGB frame The exposure time Δs converts the asynchronous event stream ε into a synchronous event frame; it should be noted that the time span of the asynchronous event stream ε is ΔsT; First, use event representation to convert ε into the event volume v of the B channel. The specific conversion method is as follows: Where R(s) normalizes the input timestamp s, and K b (a) is a bilinear interpolation function with variable a, is the value of the xth row, yth column and bth channel; Then, the event volume V is evenly divided into T consecutive blocks to obtain the synchronized event frame Step 2: Perform motion and semantic feature learning on the data aligned in step 1 simultaneously; Use the pre-trained CLIP image encoder E clip (·) and the pre-trained VideoFlow encoder E vif (·) Extract RGB frame features and supervise the event feature extractor E evt (·), which includes semantic and motion feature learning losses and Step 3: Replace the event feature extractor trained in step 2 with the existing RGB-based few-sample algorithm for fine-grained action recognition; Following the existing few-shot fine-grained human action recognition method (Shi et al.), replace its default image encoder with the event feature extractor trained in the above steps; Specifically, we first use the CLIP text encoder to extract action semantics from action text, then use a feedforward network to fine-tune the extracted event features, and calculate the semantic similarity between the extracted text features and the refined event features to classify fine-grained human actions; in the experimental part, we also combine the event feature extractor with other few-sample fine-grained human action recognition methods to demonstrate the effectiveness of the event feature extractor.

2. The method for fine-grained action recognition with few samples based on event camera according to claim 1, characterized in that: In the step 2, Step 2.1: Get the temporal motion loss through the motion learning module For fine-grained human action recognition, temporal motion should be extracted to accurately predict the action category; this method requires the event feature extractor E evt (·) with the pre-trained VideoFlow encoder E vif (·) The extracted motion features remain consistent; This method converts the RGB frame Passed to VideoFlow encoder E vif (·),get The VideoFlow encoder output is then average pooled to collapse the spatial dimensions and obtain the motion features of the RGB frame, where Yes D vif dimensional vector; At the same time, this method uses the event feature extractor E evt (·)Embedded synchronization event frame And use the task header H vif (·) Predict motion features from the extracted embeddings, while collapsing the spatial dimensions and unifying the feature dimensions to D vif ; Similarly, this method expresses the predicted motion features as in With motion features predicted from RGB frames and motion features predicted from synchronized event frames Prepare motion feature learning for event feature extractor; This method uses the Softmax function to transform motion features and Normalized to and in Can be or is the jth eigenvalue; Then, the Kullback-Leibler (KL) divergence is used to calculate and The deviation between them can be used to obtain the temporal motion loss of this method. Step 2.2: Obtain temporal semantic loss through semantic learning module Learning Synchronous Event Frames semantics to help fine-grained human action recognition; semantic features of RGB and synchronized event frames and The extraction method is: in D clip is the semantic feature dimension, H clip (·) is a task header; Direct Enforcement and The feature consistency between is suboptimal; considering that fine-grained human action recognition uses semantic changes in the temporal dimension to infer fine-grained categories, this method is respectively and Calculate the normalized temporal semantic feature correlation and in for or (i, j)∈{1,…,T}×{1,…,T}, The value of the i-th row and j-th column is ⊙ is a scalar multiplication operation; then, using the temporal semantic loss To enforce semantic consistency, Step 2.3: Using temporal motion loss and temporal semantic loss Get the overall loss function; The network passes the overall loss Optimizations are performed, including temporal motion loss and temporal semantic loss Balanced by the hyperparameter λ, the loss Defined as: