A human action detection method based on dual-mode camera driving

Abstract

The application discloses a human action detection method based on a dual-mode camera driving, and belongs to the field of computer vision, and specifically comprises the following steps: firstly, an event camera and a common camera are used respectively to collect an event sequence and a common video corresponding to a same time window; for a single time window, the event sequence is converted and stacked into a synchronous picture frame corresponding to the common camera; for the synchronous pictures corresponding to the video mode and the event mode in the same time window, respective primary features are extracted through a deep neural network; then, spatial alignment and enhancement are carried out through an adaptive feature learning module; an instance detector is used to identify and locate an instance of interest in the picture frame captured from the common camera, so that a detection frame is generated to determine the position and range of each instance in the picture; finally, each instance is interactively modeled through a multi-stage dense serial structure, and the generation of memory features is enhanced by fusing the context information of different time windows, so that action features used for classification are finally obtained. The application improves the reliability and accuracy of existing human action recognition and positioning in various complex scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically a human motion detection method based on a dual-modal camera. Background Technology

[0002] Human behavior recognition is a research hotspot in the field of computer vision. It aims to automatically identify and understand human actions and behaviors from given images, videos, or sensor data.

[0003] Event cameras are a new type of image acquisition device that works in an event-driven manner. Compared with traditional camera technology, human behavior recognition technology based on event cameras has significant advantages in dynamic range, latency, and frame rate, and can provide more accurate and detailed human behavior analysis and recognition results. However, it usually cannot provide the same appearance information as traditional image cameras. Therefore, fusing event data with traditional image data can provide a more comprehensive understanding of the scene and behavior.

[0004] In the existing technology, literature pursuing multimodal human action recognition includes: [1] Faure et al. proposed a multimodal action detection network based on RGB image stream and skeletal pose stream. By combining RGB image stream and skeletal pose stream to capture human-environment interaction information, they attempted to use the dual-modal framework of skeletal stream and video stream to achieve efficient action recognition, thereby improving the accuracy of action detection. This method utilizes a variety of feature information and surpasses the action detection capability of a single modality.

[0005] [2] A sparse-dense complementary learning (SDCL) framework was proposed, which makes full use of the complementary information of dense frames and sparse events to effectively extract identity features and improve the effect of people re-identification. This method introduces the event flow modality into the video people re-identification task for the first time. In the field of people re-identification, a collaborative learning network between event cameras and ordinary cameras is proposed to improve the recognition effect by using the complementary information of RGB frames and events.

[0006] However, the above document has the following drawbacks:

[0007] The skeleton flow used in Reference 1 requires an external pose detection module. Although this reduces the workload of dataset annotation, the detection effect directly affects the subsequent network performance. Furthermore, the skeleton-based transformer network will cause a large computational and memory overhead.

[0008] Reference 2 describes a person re-identification method based on the collaboration of RGB image sequences and event streams. Event cameras can capture brightness changes, thereby obtaining motion information at corresponding locations, giving event streams a unique advantage in extracting motion features. However, this method does not fully utilize the rich temporal information contained in the RGB image sequences themselves to extract person motion features, which is a limitation of the method.

[0009] [1]GJFaure, M.-H.Chen and S.-H.Lai, "Holistic Interaction TransformerNetwork for Action Detection," 2023 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp.3329-3339, doi:10.1109 / WACV56688.2023.00334.

[0010] [2]C.Cao et al., "Event-Guided Person Re-Identification via Sparse-Dense Complementary Learning," 2023 IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR), Vancouver, BC, Canada, 2023, pp.17990-17999, doi:10.1109 / CVPR52729.2023.01725. Summary of the Invention

[0011] This invention proposes a human motion detection method based on dual-modal camera driving through mutual learning between event cameras and ordinary cameras, which improves the robustness and accuracy of existing human motion recognition and localization in various complex scenarios.

[0012] The specific steps of the human motion detection method based on dual-modal camera driving are as follows:

[0013] Step 1: Use both event cameras and regular cameras to capture event sequences and regular video corresponding to the same time window;

[0014] Step 2: For the event sequence captured by the event camera within a single time window, the sequence is transformed and stacked into synchronized image frames corresponding to those of a regular camera.

[0015] First, a rectangular wave function is used to filter events. Based on the position and polarity of each filtered event, the intensity I(t) of the image is calculated:

[0016]

[0017] The event sequence is E = {e1, e2, ..., e}i ,…,e n}, where e i This is the i-th event, where rect(x) is the rectangular wave function, representing a unit rectangular wave in the interval [0,1]. i Δt represents the time point when the i-th event occurs, and Δt represents the width of the time window, corresponding to the frame rate of a normal camera.

[0018] All events within this time window are accumulated and stacked to generate a corresponding synchronization image.

[0019] Step 3: For synchronized images corresponding to the video modality and the event modality within the same time window, extract the features of each modality using the same deep neural network as primary features;

[0020] The deep neural network selected is SlowFast-resnet-50.

[0021] Step 4: Spatial alignment enhancement is performed on the obtained primary features through the adaptive feature learning module;

[0022] The formula for spatial alignment enhancement is:

[0023] F v =P e→v ·F e (2)

[0024] F e / v ′(i,j)=ω t ⊙{(F e / v (i,j),F v / e (i,j)} (3)

[0025] In the above formula, F e F v P represents the obtained event modal primary features and ordinary modal primary features. e→v This represents the coordinate system projection from the event mode to the ordinary mode. F e / v ′(i,j) represents the alignment of event modality / normal modality feature or attention information at position (i,j) into the spatial coordinate system of normal modality / event modality, utilizing the event modality to enhance visual information. F e / v (i,j) and F v / e (i,j) represent the initial features of the event mode and the normal mode at position (i,j), respectively.

[0026] Formula (3) expresses two cases: one is spatial attention alignment between the event modality (e) and the visual modality (v), which uses the features or attention information of the event modality (e) to enhance the visual modality (v); the other is spatial attention alignment between the visual modality (v) and the event modality (e), which uses the features or attention information of the visual modality (v) to enhance the event modality (e).

[0027] Projecting P through the coordinate system e→v Consensus is reached in the spatial coordinate system; then, spatial attention weights ω are used. t By using element-wise multiplication (⊙), the ability to represent two types of features in spatial dimensions is enhanced.

[0028] Step 5: Use an instance detector to identify and locate instances of interest in image frames captured from a regular camera, and generate a corresponding detection box for each instance to determine the location and extent of each instance in the image.

[0029] Max pooling is performed on the spatially aligned and enhanced primary features using the detection bounding box.

[0030] The instance detector employs Faster R-CNN and uses max pooling with 3D ROI Align to reduce feature dimensionality while preserving the most salient features. For the event camera modality, detection boxes obtained from ordinary camera image frames are reused to reduce computational cost.

[0031] Step 6: Interact and model each instance through a multi-stage dense serial structure, and enhance the generation of memory features by fusing contextual information from different time windows, ultimately obtaining action features for classification.

[0032] The interactive modeling fusion specifically refers to:

[0033] First, for a given instance feature P v P e Object feature O v O e Hand features H v H e and memory features M v M e The interaction block adaptively models the feature interactions for each type;

[0034] Then, by aggregating the aligned fusion structure of multiple interactions, action features for prediction are obtained. Where φε is the parameter for interactive modeling fusion.

[0035] Each interaction block takes all the outputs of the previous block and gathers them together using learnable weights, along with the outputs of the other branch, and aligns and fuses them using a similarity attention matrix.

[0036] The interactive block is expressed as follows:

[0037]

[0038] In the formula, C represents the index set of the j-th interactive block. and W is the interactive output feature of the two branches of the j-th block. ,e W represents a learnable d-dimensional vector, normalized using the Softmax function. j,v The similarity attention matrix for the two branches is used to align and fuse the enhanced output features of the two branches.

[0039] For memory features, the generation of memory features is enhanced by extracting contextual information from different time windows; specifically:

[0040] For video segments, a time window of (4L+1) is used; for event segments, a forward and backward window of (2L+1) is used for the target person; for time segments with shared coverage, weighted fusion is used to obtain... and

[0041] The characteristics of memory at the current moment are described as follows:

[0042]

[0043] ω represents the character features corresponding to the event and video at time t+L, respectively. k These are the weighting coefficients affected by the event, which follow a normal distribution. The closer to the current moment, the greater the influence on the current feature.

[0044] The advantages of this invention are:

[0045] 1) This invention provides a human motion detection method based on dual-modal camera driving. It proposes an RGB-Event dual-stream network for motion detection tasks, which guides RGB stream feature extraction through event stream, improves the spatiotemporal feature representation of the local action occurrence, and significantly enhances the accuracy of action recognition and localization.

[0046] 2) This invention proposes a human motion detection method based on dual-modal camera driving, which proposes a module to realize feature adaptive learning and enhancement. Through coordinate mapping and spatial attention mechanism, the two types of features guide each other and jointly optimize their representation ability in the spatial dimension.

[0047] 3) This invention proposes a human motion detection method based on a dual-modal camera. Starting from the motion interaction itself, it proposes a multi-level feature alignment and interaction extraction network. Through multi-level alignment and interaction aggregation strategies, it integrates information from time windows to enhance memory features and extracts deep spatiotemporal interaction relationships from the fused features, which helps to better identify actions.

[0048] 4) This invention provides a human motion detection method based on dual-modal camera driving. Compared with the previous single-modal learning method, this invention significantly improves the recognition accuracy. In addition, due to the use of event camera input, it also has strong robustness in complex visual environments. Especially under adverse conditions such as large changes in lighting or obstructed field of view, it can still provide effective information through event stream to assist in motion recognition. Attached Figure Description

[0049] Figure 1 This is a flowchart of a human motion detection method based on a dual-modal camera driven according to the present invention;

[0050] Figure 2 This is a schematic diagram of a human motion detection method based on dual-modal camera driving according to the present invention.

[0051] Figure 3 This is a schematic diagram of the interactive block structure of the present invention;

[0052] Figure 4 This is a schematic diagram of the dense serial alignment strategy of the present invention. Detailed Implementation

[0053] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0054] This invention presents a human motion detection method based on a dual-modal camera, using deep learning technology to detect and locate human actions in videos. It leverages the complementary characteristics of event cameras and ordinary cameras for feature-level collaborative processing. For the input video, it fully utilizes the visual information provided by different types of cameras, modeling the interaction between the person and the surrounding environment to identify and locate human actions. This invention aims to explore a novel human motion detection method based on event camera data, through mutual learning between event cameras and ordinary cameras, thereby improving the robustness and accuracy of existing human motion recognition and localization methods in various complex scenarios.

[0055] like Figure 2As shown, this invention takes video frames and event sequences within corresponding timestamps as input. An asynchronous event image conversion module stacks the event sequences into corresponding synchronous images, allowing both modalities to undergo primary feature extraction through the same backbone network. A feature adaptive learning enhancement module is designed to enhance the spatial representation capabilities of the two types of features through mutual learning. Subsequently, the instance detector cropps the features into corresponding human, hand, and object instance features. A multi-level alignment and interactive modeling strategy is employed, focusing on the interactive information between instance features. For memory features, considering the sparse-dense complementary characteristics of the event branch and the RGB branch, different time windows of varying lengths and normally distributed weight coefficients are used. This allows the RGB stream to primarily handle long-term global temporal representation learning, capturing the overall contour change trend of the action; simultaneously, the event stream focuses on short-term, rapid local motion, accurately describing the detailed changes in action. Finally, the action features obtained for prediction are output through a classifier.

[0056] The human motion detection method based on dual-modal camera driving, such as Figure 1 As shown, the specific steps are as follows:

[0057] Step 1: Use both event cameras and regular cameras to capture event sequences and regular video corresponding to the same time window;

[0058] This invention employs a dual-stream network structure, obtaining event sequences and videos corresponding to event camera streams and ordinary camera streams through different sensors, and extracting features of video modalities and event modalities respectively.

[0059] Step 2: For the event sequence captured by the event camera within a single time window, use the asynchronous event image conversion module to stack them into synchronous image frames corresponding to those of a regular camera;

[0060] The specific strategy will follow the formula below:

[0061]

[0062] The event sequence is E = {e1, e2, ..., e} i ,…,e n}, where e i This is the i-th event, where rect(x) is the rectangular wave function, representing a unit rectangular wave in the interval [0,1]. i Δt represents the time point when the i-th event occurs, and Δt represents the width of the time window, corresponding to the frame rate of a normal camera.

[0063] Events are filtered by rectangular wave function, and then the intensity I(t) of the image is calculated according to the position and polarity of each event, so as to accumulate and stack all events within a time window to generate the corresponding synchronized image.

[0064] Step 3: For synchronized images corresponding to the video modality and the event modality within the same time window, extract the features of each modality using the same deep neural network as primary features;

[0065] The deep neural network selected is SlowFast-resnet-50, a deep learning network based on a backbone-head structure, which extracts deep features from video and event modalities through the backbone network.

[0066] Step 4: Spatial alignment enhancement is performed on the obtained primary features through the adaptive feature learning module;

[0067] The two modalities learn and enhance each other's features through an adaptive learning module. This module uses coordinate mapping and attention mechanisms to achieve spatial alignment enhancement of video and event features. The formula for spatial alignment enhancement is:

[0068] F v =P e→v ·F e (2)

[0069] F e / v ′(i,j)=ω t ⊙{(F e / v (i,j),F v / e (i,j)} (3)

[0070] In the above formula, F e F v Pe represents the obtained event modal primary features and ordinary modal primary features. →v This represents the coordinate system projection from the event mode to the normal mode. F e / v ′(i,j) represents the alignment of event modality / normal modality feature or attention information at position (i,j) into the spatial coordinate system of normal modality / event modality, utilizing the event modality to enhance visual information. F e / v (i,j) and F v / e (i,j) represent the initial features of the event mode and the normal mode at position (i,j), respectively.

[0071] Formula (3) expresses two cases: one is spatial attention alignment between the event modality (e) and the visual modality (v), which uses the features or attention information of the event modality (e) to enhance the visual modality (v); the other is spatial attention alignment between the visual modality (v) and the event modality (e), which uses the features or attention information of the visual modality (v) to enhance the event modality (e).

[0072] Projecting P through the coordinate system e→v Consensus is reached in the spatial coordinate system; then, spatial attention weights ω are used.t By using element-wise multiplication (⊙), the ability to represent two types of features in spatial dimensions is enhanced.

[0073] Step 5: Use an instance detector to identify and locate instances of interest in image frames captured from a regular camera, and generate a corresponding detection box for each instance to determine the location and extent of each instance in the image.

[0074] Max pooling is performed on the spatially aligned and enhanced primary features using the bounding box.

[0075] The instance detector employs Faster R-CNN, which crops primary features to obtain corresponding instance features such as people, objects, and hands. Max pooling is then performed using 3D ROI Align to reduce feature dimensionality while retaining the most salient features. For event camera modalities, detection boxes obtained from ordinary camera image frames are reused to reduce computational cost.

[0076] Step 6: Interact and model each instance through a multi-stage dense serial structure to obtain action features for prediction. Enhance the generation of memory features by fusing contextual information from different time windows. Finally, perform action detection through a classifier to obtain action features for classification.

[0077] like Figure 3 As shown, the interaction block structure includes two inputs: one mapped to a key and a value, and the other used as a query. Through dot product attention, i.e., the output of the SoftMax layer in the diagram, the interaction block can select value features that are highly activated to the query features and merge them to enhance the query features. There are four types of interaction blocks: p-blocks, o-blocks, h-blocks, and m-blocks. The p-block uses the same key / value input as the query input. For the o-block (h-block), the goal is to extract the interaction between a person and an object (hand), and the key / value input is the detected object features (hand features). For the m-block, the memory features fused from sparse-dense temporal modeling are used as the key / value input of the m-block.

[0078] The interactive modeling fusion specifically refers to:

[0079] First, for a given instance feature P v P e Object feature O v O e Hand features H v H e and memory features M v M e The interaction block adaptively models the feature interactions for each type;

[0080] Then, by aggregating the aligned fusion structure of multiple interactions, action features for prediction are obtained. Where φε is the parameter for interactive modeling fusion.

[0081] like Figure 4 As shown, the dense serial alignment strategy refers to the two-stream network extracting three types of interactions using different types of interaction blocks and integrating them through a dense serial alignment structure. Specifically, each interaction block receives all the outputs of the previous block and aggregates them together using learnable weights, along with the outputs of the other branch, and aligns and fuses them using a similarity attention matrix.

[0082] The expression for the query of the i-th block in the event flow branch of the interaction block is as follows:

[0083]

[0084] In the formula, C represents the index set of the j-th interactive block. and This represents the interactive output feature of the two branches in the j-th block. ⊙ indicates element-wise multiplication, and W... j,e W represents a learnable d-dimensional vector, normalized using the Softmax function. j,v The similarity attention matrix for the two branches is used to align and fuse the enhanced output features of the two branches.

[0085] For memory features, the generation of memory features is enhanced by extracting contextual information from different time windows, which helps to better understand the causes and consequences of actions. Specifically:

[0086] The asynchronous memory pool update mechanism enhances the expressive power of features in the time dimension. The ER asynchronous memory pool update strategy includes two basic operations: READ and WRITE. In each training iteration, READ, for a given i-th video segment and its corresponding event segment, reads the memory features of surrounding segments from the memory pool Ω. For video segments, a time window of (4L+1) is used; for event segments, (2L+1) is used for the forward and backward windows of the target person. For shared time segments, weighted fusion is used to obtain... and

[0087] The characteristics of memory at the current moment are described as follows:

[0088]

[0089] ω represents the character features corresponding to the event and video at time t+L, respectively. k These are the weighting coefficients affected by the event, which follow a normal distribution. The closer to the current moment, the greater the influence on the current feature.

[0090] At the end of each iteration, the individual features of the target fragment are written back to the memory pool as estimated memory features. We evaluate the quality of feature extraction by a loss factor, highlighting the relative error between the reference loss value provided by the loss label and the actual loss value output by the current network to define a penalty factor, thereby discarding poorly estimated features and achieving a balance between GPU memory consumption and performance.

[0091] After the network structure is built, high-precision model weights are obtained by training on a spatiotemporal action detection dataset for actual action localization and recognition.

[0092] Example:

[0093] This embodiment uses PyTorch 1.6.1 as the deep learning framework. Two GeForce RTX 3090 graphics processors were used in the experiment, each with 24GB of memory. The batch size was set to 16, and the training strategy was: 14k iterations, no weight decay for the first 1400 iterations, and an initial learning rate of 0.0002.

[0094] Specifically, it consists of the following two steps:

[0095] 1) Instance feature extraction and spatial alignment:

[0096] The asynchronous event image conversion module stacks event sequences into corresponding synchronous images. Then, the video and event features extracted using the same backbone network are aligned through the adaptive feature learning module. Subsequently, the instance detector crops the features into corresponding human, hand, and object instance features.

[0097] First, the asynchronous event sequence is converted into synchronous images through stacked integration. SlowFast-ResNet-50 is selected as the backbone network for initial feature extraction. Then, a spatial attention mechanism is used to enhance the spatial representation ability of both types of features. Faster R-CNN is used for instance detection, and ROI Align operation is employed to optimize feature extraction within the region of interest.

[0098] The specific strategy will follow the formula below:

[0099]

[0100] This allows all events to be accumulated and stacked within a time window to generate a synchronized image.

[0101] Next, spatial alignment is performed on the two types of features:

[0102] F v =Pe→v ·F e (2)

[0103] F e / v ′(i,j)=ω t ⊙(F e / v (i,j),F v / e (i,j) (3)

[0104] In the above equation, the coordinate system projection P e→v Consensus is reached in the spatial coordinate system. Then, spatial attention weights ω are used. t By using element-wise multiplication (⊙), the ability to represent two types of features in spatial dimensions is enhanced.

[0105] The above summarizes the instance feature extraction and spatial alignment parts. After the corresponding features are obtained by cropping, they will be used for interactive modeling and fusion.

[0106] 2) Interaction modeling and fusion:

[0107] The obtained instance features are interactively modeled and fused through a multi-stage dense serial structure, and the temporal information is updated through an ER asynchronous memory pool update strategy, ultimately yielding action features for classification.

[0108] For a given different person P v P e Object feature O v O e Hand features H v H e and memory features M v M e The goal is to focus on information related to interactions with the target character. Interaction blocks can adaptively model each type of interaction within a unified structure, and then, by aggregating the aligned and fused structures of multiple interactions, action features for prediction are obtained. Where φε represents the structural parameters of the interactive modeling network.

[0109] Each interaction block receives all the outputs of the previous block and aggregates them together using learnable weights, along with the outputs of the other branch, which are then aligned and fused using a similarity attention matrix.

[0110] Considering the sparse-dense complementary characteristics of event branches and RGB branches, the ER asynchronous memory pool update strategy includes two basic operations: READ and WRITE. During each training iteration, READ, for a given i-th video segment and its corresponding event segment, reads the memory features of surrounding segments from the memory pool Ω. For video segments, the time window is (4L+1); for event branches, a forward and backward (2L+1) window is used for the target person. For shared time segments, weighted fusion is used to obtain... and The memory characteristics at a certain moment can be described as follows:

[0111]

[0112] Where ω k The weighting coefficients are influenced by the event. They follow a normal distribution, and the closer to the middle of the time, the greater their influence on the current feature.

[0113] At the end of each iteration, the individual features of the target fragment are written back to the memory pool as estimated memory features. The quality of feature extraction is evaluated by a loss factor. The penalty factor is defined by the relative error between the reference loss value provided by the loss label and the actual loss value output by the current network, thereby discarding poorly estimated features and achieving a balance between GPU memory consumption and performance.

[0114] After the execution was completed, the results are shown in Table 1 and Table 2.

[0115] Tables 1 and 2 show the comparative experiments of our method on three public spatiotemporal detection datasets. Compared with other models, we can see that our method achieves excellent results in terms of accuracy, and it is also demonstrated that adding the event camera modality improves existing benchmarks.

[0116] The evaluation metric used is mAP (mean Average Precision), an important performance indicator for object detection tasks. It is calculated by matching predicted bounding boxes with ground truth labels according to an IoU threshold, calculating the precision value for each class at different thresholds, then calculating the average precision for each class at different IoU thresholds, and finally calculating the average precision for all classes, which is the mAP score. There are frame-level mAP and video-level mAP. We used IoU thresholds of 0.2 and 0.5.

[0117] The performance of this method on the JHMDB and UCF101-24 datasets is shown in Table 1:

[0118] Table 1

[0119]

[0120] The performance of this method on the AVA v2.2 dataset is shown in Table 2:

[0121] Table 2

[0122]

Claims

1. A human motion detection method based on dual-modal camera driving, characterized in that, The specific steps are as follows: Step 1: Use both event cameras and regular cameras to capture event sequences and regular video corresponding to the same time window; Step 2: For the event sequence captured by the event camera within a single time window, the sequence is transformed and stacked into synchronized image frames corresponding to those of a regular camera. All events within this time window are accumulated and stacked to generate a corresponding synchronization image; Step 3: For synchronized images corresponding to the video modality and the event modality within the same time window, extract the features of each modality using the same deep neural network as primary features; Step 4: Spatial alignment enhancement is performed on the obtained primary features through the adaptive feature learning module; The formula for spatial alignment enhancement is: F v =P e→v ·F e (2) F e / v ′(i,j)=ω t ⊙{(F e / v (i,j),F v / e (i,j)} (3) In the above formula, F e F v P represents the obtained event modal primary features and ordinary modal primary features. e→v F represents the coordinate system projection from the event mode to the ordinary mode; e / v ′(i,j) means that at position (i,j), the feature or attention information of the event modality / normal modality is aligned to the spatial coordinate system of the normal modality / event modality, and the event modality is used to enhance visual information; F e / v (i,j) and F v / e (i,j) represent the initial features of the event mode and the normal mode at position (i,j), respectively; The above formula expresses two scenarios: one is spatial attention alignment between event modality e and visual modality v, which uses the features or attention information of event modality e to enhance visual modality v; the other is spatial attention alignment between visual modality v and event modality e, which uses the features or attention information of visual modality v to enhance event modality e. Projecting P through the coordinate system e→v Consensus is reached in the spatial coordinate system; then, spatial attention weights ω are used. t By using element-wise multiplication (⊙), the ability to represent the two types of features in the spatial dimension is enhanced. Step 5: Use an instance detector to identify and locate instances of interest in image frames captured from a regular camera, and generate a corresponding detection box for each instance to determine the location and extent of each instance in the image; Step 6: Interact and model each instance through a multi-stage dense serial structure, and enhance the generation of memory features by fusing contextual information from different time windows, ultimately obtaining action features for classification. The interactive modeling fusion specifically refers to: First, for a given instance feature P v P e Object feature O v O e Hand features H v H e and memory features M v M e The interaction block adaptively models the feature interactions for each type; Then, by aggregating the aligned fusion structure of multiple interactions, action features for prediction are obtained. Where φε is the parameter for interactive modeling fusion.

2. The human motion detection method based on dual-modal camera driving as described in claim 1, characterized in that, Step two specifically involves: First, a rectangular wave function is used to filter events. Based on the position and polarity of each filtered event, the intensity I(t) of the image is calculated: The event sequence is W = {e1, e2, ..., e} i ,…,e n }, where e i This is the i-th event, and rect(x) is the rectangular wave function, representing a unit rectangular wave in the interval [0,1]. t i Δt represents the time point when the i-th event occurs, and Δt represents the width of the time window, corresponding to the frame rate of a normal camera.

3. The human motion detection method based on dual-modal camera driving as described in claim 1, characterized in that, In step three, the deep neural network selected is SlowFast-resnet-50; a deep learning network based on a backbone-head structure extracts deep features from video and event modalities through the backbone network.

4. The human motion detection method based on dual-modal camera driving as described in claim 1, characterized in that, In step five, the detection box is used to perform max pooling on the spatially aligned and enhanced primary features. The instance detector uses Faster-RCNN and performs max pooling through 3D ROI Align operation to reduce the dimensionality of features while retaining the most salient features. For event camera modalities, detection boxes obtained from ordinary camera image frames are reused to reduce computation.

5. The human motion detection method based on dual-modal camera driving as described in claim 1, characterized in that, In step six, each interaction block receives all the outputs of the previous block and gathers them together using learnable weights, along with the outputs of the other branch, and aligns and fuses them using a similarity attention matrix; The interactive block is expressed as follows: In the formula, C represents the index set of the j-th interactive block. and It is the interactive output feature of the two branches of the j-th block; W j,e W represents a learnable d-dimensional vector, normalized using the Softmax function. j,v The similarity attention matrix for the two branches is used to align and fuse the enhanced output features of the two branches.

6. The human motion detection method based on dual-modal camera driving as described in claim 1, characterized in that, In step six, for memory features, the generation of memory features is enhanced by extracting contextual information from different time windows; specifically: For video segments, a time window of (4L+1) is used; for event segments, a forward and backward window of (2L+1) is used for the target person; for time segments with shared coverage, weighted fusion is used to obtain... and The characteristics of memory at the current moment are described as follows: ω represents the character features corresponding to the event and video at time t+L, respectively. k These are the weighting coefficients affected by the event, which follow a normal distribution. The closer to the current moment, the greater the influence on the current feature.

Images