Computer-implemented action detection method using modality and time-shifted features

Abstract

The invention provides a computer-aided method for action recognition of a person to be monitored, wherein the method comprises: a) Capturing a plurality of sequences of images (32) with an image sensor (22), wherein each sequence is recorded using a predetermined modality and each pair of sequences is recorded using different modalities; b) Passing each sequence through a one-stage feature learning model trained for feature extraction from image data to obtain a sequence of feature maps for each sequence of images (32), wherein each sequence of feature maps is composed of a plurality of feature maps and each feature map specifies features suitable to characterize an action of the person being monitored; c) Iterating through each sequence of feature maps by a feature-shifting module that performs a feature shift by selecting a number of feature maps along a dimension and shifting the selected feature maps along that dimension by a predetermined amount; d) Combining the sequences of feature maps resulting from step c) into an ensemble feature map (82); e) Feeding the ensemble feature card (82) into an action classification module trained to determine an action performed by the person being monitored based on the ensemble feature card (82), and determining the action most likely to have been performed by the person being monitored.

Description

TECHNICAL FIELD

[0001] The invention relates to a computer-implemented method for action recognition. The invention further relates to a system for implementing the method. BACKGROUND

[0002] Driver action recognition (DAR) is the process of automatically identifying the driver's actions within the vehicle cabin while driving. DAR is a key component for improving real-time driving safety and promoting efficient human-vehicle interactions. Research in this area primarily focuses on the extraction of spatiotemporal features to improve classification performance within single-modality features, such as infrared (IR) video images.

[0003] However, the features of a single modality may be insufficient to support accurate long-term action recognition. On the one hand, the vehicle cabin environment is complex, and the available features are limited. The driver's actions are performed by the same person, with only part of the body (e.g., the upper body) visible, and often involve very similar movements of body parts (e.g., eating and drinking). On the other hand, the lighting conditions in the vehicle cabin are unstable and depend on factors such as weather and traffic infrastructure, and can be influenced by suboptimal factors such as sunlight or noise.

[0004] In the real world, it is common for vehicle cabins to be equipped with cameras that have different modalities (e.g., color, IR, and depth) and views (e.g., front-top and top-right). Therefore, driver action recognition is inherently multimodal, with each data modality providing potentially valuable information.

[0005] US 2020 / 0 089 977 A1 discloses a method for detecting and predicting driver behavior. The system's unified representation framework may comprise two or more modalities for capturing visual phenomena and the vehicle dynamics of the associated vehicle. Various multimodal fusion architectures can be implemented using the image capture device or camera and vehicle sensor data from the CAN bus for tactical detection of driver behavior.

[0006] K. Peng, et al, “TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration”, 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 278-285, describes a transformer-based recognition of driver activity with latent space feature calibration. Furthermore, reference is made to the following documents [1] Ji Lin, Chuang Gan, and Song Han. TSM: temporal shift module for efficient video understanding. In 2019 IEEE / CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 7082-7092. IEEE, 2019. [2] Alina Roitberg, Kunyu Peng, Zdravko Marinov, Constantin Seibold, David Schneider, and Rainer Schuhehagen. A comparative analysis of decision-level fusion for multimodal driver behavior understanding. 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1438-1444, 2022. [3] Rim Trabelsi, Redouane Khemmar, Benoit Decoux, Jean-Yves Ertaud, and R'emi Boutteau. Recent advances in vision-based on-road behaviors understanding: A critical survey. Sensors, 22(7):2654-2681, 2022. [4] Zachary Wharton, Ardhendu Behera, Yonghuai Liu, and Nikolaos Bessis. Coarse temporal attention network (cta-net) for driver's activity recognition. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1278-1288, 2021. SUMMARY OF THE INVENTION

[0007] The object of the invention is to improve the action recognition of the operator when monitoring the vehicle cabin.

[0008] The object is achieved by the subject matter of the independent claims. Preferred embodiments are the subject matter of the dependent claims.

[0009] The invention provides a computer-implemented method for action recognition of a person to be monitored, preferably an operator of a vehicle, the method comprising: a) capturing a plurality of sequences of images with at least one image sensor, each sequence being recorded using a predetermined modality, at least some images containing at least in part the person to be monitored, each pair of sequences being recorded using different modalities; b) passing each sequence through at least one feature learning model in one stage, which is trained for feature extraction from image data, to obtain at least one sequence of feature maps for each sequence of images, wherein each sequence of feature maps is composed of a plurality of feature maps and each feature map indicates features suitable for characterizing an action of the person to be monitored; c) passing each sequence of feature cards through at least one feature shifting module that performs a feature shift by selecting a number of feature cards along a dimension and shifting the selected feature cards along that dimension by a predetermined amount; d) combining the sequences of feature maps resulting from step c) into an ensemble feature map; e) feeding the ensemble feature map into an action classification module trained to determine an action performed by the person to be monitored based on the ensemble feature map, and determining the action most likely performed by the person to be monitored.

[0010] The method enables the combination of different modalities in action recognition and performs action recognition using a common model for all modalities. This makes it possible to reduce the total number of parameters and increase computational efficiency. Shifting along at least one dimension can improve feature attraction and the integration of complementary features across different dimensions of the dataset, such as between modalities and / or between images. Ideally, shifting can be performed without additional computational costs, thus further increasing computational efficiency.

[0011] Preferably, the specified modality is selected from a group consisting of a viewpoint of the person to be monitored, a color channel, an infrared channel, and a depth channel. Different modalities enable the extraction of complementary features that can help reduce ambiguities or mitigate challenging environmental conditions, such as lighting. This can improve the overall accuracy of action recognition.

[0012] Preferably, step b) comprises passing each sequence of images and / or sequence of feature maps through at least one feature model in separate stages, comprising a first modality stage and a second modality stage, wherein each sequence associated with a first modality is processed by the first modality stage and each sequence associated with a second modality is processed by the second modality stage. The separate treatment, particularly at the beginning, of the different modalities during feature extraction helps prepare the data for more reliable action recognition. The separate initial stages typically enable more modality-specific treatment of the input data, which can reduce the causes of inaccurate action recognition.

[0013] Preferably, step b) comprises running each sequence of images and / or sequence of feature maps through at least one feature learning model in a joint stage, which includes parameters that can be learned jointly for all modalities. Joint processing, especially between the different modalities during feature extraction, aids in feature extraction and integration and can provide a comprehensive foundation for subsequent action recognition. Overall, the accuracy of action recognition can be improved while maintaining high computational efficiency.

[0014] Preferably, step c) comprises selecting a number of feature cards with the same timestamp. Selecting feature cards with the same timestamp contributes to reducing uncertainty along the time dimension and enables improved action recognition.

[0015] Preferably, step c) comprises shifting the selected feature maps along a modality dimension by a predetermined amount or swapping the selected feature maps between different modalities. Shifting feature maps along the modality dimension enables the interaction of features of the modality. It is possible to supplement features present in one modality with features from another modality, with all features pointing to the same object within the image.

[0016] Preferably, step c) comprises shifting the selected feature maps along a temporal dimension by a predetermined amount. Shifting feature maps along the temporal dimension or the time dimension allows for the propagation of neighboring features. It is possible to supplement features present in one image with features present in another image, e.g., to improve the detection of a temporal progression of features characterizing an action.

[0017] Preferably, a portion of the selected feature maps is shifted forward along the temporal dimension, and another portion is shifted backward along the temporal dimension. By shifting in both temporal directions simultaneously, it is possible to improve the temporal progression of the observed action or to obtain a comprehensive picture of it.

[0018] Preferably, step b) and / or step c) comprises selecting a series of consecutive images and / or feature maps along different features of a feature dimension. The selection of consecutive images enables a more continuous view of the feature maps, which improves overall action recognition.

[0019] Preferably, steps b) and c) are performed alternately for a predetermined number of iterations. Preferably, steps b) and c) are alternated such that step b) is performed a predetermined number of times before step c) is performed a predetermined number of times. This approach makes it possible to choose the balance between feature extraction and feature integration.

[0020] The invention provides a method for controlling a vehicle, comprising performing a previously described method and generating a control signal for a vehicle based on the most probable action determined in step e), and controlling the vehicle in accordance with the control signal. Upon detecting an action performed by the operator, the vehicle can be controlled to perform an action based on the operator's action.

[0021] The invention provides a system with means for carrying out the preferred method.

[0022] The invention provides a vehicle, preferably a ground or land vehicle, with the preferred system, wherein the system is designed to monitor an operator of the vehicle and / or to control the vehicle.

[0023] The invention provides a computer program containing instructions for causing the preferred system or vehicle to perform the preferred method.

[0024] The invention provides a computer-readable medium or a data carrier signal containing the computer program.

[0025] One idea is to incorporate features from multiple modalities to effectively tackle the complex and challenging task of recognizing operator actions. Multimodal and temporal features can be extracted more reliably, enabling improved action recognition.

[0026] An efficient multimodal driver action recognition system based on a dual feature shifting module (DFS) is proposed. In some embodiments, the dual feature shifting module can integrate complementary features across modalities by performing feature interaction along the modality dimension. In some embodiments, the DFS module can shift features along the temporal dimension within a single modality, thereby facilitating feature propagation between images. By using at least one, preferably both, of the modality and temporal shifting operations, better action recognition performance can be achieved with little or no additional computational cost. In some embodiments, model efficiency can be improved by designing the DFS as a hip-drum-shaped framework.In some embodiments, the DFS may have some common stages in the middle of the feature extraction model.

[0027] Multimodal fusion strategies for action recognition in vehicle cabins have been little studied so far. Only a few methods perform multimodal fusion learning through score fusion or late fusion. These methods typically train separate models or backbones for each modality, which typically results in a reduction in computational power. Compared to these methods, the invention described herein provides an efficient feature-shifting system for multimodal driver action recognition.

[0028] The proposed feature shifting module can perform both intermodality interaction and neighbor feature propagation. This can help attract and integrate complementary features across modalities and between individual images. In some embodiments, the modality and time shifting operations can transfer features without additional computational cost, which is computationally efficient. In some embodiments, shared stages are used, preferably in the middle of the feature extraction model. The structure of the framework is then typically referred to as "hip-drum-like." With the above measures, it is possible to reduce the number of model parameters and thus improve model efficiency. The proposed method can also be better adapted to real-world applications.

[0029] The idea can be illustrated using two modalities as input. It should be noted that the number of modalities does not have to be two and can be higher depending on the circumstances. For example, a system used in aircraft may have more modalities, since more than one operator may be present in the cockpit, even if they do not formally exercise control.

[0030] In a typical implementation, three to eight stages of feature learning can be used, with three to five stages being preferred, and four stages being most preferred. The feature learning stages are followed by a late fusion layer. The fusion layer combines the learned features from each modality into an ensemble. The fusion layer is typically followed by a fully connected layer, which is trained as a classifier to generate the final predicted actions.

[0031] At the beginning of each stage, the feature shifting module is deployed to integrate complementary features along the modality or temporal dimension, respectively. In some embodiments, a hip-drum-shaped framework with shared convolutional layers may be present in intermediate stages (i.e., not in the first or last stage of the model). In the case of a total of five stages, stages two and three are preferably used together, primarily for reasons of training efficiency.

[0032] In some embodiments, the feature shifting module is configured to integrate complementary features. In some embodiments, the feature shifting module includes a modality feature interaction module that can shift the features along the modality dimension. In some embodiments, the feature shifting module includes a neighbor feature propagation module that can shift the features along the temporal dimension.

[0033] Modality feature interaction can be used to learn complementary features from multiple modalities. The feature shifting mechanism preferably uses the modality feature interaction module to enable effective communication between modalities.

[0034] For example, for the p-th modality, the input video clip is X p ∈ RC×T×H×W, where C is the number of input feature channels, T is the number of input images, and H and W are the spatial resolutions of the input feature's height and width, respectively. The interaction of the modality features can transfer the feature across different modalities. The feature X p of modality p can be obtained by a shift process from the feature of modality q according to x̂ p = M shift (x p , x q ) are updated, whereby the operationM shift swaps some of the channels along the modality dimension. This operation is preferably performed without multiplication.

[0035] The neighbor feature propagation module can shift features along the temporal dimension for the feature learning part for individual modalities. Specifically, the information can be propagated along the temporal dimension in two directions (forward and backward). For example, an input video clip is Xp={xtp}t=1T with T images, where xtp∈RC×H×W with C channels and H×W the spatial resolution of the tensor containing the image data at timestamp t. The feature of xtp can be achieved by shifting the images to the timestamp xt−1p and xt+1p according to x^tp=Tshift(xt−1p,xt+1p) updated, where the operation T shift shifts a portion of the channels along the temporal dimension. This operation can be performed without multiplication.

[0036] At each stage, the updated features of the shifted images are preferably extracted by a feature extractor according to f p = H p (X p , W p ) f p = H p (X p , W p ) learned, where the operation H p the residual convolution calculation in each stage and W p is the weight matrix to be trained.

[0037] After typically three to eight, preferably three to five, and more preferably four stages of feature learning, the modality-specific spatial features integrated with modality iteration and temporal information can be obtained. In some embodiments, a late fusion strategy is used to fuse the modality-specific spatial features. The late fusion computation is preferably performed using an average-up logit approach to combine the output features from different modalities into an ensemble of feature expressions. The predicted results can be generated after a fully connected layer trained with the cross-entropy loss function for model training and optimization.

[0038] The aspects described here can be transferred to real-world applications where vehicle cabins are preferably equipped with cameras with different modalities. DFS can contribute to driver safety monitoring in these applications. DFS can be used in the automotive industry for intelligent cabin design to monitor driver safety. DFS could also be used in other areas where human actions play an important role, such as driver cab or cockpit monitoring in trains and aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] It should be noted that although the invention is described with reference to a land vehicle, e.g., a motor vehicle, the ideas disclosed herein are also applicable to vehicles in general, e.g., aircraft and watercraft. Embodiments of the invention will be described in more detail with reference to the accompanying schematic drawings. The drawings are listed below: Fig. 1 shows an embodiment of a vehicle; Fig. 2 shows an action recognition module; and Fig. 3 shows a detailed view of a dual feature-shifting module element. DETAILED DESCRIPTION OF THE EMBODIMENT

[0040] As in Fig. 1, a vehicle 10 includes a vehicle cabin 12. An operator 14, who controls the vehicle 10, is located in the vehicle cabin 12. The operator 14 is an example of a person to be monitored. The vehicle 10 further includes a system 16 for monitoring and controlling the vehicle 10.

[0041] The system 16 includes a controller 18 configured to generate control signals for controlling the vehicle 10 based on various inputs, such as foot pedals, control knobs or buttons, touch input devices, or other human-machine interfaces in general. The controller 18 is further configured to generate control signals based on sensor inputs.

[0042] The system 16 includes at least one sensor 20 arranged to output sensor data about the operator 14. Some sensors 20 may be image sensors 22 that output image data. Some image sensors 22 may be arranged to view the operator 14 from different angles. Some image sensors 22 may record image data in different colors or wavelengths. Some image sensors 22 may record image data in the optical spectrum and with the typical red, green, blue (RGB) color scheme. Some image sensors 22 may record image data in the infrared (IR) spectrum. Some sensors 20 may be configured as depth sensors 24 configured to estimate distance data indicating a distance of the operator 14 or their body parts from the depth sensor 23.

[0043] The system 16, preferably the controller 18, includes an action recognition module 24. The action recognition module 24 is configured to receive sensor data from at least one sensor 20 associated with various modalities. As used herein, a modality is the classification of a single independent input channel, such as the sensor data from sensor 20, between the system 16 and the operator 14. Such channels may differ due to their sensory nature (e.g., RGB, infrared, depth) or due to other significant differences in processing (e.g., image vs. non-image). The system 16 is configured as a multimodal system, preferably having overlapping modalities. Examples of modalities include a viewpoint of the person to be monitored, a color channel, an infrared channel, and a depth channel.

[0044] As in Fig. As shown in Figure 2, the action recognition module 24 is configured to receive sensor data 26 originating from the sensor 20. The first sensor data 28 may be associated with a first modality, such as RGB image data, and the second sensor data 30 may be associated with a second modality, such as IR image data that is different from the first modality. The first and second sensor data 28, 30 each include a sequence of images 32 constituting a video of the operator 14. At least some images 32 depict the operator 14 at least partially.

[0045] The sensor data 26 is fed into a first feature learning model in a separate stage 34. The first feature learning model in a separate stage 34 comprises a first modality stage 36 and a second modality stage 38, wherein the first modality stage 36 processes the first sensor data 28 and the second modality stage 38 processes the second sensor data 30. The first and second modality stages 36, 38 are preferably designed as feature extractors. The first and second modality stages 36, 38 are preferably configured as a residual convolutional network, e.g., ResNet50 by He et al., "Deep Residual Learning for Image Recognition," available at https: / / doi.org / 10.48550 / arXiv.1512.03385

[0046] The output of the first feature learning model in separate stage 34 is fed into a first feature shifting module 40. The feature shifting module 40 is configured to perform feature shifting along the modality dimension and the temporal dimension, which will be explained in more detail below.

[0047] The output of the first feature shifting module 40 is fed into a second feature learning model in a separate stage 42. The second feature learning model in a separate stage 42 has essentially the same structure as the first feature learning model in a separate stage 34, with separate third and fourth modality stages 44, 46. These modality stages 44, 46 can in turn be configured in the same way as the first and second modality stages 36, 38.

[0048] The output of the second feature learning model in separate stage 42 is fed into a second feature shifting module 48, which functions like the first feature shifting module 40.

[0049] The output of the second feature shifting module 48 is fed into a first feature learning model in shared stage 50. The first feature learning model in shared stage 50 processes the data associated with different modalities by using the same convolutional layer parameters for different modalities. The first feature learning model in shared stage 50 thus has common weights for all data. The first feature learning model in shared stage 50 is also preferably configured as a residual convolutional network, such as ResNet50.

[0050] The output of the first feature learning model in common stage 50 is fed into a third feature shifting module 52, which functions like the previously described feature shifting modules.

[0051] The output of the third feature shifting module 52 is fed into a second feature learning model in common stage 54, which functions in the same way as the first feature learning model in common stage 50.

[0052] The output of the second feature learning model in common stage 54 is fed into a fourth feature shifting module 56, which functions like the previously described feature shifting modules.

[0053] The output of the fourth feature shifting module 56 is fed into a third separate stage feature learning model 58, which functions like the previously described joint stage feature learning models.

[0054] The output of the third feature learning model in a separate stage 58 is fed into a late fusion module 60. The late fusion module 60 is configured to combine the data from all modalities into an ensemble feature expression using the average logit function. The late fusion module 60 is capable of generating feature expressions that indicate an action performed by operator 14.

[0055] The output of the late fusion module 60 is fed into an action classification module 62, which is configured as a classifier and outputs the most likely action performed by the operator 14. The action classification module 62 consists of a fully connected neural network trained for action classification.

[0056] The fully connected neural network comprises an input layer with a number of neurons equal to the output of the late fusion module 60. The input layer may be fully connected to a first hidden layer. The first hidden layer may contain more neurons than the input layer.

[0057] The first hidden layer can be fully connected to another hidden layer or to an output layer. Typically, the other hidden layer has the same number of neurons as the first layer or fewer. Between the other hidden layer and the output layer, there may be additional hidden layers, which generally contain the same number of neurons as the previous hidden layer or fewer.

[0058] The output layer contains a number of neurons equal to the number of action classes that can be recognized. Each neuron is therefore associated with an action. The neuron with the greatest activation typically indicates the most probable class of actions, i.e., the most probable action.

[0059] The most likely action can be processed by controller 18 to generate a control signal based on the most likely action. The control signal can initiate warning or emergency braking or other vehicle steering actions depending on the most likely action.

[0060] With reference to the Fig. 2 and Fig. 3, a method according to the invention is described. The sensor 20, e.g., the image sensor 22, records a video of the operator 14. The video is composed of an image sequence 32. Each image 32 is assigned a specific time stamp T iassigned. A first modality comprises those images 32 that were recorded with the image sensor 22 as an RGB sensor, and a second modality comprises those images 32 that were recorded with the image sensor 22 as an IR sensor.

[0061] In Fig. In Figure 3, the first modality is represented as the left branch and the second modality as the right branch. For brevity, the process for the first modality is described in more detail, and the process for the second modality is described only to the extent that it is not redundant.

[0062] The captured images 32 associated with the first modality are fed into the first feature learning model in a separate stage 34. The first feature learning model in a separate stage 34 performs a first feature extraction and outputs a sequence of first feature maps 64. The first feature maps 64 are separated by features, e.g., first to third features 66, 68, 70. The first feature maps 64 can indicate, for example, a position of the left and right hand and a position of the head.

[0063] The captured images 32 associated with the second modality, which differs from the first modality, are fed into the first feature learning model in a separate stage 34. The first feature learning model in a separate stage 34 performs a second feature extraction and outputs a sequence of second feature maps 72. The second feature maps 72 are separated by features and are generally complementary to the features of the first feature maps. The second feature maps 72 can, for example, indicate the position of the left and right hands and the position of the head, but are recorded in IR.

[0064] The first and second feature cards 64, 72 are fed into the first feature shifting module 40. The first feature shifting module 40 selects the same number of first and second feature cards 64, 72 having the same timestamp T iin the sequence and preferably follow one another. The number can be specified as constant or random during runtime. Preferably, less than 50%, more preferably equal to or less than one-third of the feature maps are selected. The selected first and second feature maps 64, 72 are swapped along the modality dimension, e.g., the RGB modality becomes the IR modality and vice versa. This results in modality-shifted first and second feature maps 74, 76. In the case of multiple modalities, it is possible to shift the selected feature maps along the modality dimension by a predetermined amount. After the swap, the sequences contain mixed modalities.

[0065] In a next step, the feature shifting module 40 shifts the modality-shifted feature maps 74, 76 along the time dimension by selecting a predetermined number of modality-shifted feature maps 74, 76 that indicate different features and shifting the selected modality-shifted feature maps 74, 76 by a predetermined amount along the time dimension. This results in first and second modality-time-shifted feature maps 78, 80. Preferably, only a minor portion, preferably less than 50%, more preferably equal to or less than one-third of the modality-shifted feature maps 74, 76 are selected. The first and second modality-time-shifted feature maps 78, 80 are output to the second feature learning model in a separate stage 42.

[0066] Further feature extractions are performed in the second feature learning model in separate stage 42, the first and second feature learning models in separate stages 50, 54, and the third feature learning model in separate stage 58.

[0067] Between these models 42, 50, 54, 58, a further modality-time feature shift is performed by the second to fourth feature shifting modules 48, 52, 56. These modules perform the same steps as the first feature shifting module 40.

[0068] The feature learning models in stages 34, 42, 50, 54, 58 perform a step b), which can be referred to as a feature extraction step or a feature learning step, while the feature shifting modules 40, 48, 52, 56 perform a step c), which can generally be referred to as a feature shifting step. As already described, the image sequence 32 is fed to the first feature learning model in a separate stage 34, which performs step b). Its output is fed to the first feature shifting module 40, which performs step c). Its output is fed to the second feature learning model in a separate stage 42, which in turn performs step b). Its output is fed to the second feature shifting module 48, which performs step c), and so on up to the fourth feature shifting module 56. Consequently, steps b) and c) are performed alternately.

[0069] The output of the fourth feature shifting module 56 is fed to the late fusion module 60. The late fusion module 60 receives feature maps indicative of the action performed by the operator 14. The late fusion module 60 combines the received feature maps into an ensemble feature map 82 based on the average logit function.

[0070] The ensemble feature map 82 is fed into the action classification module 62, which is trained for classification and predicts the action 84 performed by the operator 14 based on the ensemble feature map 82.

[0071] For example, if the operator 14 shows signs of microsleep, the system 16 detects the operator 14, recognizes the action as microsleep, and generates a control signal that triggers a warning to the operator 14 and can steer the vehicle 10 to a safe position on the side of the road.

[0072] System 16 is trained in the usual way with labeled data and performs backpropagation and optimization. System 16 is trained using ResNet-50, which is initialized with pre-trained ImageNet parameters. ImageNet is publicly available at https: / / www.image-net.org / , but requires registration. The input for each modality is a video clip uniformly sampled with a temporal step size of 8. Each image is randomly cropped and formatted to 224x224. System 16 is trained using four GPUs or an RTX 3090. Training is performed using stochastic gradient descent with an initial learning rate of 10 -4 The batch size per GPU is 8. Two commonly used metrics are used for performance evaluation: top-1 accuracy and balanced accuracy (BalAcc).

[0073] Further training is performed using the Drive&Act dataset, which is publicly available at https: / / driveandact.com / . This dataset is widely used for driver secondary activity detection and comprises 9.6 million images in three modalities (RGB, IR, and depth) and five views. There are three levels of activity labels: coarse-grained tasks, fine-grained activities, and atomic action units. System 16 was trained using the fine-grained activity labels with all three modalities from the top-right view. LIST OF REFERENCE SYMBOLS 10 vehicles 12 vehicle cabin 14 operators 16 systems 18 control units 20 sensors 22 image sensor 23 Depth sensor 24 Action recognition module 26 Sensor data 28 first sensor data 30 second sensor data 32 images 34 first feature learning model in separate stage 36 first modality level 38 second modality level 40 first feature-shifting module 42 second feature learning model in separate stage 44 third modality level 46 fourth modality level 48 second feature-shifting module 50 first feature learning model in common stage 52 third feature-shifting module 54 second feature learning model in joint stage 56 fourth feature-shifting module 58 third feature learning model in separate stage 60 Late Fusion Module 62 Action classification module 64 first feature cards 66 first feature 68 second feature 70 third feature 72 second feature cards 74 first feature card shifted by modality 76 Modality-shifted second feature map 78 first modality-time-shifted feature map 80 second modality time-shifted feature map 82 Ensemble feature card 84 predicted action QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2020 / 0 089 977 A1

[0005] Cited non-patent literature

[0000] K. Peng, et al, „TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration“, 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, S. 278-285

[0006] Ji Lin, Chuang Gan, and Song Han. TSM: temporal shift module for efficient video understanding. In 2019 IEEE / CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 7082-7092. IEEE, 2019

[0006] Alina Roitberg, Kunyu Peng, Zdravko Marinov, Constantin Seibold, David Schneider, and Rainer Stiefelhagen. A comparative analysis of decision-level fusion for multimodal driver behaviour understanding. 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1438-1444, 2022

[0006] Rim Trabelsi, Redouane Khemmar, Benoit Decoux, Jean-Yves Ertaud, and R'emi Boutteau. Recent advances in vision-based on-road behaviors understanding: A critical survey. Sensors, 22(7):2654-2681, 2022

[0006] Zachary Wharton, Ardhendu Behera, Yonghuai Liu, und Nikolaos Bessis. Coarse temporal attention network (cta-net) for driver's activity recognition. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1278-1288, 2021

[0006] He et al. „Deep Residual Learning for Image Recognition“, erhältlich unter https: / / doi.org / 10.48550 / arXiv.1512.03385

[0045]

Claims

[1] Computer-implemented method for detecting actions of a person to be monitored, preferably an operator (14) of a vehicle (10), the method comprising: a) capturing a plurality of sequences of images (32) with at least one image sensor (22), each sequence being recorded using a predetermined modality, at least some images (32) at least partially containing the person to be monitored, each pair of sequences being recorded using different modalities; b) passing each sequence through at least one feature learning model in one stage, which is trained for feature extraction from image data, to obtain at least one sequence of feature maps for each sequence of images (32), wherein each sequence of feature maps is composed of a plurality of feature maps and each feature map indicates features suitable for characterizing an action of the person to be monitored; c) passing each sequence of feature cards through at least one feature shifting module that performs a feature shift by selecting a number of feature cards along a dimension and shifting the selected feature cards along that dimension by a predetermined amount; d) combining the sequences of feature maps resulting from step c) to form an ensemble feature map (82); e) feeding the ensemble feature map (82) into an action classification module trained to determine an action performed by the person to be monitored based on the ensemble feature map (82), and determining the action most likely performed by the person to be monitored. [2] The method of claim 1, wherein the predetermined modality is selected from a group comprising a viewing angle of the person to be monitored, a color channel, an infrared channel, and a depth channel. [3] A method according to any one of the preceding claims, wherein step b) comprises passing each image sequence (32) and / or each sequence of feature maps through at least one second feature learning model in a separate stage comprising a first modality stage (36) and a second modality stage (38), wherein each sequence associated with a first modality is processed by the first modality stage (36) and each sequence associated with a second modality is processed by the second modality stage (38). [4] A method according to any one of the preceding claims, wherein step b) comprises passing each sequence of images (32) and / or sequence of feature maps through at least one common-stage feature learning model comprising shareable parameters for all modalities. [5] A method according to any one of the preceding claims, wherein step c) comprises selecting a number of feature cards representing the same timestamp. [6] A method according to any one of the preceding claims, wherein step c) comprises shifting the selected feature maps along a modality dimension by a predetermined amount or exchanging the selected feature maps between different modalities. [7] A method according to any one of the preceding claims, wherein step c) comprises shifting the selected feature maps along a temporal dimension by a predetermined amount. [8] The method of claim 7, wherein a portion of the selected feature maps is shifted forward along the temporal dimension and another portion of the selected feature maps is shifted backward along the temporal dimension. [9] Method according to one of the preceding claims, wherein step b) and / or step c) comprises selecting a number of image sequences (32) and / or feature maps along different features of a feature dimension. [10] Method according to one of the preceding claims, wherein steps b) and c) are carried out alternately for a predetermined number of iterations. [11] Control method for a vehicle (10), in which a method according to one of the preceding claims is carried out, and: f) generating a control signal for a vehicle (10) based on the most likely action determined in step e), and controlling the vehicle (10) in accordance with the control signal. [12] System (16) comprising means for carrying out the method according to any one of the preceding claims. [13] Vehicle (10), preferably a ground or land vehicle, with the system (16) according to claim 12, wherein the system (16) is designed to monitor an operator (14) of the vehicle (10) and / or to control the vehicle (10). [14] A computer program containing instructions for causing the system (16) of claim 12 or the vehicle (10) of claim 13 to carry out the method of any one of claims 1 to 11. [15] A computer-readable medium or a data carrier signal containing the computer program of claim 14.

Images