Escalator falling action recognition method based on deep neural network feature extraction and fusion

Abstract

The application discloses a kind of based on deep neural network feature extraction and fusion's escalator fall action recognition method, according to the action video of target person and skeleton point determine the action class of person.RGB video data of the action of target person on escalator is collected, and time feature, spatial feature is extracted in real time by multiple backbone neural networks and the skeleton point information generated by GCN is extracted by OpenPose tool;Multi-modal fusion module is constructed;Output action classification result, judge whether target person falls.The advantages are: using the feature extraction capability of multiple neural networks, can more quickly and more accurately process the RGB video data of person action, so as to realize more efficient action recognition.Through multi-modal feature fusion, the model can understand space-time features, making it more accurate in identifying actions.It can identify similar skeletal actions that are difficult to determine action categories when using skeletal point information alone, thus better avoiding misjudgment of action categories.The algorithm allows the model to predict human actions in real time, which enables the model to be applied to escalator scenarios while maintaining high accuracy and adaptability.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and action classification, specifically to multimodal fusion based on graphs and videos and action classification based on deep learning. Technical Background

[0002] Human motion recognition is one of the representative tasks of video understanding. Over the past decade, thanks to the emergence of deep learning, we have witnessed tremendous progress in video motion recognition. Human motion recognition is an active research area in computer vision, with wide-ranging practical applications in healthcare and rehabilitation, interactive entertainment, and video understanding. Motion recognition can be used in security monitoring systems to identify abnormal behavior or suspicious activity. For example, in surveillance cameras, deep learning can be used to detect and identify abnormal behaviors such as intrusion, theft, or violence. Motion recognition is also crucial in the health sector. It can be used to monitor patients' postures, gaits, and movements, helping medical professionals monitor patients' rehabilitation progress or the daily activities of the elderly to provide better medical care and support. In industrial automation, motion recognition can be used to monitor the movements of robots, robotic arms, and other automated equipment to ensure their proper functioning, improving productivity and safety. Advances in human skeleton detection technology have enabled computers to undertake and easily retrieve skeletal features, resulting in relatively sparse and more heterogeneous data patterns compared to existing RGB patterns. Human motion recognition has recently witnessed significant improvements in unimodal methods, such as skeleton-based methods and RGB video-based methods. For example, skeleton-based methods use graph convolutional models to represent the spatiotemporal features of skeletal joints and bones; these methods improve performance by aggregating results from similar inputs (i.e., skeletal joints and bones). Similarly, methods using RGB video input are designed to model the representation of spatiotemporal features in RGB video and the optical flow extracted from such videos.

[0003] However, single-modal methods using skeleton or RGB modes encounter obstacles. A major limitation of methods involving RGB video input is the lack of 3D structure. Skeleton-based methods are also limited by the lack of texture and appearance features. In action scenarios with similar skeletal movements, such as "reading" and "writing," "typing" and "writing," or "pointing at something" and "patting someone's back," it is difficult to distinguish them using methods based solely on skeletal point information. Fall detection is crucial in escalator scenarios. While escalator safety is currently relatively guaranteed, pedestrian falls still occur occasionally. If a fall is not detected and dealt with promptly, it can lead to serious accidents such as subsequent collisions and trampling. Introducing a multimodal fusion mechanism into a deep neural network to build an efficient action recognition model is highly feasible in this scenario. This model can achieve fine-grained action recognition, distinguish action categories, and accurately identify pedestrian fall actions under complex backgrounds and noise conditions, overcoming the shortcomings of single-modal methods. By fusing multiple modal features, such as combining skeletal information and appearance features, the recognition accuracy is greatly improved, effectively ensuring pedestrian safety on escalators, reducing accident risks, and possessing broad market application prospects and social benefits.

[0004] In summary, it is necessary to establish an efficient action recognition model, introduce a multimodal fusion mechanism into a deep neural network to achieve fine-grained action recognition and distinguish action categories; then, multiple improved deep neural network models will be used to further extract more features, construct multimodal feature fusion, and improve the accuracy of action recognition under complex backgrounds and noisy conditions. Summary of the Invention

[0005] Objective of the Invention: The technical problem to be solved by this invention is to overcome the shortcomings of existing technologies and provide an action recognition method based on multimodal fusion. This invention solves the problem of single-peak RGB video lacking 3D structural information, and the problem that single-peak skeletons are difficult to distinguish in action scenes with similar skeletal movements using methods based on skeletal point information.

[0006] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: a method for recognizing escalator fall actions based on deep neural network feature extraction and fusion, which ultimately includes the following steps:

[0007] Step 1: Process the acquired human motion video by sampling the RGB video sequence to reduce the large amount of RGB modal data. Perform temporal sampling on the video, selecting L representative frames. Specifically, for the video time interval [1, T], divide it into L intervals, and select one frame for each interval.

[0008] Step 2: Multimodal feature extraction; Extract the RGB image of each frame from the real-time acquired human motion video data, and use OpenPose to extract it into skeleton joints and skeleton bones, which are used as input to GCN (Graph Neural Network) to obtain features, and then perform RGB feature extraction through ShuffleNetV2.

[0009] Step 3: Multimodal feature fusion; Aggregate the features extracted in Step 2 from RGB video frames, skeleton joints, and skeleton bones.

[0010] Step 4: Output the action classification result. Based on the aggregated features in step 3, the probability of action classification is inferred, and the result with the highest probability is taken as the final output result.

[0011] Furthermore, the preprocessing of video data in step 1 involves first sampling the RGB video sequence to reduce the amount of RGB video data, and then converting the acquired RGB frame images into a uniform scale, i.e., image scale normalization.

[0012] Raw RGB frame data sampled from RGB video has a high pixel count, which impacts action recognition speed, reduces the real-time performance of the model's action classification output, and offers little improvement in accuracy. Therefore, the first step is to standardize the size of the RGB frame images to improve system response speed, enhance real-time performance, and facilitate action recognition.

[0013] Furthermore, the extraction of multimodal features in step 2 is carried out in the following specific steps:

[0014] S1: Extract RGB images from each frame of real-time acquired human motion video data, and use OpenPose to extract them into skeleton joints and skeleton bones, which are then used as input to GCN (Graph Neural Network).

[0015] S2: Train the GCN model using skeleton joint data.

[0016] S4: Construct spatiotemporal regions of interest from RGB modalities.

[0017] S5: Extract joint weights by inputting the skeletal joints into the trained dataset. Then, construct the spatiotemporal region of interest from the RGB modalities based on these joint weights.

[0018] S6: Use skeleton bones as input to the GCN model to obtain features.

[0019] Furthermore, the specific operations for constructing the spatiotemporal region of interest from the RGB modality in step S4 are as follows:

[0020] The RGB modalities of N video samples used for training are represented as V = {V(i)}. |.i =1,...,N}, then, an ordered video sequence of an action within the time interval [1,T] can be represented as in This is a frame at time t. To crop the spatial region of interest (ROI) from the motion video, skeleton joints retrieved using the OpenPose tool were used, which is more accurate than the skeleton retrieved by the Kinect v2 sensor. Given an RGB frame... Define a transformation function g to construct the spatial ROI of the joint, denoted as . Its formula expression is as follows:

[0021]

[0022] in It is the j-th joint of the OpenPose skeleton at time t. m1 to These are the M′0 indices of the OpenPose skeleton joints we are extracting, where M′0 is less than the total number of OpenPose skeleton joints M. o For a given We sample time, selecting L representative frames within the time interval τ = {interval × l | l = 1, ..., L, interval = T / L} and concatenating them into a square ST-ROI. The ST-ROI significantly reduces the amount of data input to the RGB video while preserving the appearance and motion information of the human body. At time t, the temporal sub-ROI will have M... o There are spatial sub-ROIs, which can be joined vertically, denoted as... Conversely, the spatial sub-ROI of the j-th joint will have L temporal sub-ROIs. They can be horizontally connected and represented as... V (i) The ST-ROI can be represented as R (i) It contains M o ×L sub-ST-ROIs, where a sub-ST-ROI is represented as...

[0023] Furthermore, in step S5, joint weights are extracted by inputting the skeleton joints into the trained dataset. Based on these joint weights, a spatiotemporal region of interest is further constructed from the RGB modality. The purpose of graph convolution is to learn the importance of the skeleton joints, thereby providing attention information to the RGB modality. The specific operations are as follows:

[0024] For a skeleton modality, the i-th training sample starts at time t=1 and ends at time T, collecting the skeleton of each frame at fixed time intervals, which can be represented as a sequence of T skeleton frames. We represent the corresponding skeletal sequence derived from skeletal joints as follows: Given M joints in a skeleton frame observed at time t, we represent them as follows: in It has C attributes. Then, we construct a spacetime graph to represent J. (i) Spatial and temporal structure. Skeleton input J (i) The skeleton diagram can be represented as G = (V, E), where V and E represent joints and bones, respectively.

[0025] To represent the sampling region of the convolution operation, node v ti The neighborhood set is defined as N(υ) ti )={υ tj |d(υ ti υ tj )≤D}, where D is d(υ)≤D. ti υ tj The maximum path length of the sampling region N(υ). ti The area is enclosed by a curve. Specifically, this sampling method uses three spatial subsets: the vertex itself; a centripetal subset containing neighboring vertices closer to the centroid; and an eccentric subset containing neighboring vertices farther from the centroid. The neighborhood set is then further divided into a fixed number of subsets, denoted as l. ti :N(v ti → {0,...,K-1}. By dividing the neighborhood set into different subsets, we can perform more refined classification and processing of a node's neighbors. This allows for better capture of the relationships and features between nodes, thereby improving the performance and effectiveness of graph convolution.

[0026] Specifically, different subsets can represent different neighborhood features or patterns. For example, a vertex itself represents the features of the node itself, while centripetal and centrifugal subsets can represent the features of neighboring nodes at different distances from the centroid. By processing these subsets separately, we can better adapt to different tasks and data characteristics.

[0027] Furthermore, a fixed number of subsets can make the model easier to learn and optimize, because the model can allocate and calculate weights according to the definition of these subsets, thereby better capturing information in the graph structure.

[0028] The formula for calculating graph convolution is as follows:

[0029]

[0030] Where f in (υ tj ) is used to obtain v tj Feature map of attribute vector, w(l(υ) tj )) is the weight function Z ti (υ tj The cardinality of the corresponding subset is used as a normalization term, and its functional expression is as follows:

[0031] Z ti (υ tj )=|υ tk |l ti (υ tk )=l ti (υ tj )|

[0032] The basic idea of ​​graph convolution is to update the feature representation of nodes through information transfer and aggregation between nodes. When graph convolution is applied to skeleton modalities, the output of each vertex in the graph can be used to infer the importance of the corresponding skeleton joint. Through such graph convolution operations, information from neighboring nodes can be aggregated to the current node, thereby learning the relationships and importance between nodes. This representation can effectively capture information in the graph structure and provide valuable features for subsequent model fusion and action recognition.

[0033] The feature map of the skeleton sequence can be represented by a tensor of dimension (C, T, M), where C represents the number of attributes of the joint vertices, T represents the time length, and M represents the number of vertices. This partitioning strategy can be represented by an adjacency matrix A, whose elements represent vertices v. tj Does it belong to N(υ)? ti A subset of ). Then, graph convolution can be performed by using classic two-dimensional convolution and multiplying the resulting tensor by a normalized adjacency matrix in the second dimension. To achieve this, there are K partitioning strategies. hour, The formula expression is as follows:

[0034]

[0035] Can be converted

[0036]

[0037] It is a diagonal matrix, W k Yes (C) in C out The weight tensor of a 1×1 convolution operation with dimensions M (1, 1). k This is an attention graph. ⊙ represents the element-wise product between two matrices. This is a tensor of size (c, t, M), where c is the number of output channels, t is the output duration, and M is the number of vertices. This tensor can be used to infer action categories and can be converted into joint weights, providing attention knowledge for RGB modalities. The formula for calculating joint weights representing the importance of their corresponding body regions is as follows:

[0038]

[0039] Where t and c are the output dimensions of the convolutional graph, representing the time length and output channels, respectively. (i) It is a vector containing the weights of M different skeletal joints.

[0040] Furthermore, the multimodal feature fusion in step 3 refers to aggregating the prediction results of steps S2, S5, and S6.

[0041] In this section, this invention will introduce the proposed deep learning architecture based on multimodal feature fusion from the perspective of sub-networks that learn features from skeleton and RGB modal features. The proposed action recognition method based on multimodal feature fusion consists of three independent networks built to learn feature representations from skeleton joints, skeleton bones, and RGB video input. The skeleton input is divided into skeleton joints and skeleton bones. Then, model-based feature fusion is performed between the ST-ROI constructed from the RGB video input using a GCN model and the joint weights learned from the skeleton joints. Given N training samples in the dataset, we represent the features of the i-th sample as {J}. (i) B (i) V (i)}, where J (i) It is the input of the skeletal joint, B (i) It is the skeleton input, V (i) It is an RGB video input; the corresponding action label is represented as y. (i) The goal is to learn feature extractors, including those with their respective parameters Θ. J Θ B and Θ V sub-model G J G B Together with GV, we infer action categories through an ensemble and fusion operation. This multimodal feature fusion operation can be written as the following function expression:

[0042]

[0043] Furthermore, step 4 involves classifying the actions, with the specific steps as follows:

[0044] This invention constructs the end-to-end format of the proposed multimodal model by summing a series of loss terms for action label supervision. Its representation is as follows:

[0045]

[0046] in and These are the loss terms for skeleton joints, skeleton bones, and RGB video input, respectively. The skeleton joint input is fed into... The graph convolutional model. Prediction of skeletal joints can be defined as... Where GJ represents graph convolution operation. Θ J This represents the learnable parameters of the GCN sub-model. J (i) These are data samples for the skeleton joint input. σ represents a linear layer that transforms the shape of the sub-model output into a one-hot representation. The skeleton input is essentially a transformation of the skeleton joint input. In the graph, the edge set is defined as ε = {(v...} ti -v tj )|v ti v tj =J tj Let A be a set of nodes, where t = 1, ..., T, i, j = 1, ..., M. This set includes all combinations of joint pairs represented in the adjacency matrix A. Based on the actual structure of the skeleton in a specific dataset, the functional relationship of the skeleton is expressed by the following formula:

[0047]

[0048] Follow the above conversion method to build the skeleton.

[0049] Since ST-ROI is essentially a 2D feature map, this invention uses ShuffleNetv2 to learn features from it. The one-hot formula for 2D features is expressed as follows:

[0050]

[0051] Where GV(R′) (i) Θ V ) represents the residual mapping to be learned, Θ V This represents the learnable parameters based on ShuffleNetv2. Based on the definition of the predictions from the aforementioned sub-models, the optimization problem is formulated according to the following objectives.

[0052]

[0053] in and These are cross-entropy losses, which are used to enhance the corresponding predictive power. Nc is the number of action categories in a specific dataset. N represents the number of samples in the training set. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method for recognizing escalator fall actions based on feature fusion extracted by deep neural networks according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the RGB feature extraction method according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the skeletal key point feature extraction method involved in the embodiments of the present invention. Detailed Implementation

[0057] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0058] A flowchart of an escalator fall action recognition method based on deep neural network feature extraction and fusion is shown below. Figure 1 As shown, the steps include:

[0059] Step 1: Import the dataset, using RGB video streams as the training dataset.

[0060] Step 2: Multimodal feature extraction. Using a dual encoder (ShuffleNetV2+GCN), RGB features are extracted separately. The RGB feature extraction process is as follows: Figure 2 As shown, the skeleton features, and the method flow for extracting skeleton features based on graph neural networks are as follows: Figure 3 As shown, these are the characteristics of these two modes.

[0061] Step 3: Multimodal features are fused by using an RGB feature encoder and a graph encoder that are jointly trained based on three types of interaction: RGB features, skeleton, and skeletal points.

[0062] Step 4: Use the trained classifier to predict the input data and obtain the final classification result. This part involves calculating the probability distribution of the input sample belonging to each category, and then selecting the category with the highest probability as the final classification result.

[0063] Step 5: Based on the classification results, determine if a fall has occurred, and issue an alarm if a fall has occurred.

[0064] 1. In step one above, the dataset is imported. In this invention, the NTU RGB+D60, NTURGB+D120, PKU-MMD, Northwestern-UCLA and Toyota Smarthome datasets are used during the training process.

[0065] 2. In step two above, multimodal feature extraction includes the following steps: RGB video sequences are sampled to construct spatiotemporal regions of interest (ST-ROI). This sampling strategy not only reduces the amount of large data in the RGB modalities but also makes it suitable for feature extraction using conventional convolutional neural network (CNN) models. Furthermore, this method, through random selection, handles the variability of the input data, thereby enhancing the system's flexibility and robustness.

[0066] 3. Step three above involves the following steps for fusing multimodal features. Given that this invention aims to acquire object and motion information from body regions including hands, feet, and head, we set M′ to 5 to construct the spatial sub-ST-ROI. In the temporal dimension, to effectively cover changes in appearance over time and avoid redundancy in appearance information due to excessively large L values, we set the parameter L to 5 based on past experience. This setting ensures both data validity and computational efficiency.

[0067] 4. In step five above, when obtaining the classification results, the trained classifier is first used to predict the input data, resulting in a probability distribution for each category (e.g., walking, running, falling, etc.). Based on this probability distribution, the category with the highest probability is determined. For example, if the classifier predicts that a certain video sequence has the highest probability of belonging to "falling," then it is initially considered that the video sequence contains the action of falling. To reduce false positives, a threshold is usually set. Only when the probability of a certain category exceeds this threshold is the category considered a correct classification result. For example, the threshold for the falling category is set to 0.8. If the probability of falling is greater than 0.8, it is confirmed as a falling action. In addition, contextual information can be combined for comprehensive judgment. For example, if several consecutive frames are predicted as falling, or if high-probability falling actions are frequently detected in a short period of time, the reliability of the judgment can be increased.

[0068] For the RGB modality, the height and width of the sub-ST-ROIs for action sequences in the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets are both 96 pixels. For the Northwestern-UCLA and Toyota Smarthome datasets, since the data was acquired using Kinect v1, the height and width of the sub-ST-ROIs are both 48 pixels. Therefore, the input sizes for the NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, Northwestern-UCLA, and Toyota Smarthome datasets are 480×480, 480×480, 480×480, 240×240, and 240×240, respectively. The ST-ROIs for all four datasets were resized to 225×225 and normalized before being input into ShuffleNetV2. Due to the relatively small data size of the Northwestern-UCLA and Toyota Smarthome datasets, we performed random selection on the RGB video frames and randomly flipped these frames. For all datasets, this invention employs an 18-layer ShuffleNetV2 model. For NTURGB+D 60, NTU RGB+D 120, and PKU-MMD, we uniformly select frames based on video length for training and testing. To further mitigate the smoothing effect of joint weights caused by using temporal position averages, we empirically select the top 15 points with the highest temporal position values ​​to calculate the joint weights, as shown in the following equation.

[0069]

[0070] This step helps improve the model's sensitivity and accuracy to key information.

[0071] In this implementation, we employ a stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.1. The learning rate is adjusted to one-tenth of its original value at the 10th and 50th epochs. Training terminates at the 80th epoch. Furthermore, we set the minibatch size to 64. This configuration ensures efficient utilization of computational resources and a stable training environment.

[0072] Accordingly, the present invention also provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described.

[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each step in the flowchart, and combinations of steps in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the steps in the flowchart. Figure 1 A device for a function specified by one or more processes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes specify the function.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0077] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing escalator fall actions based on feature extraction and fusion using deep neural networks. Its features include... Includes the following steps: Step 1: Process the acquired human motion video by sampling the RGB video sequence to reduce the amount of data in the RGB modal. Perform time sampling on the video, selecting L representative frames. Specifically, for the video time interval [1, T], divide it into L intervals, and select one frame for each interval. Step 2: Multimodal Feature Extraction; Extract RGB images from each frame of real-time acquired human motion video data, and use OpenPose to extract skeleton joints and skeleton bones, which are then used as input to a GCN (Graph Neural Network) to obtain features. RGB feature extraction is performed using ShuffleNetV2. Step 3: Multimodal feature fusion; The features extracted in step 2 from RGB video frames, skeleton joints, and skeleton bones are aggregated. Step 4: Output action classification results; Based on step 3, the probability of action classification is inferred through aggregated features, and the result with the highest probability is taken as the final output.

2. The escalator fall action recognition method based on deep neural network feature extraction and fusion according to claim 1, characterized in that... In step one, when importing data, the NTU RGB+D 60, NTURGB+D 120, PKU-MMD, Northwestern-UCLA, and Toyota Smarthome datasets can be used as commonly used video action datasets during training. For the RGB modality, the height and width of the sub-ST-ROIs of the action sequences in the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets are both 96 pixels. For the Northwestern-UCLA and Toyota Smarthome datasets, since the data acquisition uses Kinect v1, the height and width of the sub-ST-ROIs are both 48 pixels. Therefore, the input sizes of the NTURGB+D 60, NTU RGB+D 120, PKU-MMD, Northwestern-UCLA, and Toyota Smarthome datasets are 480×480, 480=480, 480=480, 240=240, and 240=240, respectively.

3. The action recognition method based on multimodal fusion according to claim 1, characterized in that... Step two, multimodal feature extraction, includes the following steps: (1) Represent the RGB modalities of the video samples with N for training as V = {V (i) |.i=1,...,N},then, an ordered video sequence of an action within the time interval [1,T] can be represented as in This is a frame at time t. To crop the spatial region of interest (ROI) from the motion video, skeleton joints retrieved using the OpenPose tool were used, which is more accurate than the skeleton retrieved by the Kinect v2 sensor. Given an RGB frame... Define a transformation function g to construct the spatial ROI of the joint, denoted as . Its formula expression is as follows: (2) Select a GCN graph encoder. Graph Convolutional Network (GCN) uses graph theory to perform convolution on a spatial topological graph and extract the spatial features of the graph. Specifically, it treats the human skeleton points and their connections as a graph and then uses the eigenvalues ​​and eigenvectors of the graph's adjacency matrix, degree matrix and Laplacian matrix to study the properties of the graph.

4. The multimodal feature fusion method for recognizing escalator fall actions based on feature fusion extracted from deep neural networks according to claim 1; aggregating the features extracted in step 2 through RGB video frames, skeleton joints, and skeleton bones. Its characteristics are as follows: We propose a spatial weighting mechanism for RGB frames to enable machines to focus on RGB features that provide discriminative information. More specifically, the machine will have stronger learning and recognition capabilities because it intuitively mimics human eye action recognition. The researchers aim to derive attention weights from the RGB modality itself. We choose to use joint weights from the skeleton modality and multiply them by the ST-ROI to regularize the RGB modality. The skeleton-focused ST-ROI of the i-th training sample can be obtained by deriving attention weights from R... (i) The mapping is obtained from the middle, where a function h is defined. Where w j It is the weight of the j-th joint. This represents the subspace ROI of the corresponding body region. And m1′,...,m M′ ′ is the index of the M′ different skeletal joints corresponding to the body regions we propose to focus on. The value of M′ is equal to the equation M′ O The value of .

5. The output action classification result of the escalator fall action recognition method based on feature fusion extracted from a deep neural network according to claim 1, based on step 3, infers the probability of action classification through aggregated features, and takes the one with the highest probability as the final output result. Step four, multimodal feature extraction includes the following steps: (1) Predict skeleton joint inputs using subnetworks: Input skeleton joint data into a graph convolutional model (such as ST-GCN), and obtain the prediction results after processing. Among them G J For graph convolution operations, Θ J J is a learnable parameter. (i) Here are the data samples, and here is a linear layer used to transform the output σ into a one-hot representation. (2) Skeleton and bone input: Skeleton and bone input is essentially a transformation of skeleton and joint input. It is constructed based on the actual structure of skeleton and bones in the dataset and uses the same graph convolution operation method as skeleton and joint input to obtain the prediction result. Among them G B For graph convolution operations, Θ B B is a learnable parameter. (i) The data sample is shown below. RGB video input: First, the RGB video is constructed as an ST-ROI, then input into ShuffleNetV2 to learn features. The prediction result is... Among them G V (R (i) ,Θ V ) represents the residual mapping to be learned, Θ V These are learnable parameters based on the number of ShuffleNetV2 layers. (3) Aggregate prediction results: Aggregate the prediction results of the three sub-networks to obtain the final classification result. For example, by summing or other fusion methods... These parameters are combined to determine the action category. During training, the model parameters are optimized by minimizing the objective function consisting of cross-entropy loss, making the predictions closer to the true labels. (4) Based on the definition of the prediction of the above sub-model, formulate the optimization problem according to the following objectives. in and These are cross-entropy losses, which are used to enhance the corresponding predictive power. Nc is the number of action categories in a specific dataset. N represents the number of samples in the training set.

Images