Escalator behavior detection method based on multi-branch human feature fusion network

Abstract

The present invention discloses an escalator behavior detection method based on a multi-branch human feature fusion network, which relates to the field of image processing technology. The method includes: using a multi-branch network containing joint flow, bone flow, velocity flow and bone angle flow to process the matrix to obtain joint, bone, velocity and angle feature matrices respectively, and fusing these features through a self-attention mechanism; using multi-scale temporal convolution to capture the dynamic changes of features in time and generate a temporal feature matrix; interacting with the temporal feature matrix through attention-weighted edge convolution to convert the result into a high-dimensional feature vector; inputting to a fully connected layer to map to the category space to predict the probability distribution of escalator behavior, and selecting the category with the highest probability as the final behavior prediction result. The present invention realizes the detection and recognition of pedestrian behavior on escalators by constructing a multi-branch human feature fusion network and combining the self-attention mechanism and multi-scale temporal convolution.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an escalator behavior detection method based on a multi-branch human feature fusion network. Background Art

[0002] Escalators are the most common means of transporting pedestrians in public places, used by people of all ages, from the elderly to infants in strollers. Therefore, escalator safety monitoring has always been a pressing public safety issue. While escalators improve passenger transportation efficiency, they also pose safety risks. In practice, escalators are prone to accidents such as falls, collisions, and crushing due to improper use by passengers, such as running too fast, walking against traffic, and children playing, as well as untimely emergency rescue. Common causes of escalator accidents are pedestrians climbing, falling, or walking against traffic.

[0003] However, in the narrow, long space of an escalator, there's high occlusion from all directions. Many human actions don't vary much, resulting in consistent spatial structure and a lack of significant geometric variation. Furthermore, temporal and spatial features are unevenly distributed, with some keyframes of the action being overly prominent while others are weaker or less distinct. This prevents the model from effectively capturing the full dynamics of the action.

[0004] Existing methods based on graph convolutional networks (GCNNs) can identify human behavior, but they still suffer from a high degree of misjudgment of similar spatial actions. The dynamic temporal changes of human motion are not fully captured, leading models to rely primarily on static spatial features for classification. Different joints or skeletal connections have varying importance in spatial dimensions. Some joints or bones contribute significantly to action recognition, while others may be redundant, impacting the efficiency and effectiveness of feature extraction. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes an escalator behavior detection method based on a multi-branch human feature fusion network. The multi-branch human feature fusion network is used to accurately identify the unsafe actions of passengers on escalators, and solves the problems of height occlusion, similarity of human spatial movements, and uneven distribution of temporal and spatial features from multiple dimensions of human features.

[0006] The specific plan is as follows:

[0007] Escalator behavior detection method based on multi-branch human feature fusion network, including:

[0008] S1, obtains the action video of a pedestrian on an escalator; extracts the coordinates of the human joint points in each frame of the action video through a human posture estimator; merges the coordinates of the human joint points in each frame into a coordinate sequence in time order, and converts the coordinate sequence into the original feature matrix;

[0009] S2, constructing a multi-branch human feature fusion network; the multi-branch human feature fusion network includes joint flow, skeleton flow, velocity flow, skeleton angle flow, attention mechanism fusion module and multiple graph convolution sub-blocks; the skeleton flow calculates the vector length and vector direction of the skeleton edge based on the original feature matrix to obtain the skeleton feature matrix; the velocity flow calculates the velocity vector of the human joint point and the skeleton edge based on the original feature matrix to obtain the velocity feature matrix; the skeleton angle flow calculates the limb angle and limb angular velocity based on the original feature matrix to obtain the skeleton angle feature matrix; each graph convolution sub-block includes graph convolution, multi-scale temporal convolution and attention-weighted edge convolution;

[0010] S3, captures the temporal dynamic changes of the original feature matrix, bone feature matrix, velocity feature matrix, and bone angle feature matrix through multi-scale temporal convolution to obtain the temporal feature matrix;

[0011] S4, performs attention weighting on the temporal feature matrix through attention-weighted edge convolution, and inputs the attention-weighted feature matrix into the global pooling layer to convert it into a high-dimensional feature vector;

[0012] S5, performs weighted fusion of high-dimensional feature vectors through the self-attention mechanism fusion module, and maps the fused high-dimensional feature vectors to the category space through the fully connected layer to calculate the probability distribution of escalator behavior in each category, and selects the category with the highest probability as the prediction result of escalator behavior.

[0013] Furthermore, in S2, the vector length and vector direction of the bone edge are calculated as follows:

[0014] Bone t,i =Joint t,i -Joint t,p ;

[0015] Magnitude t,i =||Bone t,i ||;

[0016]

[0017] Among them, Joint t,i Represents the coordinates of the i-th joint point in the t-th frame in the original feature matrix; Joint t,p Represents the parent node of the i-th joint point in the t-th frame in the original feature matrix in the bone topology graph; Bone t,iRepresents the vector length of the bone edge; Magnitude t,i Indicates that the bone amplitude is the norm of the vector length, Direction t,i The vector direction representing the bone edge.

[0018] Furthermore, in S2, the velocity vectors of the human joints and bone edges are calculated as follows:

[0019]

[0020] Among them, i∈{1,2,....,N} represents the joint number; t∈{1,2,....,T}, P i (t) represents the coordinate of joint point i at time t, T represents the total number of frames of the input video; Δt represents the time difference between adjacent frames; C i,j (t)∈{1,2,....,N} represents the coordinates of the center point of the bone segment at time t; N represents the total number of joint points and bone edge center points; v i (t) represents the velocity vector of human joints and bone edges.

[0021] Furthermore, in S2, the calculation formulas for limb angle and limb angular velocity are as follows:

[0022]

[0023] Among them, θ n,B represents the limb angle of the Bth bone edge in the nth frame, ω n,B Indicates the inter-frame limb angular velocity of the Bth skeleton edge in the nth frame; l n,B,z Indicates the component of the B-th bone edge in the positive direction of the z-axis in the n-th frame; l n,B,x Indicates the component of the B-th bone edge in the positive direction of the x-axis in the n-th frame; l n,B,y It represents the component of the B-th bone edge in the positive direction of the y-axis in the n-th frame; l represents the bone edge; t represents time.

[0024] Furthermore, in S4, the time series feature matrix is weighted by attention-weighted edge convolution, and the calculation formula is as follows:

[0025] e ij =MLP(H i ,H j -H i );

[0026]

[0027] EdgeConv(H,A)=∑ j∈N(i) φ(H i ,H j );

[0028]

[0029] Among them, MLP() represents a multi-layer perceptron; W represents a time series feature matrix; H l represents the feature matrix after attention weighting; e ij Indicates the features of key points i and j calculated based on the multi-layer perceptron MLP; α ij represents the weight coefficient of the dynamic attention mechanism; EdgeConv(H,A) represents the edge convolution; N(i) represents the neighbor set of node i; φ(H i ,H j ) represents the characteristic difference between node i and its neighbor node j; A represents the adjacency matrix; e ik represents the attention score of target node i and neighbor node k, reflecting the importance of neighbor node k to target node i; H i and H j Represents the output feature tensor o-th key point and j-th key point of the feature matrix H through graph convolution; represents element-wise multiplication; σ represents the activation function; b represents the bias.

[0030] Furthermore, in S5, the high-dimensional feature vectors are weightedly fused through the self-attention mechanism fusion module. The specific formula is as follows:

[0031] F fusion =Softmax(Q·K T )·V;

[0032] Q, K, and V represent the query matrix, key matrix, and value matrix obtained by linear transformation of the input feature stream;

[0033] Softmax(Q·K T ) represents the similarity weight distribution between different flows; F fusion Represents the weighted fusion feature.

[0034] The present invention adopts the above technical solution and has the following beneficial effects:

[0035] (1) The present invention effectively solves the problem of high occlusion in front, behind, left and right caused by the narrow and long space of the escalator and the problem of misjudgment caused by similar actions in the same spatial structure through the escalator behavior detection method based on a multi-branch human feature fusion network. Multi-stream fusion fully captures the static structure and dynamic changes of the action by combining the characteristic advantages of joints, bone segments, speed, and bone angles. The introduction of speed flow allows the motion characteristics of joints and bone segments to be expressed and the time domain characteristics can be fully utilized. The introduction of bone angle flow focuses on the geometric relationship between bones, which complements the feature flow based on position or distance of joint flow and bone flow, and can effectively enhance the ability to distinguish complex actions.

[0036] (2) The present invention constructs a multi-scale temporal convolution module (MSTCN) and implements different expansion rates to construct convolutions of multiple scales to capture information of different time spans. The multi-scale temporal convolution perceives the logic of the behavior by combining the information of the adjacent frames and restores the action trajectory of certain fragment frames during occlusion.

[0037] (3) The present invention constructs an attention-weighted edge convolution module (AEC) and introduces a dynamic attention mechanism based on edge convolution to enhance the overall network's capture of important spatial information and ignore secondary information, thereby screening out important features and improving the accuracy of unsafe action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of an escalator behavior detection method based on a multi-branch human feature fusion network according to an embodiment of the present invention;

[0039] Figure 2 This is a working architecture diagram of a system applied to an escalator according to an embodiment of the present invention;

[0040] Figure 3 This is a structural diagram of a graph convolution sub-block in a multi-branch human feature fusion network according to an embodiment of the present invention;

[0041] Figure 4 This is a diagram of the multi-scale temporal convolution structure of an embodiment of the present invention;

[0042] Figure 5 This is a diagram of the attention-weighted edge convolution structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figure 1 As shown, the escalator behavior detection method based on a multi-branch human feature fusion network of the present invention includes:

[0045] S1, obtains the action video of pedestrians on the escalator; extracts the coordinates of the human joint points in each frame of the action video through the human posture estimator; merges the coordinates of the human joint points in each frame into a coordinate sequence in time order, and converts the coordinate sequence into the original feature matrix.

[0046] Specifically, in this embodiment, the input to the network is the key point coordinates (x, y) of each person in each frame of the video, and these coordinates are used to construct joint flow, bone flow, velocity flow and bone angle flow to solve the problem of misjudgment of similar spatial actions, and use multi-scale temporal convolution modules and attention-weighted edge convolution modules to improve the graph convolution network to solve the problem of uneven distribution of spatiotemporal features; specifically, the configuration obtains escalator monitoring video (30FPS), segments the video into short videos of 10s every 300 frames, and extracts the human body key point coordinates (x, y) of each person in each frame of the video to form a text sequence in chronological order to input the network for training.

[0047] Specifically, such as Figure 2 As shown, in this embodiment, a working architecture diagram of a system applied to escalators is used, video is collected by a camera, and after preprocessing, features are extracted using a network model including joint flow, skeleton flow, velocity flow and skeleton angle flow, and then the features are integrated through a self-attention mechanism fusion module to finally determine and alarm for unsafe behavior.

[0048] S2, construct a multi-branch human feature fusion network; the multi-branch human feature fusion network includes joint flow, skeleton flow, velocity flow, skeleton angle flow, attention mechanism fusion module and multiple graph convolution sub-blocks; the skeleton flow calculates the vector length and vector direction of the skeleton edge based on the original feature matrix to obtain the skeleton feature matrix; the velocity flow calculates the velocity vector of the human joint point and skeleton edge based on the original feature matrix to obtain the velocity feature matrix; the skeleton angle flow calculates the limb angle and limb angular velocity based on the original feature matrix to obtain the skeleton angle feature matrix; each graph convolution sub-block includes graph convolution, multi-scale temporal convolution and attention-weighted edge convolution.

[0049] Specifically, the vector length and vector direction of the bone edge are calculated as follows:

[0050] Bone t,i =Joint t,i -Joint t,p ;

[0051] Magnitude t,i =||Bone t,i ||;

[0052]

[0053] Among them, Jointt,i Represents the coordinates of the i-th joint point in the t-th frame in the original feature matrix; Joint t,p Represents the parent node of the i-th joint point in the t-th frame in the original feature matrix in the bone topology graph; Bone t,i Represents the vector length of the bone edge; Magnitude t,i Indicates that the bone amplitude is the norm of the vector length, Direction t,i The vector direction representing the bone edge.

[0054] Specifically, the velocity vectors of human joints and bone edges are calculated as follows:

[0055]

[0056] Among them, i∈{1,2,....,N} represents the joint number; t∈{1,2,....,T}, P i (t) represents the coordinate of joint point i at time t, T represents the total number of frames of the input video; Δt represents the time difference between adjacent frames; C i,j (t)∈{1,2,....,N} represents the coordinates of the center point of the bone segment at time t; N represents the total number of joint points and bone edge center points; v i (t) represents the velocity vector of human joints and bone edges.

[0057] Specifically, the calculation formulas for limb angle and limb angular velocity are as follows:

[0058]

[0059] Among them, θ n,B represents the limb angle of the Bth bone edge in the nth frame, ω n,B Indicates the inter-frame limb angular velocity of the Bth skeleton edge in the nth frame; l n,B,z Indicates the component of the B-th bone edge in the positive direction of the z-axis in the n-th frame; l n,B,x Indicates the component of the B-th bone edge in the positive direction of the x-axis in the n-th frame; l n,B,y It represents the component of the B-th bone edge in the positive direction of the y-axis in the n-th frame; l represents the bone edge; t represents time.

[0060] Specifically, such as Figure 3 As shown in the figure, the joint stream (Joint Stream), bone stream (BoneStream), velocity stream (Velocity Stream) and bone angle stream (Angle Stream) in the embodiment of the present invention are used. Figure 3The improved graph convolution block shown in the figure is used to recognize human behavior. In the figure, the 1×1 convolution layer represents feature mapping, which linearly combines the input feature matrix to generate a new feature representation; the GCN graph convolution is used to aggregate the features of adjacent nodes and adjacent bone edges; the BN layer represents batch normalization, which normalizes each small batch of data to stabilize the training process, accelerate convergence, and achieve regularization effects. The ReLU layer represents the use of the ReLU activation function, which enables the neural network to learn and represent complex nonlinear relationships through simple nonlinear transformations. The GAP layer converts the 4-dimensional feature matrix into a high-dimensional feature matrix; the fully connected layer maps the high-dimensional feature matrix to the category space Softmax layer; the Softmax layer converts the category score into a probability distribution. The output layer selects the category with the highest probability as the prediction result of the human behavior based on the probability distribution output by Softmax.

[0061] S3, through multi-scale temporal convolution, captures the dynamic changes of the original feature matrix, bone feature matrix, velocity feature matrix and bone angle feature matrix in time to obtain the temporal feature matrix.

[0062] Specifically, in this embodiment, the feature extraction part of the multi-scale temporal convolution module includes six convolution branches: a max pooling branch and five one-dimensional convolution branches with dilation rates ranging from 1 to 5 and a convolution kernel of 3x1; first, a dynamic channel allocation module is constructed to perform dynamic channel allocation on each branch, and the number of remaining channels is calculated to ensure that the channel allocation of each branch in the network is reasonable. The number of intermediate channels is obtained by dividing the number of output channels by the total number of branches. The first branch is allocated to the remaining number of channels, and the remaining 5 branches are allocated to the intermediate number of channels. The features are input into the dynamic channel allocation module, divided into six groups from the channel dimension, and input into each convolution branch respectively.

[0063] In order to reduce the increase in computational complexity caused by the six branches, a residual bottleneck structure is added before the max pooling branch and the five one-dimensional convolution branches to reduce the number of channels. The basic number of channels is 64, which is reduced to 32 channels after the bottleneck structure.

[0064] The input skeleton data feature map is x i ∈R n×C×T×V , according to x1,x2,…x t To predict the results, we get the feature maps y1, y2, ...y t , as shown below:

[0065]

[0066] y t is the value of the output sequence at time step t, w iRepresents the weight of the i-th convolution kernel, k represents the length of the convolution kernel, which determines the receptive field of the convolution kernel. The feature maps y1, y2, y3, y4, y5 output by the five branches and the maximum pooling branch are spliced in the channel dimension to form a high-dimensional feature matrix C. At the same time, the spliced channels are mapped to the final number of output channels. Finally, the residual network resnet50 is used to perform high-level feature fusion C on the spliced features. L .

[0067] C=concat(y1,y2,y3,y4,y5,y max )

[0068] C L =W resnet (C)

[0069] Where L represents the number of residual blocks, W resnet Represents the weights and biases in the residual network.

[0070] Specifically, a multi-scale temporal convolution module (MSTCN) is embedded in each graph convolution block in the multi-branch human feature fusion network. Different convolution kernel sizes can extract features at different scales, and different convolution kernel sizes and dilation rates are used to process features at different time scales. The feature extraction part of the multi-scale temporal convolution consists of six convolution branches, one max pooling branch, and five one-dimensional convolution branches with 3x1 convolution kernels and dilation rates ranging from 1 to 5. It includes the following parts:

[0071] Dilation rate 1: standard 3×1 convolution with a receptive field of 3, focusing on local details.

[0072] Dilation rate 2: 3×1 convolution with an interval of 1 pixel and a receptive field of 5 to capture features in a slightly larger range.

[0073] Dilation rate 3: 3×1 convolution with a spacing of 2 pixels and a receptive field of 7.

[0074] Dilation rate 4: 3×1 convolution with a spacing of 3 pixels and a receptive field of 9.

[0075] Dilation rate 5: 3×1 convolution with an interval of 4 pixels and a receptive field of 11 to capture global information in a longer range.

[0076] like Figure 4The figure shows the multi-scale temporal convolution structure of this embodiment. The bottleneck structure in the figure represents the use of 1×1 convolution kernel to reduce the number of channels; the maximum pooling layer is used to reduce the spatial size of the data and reduce the computational complexity; the activation layer uses the ReLU activation function, which enables the neural network to learn and represent complex nonlinear relationships through simple nonlinear transformations; the dilated convolution is composed of 5 branches, each consisting of a 3×1 convolution kernel with an expansion rate of 1 to 5; the maximum pooling branch consists of a bottleneck mechanism, a maximum pooling layer, and an activation layer; the convolution branches with different expansion rates are composed of a bottleneck structure, a dilated convolution, and an activation layer; the input feature matrix: the channel allocation module re-allocates the channels according to the six convolution branches to obtain six feature matrices; the output feature matrix: after passing through the convolution layer, y1, y2, y3, y4, y5, y max Feature matrix; use the residual network resnet50 to perform high-level feature fusion C on the spliced features L Output.

[0077] S4, the time series feature matrix is attention-weighted through attention-weighted edge convolution, and the attention-weighted feature matrix is input into the global pooling layer to be converted into a high-dimensional feature vector.

[0078] Specifically, the time series feature matrix is weighted by attention-weighted edge convolution, and the calculation formula is as follows:

[0079] e ij =MLP(H i ,H j -H i );

[0080]

[0081] EdgeConv(H,A)=∑ j∈N(i) φ(H i ,H j );

[0082]

[0083] Among them, MLP() represents a multi-layer perceptron; W represents a time series feature matrix; H l represents the feature matrix after attention weighting; e ij Indicates the features of key points i and j calculated based on the multi-layer perceptron MLP; α ij represents the weight coefficient of the dynamic attention mechanism; EdgeConv(H,A) represents the edge convolution; N(i) represents the neighbor set of node i; φ(H i ,H j ) represents the characteristic difference between node i and its neighbor node j; A represents the adjacency matrix; e ikRepresents the attention score of target node i and neighbor node k, reflecting the importance of neighbor node k to target node i. i and H j Represents the output feature tensor o-th key point and j-th key point of the feature matrix H through graph convolution; represents element-wise multiplication; σ represents the activation function; b represents the bias.

[0084] In this embodiment, it is assumed that the MLP consists of L layers, and each layer performs an affine transformation and a nonlinear activation function. Then the MLP is defined as:

[0085] MLP(x)=W L σ(W L-1 σ(W L-2 +b L )…σ(W1x+b1)+b L-1 )+b L ;

[0086] W L represents the weight matrix of the Lth layer, b L represents the bias vector of the Lth layer, σ is the nonlinear activation function, x is the input vector, x=(H i ,H j -H i ).

[0087] Specifically, an attention-weighted edge convolution module (AEC) is embedded in the graph convolution block of the multi-branch human feature fusion network. By default, the edge convolution contributes equally to neighboring nodes, while the introduction of the dynamic attention mechanism can assign weights to different neighboring nodes, thereby focusing on more important edge information, such as Figure 5 As shown in the figure, the role of the adjacency matrix A is to define the graph structure and calculate the feature difference with the input feature matrix; MLP convolution represents the multi-layer perceptron used to enhance the feature expression ability; Edge convolution (EdgeConv): calculates the feature difference between the adjacency matrix and the input feature matrix; the batch normalization layer normalizes each small batch of data to stabilize the training process, accelerate convergence and regularize the effect; the role of the maximum pooling layer is to reduce the spatial size of the data and reduce the computational complexity; dynamic attention: dynamically adjusts the weights of the nodes in the feature matrix.

[0088] S5, performs weighted fusion of high-dimensional feature vectors through the self-attention mechanism fusion module, and maps the fused high-dimensional feature vectors to the category space through the fully connected layer to calculate the probability distribution of escalator behavior in each category, and selects the category with the highest probability as the prediction result of escalator behavior.

[0089] Specifically, the self-attention mechanism fusion module performs weighted fusion on high-dimensional feature vectors. The specific formula is as follows:

[0090] F fusion =Softmax(Q·K T )·V;

[0091] Q, K, and V represent the query matrix, key matrix, and value matrix obtained by linear transformation of the input feature stream; Softmax(Q·K T ) represents the similarity weight distribution between different flows; F fuwion Represents the weighted fusion feature.

[0092] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A method for detecting escalator behavior based on a multi-branch human feature fusion network, characterized in that: include: S1, obtains the motion video of pedestrians on the escalator; The human body joint coordinates of each frame in the action video are extracted through the human pose estimator; the human body joint coordinates of each frame are merged into a coordinate sequence in time order, and the coordinate sequence is converted into the original feature matrix; S2, constructing a multi-branch human feature fusion network; the multi-branch human feature fusion network includes joint flow, skeleton flow, velocity flow, skeleton angle flow, attention mechanism fusion module and multiple graph convolution sub-blocks; the skeleton flow calculates the vector length and vector direction of the skeleton edge based on the original feature matrix to obtain the skeleton feature matrix; the velocity flow calculates the velocity vector of the human joint point and the skeleton edge based on the original feature matrix to obtain the velocity feature matrix; the skeleton angle flow calculates the limb angle and limb angular velocity based on the original feature matrix to obtain the skeleton angle feature matrix; each graph convolution sub-block includes graph convolution, multi-scale temporal convolution and attention-weighted edge convolution; S3, captures the temporal dynamic changes of the original feature matrix, bone feature matrix, velocity feature matrix, and bone angle feature matrix through multi-scale temporal convolution to obtain the temporal feature matrix; S4, performs attention weighting on the temporal feature matrix through attention-weighted edge convolution, and inputs the attention-weighted feature matrix into the global pooling layer to convert it into a high-dimensional feature vector; S5: The self-attention fusion module performs weighted fusion on the high-dimensional feature vectors. The fully connected layer maps the fused high-dimensional feature vectors to the category space to calculate the probability distribution of each category of escalator behavior. The category with the highest probability is selected as the escalator behavior prediction result. The formulas for calculating limb angle and limb angular velocity are as follows: Among them, θ n,B represents the limb angle of the Bth bone edge in the nth frame, ω n,B Indicates the inter-frame limb angular velocity of the Bth skeleton edge in the nth frame; l n,B,z Indicates the component of the B-th bone edge in the positive direction of the z-axis in the n-th frame; l n,B,x Indicates the component of the B-th bone edge in the positive direction of the x-axis in the n-th frame; l n,B,y It represents the component of the B-th bone edge in the positive direction of the y-axis in the n-th frame; l represents the bone edge; t represents time; The time series feature matrix is weighted by attention-weighted edge convolution. The calculation formula is as follows: e ij =MLP(H i ,H j -H i ); EdgeConv(H,A)=∑ j∈N(i) φ(H i ,H j ); Where MLP() represents a multi-layer perceptron; W represents a time series feature matrix; H l represents the feature matrix after attention weighting; e ij Indicates the features of key points i and j calculated based on the multi-layer perceptron MLP; α ij represents the weight coefficient of the dynamic attention mechanism; EdgeConv(H,A) represents the edge convolution; N(i) represents the neighbor set of node i; φ(H i ,H j ) represents the characteristic difference between node i and its neighbor node j; A represents the adjacency matrix; e ik represents the attention score of target node i and neighbor node k, reflecting the importance of neighbor node k to target node i; H i and H j Represents the output feature tensor i-th key point and j-th key point of the feature matrix H through graph convolution; represents element-wise multiplication; σ represents the activation function; b represents the bias.

2. The escalator behavior detection method based on a multi-branch human feature fusion network according to claim 1 is characterized in that: In S2, the vector length and vector direction of the bone edge are calculated as follows: Bone t,i =Joint t,i -Joint t,p ; Magnitude t,i =||Bone t,i ||; Among them, Joint t,i Represents the coordinates of the i-th joint point in the t-th frame in the original feature matrix; Joint t,p Represents the parent node of the i-th joint point in the t-th frame in the original feature matrix in the bone topology graph; Bone t,i Represents the vector length of the bone edge; Magnitude t,i Indicates that the bone amplitude is the norm of the vector length, Direction t,i The vector direction representing the bone edge.

3. The escalator behavior detection method based on a multi-branch human feature fusion network according to claim 1 is characterized in that: In S2, the velocity vectors of human joints and bone edges are calculated as follows: Among them, i∈{1,2,....,N} represents the joint number; t∈{1,2,....,T}, P i (t) represents the coordinate of joint point i at time t, T represents the total number of frames of the input video; Δt represents the time difference between adjacent frames; C i,j (t) represents the coordinates of the center point of the bone segment at time t; N represents the total number of joint points and bone edge center points; v i (t) represents the velocity vector of human joints and bone edges.

4. The escalator behavior detection method based on a multi-branch human feature fusion network according to claim 1 is characterized in that: In S5, the self-attention mechanism fusion module performs weighted fusion on the high-dimensional feature vectors. The specific formula is as follows: F fusion =Softmax(Q·K T )·V; Q, K, and V represent the query matrix, key matrix, and value matrix obtained by linear transformation of the input feature stream; Softmax(Q·K T ) represents the similarity weight distribution between different flows; F fusion Represents the weighted fusion feature.

Images