Escalator behavior recognition method and device, computer equipment and readable storage medium

Abstract

The invention relates to an escalator behavior recognition method and device, computer equipment and a readable storage medium. The method comprises the steps of obtaining video image data of a target escalator, obtaining an escalator key point set and a human body key point set containing a human body based on the video image data, obtaining hand-object interaction features based on the escalator key point set and the human body key point set, and obtaining human body posture features according to the human body key point set, inputting the hand-object interaction features and the human body posture features into a recognition model to obtain a hand-object interaction feature map corresponding to the hand-object interaction features and a human body posture feature map corresponding to the human body posture features, and inputting the hand-object interaction feature map and the human body posture feature map into a space-time interaction module in the recognition model to obtain a fusion feature map, and according to the hand-object interaction feature map, the human body posture feature map and the fusion feature map, obtaining an escalator behavior recognition result for the target escalator. By adopting the method, the accuracy of escalator use behavior recognition is improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent monitoring and human behavior recognition, and in particular to an escalator behavior recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] As an important means of transportation in public places, escalators are found in shopping malls, stations, overpasses and other public places. However, if some dangerous behaviors on escalators are not stopped in time, they can easily lead to safety accidents, causing serious social impact and personal injury.

[0003] Traditional methods rely solely on human posture or gait analysis to identify escalator usage behavior. They identify corresponding escalator usage behavior by analyzing human posture and issue warnings for dangerous behavior.

[0004] However, the current behavior recognition method used by this type of escalator has the problem of insufficient recognition accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide an escalator behavior recognition method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of escalator usage behavior recognition in response to the above technical problems.

[0006] In a first aspect, the present application provides an escalator behavior recognition method, comprising:

[0007] Obtaining video image data of a target escalator, and obtaining an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data;

[0008] Obtain hand-object interaction features based on the escalator key point set and the human body key point set, and obtain human posture features based on the human body key point set;

[0009] The hand-object interaction features and human posture features are input into a pre-built escalator behavior recognition model. The graph convolutional network in the escalator behavior recognition model is used to obtain a hand-object interaction feature map corresponding to the hand-object interaction features, and a human posture feature map corresponding to the human posture features.

[0010] The hand-object interaction feature map and the human posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map.

[0011] Based on the hand-object interaction feature map, human posture feature map and fusion feature map, the escalator behavior recognition result for the target escalator is obtained.

[0012] In conjunction with the first aspect, in one embodiment, the human body key point set includes a hand key point set and a torso key point set; and obtaining hand-object interaction features based on the escalator key point set and the human body key point set includes:

[0013] Determine the wrist key point from the hand key point set;

[0014] Get the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, each key point in the torso key point set, and the wrist key point;

[0015] The key points, as well as the distance and angle values ​​of each key point are spliced ​​together to obtain the hand-object interaction features corresponding to each key point.

[0016] In conjunction with the first aspect, in one embodiment, obtaining the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point includes:

[0017] Get the length between the head key point and the hip key point in the torso key point set, and use the length as the scaling factor;

[0018] Normalizing each key point in the escalator key point set, the hand key point set, and the torso key point set using a scaling factor to obtain normalized key points;

[0019] Get the normalized distance and angle between each key point and the wrist key point.

[0020] In conjunction with the first aspect, in one embodiment, obtaining a human body posture feature according to a set of human body key points includes:

[0021] Determine a source key point from the human body key point set; the source key point is a key point in the human body key point set that represents the center of gravity of the human body;

[0022] Obtain the difference between the source key point and each key point in the human body key point set, and determine the difference as the skeleton information feature;

[0023] Obtaining the first coordinate difference between two adjacent frames of each key point in the human body key point set, and determining the first coordinate difference as a motion information feature;

[0024] Obtaining the second coordinate difference of each key point in the human body key point set within two consecutive time periods, and obtaining the speed difference information feature based on the second coordinate difference;

[0025] Each key point, skeleton information feature, motion information feature and speed difference information feature in the human body key point set are spliced ​​and fused to obtain the human body posture feature corresponding to each key point in the human body key point set.

[0026] In conjunction with the first aspect, in an exemplary embodiment, the hand-object interaction feature map and the human posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map, including:

[0027] Perform average pooling operations on the hand-object interaction feature map and the human body posture feature map to obtain a pooled hand-object interaction feature map and a pooled human body posture feature map;

[0028] Obtain the first attention score corresponding to the pooled hand-object interaction feature map and the second attention score corresponding to the pooled human posture feature map;

[0029] A fused feature map is obtained based on the first attention score, the second attention score, the hand-object interaction feature map, and the human posture feature map.

[0030] In conjunction with the first aspect, in one embodiment, the first attention score includes a first temporal attention score and a first spatial attention score, and the second attention score includes a second temporal attention score and a second spatial attention score;

[0031] According to the first attention score, the second attention score, the hand-object interaction feature map and the human posture feature map, a fusion feature map is obtained, including:

[0032] The first spatial attention score and the second temporal attention score are multiplied by the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain the first attention feature map;

[0033] The first temporal attention score and the second spatial attention score are multiplied by the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain the second attention feature map;

[0034] Merge the first attention feature map and the second attention feature map to obtain a fused feature map.

[0035] In combination with the first aspect, in one embodiment, an escalator behavior recognition result for a target escalator is obtained based on the hand-object interaction feature map, the human posture feature map, and the fusion feature map, including:

[0036] Obtain the first escalator probability corresponding to the hand-object interaction feature map, the second escalator probability corresponding to the human body posture feature map, and the third escalator probability corresponding to the fusion feature map;

[0037] The product of the first escalator probability, the second escalator probability, and the third escalator probability is obtained, and the product is determined as the escalator behavior recognition result for the target escalator.

[0038] In a second aspect, the present application further provides an escalator behavior recognition device, comprising:

[0039] A set acquisition module is used to acquire video image data of a target escalator, and acquire an escalator key point set of the target escalator based on the video image data, as well as a human body key point set of a human body contained in the video image data;

[0040] A feature acquisition module, used to acquire hand-object interaction features based on the escalator key point set and the human body key point set, and to acquire human posture features based on the human body key point set;

[0041] The feature map acquisition module is used to input hand-object interaction features and human posture features into a pre-built escalator behavior recognition model. The graph convolutional network in the escalator behavior recognition model is used to obtain a hand-object interaction feature map corresponding to the hand-object interaction features, and a human posture feature map corresponding to the human posture features.

[0042] A fusion module is used to input the hand-object interaction feature map and the human posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map;

[0043] The behavior recognition module is used to obtain escalator behavior recognition results for the target escalator based on the hand-object interaction feature map, the human posture feature map and the fusion feature map.

[0044] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0045] Obtaining video image data of a target escalator, and obtaining an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data;

[0046] Obtain hand-object interaction features based on the escalator key point set and the human body key point set, and obtain human posture features based on the human body key point set;

[0047] The hand-object interaction features and human posture features are input into a pre-built escalator behavior recognition model. The graph convolutional network in the escalator behavior recognition model is used to obtain a hand-object interaction feature map corresponding to the hand-object interaction features, and a human posture feature map corresponding to the human posture features.

[0048] The hand-object interaction feature map and the human posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map.

[0049] Based on the hand-object interaction feature map, human posture feature map and fusion feature map, the escalator behavior recognition result for the target escalator is obtained.

[0050] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0051] Obtaining video image data of a target escalator, and obtaining an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data;

[0052] Obtain hand-object interaction features based on the escalator key point set and the human body key point set, and obtain human posture features based on the human body key point set;

[0053] The hand-object interaction features and human posture features are input into a pre-built escalator behavior recognition model. The graph convolutional network in the escalator behavior recognition model is used to obtain a hand-object interaction feature map corresponding to the hand-object interaction features, and a human posture feature map corresponding to the human posture features.

[0054] The hand-object interaction feature map and the human posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map.

[0055] Based on the hand-object interaction feature map, human posture feature map and fusion feature map, the escalator behavior recognition result for the target escalator is obtained.

[0056] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0057] Obtaining video image data of a target escalator, and obtaining an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data;

[0058] Obtain hand-object interaction features based on the escalator key point set and the human body key point set, and obtain human posture features based on the human body key point set;

[0059] The hand-object interaction features and human posture features are input into a pre-built escalator behavior recognition model. The graph convolutional network in the escalator behavior recognition model is used to obtain a hand-object interaction feature map corresponding to the hand-object interaction features, and a human posture feature map corresponding to the human posture features.

[0060] The hand-object interaction feature map and the human posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map.

[0061] Based on the hand-object interaction feature map, human posture feature map and fusion feature map, the escalator behavior recognition result for the target escalator is obtained.

[0062] The above-mentioned escalator behavior recognition method, device, computer equipment, computer-readable storage medium and computer program product obtain video image data of the target escalator, obtain the escalator key point set of the target escalator based on the video image data, and the human body key point set of the human body contained in the video image data, obtain hand-object interaction features based on the escalator key point set and the human body key point set, and obtain human body posture features based on the human body key point set, input the hand-object interaction features and the human body posture features into a pre-built escalator behavior recognition model, obtain a hand-object interaction feature map corresponding to the hand-object interaction features and a human body posture feature map corresponding to the human body posture features through the graph convolutional network in the escalator behavior recognition model, input the hand-object interaction feature map and the human body posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map, and obtain the escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human body posture feature map and the fused feature map. By introducing hand-object interaction recognition, the escalator behavior recognition model can more accurately capture whether the operator holds the handle when going up and down the escalator. Combining human posture recognition with hand-object interaction recognition, a multi-dimensional feature fusion method is adopted, and the spatiotemporal attention module is used to further capture the relationship between human posture features and hand-object interaction features at time and space points, thereby improving the accuracy of escalator usage behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A diagram showing an application environment of an escalator behavior recognition method in one embodiment;

[0065] Figure 2 1 is a flow chart of an escalator behavior recognition method according to an embodiment;

[0066] Figure 3 Schematic diagram of a flow chart of an escalator behavior recognition method in another embodiment;

[0067] Figure 4 A schematic diagram of extracting key points for workers getting on and off an escalator in one embodiment;

[0068] Figure 5 Schematic diagram of the key point spatiotemporal interaction attention mechanism structure in one embodiment;

[0069] Figure 6This is a structural block diagram of an escalator behavior recognition device in one embodiment;

[0070] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0072] Through observation, it was found that when the maintenance depot workers go up and down the escalator, the whole body posture is different when holding the handrail and when not holding the handrail. When going up the stairs, the workers' hands and legs will swing as they climb. When not holding the handrail, only the legs swing and the hands do not move (the hands are holding something). When going down the stairs, only the legs swing and the hands do not move, whether the workers hold the handrail or not. Based on this, the present invention uses human body posture recognition and hand-object interaction recognition methods to identify and detect whether the workers are holding the handrail when going up and down the stairs.

[0073] Traditional methods may rely solely on human posture or gait analysis to identify escalator usage behavior, but this method often ignores the key information of the interaction between the hands and the handrails. By introducing hand-object interaction recognition, the network can more accurately capture whether the operator holds the handle when going up and down the escalator, because the contact between the hand and the handrail is an important basis for judging whether they have taken safe behavior. Compared with the traditional single behavior recognition method, this application is based on the key point calculation of people and targets. By combining human posture recognition with hand-object interaction recognition methods, it adopts a multi-dimensional feature fusion method and proposes a key point spatiotemporal interaction attention module to further capture the relationship between human posture features and hand-object interaction features in time and space, thereby improving the accuracy and safety of escalator usage behavior recognition and making up for the shortcomings of traditional posture or gait analysis methods.

[0074] The escalator behavior recognition method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, a video monitor 102 communicates with a server 104 via a network. Video monitor 102 covers the range of a target escalator and monitors the operation of the target escalator. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed in the cloud or other network servers. Server 104 obtains video image data of a target escalator uploaded by video monitor 102, obtains an escalator key point set of the target escalator based on the video image data, and a human key point set of the human body contained in the video image data, obtains hand-object interaction features based on the escalator key point set and the human body key point set, and obtains human posture features based on the human body key point set, inputs the hand-object interaction features and the human body posture features into a pre-built escalator behavior recognition model, obtains a hand-object interaction feature map corresponding to the hand-object interaction features and a human body posture feature map corresponding to the human body posture features through a graph convolutional neural network in the escalator behavior recognition model, inputs the hand-object interaction feature map and the human body posture feature map into a spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map, and obtains an escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human body posture feature map, and the fused feature map. Server 104 can be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0075] In an exemplary embodiment, Figure 2 As shown, a method for identifying escalator behavior is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps S201 to S205.

[0076] Step S201 : obtaining video image data of a target escalator, and obtaining an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data.

[0077] Among them, the escalator key point set can be understood as the set of key points in the escalator handle area. The center of the handle is taken as the center point, and the points between the upper and lower endpoints to the center point are marked as basic key points. Finally, the key points are arranged in a logical order to obtain the key point set; the human body key point set can be understood as the set of points representing the human torso and hands.

[0078] Optionally, the server 104 receives the video image data of the target escalator uploaded by the video monitor 102, uses open source pose estimation to perform full-body pose estimation and key point extraction on the human body contained in the video image data, obtains a corresponding set of human key points, and accurately separates the escalator handle area from the video image data containing the target escalator through instance segmentation technology, and then determines the center of gravity of the handle as the center point, and marks the upper and lower endpoints and the points between the center point as basic key points; finally, arranges these key points in a logical order to construct a set of escalator key points that comprehensively characterizes the shape and position of the escalator handle.

[0079] Based on the aforementioned implementation, open-source pose estimation is used to estimate the full-body pose and extract key points of the human body contained in the video image data. Instance segmentation is then used to separate the escalator handrail area and determine the corresponding escalator key point set. This ensures the accuracy of the key points in both the human key point set and the escalator key point set, thereby improving the accuracy of escalator behavior recognition.

[0080] Step S202: obtaining hand-object interaction features based on the escalator key point set and the human body key point set, and obtaining human body posture features based on the human body key point set.

[0081] Exemplarily, the server 104 determines the wrist key point from the human body key point set, calculates the distance and angle value between each key point included in the escalator key point set and the human body key point set and the wrist key point, splices each key point, and the distance and angle value of each key point to obtain the hand-object interaction feature corresponding to each key point, and obtains the skeleton information feature, motion information feature and speed difference information feature respectively based on each key point included in the human body key point set, splices and fuses each key point, skeleton information feature, motion information feature and speed difference information feature included in the human body key point set, and obtains the human body posture feature corresponding to each key point included in the human body key point set.

[0082] Based on the above implementation, by obtaining hand-object interaction features based on the escalator key point set and the human body key point set, the interaction behavior between the human body and the escalator handle can be more comprehensively described. Secondly, the multi-dimensional features are integrated to obtain the human body posture features, which enhances the expressive ability of the features, thereby improving the accuracy and robustness of escalator behavior recognition.

[0083] In step S203, the hand-object interaction features and the human body posture features are input into a pre-built escalator behavior recognition model, and a hand-object interaction feature graph corresponding to the hand-object interaction features and a human body posture feature graph corresponding to the human body posture features are obtained through the graph convolutional network in the escalator behavior recognition model.

[0084] In step S204, the hand-object interaction feature map and the human body posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map.

[0085] Among them, the spatiotemporal interactive attention module can be understood as a module that uses the spatiotemporal joint attention mechanism to focus on the importance of human joints in space and time.

[0086] Exemplarily, the server 104 inputs the hand-object interaction features and the human posture features into a pre-built escalator behavior recognition model, performs further feature extraction on the hand-object interaction features through the graph convolutional network in the escalator behavior recognition model, and obtains a hand-object interaction feature map corresponding to the hand-object interaction features, and further extracts the human posture features to obtain a human posture feature map corresponding to the human posture features, and then inputs the hand-object interaction feature map and the human posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to capture the spatial relationship and time series information between the key points, and performs corresponding feature fusion to obtain a fused feature map.

[0087] According to the aforementioned implementation, by using a graph convolutional network to further extract the extracted features, a corresponding feature map is obtained, thereby further improving the feature expression capability of the feature map. Secondly, the spatiotemporal interactive attention module is used to capture spatial and temporal information, and feature fusion is performed, thereby improving the accuracy and efficiency of escalator behavior recognition.

[0088] Step S205 , obtaining an escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human body posture feature map, and the fusion feature map.

[0089] Optionally, the server 104 inputs the hand-object interaction feature map into the fully connected layer in the escalator behavior recognition model for escalator behavior recognition to obtain the corresponding first escalator probability, inputs the human body posture feature map into the fully connected layer for escalator behavior recognition to obtain the corresponding second escalator probability, inputs the fusion feature map into the fully connected layer for escalator behavior recognition to obtain the corresponding third escalator probability, and determines the product of the first, second, and third escalator probabilities as the escalator behavior recognition result for the target escalator.

[0090] Based on the above implementation, escalator behavior recognition is performed on the object interaction feature map, human posture feature map and fusion feature map respectively to obtain the corresponding escalator probability, and the final escalator recognition result is multiplied by the probabilities, which improves the comprehensive coverage of escalator recognition and enhances the credibility of the recognition results.

[0091] In the above-mentioned escalator behavior recognition method, video image data of the target escalator is obtained, the escalator key point set of the target escalator and the human body key point set of the human body contained in the video image data are obtained, hand-object interaction features are obtained based on the escalator key point set and the human body key point set, and human body posture features are obtained according to the human body key point set, the hand-object interaction features and the human body posture features are input into a pre-built escalator behavior recognition model, the hand-object interaction feature map corresponding to the hand-object interaction features and the human body posture feature map corresponding to the human body posture features are obtained through the graph convolutional network in the escalator behavior recognition model, the hand-object interaction feature map and the human body posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fusion feature map, and the escalator behavior recognition result for the target escalator is obtained according to the hand-object interaction feature map, the human body posture feature map and the fusion feature map. By introducing hand-object interaction recognition, the escalator behavior recognition model can more accurately capture whether the operator holds the handle when going up and down the escalator. Combining human posture recognition with hand-object interaction recognition, a multi-dimensional feature fusion method is adopted, and the spatiotemporal attention module is used to further capture the relationship between human posture features and hand-object interaction features at time and space points, thereby improving the accuracy of escalator usage behavior recognition.

[0092] In one embodiment, the human body key point set includes a hand key point set and a torso key point set; obtaining hand-object interaction features based on the escalator key point set and the human body key point set includes: determining the wrist key point from the hand key point set; obtaining the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point; splicing each key point, as well as the distance and angle values ​​of each key point, to obtain the hand-object interaction features corresponding to each key point.

[0093] For example, the server 104 determines the wrist key point from the hand key point set in the human body key point set. Then, we calculate the distance and angle between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set in the human body key point set, and the wrist key point. We then concatenate each key point and the distance and angle values ​​of each key point to obtain the hand-object interaction features corresponding to each key point:

[0094] Given ,in , as the set of key points of the human hand, torso and escalator. Calculation is based on the wrist reference point The distance d to each key point i,t and angle a i,t for:

[0095]

[0096]

[0097] in, Represents the coordinates of the wrist key point. Finally, each key point in the set M, the distance and angle value of each key point are concatenated into the input feature vector .

[0098] Based on the aforementioned implementation, a method is adopted to calculate the angles and distances between the key points of the human hand and the key points of the body torso and escalator handles to capture the interaction features between people and escalators when going up and down escalators, thereby improving the feature expression capability of the acquired hand-object interaction features and laying a data foundation for the subsequent recognition of escalator behavior.

[0099] In one embodiment, obtaining the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point includes: obtaining the length between the head key point and the hip key point in the torso key point set, and using the length as a scaling factor; normalizing each key point in the escalator key point set, the hand key point set, and the torso key point set using the scaling factor to obtain normalized key points; and obtaining the distance and angle values ​​between each normalized key point and the wrist key point.

[0100] Optionally, the server 104 first translates the positions of all key points to a coordinate system relative to the upper left corner of the human body bounding box. Assuming that the coordinates of the upper left corner of the human body bounding box are (x0, y0), then for any key point (x i ,y i ), the coordinates after translation are:

[0101]

[0102] In order to eliminate the size difference caused by the distance between the human body and the camera, the key point position needs to be scaled. The scaling factor is calculated by the length L of the key point between the head and hip of the human body. Assuming that the coordinates of the key point of the head are (x head ,y head ), the coordinates of the hip key point are (x hip ,y hip ),but:

[0103]

[0104] The final normalized key point coordinates are:

[0105]

[0106] Finally, the distance and angle values ​​between each normalized key point and the wrist key point are obtained.

[0107] According to the above embodiment, the coordinates of each key point are normalized by using the length between the head key point and the hip key point, which effectively avoids the errors caused by the height and depth differences between the person and the camera, improves the position accuracy of each key point, and further improves the accuracy of the hand-object interaction features, so that the model can perform detection more robustly.

[0108] In an exemplary embodiment, human body posture features are obtained based on a human body key point set, including: determining a source key point from the human body key point set; the source key point is a key point in the human body key point set that represents the center of gravity of the human body; obtaining the difference between the source key point and each key point in the human body key point set, and determining the difference as a skeleton information feature; obtaining the first coordinate difference of each key point in the human body key point set between two adjacent frames, and determining the first coordinate difference as a motion information feature; obtaining the second coordinate difference of each key point in the human body key point set within two consecutive time periods, and obtaining a speed difference information feature based on the second coordinate difference; splicing and fusing each key point, skeleton information feature, motion information feature and speed difference information feature in the human body key point set to obtain the human body posture features corresponding to each key point in the human body key point set.

[0109] Exemplarily, the server 104 uses posture recognition based on key points of the human body to calculate skeleton information features, motion information features, and speed difference information features to represent gait features.

[0110] Skeleton information features can be calculated by calculating the difference between the source joint and the target joint. Specifically, the source joint is usually defined as the joint closest to the center of gravity of the human body. For example, when holding the handrail when going up and down the escalator, the target joint coordinates are the coordinates in the set of key points of the human body, and the source joint coordinates are defined as The skeleton information features are calculated by analyzing the relative positions between the source joint and the target joint. The formula is as follows:

[0111]

[0112] Motion information features, by calculating the coordinate difference between the same joint in two adjacent frames, can directly reflect the position change of a joint over a short period of time, namely, the instantaneous velocity or displacement. By analyzing motion information features, we can identify the dynamic behavior patterns of joints when a person ascends or descends stairs with and without a handrail. For example, acceleration, deceleration, and direction changes differ when holding a handrail or not. This is very helpful for understanding the coherence and fluidity of the entire motion sequence.

[0113]

[0114] in, represents the coordinates of key point i in frame t, Indicates the coordinates of key point i at frame t+1. When t=T, Take 0.

[0115] The velocity difference feature calculates the coordinate difference between the previous (T-1) frame and the next (T-1) frame, and expands the matrix to compensate for the change in dimensionality, providing information about the movement trends of joints over a longer period of time. In the context of grabbing handles when getting on and off a vehicle, we pay special attention to the movement patterns of the hands and legs. By calculating the velocity difference features of these joints between different time frames, we can observe how their movement states change over time, further confirming whether the person is holding the handles and moving steadily. The specific formula is:

[0116]

[0117] in, Indicates the change of the i-th key point in the coordinate system of the previous T-1 frame, Represents the change of the i-th key point in the coordinate system of the next T-1 frame.

[0118] Finally, the human body key points, skeleton information features, motion information features and speed difference information features are spliced ​​and fused into the input feature vector .

[0119] Based on the above implementation, key point information directly reflects the basic posture of the human body; bone information describes the human skeleton structure through the connection between key points; motion information and velocity difference information further reveal the dynamic changes in human posture. By integrating information from multiple dimensions to more comprehensively describe human posture characteristics, the recognition accuracy of escalator behavior recognition is improved.

[0120] In one embodiment, the hand-object interaction feature map and the human body posture feature map are input into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map, including: performing an average pooling operation on the hand-object interaction feature map and the human body posture feature map to obtain a pooled hand-object interaction feature map and a pooled human body posture feature map; obtaining a first attention score corresponding to the pooled hand-object interaction feature map and a second attention score corresponding to the pooled human body posture feature map; and obtaining a fused feature map based on the first attention score, the second attention score, the hand-object interaction feature map and the human body posture feature map.

[0121] Among them, the first attention score can be understood as the attention score of the frame (time) dimension and the attention score of the node (space) dimension. Similarly, the second attention score can also be understood as the attention score of the frame dimension and the attention score of the node dimension.

[0122] Optionally, server 104 first performs average pooling operations on the hand-object interaction feature map and the human posture feature map at the frame level and joint level, respectively. These pooled feature vectors are then concatenated and fed through a fully connected (FC) layer to compress the information. Four independent FC layers are used to obtain two sets of attention scores, namely the first attention score and the second attention score, for the frame dimension and the node dimension, respectively. A fused feature map is then obtained based on the first attention score, the second attention score, the overall hand-object interaction feature map, and the human posture feature map.

[0123] According to the aforementioned implementation, average pooling is performed at the frame level and joint level, which helps to extract stable and representative human-hand-object interaction features, reduce noise interference, and combine the first and second attention scores as well as the hand-object interaction feature map and the human posture feature map for multi-level fusion, thereby enhancing the model's ability to capture complex actions and interactions and improving the recognition accuracy of escalator behavior recognition.

[0124] In one embodiment, the first attention score includes a first temporal attention score and a first spatial attention score, and the second attention score includes a second temporal attention score and a second spatial attention score;

[0125] A fused feature map is obtained according to the first attention score, the second attention score, the hand-object interaction feature map and the human posture feature map, including: using the first spatial attention score and the second temporal attention score, multiplying the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain the first attention feature map; using the first temporal attention score and the second spatial attention score, multiplying the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain the second attention feature map; merging the first attention feature map and the second attention feature map to obtain the fused feature map.

[0126] Exemplarily, the server 104 uses an interactive method to multiply the temporal attention scores and spatial attention scores of the hand-object interaction features and the human body posture features by their respective channel outer products, specifically including: using the first spatial attention score and the second temporal attention score, multiplying the channel outer product of the hand-object interaction feature map and the human body posture feature map to obtain the first attention feature map; using the first temporal attention score and the second spatial attention score, multiplying the channel outer product of the hand-object interaction feature map and the human body posture feature map to obtain the second attention feature map; then merging the obtained first attention feature map and the second attention feature map. The entire result can be regarded as the attention score of the escalator handrail action combined with the hand-object interaction features and the human body posture features. The proposed key point spatiotemporal interaction attention module can be expressed as:

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] in, and Represents the internal feature vector after processing by the FC layer, which is the basis for subsequent generation of attention scores; and Represents the average pooling operation in the time (frame) dimension and the spatial (joint) dimension respectively; The weight matrices representing the hand-object interaction features and the human body posture features are used to linearly transform the concatenated feature vectors through the fully connected layer to compress and integrate the information; and Represent HardSwish and Sigmoid activation functions respectively; Respectively represent the human body posture features used to Extract the attention scores of time dimension and spatial dimension; and denote element-wise product and channel-wise outer product, respectively. Represents a concatenation operation.

[0133] Based on the aforementioned implementation method, by generating attention scores in the time and space dimensions respectively, the dynamic changes and spatial layout characteristics of hand-object interaction actions can be captured more accurately. Secondly, the multiple attention mechanism makes the model better at adapting to different scenarios and action changes, thereby improving the recognition consistency and stability of escalator behavior recognition.

[0134] In one embodiment, an escalator behavior recognition result for a target escalator is obtained based on a hand-object interaction feature map, a human posture feature map, and a fusion feature map, including: obtaining a first escalator probability corresponding to the hand-object interaction feature map, a second escalator probability corresponding to the human posture feature map, and a third escalator probability corresponding to the fusion feature map; obtaining the product of the first escalator probability, the second escalator probability, and the third escalator probability, and determining the product as the escalator behavior recognition result for the target escalator.

[0135] Optionally, the server 104 converts the hand-object interaction feature map Input the fully connected layer for behavior prediction: , the human body posture feature map Input the fully connected layer for behavior prediction: , the fusion feature map Input the fully connected layer for behavior prediction: .in, represents the sigmoid function, FC is a fully connected network, (i.e. the aforementioned first escalator probability), (i.e., the aforementioned second escalator probability) and (i.e., the third escalator probability mentioned above) is the probability of whether a person holds the handrail when going up or down the escalator. Its dimension is 2, representing the number of behavioral actions, i.e., holding the handrail and not holding the handrail. Finally, each prediction result is multiplied to obtain the final escalator behavior recognition result: .

[0136] Based on the above implementation, escalator behavior recognition is performed on the object interaction feature map, human posture feature map and fusion feature map respectively to obtain the corresponding escalator probability, and the final escalator recognition result is multiplied by the probabilities, which improves the comprehensive coverage of escalator recognition and enhances the credibility of the recognition results.

[0137] In an exemplary embodiment, Figure 3 As shown in the figure, a specific implementation method of the escalator behavior recognition method is provided. First, the key points of the human body and the key points of the escalator handle are extracted. Then, the extracted key points of the human hand and torso are used to identify the hand-object interaction behavior with the key points of the escalator handle. The wrist key point dimension is used as the reference point. The angle and distance between the wrist key point and other parts of the hand (elbow, shoulder) and the key points of the escalator handle are calculated. The 4-dimensional feature vector (the horizontal and vertical coordinates, angles, and distances of each key point) is input into the graph convolutional network. Finally, the probability P of whether to hold the handle is output through the fully connected layer. o At the same time, based on the key points of the human body, the key point information features, bone information features, motion information features and speed difference information features are calculated respectively, and then the obtained features are spliced ​​and fused and input into the graph convolutional network, and the probability P of the handrail is output through the fully connected layer. h The hand-object interaction feature map output by the graph convolutional network and the feature map of the human body posture are further fused through the key point spatiotemporal attention module to obtain a new feature map, which is then output through the fully connected layer to obtain the handlebar probability P. ho , and finally multiply the three probabilities to get the probability of holding the handle at the end.

[0138] Step 1: Key point extraction:

[0139] OpenPose (open source pose estimation) is used to estimate the full-body pose and extract key points from the human body in the video. The extracted key points are then used to construct a human key point set. For escalator handle key point extraction, instance segmentation is first used to accurately isolate the escalator handle region from the image containing the escalator. Next, the center of gravity of the handle is determined as the center point, and the upper and lower endpoints and points between the center point are marked as basic key points. Finally, these key points are arranged in a logical order to construct a key point set that comprehensively represents the shape and position of the escalator handle.

[0140] Use the key points of the human body and the target to construct the spatial feature map G=(V,E), where V represents the key point of the human and the escalator handle. h and V o A set of human skeleton sequences, where the human key point set Represented as a set of human hands, torso and other key points, respectively, where , , ,and ; The escalator key point set is expressed as , where N represents the number of key points and T represents the total number of frames in the video sequence. E represents the edges connecting people, objects, and people-object nodes. Figure 4 As shown, circles represent key points of the human body and squares represent key points of the escalator.

[0141] Step 2: Calculate based on key point features:

[0142] Step 2.1: Calculation of hand-object interaction features:

[0143] The method of calculating the angle and distance between the key points of the human hand and the key points of the body torso and the escalator handle is adopted to capture the interaction characteristics between the person and the escalator when going up and down the escalator to determine whether the person is holding the handle. Taking into account the errors caused by the height and depth differences between the person and the camera, the key point positions need to be normalized before calculating the angle and distance so that the model can perform detection more robustly. First, the positions of all key points are translated to the coordinate system relative to the upper left corner of the human bounding box. Assuming that the coordinates of the upper left corner of the human bounding box are (x0, y0), then for any key point (x i ,y i ), the coordinates after translation are:

[0144] ( )

[0145] In order to eliminate the size difference caused by the distance between the human body and the camera, the key point position needs to be scaled. The scaling factor is calculated by the length L of the key point between the head and hip of the human body. Assuming that the coordinates of the key point of the head are (x head ,y head ), the coordinates of the hip key point are (x hip ,y hip ),but:

[0146] ( )

[0147] The final normalized key point coordinates are:

[0148] ( )

[0149] Through observation, it is found that the wrist is usually in direct physical contact with the escalator handle, and the position of the wrist is relatively stable, unlike the fingers, which are flexible and difficult to detect. Therefore, this application uses the wrist key point as the reference point And calculate the angle and distance based on this point. ,in , as the set of key points of the human hand, torso and escalator. Calculation is based on the wrist reference point The distance d to each key point i,t and angle a i,t for:

[0150] ( )

[0151] ( )

[0152] in, Represents the coordinates of the wrist key point. Finally, each key point in the set M, the distance and angle value of each key point are concatenated into the input feature vector , and then input into the graph convolutional network to further extract features.

[0153] Step 2.2: Multi-feature fusion calculation of human body key points:

[0154] The system uses posture recognition based on key points to calculate skeleton information features, motion information features, and speed difference information features to represent gait characteristics. Key point information directly reflects the basic posture of the human body; bone information describes the human skeleton structure through the connection between key points; motion information and speed difference information further reveal the dynamic changes in human posture. By integrating information from multiple dimensions, a more comprehensive description of handrail behavior recognition is achieved.

[0155] Skeleton information features can be calculated by calculating the difference between the source joint and the target joint. Specifically, the source joint is usually defined as the joint closest to the center of gravity of the human body. For example, when holding the handrail when going up and down the escalator, the target joint coordinates are the coordinates in the set of key points of the human body, and the source joint coordinates are defined as The skeleton information features are calculated by analyzing the relative positions between the source joint and the target joint. The formula is as follows:

[0156] ( )

[0157] Motion information features, by calculating the coordinate difference between the same joint in two adjacent frames, can directly reflect the position change of a joint over a short period of time, namely, the instantaneous velocity or displacement. By analyzing motion information features, we can identify the dynamic behavior patterns of joints when a person ascends or descends stairs with and without a handrail. For example, acceleration, deceleration, and direction changes differ when holding a handrail or not. This is very helpful for understanding the coherence and fluidity of the entire motion sequence.

[0158] ( )

[0159] in, represents the coordinates of key point i in frame t, Indicates the coordinates of key point i at frame t+1. When t=T, Take 0.

[0160] The velocity difference feature calculates the coordinate difference between the previous (T-1) frame and the next (T-1) frame, and expands the matrix to compensate for the change in dimensionality, providing information about the movement trends of joints over a longer period of time. In the context of grabbing handles when getting on and off a vehicle, we pay special attention to the movement patterns of the hands and legs. By calculating the velocity difference features of these joints between different time frames, we can observe how their movement states change over time, further confirming whether the person is holding the handles and moving steadily. The specific formula is:

[0161] ( )

[0162] in, Indicates the change of the i-th key point in the coordinate system of the previous T-1 frame, Represents the change of the i-th key point in the coordinate system of the next T-1 frame.

[0163] Finally, the human body key points, skeleton information features, motion information features and speed difference information features are spliced ​​and fused into the input feature vector , and then input into the convolutional network.

[0164] Step 3: Graph Convolutional Network based on Spatiotemporal Interactive Attention

[0165] The extracted features are processed using a graph convolutional network (GCN), and the ST-JointAtt (Spatio-Temporal Joint Attention) spatiotemporal key point attention mechanism is added after the GCN to better capture the spatial relationship and time series information between the key points of the human body, thereby improving the accuracy and efficiency of recognition. The feature vector and Input graph convolution operation, where , the output feature vector is .

[0166] The final hand-object interaction feature map is obtained by superimposing multiple GCN networks and ST-JointAtt spatiotemporal key point attention modules. and human posture feature map Then, the feature map is input into the fully connected layer FC to perform behavior prediction respectively:

[0167] ( )

[0168] ( )

[0169] in, represents the sigmoid function, FC is a fully connected network, and is the probability of whether a person holds the handrail when going up or down the escalator. Its dimension is 2, which represents the number of behavioral actions, i.e., holding the handrail and not holding the handrail.

[0170] Hand-object interaction features can assist in human posture recognition of the operator's movements when holding the handle in different situations, and provide fine-grained information about the hand and torso posture and the contact between the hand and the handrail. Therefore, this application proposes a key point spatio-temporal interactive attention mechanism (STIA), which uses the spatio-temporal information interaction between hand-object interaction features and human posture features to further refine the effective extraction and utilization of key features. The attention module structure is as follows: Figure 5 shown.

[0171] Similar to the ST-JointAtt spatiotemporal keypoint attention module, the input features are first average pooled at the frame level and joint level respectively. Then, these pooled feature vectors are concatenated together and fed through the FC layer to compress the information. Four independent FC layers are used to obtain two sets of attention scores for the frame dimension and the node dimension respectively. Finally, an interactive method is used to multiply the temporal attention scores and spatial attention scores of the hand-object interaction features and the human posture features by their respective channel outer products, and then the two obtained attention feature maps are merged. The whole result can be regarded as the attention score of the escalator handrail action combined with the hand-object interaction features and the human posture features. The proposed keypoint spatiotemporal interaction attention module can be expressed as:

[0172] ( )

[0173] ( )

[0174] ( )

[0175] ( )

[0176] ( )

[0177] in, and Represents the internal feature vector after processing by the FC layer, which is the basis for subsequent generation of attention scores; and Represents the average pooling operation in the time (frame) dimension and the spatial (joint) dimension respectively; The weight matrices representing the hand-object interaction features and the human body posture features are used to linearly transform the concatenated feature vectors through the fully connected layer to compress and integrate the information; and Represent HardSwish and Sigmoid activation functions respectively; Respectively represent the human body posture features used to Extract the attention scores of time dimension and spatial dimension; and denote element-wise product and channel-wise outer product, respectively. Represents a concatenation operation.

[0178] The fused feature map is then input into the fully connected layer for behavior prediction:

[0179] ( )

[0180] Finally, multiply each prediction result to get the final behavior recognition result:

[0181] ( )

[0182] Compared with the existing technology, this application has the following technical advantages:

[0183] 1. Based on skeleton key point modeling, we focus on the spatiotemporal relationship and interaction between the human skeleton key points and the target skeleton key points, capture the interaction and relative position between the key points, and improve the performance of ladder climbing risk behavior recognition.

[0184] 2. Combining human posture recognition and hand-object interaction recognition, the system extracts multi-dimensional features of human key points and escalator handle key points and performs feature fusion. This enables a more comprehensive description of the interaction between the human body and escalator handles, thereby improving recognition accuracy and robustness.

[0185] 3. A key point spatiotemporal interaction attention module is proposed. By interactively fusing the temporal and spatial attention scores of the hand-object interaction features and the human posture features respectively, two attention interaction feature maps are obtained. The model can more accurately capture the intrinsic connection between human posture and hand-object interaction. This cross-modal fusion method also further makes full use of the complementarity between the two features, thereby improving the recognition accuracy of complex actions and enhancing the robustness of the model.

[0186] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0187] Based on the same inventive concept, embodiments of the present application also provide an escalator behavior recognition device for implementing the aforementioned escalator behavior recognition method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more escalator behavior recognition device embodiments provided below can be found in the above-mentioned limitations of the escalator behavior recognition method and will not be further elaborated here.

[0188] In an exemplary embodiment, Figure 6 As shown, an escalator behavior recognition device is provided, including: a set acquisition module 601, a feature acquisition module 602, a feature map acquisition module 603, a fusion module 604 and a behavior recognition module 605, wherein:

[0189] A set acquisition module 601 is used to acquire video image data of a target escalator, and acquire an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data;

[0190] A feature acquisition module 602 is configured to acquire hand-object interaction features based on the escalator key point set and the human body key point set, and to acquire human posture features based on the human body key point set;

[0191] Feature map acquisition module 603, configured to input hand-object interaction features and human posture features into a pre-built escalator behavior recognition model, and obtain a hand-object interaction feature map corresponding to the hand-object interaction features and a human posture feature map corresponding to the human posture features through the graph convolutional network in the escalator behavior recognition model;

[0192] A fusion module 604 is configured to input the hand-object interaction feature map and the human posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map;

[0193] The behavior recognition module 605 is used to obtain an escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human body posture feature map and the fusion feature map.

[0194] In one embodiment, the human body key point set includes a hand key point set and a torso key point set, and the feature acquisition module 602 is further used to determine the wrist key point from the hand key point set; obtain the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point; splice each key point, as well as the distance and angle values ​​of each key point, to obtain the hand-object interaction features corresponding to each key point.

[0195] In one embodiment, the feature acquisition module 602 is further used to obtain the length between the head key point and the hip key point in the torso key point set, and use the length as a scaling factor; use the scaling factor to normalize each key point in the escalator key point set, the hand key point set, and the torso key point set to obtain each normalized key point; obtain the distance and angle value between each normalized key point and the wrist key point.

[0196] In an exemplary embodiment, the feature acquisition module 602 is also used to determine the source key point from the human body key point set; the source key point is the key point in the human body key point set that represents the center of gravity of the human body; the difference between the source key point and each key point in the human body key point set is obtained, and the difference is determined as a skeleton information feature; the first coordinate difference of each key point in the human body key point set between two adjacent frames is obtained, and the first coordinate difference is determined as a motion information feature; the second coordinate difference of each key point in the human body key point set in two consecutive time periods is obtained, and the speed difference information feature is obtained based on the second coordinate difference; each key point, skeleton information feature, motion information feature and speed difference information feature in the human body key point set are spliced ​​and fused to obtain the human body posture feature corresponding to each key point in the human body key point set.

[0197] In one embodiment, the fusion module 604 is also used to perform an average pooling operation on the hand-object interaction feature map and the human body posture feature map to obtain a pooled hand-object interaction feature map and a pooled human body posture feature map; obtain a first attention score corresponding to the pooled hand-object interaction feature map, and a second attention score corresponding to the pooled human body posture feature map; and obtain a fused feature map based on the first attention score, the second attention score, the hand-object interaction feature map, and the human body posture feature map.

[0198] In one embodiment, the first attention score includes a first temporal attention score and a first spatial attention score, and the second attention score includes a second temporal attention score and a second spatial attention score; the fusion module 604 is further used to use the first spatial attention score and the second temporal attention score to multiply the channel outer product of the hand-object interaction feature map and the human body posture feature map to obtain a first attention feature map; use the first temporal attention score and the second spatial attention score to multiply the channel outer product of the hand-object interaction feature map and the human body posture feature map to obtain a second attention feature map; merge the first attention feature map and the second attention feature map to obtain a fused feature map.

[0199] In an exemplary embodiment, the behavior recognition module 605 is also used to obtain a first escalator probability corresponding to the hand-object interaction feature map, a second escalator probability corresponding to the human body posture feature map, and a third escalator probability corresponding to the fusion feature map; obtain the product of the first escalator probability, the second escalator probability and the third escalator probability, and determine the product as the escalator behavior recognition result for the target escalator.

[0200] Each module in the escalator behavior recognition device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device as hardware, or stored in a computer device memory as software, allowing the processor to call and execute the corresponding operations of each module.

[0201] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store video image data, escalator key point sets, human body key point sets, hand-object interaction features, hand-object interaction feature maps, human body posture features, human body posture feature maps, and escalator behavior recognition results. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an escalator behavior recognition method.

[0202] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0203] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the escalator behavior recognition method of the above embodiment when executing the computer program.

[0204] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the escalator behavior recognition method of the above embodiment is implemented.

[0205] In one embodiment, a computer program product is provided, including a computer program, which implements the escalator behavior recognition method of the above embodiment when executed by a processor.

[0206] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0207] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0208] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0209] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for identifying escalator behavior, characterized in that: The method comprises: Acquire video image data of a target escalator, and acquire an escalator key point set of the target escalator and a human body key point set of a human body contained in the video image data based on the video image data; Acquire hand-object interaction features based on the escalator key point set and the human body key point set, and acquire human body posture features based on the human body key point set; Inputting the hand-object interaction features and the human posture features into a pre-built escalator behavior recognition model, and obtaining a hand-object interaction feature graph corresponding to the hand-object interaction features and a human posture feature graph corresponding to the human posture features through a graph convolutional network in the escalator behavior recognition model; Inputting the hand-object interaction feature map and the human body posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map; An escalator behavior recognition result for the target escalator is obtained according to the hand-object interaction feature map, the human body posture feature map, and the fusion feature map.

2. The method according to claim 1, characterized in that The human body key point set includes a hand key point set and a torso key point set; and obtaining hand-object interaction features based on the escalator key point set and the human body key point set includes: Determining a wrist key point from the hand key point set; Obtaining distances and angles between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point; The key points, as well as the distance and angle values ​​of the key points are spliced ​​together to obtain the hand-object interaction features corresponding to the key points.

3. The method according to claim 2, characterized in that The obtaining of the distance and angle values ​​between each key point in the escalator key point set, each key point in the hand key point set, and each key point in the torso key point set and the wrist key point includes: Obtaining the length between the head key point and the hip key point in the torso key point set, and using the length as a scaling factor; Normalizing each key point in the escalator key point set, the hand key point set, and the torso key point set using the scaling factor to obtain normalized key points; Obtain the distance and angle values ​​between each key point after the normalization processing and the wrist key point.

4. The method according to claim 1, wherein The acquiring of human body posture features according to the human body key point set includes: Determine a source key point from the human body key point set; the source key point is a key point in the human body key point set that represents the center of gravity of the human body; Obtaining a difference between the source key point and each key point in the human body key point set, and determining the difference as a skeleton information feature; Obtaining a first coordinate difference between two adjacent frames of each key point in the set of human key points, and determining the first coordinate difference as a motion information feature; Obtaining a second coordinate difference of each key point in the set of key points of the human body within two consecutive time periods, and obtaining a speed difference information feature based on the second coordinate difference; Each key point, skeleton information feature, motion information feature and speed difference information feature in the human body key point set are spliced ​​and fused to obtain a human body posture feature corresponding to each key point in the human body key point set.

5. The method according to claim 1, wherein The step of inputting the hand-object interaction feature map and the human body posture feature map into the spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map includes: performing an average pooling operation on the hand-object interaction feature map and the human body posture feature map to obtain a pooled hand-object interaction feature map and a pooled human body posture feature map; Obtaining a first attention score corresponding to the pooled hand-object interaction feature map and a second attention score corresponding to the pooled human posture feature map; The fused feature map is obtained according to the first attention score, the second attention score, the hand-object interaction feature map and the human posture feature map.

6. The method according to claim 5, characterized in that The first attention score includes a first time attention score and a first spatial attention score, and the second attention score includes a second time attention score and a second spatial attention score; The obtaining of the fused feature map according to the first attention score, the second attention score, the hand-object interaction feature map, and the human posture feature map includes: Multiplying the first spatial attention score and the second temporal attention score by the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain a first attention feature map; Multiplying the first temporal attention score and the second spatial attention score by the channel outer product of the hand-object interaction feature map and the human posture feature map to obtain a second attention feature map; The first attention feature map and the second attention feature map are merged to obtain the fused feature map.

7. The method according to any one of claims 1 to 6, characterized in that Obtaining an escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human body posture feature map, and the fusion feature map includes: Obtaining a first escalator probability corresponding to the hand-object interaction feature map, a second escalator probability corresponding to the human body posture feature map, and a third escalator probability corresponding to the fusion feature map; A product of the first escalator probability, the second escalator probability, and the third escalator probability is obtained, and the product is determined as an escalator behavior recognition result for the target escalator.

8. An escalator behavior recognition device, characterized in that: The device comprises: a set acquisition module, configured to acquire video image data of a target escalator, and acquire, based on the video image data, a set of escalator key points of the target escalator and a set of human body key points of a human body contained in the video image data; a feature acquisition module, configured to acquire hand-object interaction features based on the escalator key point set and the human body key point set, and to acquire human body posture features based on the human body key point set; a feature map acquisition module, configured to input the hand-object interaction features and the human posture features into a pre-built escalator behavior recognition model, and obtain a hand-object interaction feature map corresponding to the hand-object interaction features and a human posture feature map corresponding to the human posture features through a graph convolutional network in the escalator behavior recognition model; a fusion module, configured to input the hand-object interaction feature map and the human posture feature map into a spatiotemporal interaction attention module in the escalator behavior recognition model to obtain a fused feature map; The behavior recognition module is used to obtain an escalator behavior recognition result for the target escalator based on the hand-object interaction feature map, the human posture feature map and the fusion feature map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images