A behavior recognition method, system, device and medium for passenger station groups

Abstract

The present application relates to the field of intelligent monitoring and computer vision technology, and discloses a behavior recognition method, system, device and medium for passenger station groups, the method comprising: obtaining a video frame image of a target passenger station group and preprocessing it, respectively extracting the corresponding RGB image, depth map, skeleton map and optical flow map; extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and obtaining early fusion features according to the RGB features and the depth map features; capturing skeleton features according to the skeleton map, capturing optical flow features according to the optical flow map, and obtaining mid-term fusion features according to the skeleton features and the optical flow features; obtaining late fusion features according to the early fusion features and the mid-term fusion features, and obtaining the behavior category prediction result corresponding to the video frame image according to the late fusion features. The method can accurately identify intensive group behaviors using multi-feature data in complex environments such as railway passenger stations.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring and computer vision technology, and in particular to a behavior recognition method, system, equipment and medium for passenger station groups. Background Art

[0002] With the rapid development of railway transportation, the density of passenger flow and the need for safety management in railway passenger stations are also increasing, especially during peak hours or holiday travel periods, when the passenger flow in passenger stations has increased significantly. As a transportation hub, passenger stations include multiple key areas such as escalators, waiting halls, entrances and exits. These areas often gather a large number of passengers during peak hours, increasing potential safety hazards. For example, escalators are an important passage for passengers to enter and exit the station. With the increase in the density of people and the increase in the number of passengers carrying luggage, it is easy for passengers to fall or be tripped by luggage on the escalator. Once such an emergency occurs, the passenger's fall or stay may be covered by the surrounding obstructions, making it impossible for staff to discover it in time, thereby delaying the emergency response time.

[0003] Existing monitoring systems mainly rely on video surveillance and manual observation, but this method is inefficient and slow to respond, making it difficult to achieve real-time monitoring and analysis of group behavior. Most existing automated behavior recognition systems rely only on single feature information (such as RGB images), and have low recognition accuracy in complex scenarios (such as insufficient light, occlusion, etc.), which cannot meet the security needs of complex places such as railway passenger stations. Therefore, how to use multi-feature data to accurately identify intensive group behavior in complex environments such as railway passenger stations has become an urgent problem to be solved. Summary of the invention

[0004] Based on this, the present application proposes a behavior recognition method, system, device and medium for passenger station groups, aiming to accurately identify intensive group behaviors using multi-feature data in complex environments such as railway passenger stations.

[0005] The first aspect of the present application provides a behavior recognition method for a passenger station group, the method comprising:

[0006] Obtain the video frame images of the target passenger station group and perform preprocessing to extract the corresponding RGB image, depth map, skeleton map and optical flow map respectively;

[0007] Extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and obtaining early fusion features according to the RGB features and the depth map features;

[0008] Capturing skeleton features according to the skeleton map, capturing optical flow features according to the optical flow map, and obtaining mid-term fusion features according to the skeleton features and the optical flow features;

[0009] Obtaining a late fusion feature according to the early fusion feature and the mid-term fusion feature;

[0010] The behavior category prediction result corresponding to the video frame image is obtained according to the late fusion feature.

[0011] As an optional implementation manner of the first aspect, the steps of obtaining a video frame image of a group of a target passenger station and preprocessing it, and respectively extracting corresponding RGB images, depth maps, skeleton maps, and optical flow maps include: obtaining a group video of the target passenger station, intercepting a video segment to be analyzed from the group video, and extracting a video frame image based on the video segment to be analyzed; preprocessing the video frame image, wherein the preprocessing includes resizing, normalization, and data enhancement; extracting RGB image channels based on the preprocessed video frame image to obtain the RGB image; extracting the depth map based on the RGB image using a stereo vision method; extracting the skeleton map based on the RGB image using a human posture estimation algorithm; and calculating the optical flow map between adjacent video frame images using an optical flow algorithm.

[0012] As an optional implementation manner of the first aspect, the steps of extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and obtaining early fusion features according to the RGB features and the depth map features include: constructing an early feature extraction and fusion module for generating the early fusion features according to the RGB image and the depth map, wherein the early feature extraction and fusion module includes an EfficientNet network layer, a MiDaS module, an SE module, a spatial transformation network layer, a spatial attention mechanism layer and a splicing layer; the EfficientNet network layer is used to extract the RGB features according to the RGB image; the MiDaS module is used to extract the depth map features according to the depth map; the SE module is used to perform adaptive weighted fusion of the RGB features and the depth map features to obtain weighted fusion features; the spatial transformation network layer is used to perform feature alignment on the weighted fusion features to obtain aligned features; the spatial attention mechanism layer is used to perform feature weighting on the aligned features to obtain a weighted feature map; the splicing layer is used to obtain the early fusion features according to the weighted feature map.

[0013] As an optional implementation of the first aspect, the steps of capturing skeleton features according to the skeleton map, capturing optical flow features according to the optical flow map, and obtaining mid-term fusion features according to the skeleton features and the optical flow features include: constructing a skeleton and optical flow joint modeling module for generating the mid-term fusion features according to the skeleton map and the optical flow map, wherein the skeleton and optical flow joint modeling module includes an adaptive adjacency matrix layer, an ST-GCN layer, a self-attention mechanism layer and a splicing layer; the adaptive adjacency matrix layer is used to capture the dynamic relationship of skeleton points according to the skeleton map; the ST-GCN layer is used to capture spatiotemporal dependencies based on the dynamic relationship of the skeleton points to obtain the skeleton features; the self-attention mechanism layer is used to capture motion information and enhance key action features according to the optical flow map to obtain the optical flow features; the splicing layer is used to perform feature splicing of the skeleton features and the optical flow features to obtain the mid-term fusion features.

[0014] As an optional implementation of the first aspect, the step of obtaining a late fusion feature based on the early fusion feature and the mid-term fusion feature includes: constructing a late fusion and dynamic weighting module, the late fusion and dynamic weighting module including a splicing layer, a multi-head attention mechanism layer, a feature conversion layer, a normalization layer and a feedforward neural network layer; the splicing layer is used to perform feature splicing on the early fusion feature and the mid-term fusion feature to obtain a splicing feature; the multi-head attention mechanism layer is used to perform adaptive dynamic weighting according to the splicing feature to extract the correlation between different features in the spatiotemporal information; the feature conversion layer is used to perform residual connection on the correlation between different features in the spatiotemporal information to obtain a residual connection feature map; the normalization layer is used to perform normalization on the residual connection feature map to obtain a normalized feature map; the feedforward neural network layer is used to obtain the late fusion feature according to the normalized feature map.

[0015] As an optional implementation manner of the first aspect, the behavior recognition method further includes: designing a final loss function regarding the adaptive dynamic weighting: ,in, Represents the weight of adaptive learning, indicating the contribution of the i-th feature to the final loss, is the loss corresponding to the i-th feature.

[0016] As an optional implementation of the first aspect, the step of obtaining a behavior category prediction result corresponding to the video frame image based on the late fusion features includes: inputting the late fusion features into multiple Transformer encoding layers to obtain behavior category predictions; inputting the behavior category predictions into a fully connected layer to convert them into a probability distribution of behavior categories; and selecting the category corresponding to the highest probability in the probability distribution of the behavior categories as the final behavior category prediction result.

[0017] A second aspect of the present application provides a behavior recognition system for passenger station groups, the behavior recognition system comprising:

[0018] An image acquisition module is used to acquire video frame images of the target passenger station group and perform preprocessing to extract the corresponding RGB image, depth map, skeleton map and optical flow map respectively;

[0019] An early fusion module, used to extract RGB features according to the RGB image, extract depth map features according to the depth map, and obtain early fusion features according to the RGB features and the depth map features;

[0020] A mid-term fusion module, used for capturing skeleton features according to the skeleton map, capturing optical flow features according to the optical flow map, and obtaining mid-term fusion features according to the skeleton features and the optical flow features;

[0021] A late fusion module, used for obtaining a late fusion feature according to the early fusion feature and the mid-term fusion feature;

[0022] The behavior category prediction module is used to obtain the behavior category prediction result corresponding to the video frame image according to the late fusion feature.

[0023] The third aspect of the present application provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned behavior recognition method for passenger station groups.

[0024] A fourth aspect of the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned behavior recognition method for passenger station groups.

[0025] Compared with the prior art, the beneficial effects of the present application are as follows.

[0026] 1. Multi-feature extraction: Existing technologies mainly rely on single feature information, such as RGB images or skeleton structures, which are easily disturbed by environmental factors (such as lighting changes, occlusions), and the recognition effect is limited. The innovation of this application lies in the introduction of multi-feature (RGB features, depth map features, skeleton features and optical flow features) extraction technology, combining multiple behavioral information such as appearance, structure and movement, to overcome the problem of insufficient recognition accuracy of existing technologies in complex scenes, especially when people are occluding each other.

[0027] 2. Adaptive multi-feature fusion strategy: Traditional fusion methods usually only use single-step fusion (such as simple splicing of features in the initial stage or later stage), lack of multi-level feature integration, resulting in the inability to fully capture individual behavior changes in complex scenes. The three-step fusion strategy of the present invention achieves a more accurate combination of multiple features by refining the features at different times. Early fusion first splices the features of the RGB image and the depth map in the channel dimension; mid-term fusion splices the skeleton and optical flow features; and the feature weights of each feature are adaptively adjusted in the later stage through the attention mechanism. When dealing with group behavior, this strategy achieves efficient combination of multiple features and adaptive weight allocation through staged fusion processing, and can dynamically adapt to feature changes in different scenarios.

[0028] 3. Dynamic weighting based on multi-head attention mechanism: Current group behavior recognition technology usually adopts fixed weights or simple weighting mechanisms, which makes it difficult to adaptively adjust the weights of features. The present invention uses a multi-head attention mechanism to dynamically adjust weights according to the scene, especially when dealing with complex behaviors, and can perform targeted weighting for different features, thereby improving the accuracy and flexibility of recognition.

[0029] Additional aspects and advantages of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be understood through the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a behavior recognition method for passenger station groups proposed in the first embodiment of the present application;

[0031] Figure 2 A network framework diagram of three key phase modules proposed in the first embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of a behavior recognition system for passenger station groups proposed in the second embodiment of the present application.

[0033] The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0036] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0037] Example 1

[0038] See also Figure 1 , is a flow chart of a behavior recognition method for passenger station groups proposed in the first embodiment of the present application, the proposed behavior recognition method includes S01 to S05. In this embodiment, the behavior recognition method can be divided into three key phase modules, see Figure 2 , which is a network framework diagram of the three key phase modules proposed in the first embodiment of the present application.

[0039] S01: Obtain the video frame images of the target passenger station group and perform preprocessing to extract the corresponding RGB image, depth map, skeleton map and optical flow map respectively.

[0040] This step specifically includes the following steps S011 to S013.

[0041] S011: Acquire a group video of a target passenger station, intercept a video segment to be analyzed from the group video, and extract a video frame image according to the video segment to be analyzed;

[0042] For example, a 30-second video of a railway passenger falling down is imported into the processing system, and frame images are extracted at a rate of 30 frames per second. This step extracts a total of 900 frames of images, and each frame is used as input for subsequent image preprocessing. Frame-by-frame extraction can capture the passenger's continuous movements, especially the key moment of falling down, and provide complete data input for subsequent behavior recognition.

[0043] S012: Preprocessing the video frame image, wherein the preprocessing includes resizing and normalization and data enhancement;

[0044] First, in order to ensure that all images are of the same size and proportion and meet the input requirements of the deep learning model, all frames will be resized to a fixed size of 224×224 pixels. Subsequently, the image is normalized to convert each channel value of the RGB image into a standardized value to fit the numerical range of the model training. The normalization formula is as follows:

[0045] (1)

[0046] in, Represents the output RGB image pixel value, Represents the input RGB image pixel value, is the mean value of each channel, is the standard deviation of each channel. This normalization operation helps improve the training efficiency of the model and ensures that the input image has consistent feature expression under different lighting conditions.

[0047] Finally, in the implementation process, data augmentation is combined with random rotation operations to enhance the model's ability to recognize behaviors from different perspectives. Due to the limited viewing angles of the video dataset, especially in the railway passenger station scene, the fixed position of the camera will lead to insufficient understanding of multiple angles of the same scene. Therefore, in order to improve the robustness of the model, we use random rotation to enhance the data.

[0048] By randomly rotating the image, we simulate different shooting angles so that the model can better adapt to the multi-view changes in the actual scene. Specifically, the image will be randomly rotated between -10 degrees and +10 degrees around the center point. The rotation operation uses the following rotation matrix for transformation:

[0049] (2)

[0050] in, represents the rotation matrix, Indicates the angle of rotation. For each frame, the rotation angle is randomly selected between -10 and +10 degrees, and data enhancement is achieved through rotation matrix transformation. The introduction of random rotation can improve the performance of the model under different viewing angles, especially when dealing with passenger falls, the camera may capture the same behavior scene from different angles. Through data enhancement, the model can learn more diverse behavior characteristics on a limited data set and improve its generalization ability.

[0051] S013: Based on the preprocessed video frame image, extract the RGB image channel to obtain the RGB image; based on the RGB image, use the stereo vision method to extract the depth map; based on the RGB image, use the human body posture estimation algorithm to extract the skeleton map; use the optical flow algorithm to calculate the optical flow map between adjacent video frame images.

[0052] It should be noted that the preprocessed video frame image is usually a color image containing three color channels: red (R), green (G), and blue (B). In order to extract these three color channels separately, we need to decode the image and separate them into separate R, G, and B images.

[0053] Based on RGB images, we can use stereo vision technology to estimate depth information. Common stereo matching algorithms include semi-global matching (SGM), Block Matching, etc., or use deep learning methods such as MVSNet, DeepStereo, etc. These methods use the disparity information of the left and right eyes (or the front and back viewpoints) to calculate the depth value of each pixel and form a depth map.

[0054] Human pose estimation usually involves identifying and tracking key points of the human body from RGB images to form a so-called "skeleton graph". This can be achieved through traditional machine learning methods, such as HOG+SVM, or using deep learning models such as OpenPose, PoseNet, etc. These algorithms can identify the main joints of the human body, such as the head, shoulders, elbows, wrists, waist, knees and ankles, and connect them to form a skeleton.

[0055] Finally, we use the RGB images of adjacent video frames to calculate the optical flow map. Common optical flow algorithms include Lucas-Kanade, Optical Flow Network (FlowNet), SPyNet, etc. These algorithms are based on the optical flow assumption that the pixel positions of objects in the image do not change much within a short time interval. By comparing the pixel differences between adjacent frames, we can obtain the optical flow vector field, which represents the movement of each point in time and forms an optical flow map.

[0056] S02: extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and obtaining early fusion features according to the RGB features and the depth map features.

[0057] In traditional methods of extracting RGB features, such as the Res-Net method, the problem of gradient vanishing is alleviated by skip connections, but it also increases the complexity and computational overhead of the network. Although this method is simple, it has many deficiencies in the comprehensive understanding of scenes and behaviors. In order to overcome this limitation, the present invention proposes a new early feature extraction and fusion module (ADFN), whose network architecture is as follows Figure 2 As shown in the figure, this module uses an adaptive weighting mechanism to adaptively adjust the feature weights of RGB images and depth maps, fuse the information of the two features, and improve the comprehensive understanding of scenes and behaviors.

[0058] In this step, the first key phase module, namely the early feature extraction and fusion module, is constructed to generate the early fusion features according to the RGB image and the depth map. The early feature extraction and fusion module includes an EfficientNet network layer, a MiDaS module, an SE module, a spatial transformation network layer, a spatial attention mechanism layer, and a splicing layer.

[0059] Among them, the EfficientNet network layer is used to extract the RGB features according to the RGB image; the MiDaS module is used to extract the depth map features according to the depth map; the SE module is used to perform adaptive weighted fusion of the RGB features and the depth map features to obtain weighted fusion features; the spatial transformation network layer is used to align the weighted fusion features to obtain aligned features; the spatial attention mechanism layer is used to perform feature weighting on the aligned features to obtain a weighted feature map; the splicing layer is used to obtain the early fusion features according to the weighted feature map.

[0060] Specifically, RGB images can provide appearance and texture information, and depth maps can capture the geometric structure of the scene. The ADFN module extracts RGB features through EfficientNet, and uses the MiDaS module to extract depth map features, and performs weighted processing through the SE module, which can highlight the most important features for behavior recognition in each feature and reduce the interference between the two features. Then, the spatial transformation network is used to align the features, and finally, the spatial attention mechanism is introduced in the process of feature splicing to ensure effective information fusion. The specific process is as follows:

[0061] First, two features are extracted. The feature extraction formula is as follows:

[0062] (3)

[0063] (4)

[0064] Among them, EfficientNet () represents the EfficientNet network, which is used to extract the spatial features of RGB images. MiDaS () represents the MiDaS module, which is used to extract the geometric structure features of the depth map. Among them, Represent the input RGB image and depth map respectively, Respectively represent the extracted RGB features and depth map features, Respectively represent the height, width and number of channels of RGB features, Represent the height, width and number of channels of the depth map features respectively.

[0065] At the same time, in order to improve real-time performance, the number of parameters of EfficientNet is further optimized through pruning methods. Unimportant network connections are pruned to reduce model complexity, and the amount of calculation is reduced by quantizing model weights. These optimizations not only maintain the accuracy of EfficientNet, but also reduce the computational cost, making the system more efficient when processing large-scale videos.

[0066] Secondly, the weighted processing of the SE module can highlight the most important features for behavior recognition in each feature and reduce the interference between the two features. The SE module mainly includes compression and excitation. The steps are as follows:

[0067] 1. Compression operation: for input feature map First, the spatial dimension of each channel is compressed through the global average pooling operation to obtain the global feature description of each channel. The final result is , the formula for global average pooling is as follows:

[0068] (5)

[0069] in, is the global feature of the ith channel, Represents the (x, y)th pixel value of the input feature map on channel i.

[0070] 2. Excitation operation: After obtaining the compressed global feature z, a nonlinear transformation is performed through a fully connected layer to generate the weight of each channel. This process includes two fully connected layers, with the ReLU activation function used in the middle and the Sigmoid activation function used in the end to generate the weight of each channel. The formula is as follows:

[0071] (6)

[0072] in, are the weight matrices of the first and second fully connected layers, respectively. Represents the Sigmoid activation function, which is used to limit the weight to the range of [0,1], and s represents the weight vector of each channel, which is .

[0073] 3. Apply the weight s obtained by the Excitation operation to the input feature map F to enhance important channel features. The re-weighted formula is:

[0074] (7)

[0075] Where i is the channel index, is the weight of the ith channel, Represents the weighted (x, y)th pixel value on channel c.

[0076] Subsequently, there may be spatial offsets and scale differences between the RGB features and the depth map after feature extraction. Therefore, feature alignment and scale normalization can be performed before fusion. The coordinate system of the depth map is adjusted through the spatial transformation network to ensure that the two types of features are aligned in the spatial dimension. Then, the normalization method is used to unify the two types of features in the numerical scale to avoid interference caused by mismatched numerical ranges. The processing flow of the spatial transformation network is as follows:

[0077] Due to the RGB characteristics and depth map features There may be offset and scale differences in the spatial dimension. The spatial transformation network is used to geometrically transform the depth map features to align them with the RGB features. The transformed depth map features are recorded as The formula is as follows:

[0078] (8)

[0079] in, represents the RGB features after spatial transformation, STN () represents the spatial transformation network, by predicting the affine transformation parameters Spatial correction of depth map features.

[0080] Localization network and transformation parameter prediction:

[0081] (9)

[0082] Among them, CNN ( ) represents a series of convolutional layers, which are used to extract high-level representations of input features, and MLP ( ) represents a fully connected layer, which is used to generate affine transformation parameters. .

[0083] Unify RGB features through normalization method And the aligned depth map features Range on the numerical scale to avoid interference of numerical differences on feature fusion:

[0084] (10)

[0085] (11)

[0086] in, represents the normalized RGB feature, represents the normalized depth map features. These two types of normalized features are the alignment features. and are the mean and standard deviation of the RGB feature channels respectively, and are the mean and standard deviation of the depth map feature channels, respectively.

[0087] Finally, in the process of feature concatenation, the spatial attention mechanism is introduced to help the model automatically select and focus on the most important areas based on the spatial distribution of the input feature map. This mechanism helps to further improve the model's expressiveness during feature fusion, especially when RGB features and depth map features may contain redundant information after concatenation. The spatial attention mechanism can suppress irrelevant features and enhance the representation of useful information. The main process is as follows:

[0088] Concatenate the normalized features in the channel dimension:

[0089] (12)

[0090] Among them, Concat() represents the concatenation operation in the channel dimension.

[0091] Introduce the spatial attention mechanism to fusion feature map Perform weighted processing to generate a weighted feature map:

[0092] (13)

[0093] in, represents the element-by-element multiplication operation, M is the attention map, It is the weighted early fusion feature output, which enhances the feature expression of important areas.

[0094] S03: capturing skeleton features according to the skeleton map, capturing optical flow features according to the optical flow map, and obtaining mid-term fusion features according to the skeleton features and the optical flow features.

[0095] After the early fusion is completed, mid-term fusion is required. In this step, the second key stage module, namely the skeleton and optical flow joint modeling module, is constructed to generate the mid-term fusion features according to the skeleton map and the optical flow map. This module can capture individual posture changes and dynamic motion information by combining skeleton data and optical flow data, and is particularly suitable for identifying complex actions and interactions in group behaviors. In crowded scenes at passenger stations, the problem of insufficient recognition accuracy caused by mutual occlusion of people can be overcome. The skeleton and optical flow joint modeling module includes an adaptive adjacency matrix layer, an ST-GCN layer, a self-attention mechanism layer, and a splicing layer.

[0096] Among them, the adaptive adjacency matrix layer is used to capture the dynamic relationship of skeleton points according to the skeleton graph; the ST-GCN layer is used to capture the spatiotemporal dependency based on the dynamic relationship of the skeleton points to obtain the skeleton features; the self-attention mechanism layer is used to capture motion information and enhance key action features according to the optical flow graph to obtain the optical flow features; the splicing layer is used to perform feature splicing on the skeleton features and the optical flow features to obtain the mid-term fusion features.

[0097] Specifically, firstly, the skeleton modeling dynamically learns the dependencies between joints through the adaptive adjacency matrix, making the model more flexible between different individuals and actions. The calculation formula of the adaptive matrix is ​​as follows:

[0098] (14)

[0099] in, represents the adaptive matrix, represents the initial adjacency matrix, and is a learnable weight matrix, ReLU is an activation function, and adds nonlinearity. Softmax is used to normalize the adjacency matrix so that the sum of the elements in each row is 1, ensuring the rationality of the adjacency relationship.

[0100] After that, the skeleton features are extracted through ST-GCN. The formula is as follows:

[0101] (15)

[0102] in, represents the skeleton feature, and ST-GCN represents the graph convolution operation, which captures the dynamic relationship between joints and extracts the individual motion features. The skeleton data dimension is , T is the length of the time series, V is the number of joint points, and D is the coordinate dimension of each joint point.

[0103] At the same time, the self-attention mechanism is used to process the optical flow map, which can capture the motion trajectory and focus on the key area where the action occurs through the attention mechanism, filtering out irrelevant background motion. The calculation of the self-attention mechanism is as follows: (16)

[0104] in, represents the optical flow feature, is a learnable weight matrix used to generate query (Q), key (K), and value (V) vectors, is the dimension of the key vector, used to scale the dot product result. Softmax() is used to ensure the normalization of the attention weights.

[0105] Finally, using the joint modeling method, the joint point position information in the skeleton data is used as a priori and embedded into the optical flow calculation, so that the optical flow features focus on the key positions where the action occurs. Then, the skeleton features and optical flow features are spliced ​​in the channel dimension and fused through convolution operations, and finally the joint features are output for more accurate behavior recognition. The joint feature output formula is as follows:

[0106] (17)

[0107] in, represents the mid-term fusion feature, Conv( ) represents the convolution operation, Represents the features after concatenation in the channel dimension.

[0108] S04: Obtaining a late fusion feature according to the early fusion feature and the mid-term fusion feature.

[0109] In the present invention, during the late fusion, the dynamic weighting strategy based on the multi-head attention mechanism can further improve the fusion effect of multiple features. The core goal of this step is to dynamically weight the features from different features (ADFN output of RGB+depth map and SFN output of skeleton+optical flow) through the multi-head attention mechanism, highlight the most relevant spatiotemporal information, and thus enhance the accuracy of group behavior recognition.

[0110] In this step, the third key stage module is constructed, namely, the late fusion and dynamic weighting module, which includes a splicing layer, a multi-head attention mechanism layer, a feature conversion layer, a normalization layer and a feedforward neural network layer.

[0111] Among them, the splicing layer is used to perform feature splicing on the early fusion features and the mid-term fusion features to obtain splicing features; the multi-head attention mechanism layer is used to perform adaptive dynamic weighting according to the splicing features to extract the correlation between different features in the spatiotemporal information; the feature conversion layer is used to perform residual connection on the correlation between different features in the spatiotemporal information to obtain a residual connection feature map; the normalization layer is used to perform normalization on the residual connection feature map to obtain a normalized feature map; the feedforward neural network layer is used to obtain the late fusion features according to the normalized feature map.

[0112] Specifically, first, feature concatenation: In this step, the output features from the ADFN module (RGB+depth map features) and the SFN module (skeleton+optical flow features) are concatenated in the channel dimension.

[0113] Secondly, the introduction of the multi-head attention mechanism: the input features are weighted by the multi-head attention mechanism, and each attention head independently learns the relationship between features and dynamically assigns feature weights. Specifically, through parallel processing of multiple attention heads, the correlation between different features in spatiotemporal information is extracted. The output results are spliced ​​by each attention head:

[0114] (18)

[0115] in, Indicates multiple attentions, Represents a splicing operation, represents the result of the i-th attention head, is the linear transformation matrix.

[0116] After multi-head attention calculation, the input features and output features are residually connected (ResidualConnection).

[0117] Finally, layer normalization is performed to improve the numerical stability of the model. This is also beneficial for subsequent classification. The formula is as follows:

[0118] (19)

[0119] (20)

[0120] in, represents the normalized feature map, represents the input feature map, Represents the late fusion features of the output, LayerNorm is a regularization technique, and FFN is a feed-forward neural network.

[0121] Through hierarchical interactive feature extraction, the present invention can comprehensively utilize the advantages of multi-feature data (RGB, depth map, skeleton, optical flow) to perform comprehensive information fusion. With the help of adaptive dynamic weighting mechanism, the system highlights the most relevant spatiotemporal information and improves the accuracy and robustness of behavior recognition in complex scenes.

[0122] S05: Obtaining a behavior category prediction result corresponding to the video frame image according to the late fusion feature.

[0123] In some embodiments, step S05 includes: inputting the late fusion features into multiple Transformer encoding layers to obtain behavior category predictions; inputting the behavior category predictions into a fully connected layer to convert them into probability distributions of behavior categories; and selecting the category corresponding to the highest probability in the probability distribution of the behavior categories as the final behavior category prediction result.

[0124] It should be noted that in the present invention, the input features come from multiple features, including RGB images, depth maps, skeleton data and optical flow features. These multiple features are extracted and fused through the previous features to generate a unified spatiotemporal feature representation. After being processed by multiple Transformer encoding layers, the final behavior category prediction is output. The output features of the last layer of Transformer are converted into the probability distribution of behavior categories through the fully connected layer, and the category with the highest probability is selected as the final behavior prediction result.

[0125] In some embodiments, since the present invention involves the fusion of multiple features, features such as RGB, depth map, skeleton, optical flow, etc. may have different contributions to the final behavior recognition results. In order to better integrate these features, an adaptive weighted loss function is designed, and the weights of different features are learned to dynamically adjust the various parts of the loss function. Therefore, the final loss function of the adaptive dynamic weighting can also be designed:

[0126] (twenty one)

[0127] in, Represents the weight of adaptive learning, indicating the contribution of the i-th feature to the final loss, is the loss corresponding to the i-th feature.

[0128] In this embodiment, the overall framework of the above steps S01 to S05 is described as follows:

[0129] 1. Multi-feature data acquisition: The acquired video data includes RGB images, depth maps, skeleton point data (skeleton maps), and optical flow information (optical flow maps). These features provide different perspectives of the scene, including appearance, geometry, posture information, and motion trajectory, ensuring full support for behavior recognition.

[0130] 2. Early feature extraction and fusion module (ADFN module): The ADFN module extracts features from the RGB image and the depth image respectively. The RGB image extracts spatial and texture information through a convolutional neural network (such as EfficientNet), and the depth image extracts geometric information through the MiDaS module. Subsequently, these features are concatenated in the channel dimension, and the contributions of the two are dynamically adjusted through an adaptive weighting mechanism (Squeeze-and-Excitation module) to ensure the effective fusion of key information.

[0131] 3. Joint skeleton and optical flow modeling module (SFN module): This module processes data from skeleton points and optical flow. Skeleton features are extracted through adaptive adjacency matrix and graph convolution network (ST-GCN) to capture the spatiotemporal dependencies between joint points. Optical flow features model the motion information between video frames through the self-attention mechanism, and introduce the skeleton point position as prior information to help the optical flow focus on the key areas where the action occurs.

[0132] 4. Late fusion and dynamic weighting: The feature outputs of ADFN and SNF are spliced ​​in the channel dimension to form features containing RGB, depth, skeleton and optical flow data. Then, the system weights the spliced ​​features through a multi-head attention mechanism, adaptively adjusting the contribution of each feature to highlight key spatiotemporal information. Finally, the spliced ​​features are further optimized through deep learning layers such as residual connections to ensure the effective fusion of low-level local information and high-level global information.

[0133] 5. Behavior recognition and decision-making: The final fused features are input into the classifier or behavior recognition network to identify and analyze complex group behaviors. Through this module, the system can detect abnormal behaviors, key posture changes, or identify dynamic interactions between individuals and groups.

[0134] Based on the late fusion features, real-time detection and early warning of abnormal behaviors (such as passengers falling) can also be achieved. This network is particularly suitable for scenes with dense crowds and complex environments such as railway passenger stations, which significantly improves the accuracy and response efficiency of security monitoring, and ensures the safety of passengers and smooth operations within the station. Through the above steps, the system can achieve a global understanding and dynamic monitoring of group behavior. Through the fusion and dynamic weighting of multiple features, the system can accurately identify individual and group behaviors in complex scenes. The combination of multi-feature data improves the robustness of the system in complex scenes, and can cope with challenges such as occlusion, low light, and rapid movement, and shows excellent recognition capabilities in applications such as public place monitoring and intelligent transportation.

[0135] In summary, the innovation of this embodiment and the difference from the prior art are summarized as follows:

[0136] 1. Multi-feature extraction

[0137] Advantages: Most current group behavior recognition methods mainly rely on a single feature (such as RGB images or skeleton architectures), which are easily affected by factors such as the environment, lighting, and occlusion when dealing with complex behaviors, resulting in reduced recognition accuracy. The uniqueness of the present invention lies in the use of multi-feature extraction to enhance the capture of behavioral details, and the use of a hierarchical fusion strategy to achieve complementary and collaborative processing of information. Specifically, EfficientNet and MaDis are used to extract the spatial features of RGB images and depth maps, ST-GCN is used to extract spatiotemporal features of skeleton data, and optical flow algorithms (such as 3D CNN) are used to extract motion features from optical flow information. Each feature targets different types of information sources and can fully capture the appearance, structure, and motion features of the scene, ultimately improving the accuracy of group behavior recognition through multi-feature fusion.

[0138] Technological improvements and innovations:

[0139] Efficient feature extraction: EfficientNet is used to extract features from RGB images and depth maps. The network uses a compound scaling strategy (including network width, depth, and resolution) to achieve high performance with low computing resources. Compared with traditional convolutional neural networks, EfficientNet can provide more refined image features while reducing computational complexity.

[0140] Spatiotemporal feature modeling: The ST-GCN network not only takes into account the spatial information of the skeletal joints, but also tracks the dynamic changes of the joints in time, so that the model can accurately capture the movement trajectory and posture changes of individuals. Compared with static 2D images or single-frame data, spatiotemporal feature modeling can effectively capture subtle differences in group behavior.

[0141] Differences from existing technologies:

[0142] The existing technology mainly relies on single feature information, such as RGB images or skeleton structures, which are easily interfered by environmental factors (such as lighting changes, occlusions), and the recognition effect is limited. The innovation of the present invention lies in the introduction of multi-feature extraction technology, combining multiple behavioral information such as appearance, structure and movement, to overcome the problem of insufficient recognition accuracy of the existing technology in complex scenes, especially when people are occluding each other.

[0143] 2. Adaptive multi-feature fusion strategy

[0144] Advantages: In the feature fusion process, a hierarchical fusion strategy is adopted: the early fusion first splices the features of the RGB image and the depth map in the channel dimension; the mid-term fusion splices the skeleton and optical flow features; and the attention mechanism is used to adaptively adjust the feature weights of each feature in the later stage. When dealing with group behavior, this strategy realizes the efficient combination and adaptive weight allocation of multiple features through staged fusion processing, and can dynamically adapt to feature changes in different scenarios.

[0145] Technological innovation:

[0146] Early fusion: The RGB image and the depth map are fused early after the feature extraction stage, and a higher-dimensional feature vector is formed by splicing the two features in the channel dimension. This fusion method can combine the appearance information (RGB image) and geometric information (depth map) of the scene, improving the overall understanding of the scene.

[0147] Mid-term fusion: After the skeleton data and optical flow information are extracted separately, they are spliced ​​in the channel dimension to form mid-term fusion features. This step of fusion enhances the perception of individual motion status, capturing individual posture changes and combining the motion trajectory information of joint points, making the system perform well in fast motion and dynamic behavior recognition.

[0148] Late fusion and adaptive weighting: The feature vectors after early and mid-term fusion are finally fused in the late stage through the multi-head attention mechanism. This mechanism can adaptively adjust the weight of each feature according to its spatiotemporal dependency, ensuring that the system can dynamically select the most important features for behavior recognition in different scenarios.

[0149] Differences from existing technologies:

[0150] Traditional fusion methods usually only use single-step fusion (such as simple splicing of features in the initial stage or later stage), lack multi-level feature integration, and fail to fully capture individual behavior changes in complex scenes. The three-step fusion strategy of the present invention achieves more accurate multi-feature combination by refining features at different stages.

[0151] 3. Dynamic weighting based on multi-head attention mechanism

[0152] Advantages: The multi-head attention mechanism of the present invention can adaptively adjust the weights of different features according to the spatiotemporal changes of scene features, ensuring that the system can dynamically select the most important features for processing based on the current environment and behavior characteristics. Different from the traditional fixed weight allocation method, the multi-head attention mechanism can process multiple features in parallel and capture the spatiotemporal dependencies between them.

[0153] Technological innovation and improvement:

[0154] Introduction of multi-head attention mechanism: Compared with the traditional single attention mechanism, this invention introduces multiple parallel attention heads, each of which can focus on different features and perform weighted processing on them at different times. This parallel processing method can enhance the system's ability to capture complex behaviors.

[0155] Adaptive feature weighting: Through the multi-head attention mechanism, the system can dynamically adjust the weights of different features according to their relevance, thereby ensuring that the most recognizable features can be selected for behavior recognition in complex environments. Each attention head assigns different weights to each feature in different time periods or scenes, ensuring that the final fused feature has higher discriminative power.

[0156] Differences from existing technologies:

[0157] Current group behavior recognition technologies usually use fixed weights or simple weighting mechanisms, which makes it difficult to adaptively adjust the weights of features. The present invention uses a multi-head attention mechanism to dynamically adjust weights according to the scene, especially when dealing with complex behaviors, and can perform targeted weighting for different features, thereby improving the accuracy and flexibility of recognition.

[0158] Therefore, the above innovations jointly build a new group behavior recognition framework, which is mainly divided into three modules:

[0159] Early Feature Extraction and Fusion Module (ADFN): In the complex environment of a railway passenger station, lighting conditions may change. Therefore, the feature alignment and normalization process in the ADFN module can enhance the adaptability to lighting changes. At the same time, the use of depth maps can assist in identifying individual appearance information under occlusion;

[0160] Skeleton and optical flow joint modeling module (SFN): In the railway passenger station scene, the dense passenger flow leads to frequent occlusion. For example, in crowded platforms or escalator areas, skeleton features can help identify individual posture changes under occlusion conditions, while optical flow features can capture key motion trajectory information for dynamic monitoring;

[0161] Late fusion and dynamic weighting module: In passenger stations, different behavioral scenarios (such as escalators, waiting halls, entrances and exits, etc.) may have different requirements for features. By introducing a multi-head attention mechanism, the weights of features can be adaptively adjusted according to different scenarios. The accuracy of behavior recognition and the robustness of the system are significantly improved. The present invention exhibits strong generalization capabilities in complex scenarios. This framework can not only cope with individual interactions in group behaviors, but also maintain efficient recognition performance under challenging conditions such as occlusion, light changes, and motion blur. It is particularly suitable for crowded scenes such as railway passenger stations. For example, in an application scenario where a passenger suddenly falls on an escalator.

[0162] See also Figure 3 , which is a schematic diagram of a behavior recognition system for passenger station groups proposed in the second embodiment of the present application, and the system includes:

[0163] An image acquisition module 10 is used to acquire video frame images of a target passenger station group and perform preprocessing to extract corresponding RGB images, depth maps, skeleton maps and optical flow maps respectively;

[0164] An early fusion module 20, configured to extract RGB features according to the RGB image, extract depth map features according to the depth map, and obtain early fusion features according to the RGB features and the depth map features;

[0165] A mid-term fusion module 30, configured to capture skeleton features according to the skeleton map, capture optical flow features according to the optical flow map, and obtain mid-term fusion features according to the skeleton features and the optical flow features;

[0166] A late fusion module 40, configured to obtain a late fusion feature according to the early fusion feature and the mid-term fusion feature;

[0167] The behavior category prediction module 50 is used to obtain the behavior category prediction result corresponding to the video frame image according to the late fusion feature.

[0168] On the other hand, the present application also proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned behavior recognition method for passenger station groups.

[0169] On the other hand, the present application also proposes a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to execute the above-mentioned behavior recognition method for passenger station groups.

[0170] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A behavior recognition method for passenger station groups, characterized in that: The behavior recognition method comprises: Obtain video frame images of the target passenger station group and perform preprocessing to extract corresponding RGB images, depth maps, skeleton maps, and optical flow maps, respectively. The video frame images include continuous action images of passengers falling and occluding. The preprocessing includes a data enhancement method in which the image is randomly rotated between -10 degrees and +10 degrees around the center point; Extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and constructing an early feature extraction and fusion module for generating early fusion features according to the RGB image and the depth map, wherein the early feature extraction and fusion module includes an EfficientNet network layer, a MiDaS module, an SE module, a spatial transformation network layer, a spatial attention mechanism layer, and a splicing layer; Capturing skeleton features according to the skeleton graph, capturing optical flow features according to the optical flow graph, and constructing a skeleton and optical flow joint modeling module for generating mid-term fusion features according to the skeleton graph and the optical flow graph, wherein the skeleton and optical flow joint modeling module includes an adaptive adjacency matrix layer, an ST-GCN layer, a self-attention mechanism layer, and a splicing layer; Constructing a late fusion and dynamic weighting module, which is used to obtain late fusion features according to the early fusion features and the mid-term fusion features, wherein the late fusion and dynamic weighting module includes a splicing layer, a multi-head attention mechanism layer, a feature conversion layer, a normalization layer and a feedforward neural network layer; The behavior category prediction result corresponding to the video frame image is obtained according to the late fusion feature.

2. The method for identifying behavior of a group at a passenger station according to claim 1, characterized in that: The steps of obtaining the video frame images of the target passenger station group and preprocessing them, and extracting the corresponding RGB image, depth map, skeleton map and optical flow map respectively include: Acquire a group video of a target passenger station, intercept a video segment to be analyzed from the group video, and extract a video frame image according to the video segment to be analyzed; Preprocessing the video frame image, wherein the preprocessing includes resizing, normalization and data enhancement; Based on the preprocessed video frame image, extracting RGB image channels to obtain the RGB image; Based on the RGB image, using a stereo vision method, extracting the depth map; Based on the RGB image, using a human posture estimation algorithm, extracting the skeleton graph; An optical flow algorithm is used to calculate the optical flow map between adjacent video frame images.

3. The method for identifying behavior of passenger station groups according to claim 1, characterized in that: The steps of extracting RGB features according to the RGB image, extracting depth map features according to the depth map, and obtaining early fusion features according to the RGB features and the depth map features include: The EfficientNet network layer is used to extract the RGB features according to the RGB image; The MiDaS module is used to extract the depth map features according to the depth map; The SE module is used to perform adaptive weighted fusion on the RGB features and the depth map features to obtain weighted fusion features; The spatial transformation network layer is used to perform feature alignment on the weighted fusion features to obtain aligned features; The spatial attention mechanism layer is used to perform feature weighting on the alignment features to obtain a weighted feature map; The concatenation layer is used to obtain the early fusion feature according to the weighted feature map.

4. The method for identifying behavior of a group at a passenger station according to claim 1, characterized in that: The steps of capturing skeleton features according to the skeleton graph, capturing optical flow features according to the optical flow graph, and obtaining mid-term fusion features according to the skeleton features and the optical flow features include: The adaptive adjacency matrix layer is used to capture the dynamic relationship of skeleton points according to the skeleton graph; The ST-GCN layer is used to capture the spatiotemporal dependency based on the dynamic relationship of the skeleton points to obtain the skeleton features; The self-attention mechanism layer is used to capture motion information and enhance key action features according to the optical flow map to obtain the optical flow features; The splicing layer is used to perform feature splicing on the skeleton features and the optical flow features to obtain the mid-term fusion features.

5. The method for identifying behavior of a group at a passenger station according to claim 1, characterized in that: The step of obtaining the late fusion feature according to the early fusion feature and the mid-term fusion feature comprises: The splicing layer is used to perform feature splicing on the early fusion feature and the mid-term fusion feature to obtain a splicing feature; The multi-head attention mechanism layer is used to perform adaptive dynamic weighting according to the splicing features to extract the correlation between different features in the spatiotemporal information; The feature conversion layer is used to perform residual connection on the correlation between different features in the spatiotemporal information to obtain a residual connection feature map; The normalization layer is used to perform a normalization operation on the residual connection feature map to obtain a normalized feature map; The feedforward neural network layer is used to obtain the late fusion feature according to the normalized feature map.

6. The method for identifying behavior of a group at a passenger station according to claim 5, characterized in that: The behavior recognition method further includes: Design the final loss function with respect to the adaptive dynamic weighting: ， in, Represents the weight of adaptive learning, indicating the contribution of the i-th feature to the final loss, is the loss corresponding to the i-th feature.

7. The method for identifying behavior of a group at a passenger station according to claim 1, characterized in that: The step of obtaining the behavior category prediction result corresponding to the video frame image according to the late fusion feature comprises: Inputting the late fusion features into multiple Transformer encoding layers to obtain behavior category prediction; Inputting the behavior category prediction into a fully connected layer to convert it into a probability distribution of the behavior category; The category corresponding to the highest probability in the probability distribution of the behavior category is selected as the final behavior category prediction result.

8. A behavior recognition system for passenger station groups, characterized in that: The behavior recognition system comprises: An image acquisition module is used to acquire video frame images of the target passenger station group and perform preprocessing to extract corresponding RGB images, depth maps, skeleton maps and optical flow maps respectively, wherein the video frame images include continuous action images of passengers falling and occluding, and the preprocessing includes a data enhancement method of randomly rotating the image around the center point between -10 degrees and +10 degrees; An early fusion module, used to extract RGB features according to the RGB image, extract depth map features according to the depth map, and construct an early feature extraction and fusion module, used to generate early fusion features according to the RGB image and the depth map, wherein the early feature extraction and fusion module includes an EfficientNet network layer, a MiDaS module, an SE module, a spatial transformation network layer, a spatial attention mechanism layer, and a splicing layer; A mid-term fusion module, used to capture skeleton features according to the skeleton map, capture optical flow features according to the optical flow map, and construct a skeleton and optical flow joint modeling module, used to generate mid-term fusion features according to the skeleton map and the optical flow map, wherein the skeleton and optical flow joint modeling module includes an adaptive adjacency matrix layer, an ST-GCN layer, a self-attention mechanism layer and a splicing layer; A late fusion module, used to construct a late fusion and dynamic weighting module, used to obtain a late fusion feature according to the early fusion feature and the mid-term fusion feature, the late fusion and dynamic weighting module comprising a splicing layer, a multi-head attention mechanism layer, a feature conversion layer, a normalization layer and a feedforward neural network layer; The behavior category prediction module is used to obtain the behavior category prediction result corresponding to the video frame image according to the late fusion feature.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the executable instructions to implement a behavior recognition method for passenger station groups as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute a behavior recognition method for passenger station groups as described in any one of claims 1 to 7.

Images