Escalator scene passenger behavior detection method and device based on global information

Abstract

The application discloses a passenger behavior detection method and device in an escalator scene based on global information, and relates to the technical field of safety monitoring.The method comprises the following steps: creating a target detection model, taking a labeled image data set as a convolutional neural network input to extract image high-level semantic features, and reserving spatial relationships between feature maps through a multi-layer convolutional attention module; taking the high-level semantic feature map as an input of a coordinate attention module to model long-range dependency relationships of the feature map; and outputting a prediction image through multi-scale feature fusion, so that a passenger behavior detection model in an escalator scene based on global information is obtained.The application combines the coordinate attention and the multi-layer convolutional attention module to reflect information and positions of interested places, enhances the recognition ability of the network to partial body information of an occluded target, constructs long-range dependency of the occluded target and position information of the occluded target, alleviates the problem that a foreground target suppresses the occluded target, and improves the detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety monitoring, in particular to a passenger behavior detection method and device in an escalator scene based on global information. BACKGROUND

[0002] An escalator is a fixed power-driven device with circulating running steps for upward or downward inclined passenger transportation, and is the most common transportation tool for pedestrians in public places, and is widely used in places with high passenger flow such as shopping malls, train stations, and subways. The escalator is in a situation where passengers are exposed to the outside, and it is difficult to match the safety brought by the elevator, so the mortality rate is high when accidents occur, causing significant social and economic losses and casualties. Escalator accidents are mostly falling, falling accidents, and collision and crushing accidents with objects. If the unsafe behavior of pedestrians when riding an escalator can be quickly discovered and timely protective measures are taken, the degree of accident danger can be reduced as much as possible, and secondary harm to passengers can be reduced.

[0003] When the escalator is running, the moving planes of the conveyor belt fold each other to form an inclined upward stair platform. Such a design will present a situation of passenger blocking in an angle in the monitoring angle at the entrance or exit of the escalator, that is, the front row of passengers blocks most of the bodies of the rear row of passengers. When there are many passengers on the escalator, the front row of passengers in the monitoring screen will block a large area of the rear row of passengers.

[0004] Therefore, it is difficult for the neural network to extract features, and when detecting the unsafe behavior of pedestrians, the target boxes generated for multiple targets that are too close are very close to each other, and the overlapping parts are large, so there is mutual inhibition between the target boxes, and the detection result is often poor, causing false detection and missed detection of overlapping targets, and seriously damaging the accuracy of detection. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a passenger behavior detection method and device in an escalator scene based on global information, to optimize the detection of passenger behavior in an escalator scene with occlusion problems, to improve the attention of the model to overlapping targets, to obtain attention targets from global information, and to improve the recognition rate and accuracy of detection.

[0006] In a first aspect, the present application provides a passenger behavior detection method in an escalator scene based on global information, comprising:

[0007] Step 10, obtaining a passenger unsafe behavior data set in the escalator, labeling the class and position of the unsafe behavior in the data set, and generating a labeled image data set;

[0008] Step 20, create a target detection model to label the image dataset as a training sample input of the target detection model;

[0009] Step 30, extract feature information layer by layer from the labeled image dataset in the backbone part of the convolutional neural network, generate high-dimensional semantic feature maps by reducing image size and deepening image dimension, and encode the information in the horizontal and vertical directions of the high-dimensional semantic feature maps through the coordinate attention module, and build a long-range dependency relationship between adjacent pixel points to obtain a global information feature map;

[0010] Step 40, in the neck part of the convolutional neural network, the global information feature map is up-sampled and fused with multiple low-dimensional feature maps through PANet from bottom to top, the expression of feature information in each dimension is increased, and the feature information of the coordinate attention feature map in space and channel is fused through the multi-layer convolution attention module; the output feature map of PANet and multiple high-dimensional feature maps are fused again through FPN from bottom to top to obtain multiple dimensional feature maps including low-dimensional feature maps, medium-dimensional feature maps and high-dimensional feature maps.

[0011] Step 50, in the head part of the convolutional neural network, multiple dimensional feature maps are fused through multi-scale feature fusion, and then the image predicted based on global information is output through the convolutional neural network.

[0012] Further, in the step 30, the specific processing process of the coordinate attention module is:

[0013] For a given input feature map x, two spatial range pooling kernels (H, 1) and (1, W) are used to encode each channel along the horizontal and vertical coordinates, respectively;

[0014] The output of the cth channel at height h is represented as:

[0015]

[0016] The output of the cth channel at width w is represented as:

[0017]

[0018] The above two transformations respectively aggregate features along two spatial directions to produce a pair of direction-aware feature maps and So that the coordinate attention module block captures long-range dependencies along one spatial direction and preserves precise location information along the other spatial direction.

[0019] Further, in the step 40, the multi-layer convolution attention module includes a channel attention M c (F) and a spatial attention M s(F);

[0020] The channel attention M c (F) generates a channel attention map using the inter-channel relationship of features, aggregates the spatial information of the feature map using max pooling and average pooling, then forwards the two pooling results to the shared network MLP to generate the channel attention map, and uses element-wise summation to combine the output feature vector;

[0021] The spatial attention M s (F) generates a spatial attention map using the spatial relationship of features, encodes the positions to be emphasized or suppressed, aggregates the channel information of the feature map by using two pooling operations of average pooling and max pooling, generates two two-dimensional images representing the average pooled features and the maximum pooled features in the channel respectively; then they are connected and convolved through a standard convolution layer to generate a two-dimensional spatial attention map.

[0022] Further, in the step 40, the FPN up-samples the feature map of the previous dimension by the Concat method, trims the dimension, and splices the feature map of the next dimension after spatial and dimensional matching.

[0023] Further, the step 10 further comprises Mosaic data enhancement on the passenger unsafe behavior dataset, randomly selects 4 pictures in the dataset, performs random cropping, scaling and rotating operations on them, and randomly selects a point in the range of 1 / 3-3 / 2 of the size of the first input picture as a center point to splice, to synthesize one image.

[0024] In a second aspect, the present application provides a passenger behavior detection device in an escalator scene based on global information, comprising: a training sample processing module, a model creation module, a first processing module, a second processing module and a third processing module.

[0025] The training sample processing module is used to obtain a passenger unsafe behavior dataset in an escalator, label the categories and positions of unsafe behaviors in the dataset, and generate a labeled image dataset.

[0026] The model creation module is used to create a target detection model, and the labeled image dataset is used as the training sample input of the target detection model.

[0027] The first processing module is used to extract feature information from the labeled image dataset layer by layer in the backbone part of the convolutional neural network, generate a high-dimensional semantic feature map by reducing the image size and deepening the image dimension, and obtain a global information feature map by globally encoding the information in the horizontal and vertical directions of the high-dimensional semantic feature map and constructing a long-range dependency relationship between adjacent pixel points.

[0028] The second processing module is configured to: in a neck part of the convolutional neural network, perform upsampling fusion of a plurality of low-dimensional feature maps on the global information feature map by using a PANet from bottom to top, increase expression of feature information in each dimension, and then perform fusion of feature information in space and channels of a coordinate attention feature map by using a multi-layer convolutional attention module; and perform fusion of the output feature map of the PANet and a plurality of high-dimensional feature maps again by using a FPN from bottom to top to obtain a plurality of dimensional feature maps including low-dimensional feature maps, medium-dimensional feature maps and high-dimensional feature maps.

[0029] The third processing module is configured to: in a head part of the convolutional neural network, perform fusion of the plurality of dimensional feature maps by using multi-scale feature, and then output an image predicted based on global information by using the convolutional neural network.

[0030] Further, in the first processing module, a specific processing process of the coordinate attention module is as follows:

[0031] For a given input feature map x, two spatial range pooling kernels (H, 1) and (1, W) are used to encode each channel along the horizontal coordinate and the vertical coordinate, respectively.

[0032] The output of the cth channel at the height h is represented as:

[0033]

[0034] The output of the cth channel at the width w is represented as:

[0035]

[0036] The above two transformations respectively aggregate features along two spatial directions to generate a pair of direction perception feature maps and So that the coordinate attention module block captures long-range dependencies along one spatial direction and preserves precise location information along the other spatial direction.

[0037] Further, the multi-layer convolutional attention module includes a channel attention M c (F) and a spatial attention M s (F);

[0038] The channel attention M c (F) uses the inter-channel relationship of features to generate a channel attention map, and uses maximum pooling and average pooling to aggregate spatial information of the feature map, and then forwards the two pooling results to a shared network MLP to generate a channel attention map, and uses element-wise summation to combine the output feature vectors;

[0039] The spatial attention M s(F) generate a spatial attention map using the spatial relationship of features, encode the positions to be emphasized or suppressed, aggregate the channel information of the feature map by using two pooling operations of average pooling and maximum pooling, generate two two-dimensional images representing the average pooled features and the maximum pooled features in the channel respectively, then connect them and convolve them through a standard convolution layer to generate a two-dimensional spatial attention map.

[0040] Further, in the second processing module, the FPN up-samples the feature map of the previous dimension, trims the dimension, and splices the feature map of the next dimension after spatial and dimensional matching.

[0041] Further, the training sample processing module is further used for Mosaic data enhancement on the passenger unsafe behavior data set, randomly selects 4 pictures in the data set, performs random cropping, scaling and rotating operations on the 4 pictures, and splices a synthesized image by randomly selecting a center point in the range of 1 / 3-3 / 2 of the size of the first input picture.

[0042] The one or more technical solutions provided in the embodiments of the application have at least the following technical effects or advantages:

[0043] 1. By introducing the coordinate attention module (CA) into the backbone network part of the convolutional neural network, the dependence relationship between pixels after long-distance feature extraction is maintained, and the spatial relationship of the occluded body part in the horizontal and vertical directions is established. CA alleviates the problem of loss of feature position context information after long-distance feature extraction, which helps the network to locate the information of the occluded part. Through the multi-layer convolutional attention module (C3CBAM), the channel attention and spatial attention of the feature map in the local are improved, and the information of the occluded body part in the picture is strengthened to become the information on which the network focuses attention. C3CBAM enhances the attention of the position information of the occluded part provided by CA in the channel and the space, combines the feature position information to improve the attention of small targets and occluded targets when generating the predicted target frame, and effectively feeds back the position and channel of the occluded part, thereby improving the detection performance of the network on the occluded target.

[0044] 2. By using the structure of FPN and PAN in the Neck part of the convolutional neural network, the FPN layer conveys strong semantic features from top to bottom. The high-level semantic features are the feature information obtained after feature extraction, have a large receptive field, and are abstract, which is beneficial to the classification of objects but will lose detailed information, and is not conducive to accurate segmentation. The high-level semantic features are abstract features. The feature pyramid conveys strong positioning features from bottom to top. The two work together to aggregate parameters for different detection layers from different backbone layers, further improving the feature extraction capability.

[0045] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, which can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0046] The present application will be further described below with reference to the accompanying drawings in conjunction with the embodiments.

[0047] Figure 1 The network overall structure schematic diagram of the embodiment of the present application is shown in the figure.

[0048] Figure 2 The coordinate attention module (CA) schematic diagram of the embodiment of the present application is shown in the figure.

[0049] Figure 3 The multi-layer convolutional attention module (C3CBAM) structure schematic diagram of the embodiment of the present application is shown in the figure.

[0050] Figure 4 The multi-scale feature fusion schematic diagram of the embodiment of the present application is shown in the figure.

[0051] Figure 5 The flow chart in the method of the embodiment of the present application is shown in the figure.

[0052] Figure 6 The structure schematic diagram of the device in the second embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] The technical scheme of the embodiment of the present application provides a passenger behavior detection method and device based on global information in an escalator scene, which optimizes the detection of passenger behavior in the escalator scene with occlusion problems, improves the attention of the model to overlapping targets, obtains attention targets from global information, and improves the recognition rate and accuracy of detection.

[0054] The technical scheme of the embodiment of the present application is as follows:

[0055] An unsafe behavior data set of passengers in an escalator is acquired; the categories and positions of the unsafe behaviors in the data set are labeled, and a corresponding label set is generated; a target detection model is created, and the data set and the label set are used as inputs of a convolutional neural network to extract high-level semantic features of images, and a multi-layer convolutional attention module is used to retain the spatial relationship between feature maps; the high-level semantic feature maps are used as inputs of a coordinate attention module to model the long-range dependency of the feature maps; a prediction image is output through multi-scale feature fusion, and a detection model of passenger behaviors in an escalator scene with global information is obtained.

[0056] The training steps of the detection model are as follows:

[0057] S1, an unsafe behavior data set of passengers in an escalator is acquired, including images of unsafe behaviors such as passenger falling, reverse direction, and climbing the escalator; the data set is labeled according to the categories and positions of the unsafe behaviors, and a labeled image set is generated.

[0058] In order to expand the training samples, Mosaic data enhancement can be performed on the data set, four images in the data set are randomly selected, and random cropping, scaling, rotation and other operations are performed on them, and a center point in the range of 1 / 3-1 / 2 of the size of the first input image is randomly selected as a center point for splicing, and a synthesized image is obtained.

[0059] S2, a target detection model is created, and the overall structure of the model can be as shown in Figure 1 The labeled image set is used as the training sample input of the target detection model.

[0060] S3, each training sample of the labeled image set is trained through a convolutional neural network (such as YOLOv5), the backbone part extracts feature information of each part of the sample, the image size is reduced by 5 times at a rate of 2, the image dimension is increased by 4 times at a rate of 2 from 64 dimensions, a high-dimensional semantic feature map of 20x20x1024 is generated, the horizontal and vertical information of the high-dimensional semantic feature map is globally encoded through a coordinate attention module (Coordinate Attention), and a long-range dependency relationship between adjacent pixel points is constructed to obtain a global information feature map.

[0061] As shown in Figure 2 The execution process of step S3 can be as follows:

[0062] S31, input a two-dimensional feature map;

[0063] S32, use adaptive pooling operation to pool each channel along the horizontal coordinate direction and the vertical coordinate direction respectively, and a pair of direction perception feature maps are generated by the transformation of the two directions;

[0064] Given a feature map F∈R C*H*W As input, it is reduced to 1 / 2(HxW) size, and then the feature map size is restored, the receptive field is reduced, and the area on which the convolution kernel slides on the feature map is expanded; the maximum pooling and average pooling operations are used on the input feature map at the same time, the information on the feature map space is compressed, and the feature information generated by the pooling operation is shared through the network to generate the attention map M C ∈R C*1*1 ; finally, the two pooled features sent into the shared network are element-wise summed, and the generated feature vector is the channel attention map M c (F), the calculation method is as follows:

[0065]

[0066] Where F represents the input feature map, AvgPool represents the average pooling operation, MaxPool represents the maximum pooling operation, MLP represents the multi-layer perception, and the two MLs share the weights W0 and W1, W0∈R C / r*C , W1∈R C*r / C σ, σ represents the Sigmoid function, represents the feature map generated after the average pooling operation in the channel direction, represents the feature map generated after the maximum pooling operation in the channel direction.

[0067] The spatial attention generates a spatial attention map using the spatial relationship of the features, which encodes the positions to be emphasized or suppressed. The spatial attention focuses on "where" an information part is, which is a supplement to the channel attention; by using two pooling operations of average pooling and maximum pooling to aggregate the channel information of the feature map, two two-dimensional images are generated, which represent the average pooling features and the maximum pooling features in the channel respectively; then they are connected and convolved through a standard convolution layer to generate a two-dimensional spatial attention map; the calculation method is as follows:

[0068]

[0069] Where M c (F) is the channel attention map, 7*7 represents the convolution calculation with a size of 7x7, represents the feature map generated after the average pooling operation in the spatial direction, represents the feature map generated after the maximum pooling operation in the spatial direction.

[0070] S33, convolution is performed on the two direction perception feature maps splicing;

[0071] S34, a normalization layer and a nonlinear mapping activation function are passed through;

[0072] S35, the feature maps are divided and convolution is performed respectively, that is, the regions of interest in the feature maps are paid attention to in the horizontal and vertical directions at the same time;

[0073] S36, finally, a Sigmoid activation function is entered to generate a global information feature map.

[0074] The feature information is extracted layer by layer from the labeled image set by using the convolutional neural network, the image size is reduced, the image dimension is deepened, the spatial information of the pooled feature maps is aggregated by using the maximum pool output and the average pool, then the two pooling results are forwarded to the shared network to generate a channel attention map, the output feature vectors are combined by using element-wise summation, and the spatial information of the high-dimensional semantic feature map is globally encoded by using the coordinate attention module, and the long-range dependency between the pixel points is modeled.

[0075] S4, the neck part is composed of a PANet and an FPN network, the global information feature map is up-sampled and fused with multiple low-dimensional feature maps from bottom to top by the PANet, the expression of the feature information in each dimension is increased, and then the feature information of the coordinate attention feature map in space and channel is fused by a multi-layer convolution attention module (C3CBAM). The output feature map of the PANet and multiple high-dimensional feature maps are fused again from bottom to top by the FPN to obtain three-dimensional feature maps, which are 80x80x256, 40x40x512 and 20x20x1024, respectively.

[0076] As shown in Figure 4 , the execution process of step S4 can be as follows:

[0077] S41, the global information feature map of the backbone is obtained as the input in the PANet;

[0078] S42, the input high-dimensional feature map P3 is expanded in space scale by one up-sampling, the number of dimensions is reduced, and after matching with the 40x40x512 dimensional feature map, the dimensions are spliced, the feature fusion is completed by the multi-layer convolution attention module (C3CBAM), and the medium-dimensional feature fusion map P2 is generated;

[0079] S43, the medium-dimensional feature map P2 is continuously up-sampled and fused with the 80x80x256 dimensional feature map after matching, and the low-dimensional feature fusion map P1 is generated;

[0080] S44, the feature map P1 is taken as the input F1 of the FPN, is down-sampled to reduce the spatial scale, is expanded in dimension, and is fused with the feature map P2 to generate a medium-scale prediction output feature map F2;

[0081] S45, the feature map F2 is down-sampled to reduce the spatial scale, is expanded in dimension, and is fused with the feature map P3 to generate a feature map F3.

[0082] As shown in Figure 3 the processing procedure of the multi-layer convolution attention module (C3CBAM) in step S42 includes:

[0083] S421, a given feature map is taken as an input, is reduced to 1 / 2 size, is restored to the size of the feature map, is reduced in receptive field, and is expanded in the area where the convolution kernel slides on the feature map; the global maximum pooling and the global average pooling operations are simultaneously applied to the input feature map, the information on the space of the feature map is compressed, the feature information generated through the pooling operation is shared through the network to generate an attention map in the channel direction; and finally, the two pooled features sent into the shared network are spliced, and the feature vector generated thereby is taken as the attention map of the channel after convolution and activation;

[0084] S422, the channel attention module performs spatial pooling on the average value and the maximum value of each row along the channel axis in the form of row vectors, and after being convolved, normalized and activated by a 7x7 standard convolution layer, is connected in residual with the channel attention map to generate a feature map, which is a multi-layer convolution attention (C3CBAM) feature map.

[0085] The PANet up-samples the global information feature map to fuse multiple low-dimensional feature maps, increases the expression of feature information in each dimension, and effectively reflects what information in the feature map is of interest and where the region of interest is through the high-dimensional semantic feature map generated by the multi-layer convolution attention.

[0086] S5, the low-dimensional feature map and the high-dimensional feature map are fused through the multi-scale feature fusion, and an image based on global information prediction is output through the convolutional neural network. The three results of the FPN can be output through a 1x1x(5+total number of recognition categories)x3 convolution in the Head part.

[0087] Embodiment one

[0088] The embodiment provides a passenger behavior detection method in an escalator scene based on global information, as shown in Figure 5 , which comprises the following steps.

[0089] Step 10, an unsafe behavior data set of passengers in an escalator is acquired, the categories and positions of the unsafe behaviors in the data set are labeled, and a labeled image data set is generated.

[0090] Step 20, create a target detection model to label the image dataset as a training sample input of the target detection model;

[0091] Step 30, extract feature information layer by layer from the labeled image dataset in the backbone part of the convolutional neural network, generate high-dimensional semantic feature maps by reducing image size and deepening image dimension, and encode the information in the horizontal and vertical directions of the high-dimensional semantic feature maps through the coordinate attention module, and build a long-range dependency relationship between adjacent pixel points to obtain a global information feature map;

[0092] Step 40, in the neck part of the convolutional neural network, the global information feature map is up-sampled and fused with multiple low-dimensional feature maps through PANet from bottom to top, the expression of feature information in each dimension is increased, and the feature information of the coordinate attention feature map in space and channel is fused through the multi-layer convolution attention module; the output feature map of the PANet and multiple high-dimensional feature maps are fused again through FPN from bottom to top to obtain multiple dimensional feature maps including low-dimensional feature maps, medium-dimensional feature maps and high-dimensional feature maps.

[0093] Step 50, in the head part of the convolutional neural network, multiple dimensional feature maps are fused through multi-scale feature fusion, and then the image based on global information prediction is output through the convolutional neural network.

[0094] In a possible implementation, in the step 30, the specific processing process of the coordinate attention module is as follows:

[0095] For a given input feature map x, two spatial range pooling kernels (H, 1) and (1, W) are used to encode each channel along the horizontal and vertical coordinates, respectively.

[0096] The output of the cth channel at height h is represented as:

[0097]

[0098] The output of the cth channel at width w is represented as:

[0099]

[0100] The above two transformations respectively aggregate features along two spatial directions to produce a pair of direction-aware feature maps And So that the coordinate attention module block captures long-range dependencies along one spatial direction and preserves precise location information along the other spatial direction.

[0101] In the step 40, the multi-layer convolution attention module includes channel attention M c (F) and spatial attention M s(F);

[0102] The channel attention M c (F) The channel attention map is generated by using the inter-channel relationship of the features, and the spatial information of the feature map is aggregated by using max-pooling and average-pooling, and then the two pooling results are forwarded to a shared network MLP to generate the channel attention map, and an element-wise sum is used to combine the output feature vectors;

[0103] The spatial attention M s (F) The spatial attention map is generated by using the spatial relationship of the features, and the positions to be emphasized or suppressed are encoded, and the channel information of the feature map is aggregated by using two pooling operations of average-pooling and max-pooling, and two two-dimensional images are generated, which represent the average-pooled features and the max-pooled features in the channel, respectively; then they are connected and convolved through a standard convolution layer to generate a two-dimensional spatial attention map.

[0104] In the step 40, the FPN is up-sampled to the feature map of the previous dimension, the dimension is cropped, and then spliced with the feature map of the next dimension after spatial and dimensional matching.

[0105] The step 10 further includes Mosaic data enhancement on the passenger unsafe behavior data set, randomly selecting 4 images in the data set, performing random cropping, scaling and rotating operations, and randomly selecting a point in the range of 1 / 3-3 / 2 of the size of the first input image as a center point for splicing to synthesize one image.

[0106] Based on the same inventive concept, the application also provides a device corresponding to the method in Embodiment One, which is described in detail in Embodiment Two.

[0107] Embodiment Two

[0108] In this embodiment, a passenger behavior detection device in an escalator scene based on global information is provided, as shown in Figure 6 The device includes a training sample processing module, a model creating module, a first processing module, a second processing module and a third processing module.

[0109] The training sample processing module is configured to obtain a passenger unsafe behavior data set in an escalator, label the categories and positions of the unsafe behaviors in the data set, and generate a labeled image data set.

[0110] The model creating module is configured to create a target detection model, and use the labeled image data set as the training sample input of the target detection model.

[0111] The first processing module is configured to extract feature information from the annotated image dataset layer by layer in a backbone part of a convolutional neural network, generate a high-dimensional semantic feature map by reducing image size and deepening image dimension, and encode information in horizontal and vertical directions of the high-dimensional semantic feature map by a coordinate attention module to build a long-range dependency relationship between adjacent pixel points and obtain a global information feature map.

[0112] The second processing module is configured to up-sample and fuse a plurality of low-dimensional feature maps in a neck part of the convolutional neural network by PANet from bottom to top, increase expression of feature information in each dimension, and fuse feature information of the coordinate attention feature map in space and channels by a multi-layer convolutional attention module.

[0113] The third processing module is configured to fuse a plurality of dimensional feature maps by multi-scale feature in a head part of the convolutional neural network, and output an image predicted based on global information by the convolutional neural network.

[0114] In a possible implementation, the specific processing process of the coordinate attention module in the first processing module is as follows.

[0115] For a given input feature map x, two spatial range pooling kernels (H, 1) and (1, W) are used to encode each channel along the horizontal coordinate and the vertical coordinate, respectively.

[0116] The output of the cth channel at the height h is represented as:

[0117]

[0118] The output of the cth channel at the width w is represented as:

[0119]

[0120] The above two transformations respectively aggregate features along two spatial directions to generate a pair of direction perception feature maps and so that the coordinate attention module block captures a long-range dependency relationship along one spatial direction and retains accurate position information along the other spatial direction.

[0121] The multi-layer convolutional attention module includes a channel attention M c (F) and a spatial attention M s (F);

[0122] The channel attention M c(F) the inter-channel relationship of features is used to generate a channel attention map, and the spatial information of the feature map is aggregated using max-pooling and average-pooling, then the two pooling results are forwarded to a shared network MLP to generate a channel attention map, and an element-wise sum is used to combine the output feature vectors;

[0123] The spatial attention M s (F) the spatial relationship of features is used to generate a spatial attention map, which encodes the positions to be emphasized or suppressed, two two-dimensional images are generated by using average-pooling and max-pooling to aggregate the channel information of the feature map, which represent the average-pooling features and the max-pooling features in the channel respectively; then they are connected and convolved through a standard convolution layer to generate a two-dimensional spatial attention map.

[0124] In the second processing module, the FPN is used to up-sample the feature map of the previous dimension, cut the dimension, and splice the feature map of the next dimension after spatial and dimensional matching.

[0125] The training sample processing module is also used for Mosaic data enhancement on the passenger unsafe behavior data set, randomly selects four pictures in the data set, performs random cropping, scaling and rotating operations on the four pictures, and splices a new image by randomly selecting a point in the range of 1 / 3-3 / 2 of the size of the first input picture as a center point.

[0126] Since the device introduced in the second embodiment of the present application is the device used to implement the method of the first embodiment of the present application, the specific structure and deformation of the device can be understood based on the method introduced in the first embodiment of the present application, and therefore will not be described here. Any device used in the method of the first embodiment of the present application belongs to the scope of protection of the present application.

[0127] The application maintains the dependence relationship between pixels after long-distance feature extraction by introducing a coordinate attention module (CA) in the backbone network part of the convolutional neural network, establishes the spatial relationship of the occluded body part in the horizontal and vertical directions, the CA alleviates the problem of loss of feature position context information after long-distance feature extraction, which helps the network to locate the information of the occluded part; the channel attention and spatial attention of the feature map in the local are improved through the multi-layer convolutional attention module (C3CBAM), the information of the occluded body part in the picture is strengthened, and the information becomes the focus of network attention; the C3CBAM enhances the attention of the position information of the occluded part provided by the CA in the channel and the space, combines the feature position information to improve the attention of small targets and occluded targets when generating a predicted target frame, and effectively feeds back the position and channel of the occluded part, thereby improving the detection performance of the network on the occluded target. The FPN and PAN structure is used in the Neck part of the convolutional neural network, the FPN layer conveys strong semantic features from top to bottom, the high-level semantic features are feature information obtained after feature extraction, the receptive field is large, the extracted features are abstract, and the high-level semantic features are conducive to object classification but are not conducive to accurate segmentation. The high-level semantic features are abstract features. The feature pyramid conveys strong positioning features from bottom to top, and the two work together to aggregate parameters for different detection layers from different backbone layers, and further improve the feature extraction capability.

[0128] Those skilled in the art will understand that embodiments of the application can be provided as methods, apparatus, or computer program products. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0129] The application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0130] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0131] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0132] While the above describes a particular embodiment of the application, those skilled in the art will understand that the specific embodiments described are only illustrative of the present application and are not intended to limit the scope of the present application, which is defined by the following claims.

Claims

1. A method for detecting passenger behavior in an escalator context based on global information, characterized in that, The method comprises the following steps: Step 10, obtaining a passenger unsafe behavior data set in an escalator, marking the category and position of the unsafe behavior in the data set, and generating a labeled image data set; Step 20, creating a target detection model, and taking the labeled image data set as the training sample input of the target detection model; Step 30, extracting feature information of the labeled image data set layer by layer in the backbone part of the convolutional neural network, generating a high-dimensional semantic feature map by reducing the image size and deepening the image dimension, and obtaining a global information feature map by constructing a long-range dependency relationship between adjacent pixel points through a coordinate attention module to globally encode information in the horizontal and vertical directions of the high-dimensional semantic feature map; Step 40, in the neck part of the convolutional neural network, a plurality of low-dimensional feature maps are fused by upsampling the global information feature map from bottom to top through PANet, the expression of feature information in each dimension is increased, and the feature information of the coordinate attention feature map in space and channel is fused through a multi-layer convolutional attention module; the output feature map of the PANet and a plurality of high-dimensional feature maps are fused again from bottom to top through FPN to obtain a plurality of dimensional feature maps including low-dimensional feature maps, medium-dimensional feature maps and high-dimensional feature maps; Step 50, in the head part of the convolutional neural network, a plurality of dimensional feature maps are fused through multi-scale feature fusion, and then an image based on global information prediction is output through the convolutional neural network; In the step 30, the specific processing process of the coordinate attention module is as follows: For a given input feature map x , two spatial range pooling kernels H (1, 1) and (1, W ) are used to encode each channel along the horizontal and vertical coordinates, respectively; height h the output of the first c channel at the second width is w the output of the c first channel is represented as: The above two transformations respectively aggregate features along two spatial directions, generating a pair of direction-aware feature maps and such that the coordinate attention module captures long-range dependencies along one spatial direction and preserves precise positional information along the other spatial direction; The multi-layer convolutional attention module in the step 40 includes channel attention M c ( F ) and spatial attention M s ( F ). The channel attention M c ( F ) generates a channel attention map using the inter-channel relationship of features, and uses maximum pooling and average pooling to aggregate the spatial information of the feature map, and then forwards the two pooling results to a shared network MLP to generate a channel attention map, and uses element-wise summation to combine the output feature vectors; The spatial attention M s ( F )The spatial attention map is generated by using the spatial relationship of the features, the positions to be emphasized or suppressed are encoded, two two-dimensional images are generated by using two pooling operations of average pooling and maximum pooling to aggregate the channel information of the feature map, and the average pooled features and the maximum pooled features in the channel are respectively represented. Then they are connected and convolved through a standard convolutional layer to generate a two-dimensional spatial attention map.

2. The method of claim 1, wherein: In the step 40, the FPN upsamples and trims the dimension of the feature map of the previous dimension, and splices the feature map after matching the space and dimension with the feature map of the next dimension.

3. The method of claim 1, wherein: The step 10 further comprises: performing Mosaic data enhancement on the passenger unsafe behavior data set, randomly selecting four images in the data set, performing random cropping, scaling and rotating operations on the four images, and splicing the four images to generate one image by randomly selecting a point in the range of 1 / 3-3 / 2 of the size of the first input image as a center point.

4. An escalator scene based passenger behavior detection apparatus based on global information, characterized by, The method comprises the following steps: The training sample processing module is used for obtaining a passenger unsafe behavior data set in an escalator, marking the category and position of the unsafe behavior in the data set, and generating a labeled image data set; The model creating module is used for creating a target detection model, and taking the labeled image data set as the training sample input of the target detection model; The first processing module is used for extracting feature information of the labeled image data set layer by layer in the backbone part of the convolutional neural network, generating a high-dimensional semantic feature map by reducing the image size and deepening the image dimension, and obtaining a global information feature map by constructing a long-range dependency relationship between adjacent pixel points through a coordinate attention module to globally encode information in the horizontal and vertical directions of the high-dimensional semantic feature map; The first processing module is used for extracting feature information of the labeled image data set layer by layer in the backbone part of the convolutional neural network, generating a high-dimensional semantic feature map by reducing the image size and deepening the image dimension, and obtaining a global information feature map by constructing a long-range dependency relationship between adjacent pixel points through a coordinate attention module to globally encode information in the horizontal and vertical directions of the high-dimensional semantic feature map; The second processing module is configured to: in a neck part of the convolutional neural network, perform upsampling fusion of a plurality of low-dimensional feature maps on the global information feature map by using a PANet from bottom to top, increase expression of feature information in each dimension, and then perform fusion of feature information in space and channels of a coordinate attention feature map by using a multi-layer convolutional attention module; and perform again fusion of an output feature map of the PANet and a plurality of high-dimensional feature maps by using a FPN from bottom to top to obtain a plurality of dimensional feature maps including low-dimensional feature maps, medium-dimensional feature maps and high-dimensional feature maps. The third processing module is configured to: in a head part of the convolutional neural network, perform fusion of the plurality of dimensional feature maps by using a multi-scale feature, and then output an image predicted based on global information by using the convolutional neural network.

5. The apparatus of claim 4, wherein: In the first processing module, a specific processing process of the coordinate attention module is as follows: For a given input feature map x , two spatial range pooling kernels H , 1) and (1, W ) are used to encode each channel along the horizontal and vertical coordinates, respectively; height h the output of the first c channel at the second width is w the output of the c first channel is represented as: The above two transformations respectively aggregate features along two spatial directions, resulting in a pair of direction-aware feature maps and such that the coordinate attention module captures long-range dependencies along one spatial direction and preserves precise positional information along the other spatial direction.

6. The apparatus of claim 5, wherein: The multi-layer convolutional attention module includes channel attention M c ( F ) and spatial attention M s ( F ); The channel attention M c ( F ) generates a channel attention map using the inter-channel relationship of features, and uses maximum pooling and average pooling to aggregate the spatial information of the feature map, and then forwards the two pooling results to a shared network MLP to generate a channel attention map, and uses element-wise summation to combine the output feature vectors; The spatial attention M s ( F )The spatial attention map is generated by using the spatial relationship of the features, the positions to be emphasized or suppressed are encoded, two two-dimensional images are generated by using two pooling operations of average pooling and maximum pooling to aggregate the channel information of the feature map, and the average pooled features and the maximum pooled features in the channel are respectively represented. Then, they are connected and convolved by a standard convolutional layer to generate a two-dimensional spatial attention map.

7. The apparatus of claim 5, wherein: In the second processing module, the FPN performs upsampling and dimension clipping on a feature map of a previous dimension, and performs splicing after spatial and dimensional matching of the feature map with a feature map of a next dimension.

8. The apparatus of claim 5, wherein: The training sample processing module is further configured to: perform Mosaic data enhancement on the passenger unsafe behavior dataset, randomly select four pictures in the dataset, perform random cropping, scaling and rotating operations on the four pictures, randomly select a point in a range of 1 / 3-3 / 2 of a size of a first input picture as a center point for splicing, and synthesize one image.

Images