Pedestrian direction detection method based on multilayer fusion feature pyramid network

Abstract

The invention relates to the technical field of pedestrian direction detection, and discloses a pedestrian direction detection method based on a multi-layer fusion feature pyramid network, which comprises the following steps: S1, splicing multiple windows: splitting a multi-level prediction feature map in a backbone network according to different window sizes, then splicing features in windows at corresponding positions in different feature maps together to form a window block sequence; and S2, GRU fusion: inputting all the prediction feature maps into GRU to perform fusion once. According to the method, a novel multi-layer fusion feature pyramid (MLF-FPN) structure is designed, a new scheme is provided for an FPN feature fusion mode, shallow context information and deep semantic information can be considered in the fusion mode, and the problem of target detection of various scales can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian direction detection, and in particular to a pedestrian direction detection method based on a multi-layer fusion feature pyramid network. Background Art

[0002] Pedestrian direction detection belongs to the category of target detection. Although today's target detection algorithms are very mature, especially those based on deep learning have achieved high performance, and there are many excellent models, such as the R-CNN series, YOLO series, SSD, and Mask R-CNN. However, there is no mature method in the field of pedestrian direction detection. Since pedestrian direction detection pays more attention to the detailed features of pedestrians, and the feature differences between different directions are small, the requirements for feature extraction are higher. Therefore, it is necessary to develop a dedicated detection method for pedestrian direction detection.

[0003] At present, the feature description of the target is mainly automatically extracted by convolutional neural network, which is more accurate and powerful than manually designed features. However, as the number of network layers increases, not only the features become more abstract, but also the loss will increase. In order to solve this problem, the feature pyramid network (FPN) was proposed, which adds a feature fusion structure after the backbone network to realize the reverse feature transfer from deep to shallow layers, thereby compensating for the problem of feature loss in the forward process. However, since the layer-by-layer transfer method is still used, the loss of features is still inevitable. In addition, the efficiency of layer-by-layer fusion is low. The above problems show that the feature fusion method of FPN still has many limitations and needs further improvement.

[0004] In addition, identifying the direction of pedestrians pays more attention to the pedestrians' limb features, because pedestrians in different directions often show different limb movements. Therefore, it is necessary to extract the pedestrians' limb features from the image to identify the pedestrians' direction. Therefore, a pedestrian direction detection method based on a multi-layer fusion feature pyramid network is proposed. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a pedestrian direction detection method based on a multi-layer fusion feature pyramid network, which solves the problem that the feature fusion of FPN usually adopts the maximum or average pooling method, but this method will lead to the loss of feature information, and the shallow prediction feature map has rich context information, but the semantic information is relatively missing, while the deep prediction feature map is the opposite. Therefore, in the prediction feature maps at each level, both types of information have different degrees of loss.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A pedestrian direction detection method based on a multi-layer fusion feature pyramid network includes the following steps:

[0010] S1: Multi-window splicing: split the multi-level prediction feature maps in the backbone network according to different window sizes, and then splice the features in the windows at corresponding positions in different feature maps together to form a window block sequence;

[0011] S2: GRU fusion, all predicted feature maps are input into GRU to perform a fusion, so that each level of predicted feature map has context information and deep semantic information, and can be executed in parallel, taking the entire sequence as input without having to input them in sequence;

[0012] S3: Window block splitting: split the points on the GRU output feature map according to the window, and restore these feature points to the corresponding positions of the original prediction feature map. A body part map is constructed on the feature map to represent the relationship between body parts, and a graph recurrent encoder and a graph recurrent attention decoder are proposed to propagate node features on the body part map.

[0013] S4: Graph Recurrent Attention Network, a body part graph is constructed on the feature graph output by RPN, the relationship between body parts is represented in the form of a graph structure, and then the body part graph is input into the encoder module. A fixed adjacency matrix is ​​used in the encoder module to represent the node relationship. In the decoder module, an adjacency matrix with attention edge weights is used. GRU is used in the encoder and decoder to transfer information between nodes. The graph features output by the decoder are passed to the classifier to achieve the final pedestrian direction classification.

[0014] As a further solution of the present invention, the stitching of windows in S1 first flattens the features in each window.

[0015] Furthermore, the S1 includes a multi-window stitching module, and the multi-window stitching module is connected to a feature map information acquisition module.

[0016] On the basis of the above solution, S2 includes a parallel fusion module, and the parallel fusion module is connected to the multi-window splicing module.

[0017] Furthermore, S3 includes a window block splitting module, and the window block splitting module is connected to the parallel fusion module.

[0018] As a further solution of the present invention, the S4 includes a decoder module, an encoder module and an information transmission module, and the information transmission module is connected to the window block splitting module.

[0019] (III) Beneficial effects

[0020] Compared with the prior art, the present invention provides a pedestrian direction detection method based on a multi-layer fusion feature pyramid network, which has the following beneficial effects:

[0021] 1. In the present invention, the multi-window splicing operation can convert the feature map into a 1D sequence so that it can be used as the input of GRU, and the window block splitting operation can restore the 1D sequence output by GRU to the size of the original predicted feature map, so as to set the anchor on the new feature map.

[0022] 2. The present invention designs a novel multi-layer fusion feature pyramid (MLF-FPN) structure, which provides a new solution for the FPN feature fusion method. This fusion method can take into account both shallow contextual information and deep semantic information, and can cope with target detection problems at various scales.

[0023] 3. The present invention proposes an adjacency matrix with attention edge weights in the graph recurrent attention decoder, instead of using an artificially designed fixed adjacency matrix. This weight can be learned during training, thereby improving the learning ability of the graph network to understand the characteristics of different body parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the steps of a pedestrian direction detection method based on a multi-layer fusion feature pyramid network proposed by the present invention;

[0025] Figure 2 A schematic diagram showing a splicing display of a pedestrian direction detection method based on a multi-layer fusion feature pyramid network proposed in the present invention; DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Reference Figure 1-2 , a pedestrian direction detection method based on a multi-layer fusion feature pyramid network, comprising the following steps:

[0028] S1: Multi-window splicing. In order to input the prediction feature map into GRU, the following two issues need to be considered first: (a) GRU receives a 1D sequence as input, while the prediction feature map in the pyramid is 2D; (b) Simply splicing all the prediction feature maps together will produce a particularly long 1D sequence, which is bound to lead to higher computational complexity. Based on the above two points, the present invention first splits the prediction feature map according to the window and then splices it to form a 1D window block sequence. Specifically, a sliding window is placed on the prediction feature map at each level. The size of the window and the sliding step size are set differently for each prediction feature map so that it can evenly cover each position of the prediction feature map without overlapping. Then, each window is flattened and the windows at corresponding positions in the prediction feature map at each level are spliced ​​together to form a 1D window block sequence, such as Figure 2 As shown in the figure, taking the 3-layer prediction map as an example, the multi-window splicing method is demonstrated. For the N prediction feature maps output by the backbone network, n∈{1,2,...,N} is used to represent the layer, and n=1 is specified as the deepest layer in the pyramid. Then the sliding window size of the nth layer can be expressed as 2 n-1 ×2 n-1 , the sliding step is 2 n-1 , window sliding execution h 1 × 1 You can traverse all positions of the predicted feature map in one step, where h 1 and w 1 are the height and width of the deepest predicted feature map, t∈{1,...,h 1 × 1} is the number of sliding windows, all windows at the same position in the feature map are connected into a window block, and then all wp t Connect into a window block diagram (wpm), wp t The splicing method of and wpm can be expressed as

[0029]

[0030] in It represents the feature area covered by the t-th sliding of the window on the n-th layer prediction feature map. It is formed by concatenating the features covered by the t-th window sliding on all prediction images;

[0031] S2: GRU fusion, for efficient execution, in each wp t Internally, multiple layers of fusion are performed, because each wp t Contains all window features at the same position from deep to shallow, so in each wp t In the above example, we can use GRU’s ability to extract global information to fuse the features of each layer. In addition, each wp tThey are independent of each other, so they can be executed in parallel, which improves efficiency. For efficient feature fusion, a standard GRU encoder is used;

[0032] S3: Window block splitting. The window block sequence output by GRU already contains the fusion features of each predicted feature map. The next step is to restore the GRU output to the predicted feature map of the pyramid. Specifically, each output wp t Split by window and restore to the corresponding window position of the corresponding prediction feature map. This process can be regarded as the inverse process of multi-window splicing, which is called window block splitting;

[0033] S4: Graph Recurrent Attention Network, a body part graph is constructed to model the dependencies between them. The body part graph structure is built on the region proposal box generated by RPN. Specifically, the region proposal box is first uniformly scaled to a feature map with a fixed size of 3×2 through the ROIPooling layer. Each point on this feature map represents a node on the graph, which can be defined as G=(V,E), where V={v i |i∈(1,2,3,4,5,6)} represents 6 body parts; It is called an edge, which represents the internal relationship between body parts, that is, the relationship between the head, arms and legs. The initial feature description of the node is obtained by ROIPooling and is defined as X∈R 6×d , where d represents the feature dimension. In order to enhance the learning ability of the model, an adjacency matrix with attention edge weights is used in GRAN to represent the relationship between graph nodes.

[0034] In the GRAN-based module, two GRAN layers are stacked. It has two inputs, one is the node’s own information, and the other is the aggregated information from neighboring nodes. The information aggregation of neighboring nodes is obtained by calculating the weighted sum of its neighboring nodes using the adjacency matrix with attention edge weights. The state update of the node is implemented through GRU.

[0035] In the description of this article, it should be noted that relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0036] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A pedestrian direction detection method based on a multi-layer fusion feature pyramid network, characterized in that: The following steps are involved: S1: Multi-window splicing: split the multi-level prediction feature maps in the backbone network according to different window sizes, and then splice the features in the windows at corresponding positions in different feature maps together to form a window block sequence; S2: GRU fusion, all predicted feature maps are input into GRU to perform a fusion, so that each level of predicted feature map has context information and deep semantic information, and can be executed in parallel, taking the entire sequence as input without having to input them in sequence; S3: Window block splitting: split the points on the GRU output feature map according to the window, and restore these feature points to the corresponding positions of the original prediction feature map. A body part map is constructed on the feature map to represent the relationship between body parts, and a graph recurrent encoder and a graph recurrent attention decoder are proposed to propagate node features on the body part map. S4: Graph Recurrent Attention Network, a body part graph is constructed on the feature graph output by RPN, the relationship between body parts is represented in the form of a graph structure, and then the body part graph is input into the encoder module. A fixed adjacency matrix is ​​used in the encoder module to represent the node relationship. In the decoder module, an adjacency matrix with attention edge weights is used. GRU is used in the encoder and decoder to transfer information between nodes. The graph features output by the decoder are passed to the classifier to achieve the final pedestrian direction classification.

2. According to claim 1, a pedestrian direction detection method based on a multi-layer fusion feature pyramid network is characterized in that: The stitching of windows in S1 first flattens the features in each window.

3. According to claim 2, a pedestrian direction detection method based on a multi-layer fusion feature pyramid network is characterized in that: The S1 includes a multi-window splicing module, and the multi-window splicing module is connected to a feature map information acquisition module.

4. The method for pedestrian direction detection based on a multi-layer fusion feature pyramid network according to claim 3 is characterized in that: The S2 includes a parallel fusion module, and the parallel fusion module is connected to the multi-window splicing module.

5. The method for pedestrian direction detection based on a multi-layer fusion feature pyramid network according to claim 4 is characterized in that: The S3 includes a window block splitting module, and the window block splitting module is connected to the parallel fusion module.

6. The method for pedestrian direction detection based on a multi-layer fusion feature pyramid network according to claim 5, characterized in that: The S4 includes a decoder module, an encoder module and an information transmission module, and the information transmission module is connected to the window block splitting module.

Images