Improved ST-GCN automatic driving vehicle pedestrian intention recognition method

Abstract

The invention discloses an improved ST-GCN-based pedestrian intention recognition method for an automatic driving vehicle, and particularly relates to a method which is based on a space-time graph convolutional network ST-GCN, introduces an improved adaptive graph convolutional network in a spatial dimension, enhances a dependency relationship between arms and legs of a pedestrian, and enhances extraction of effective spatial features in pedestrian actions. According to the pedestrian intention recognition method, in the time dimension, multi-branch time convolution is adopted, the remote dependency relationship of pedestrian intentions is captured, the attention of a model to pedestrian channel information is improved, and the time change relation of the model to pedestrian actions is enhanced. The invention aims to quickly and accurately identify the street crossing intention of pedestrians, so that an advanced auxiliary driving system or an automatic driving vehicle can provide sufficient time for a driver to make a response or directly intervene in driving to prevent collision. The method is realized by the following steps of: 1, capturing a pedestrian video stream in a traffic scene through a vehicle-mounted monocular camera; 2, predicting pedestrian key points by using a human body posture estimation network, extracting coordinates of the pedestrian key points, and generating a pedestrian 2D skeleton structure chart; and step 3, fusing the adaptive graph convolution network and the multi-branch time convolution with the ST-GCN network to realize pedestrian intention recognition of the automatic driving vehicle. The method is suitable for pedestrian crossing intention recognition in an automatic driving vehicle scene.

Description

Technical Field

[0001] The present invention relates to the field of graph convolutional neural networks, and in particular to a method for pedestrian intention recognition of an autonomous driving vehicle based on an improved ST-GCN. Background Art

[0002] With the development of computer vision in the field of autonomous driving, the prospects for autonomous vehicles are very bright and their economic investment is booming. However, according to the "Global Status Report on Road Safety" released by the World Health Organization, the number of road deaths worldwide has reached an all-time high, with 1.35 million deaths per year. Among them, nearly half of the victims of traffic accidents are vulnerable road users, such as pedestrians crossing the street. Avoiding vehicle-pedestrian collisions is a key requirement for today's advanced driver assistance systems and future autonomous vehicles. Against this background, it is particularly important for advanced driver assistance systems to predict pedestrians' intentions as early as possible, provide drivers with sufficient time to react, or directly intervene to prevent collisions.

[0003] At present, pedestrian recognition in road traffic scenarios is usually used for fixed area monitoring of important traffic intersections. The field of view is fixed and cannot meet the pedestrian intention recognition of autonomous driving vehicles in different environments and scenarios. Due to the life safety issues involved, pedestrian posture estimation under autonomous driving vehicles requires very high precision, accuracy and detection speed. Therefore, in order to meet the precision, accuracy and detection speed of pedestrian intention detection under autonomous driving vehicles, the present invention proposes an improved ST-GCN method for pedestrian intention recognition of autonomous driving vehicles, which provides theoretical guidance and solutions for autonomous driving vehicles and advanced driver assistance systems. The purpose of the present invention is to predict the intention of pedestrians as early as possible, provide the driver with sufficient time to react or directly intervene in the driver to prevent collisions with pedestrians. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes an improved ST-GCN pedestrian intention recognition method for autonomous driving vehicles, which realizes the prediction and recognition of pedestrian crossing intention in autonomous driving vehicles.

[0005] The present invention is implemented by the following technical solutions:

[0006] Improve the ST-GCN method for pedestrian recognition in autonomous driving vehicles. The method is implemented using the following steps:

[0007] Step 1: Capture the pedestrian video stream in the traffic scene through the vehicle-mounted monocular camera;

[0008] Step 2: Use the human posture estimation network to predict the key points of pedestrians, extract the coordinates of the key points of pedestrians, and generate a 2D skeleton structure diagram of pedestrians;

[0009] Step three, by integrating the adaptive graph convolutional network and multi-branch temporal convolution with the ST-GCN network, pedestrian intention recognition for autonomous driving vehicles is achieved.

[0010] Specifically, step one includes: the monocular USB camera carried by the autonomous driving vehicle collects the color video stream of the intelligent driving vehicle in the traffic scene with a resolution of up to 2K, transmits the video stream to the on-board computing platform, and obtains the video resolution to define the pixel coordinate system.

[0011] The specific steps in step 2 include: (1) Use the 5×5 depthwise separable convolution introduced in the basic components of CSPDarkNet to build a single-stage object detector to detect pedestrians and obtain the position of pedestrians in the pixel coordinate system of the input image frame, and extract the human image based on the located pedestrian frame; wherein, the pedestrian dataset is the public JAAD dataset and a self-made dataset based on actual traffic scenes. (2) The pedestrian dataset is annotated in MSCOCO format to train the human posture estimation network. Through the dataset training, the human posture network learns the coordinates of the X-axis and Y-axis of the pedestrian's joints and generates a pedestrian skeleton graph; wherein, the generated pedestrian joints include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0012] The specific steps in step three include:

[0013] (1) Construct the temporal and spatial structure of the pedestrian skeleton graph; each frame of the pedestrian skeleton is represented by an undirected spatiotemporal graph G = (V, E), where V represents the joint point and E represents the bone edge. The joint point set in the pedestrian skeleton spatiotemporal graph is set to V = {v ti |t=1,...,T,i=1,...,N}, t represents the time sequence number of the skeleton frame, and i represents the spatial sequence number of the joint point in each frame. The feature vector of the pedestrian skeleton key point i in the tth frame is F(v ti ), which consists of the x, y coordinates and confidence of the point. Divide all edges E into two sets. In the spatial domain, it represents the skeleton information of the human skeleton, and E S = {v ti v tj |(i,j)∈H}, where H represents the connection of the key points of the pedestrian skeleton; in the time domain, it represents the skeleton time motion information, denoted by E F = {v ti v (t+1)i |t=1,...,T-1}, for joint point i, if E F The time edges in the graph convolution network are connected to form the time motion trajectory of the joint point i. tiThe calculation formula for the graph convolution operation is as follows:

[0014]

[0015] Where v is the human joint point of the graph, and f is the feature graph; B ti Indicates v ti The convolution sampling area, v tj Corresponding to v ti The neighboring nodes with a distance of 1 in the sampling area; ω is the weight function. B ti The number of vertices in is variable; the mapping function m ti Make joints have unique weights.

[0016] The graph convolution formula can be summarized as follows:

[0017]

[0018] Among them, K is the number of subset categories; represents element-wise multiplication; M is an attention map representing the importance of each joint point; Represents the adjacency matrix of the undirected graph connected in the object; I is the identity matrix; W k is the trainable weight matrix. Adding the characteristics of its own node, it is expressed as D k is a normalized degree matrix;

[0019] (2) Improved adaptive graph convolutional network: In the spatial dimension, the characteristics of pedestrians are expressed as the coordinates of the key points of the pedestrian skeleton in a single frame. The spatial graph convolution of the original ST-GCN network is based on the natural connection relationship of the pedestrian skeleton. The topological structure of the pedestrian skeleton is fixed, and there is no direct connection between the arms and legs of the pedestrian. For pedestrian intention recognition, it is impossible to assign weights to them, causing the network to ignore the relationship between the arms and legs. It is impossible to learn well actions such as walking and running that require consideration of the interaction between the upper and lower limbs of pedestrians. Therefore, this paper improves the spatial graph convolution layer of the original ST-GCN network, uses the improved adaptive graph convolutional network, learns the connection between pedestrian nodes, and optimizes unnecessary redundant parameters, which are specifically expressed as follows:

[0020]

[0021] Among them, f in To input pedestrian skeleton features, W k is the convolution weight. K v is the convolution kernel size. Since the spatial partition strategy of dividing nodes into three groups, namely root, centripetal, and centrifugal, is adopted, the convolution kernel size is set to 3. B kIt is a parameterized graph that can be learned from data during training. The initial parameters are the normalized adjacency matrix of the physical connection of the pedestrian skeleton structure. α represents a trainable coefficient, which represents Figure C k The importance in different blocks is initialized to 0. C k The calculation method is as follows:

[0022]

[0023] Among them, W θk Denotes the weight of the embedded convolution operation θ, and V represents the number of pedestrian joints. Use the convolution operation θ to embed the input feature f in The correlation between two pedestrian joints is measured using the embedded feature map and its deconvolution, resulting in the matrix And use tanh to normalize C k Therefore, C k The element in the i-th row and j-th column represents the pedestrian joint point v i and joint point v j The correlation between

[0024] (3) Multi-branch temporal convolution: In the temporal dimension, the original ST-GCN network uses a one-dimensional convolution with a convolution kernel of 9 to model temporal information. The large convolution kernel covers a wide temporal receptive field. This design lacks flexibility and will lead to redundancy in model parameters and computation. In order to more effectively capture the long-range dependencies of pedestrian intentions, multi-branch temporal convolution is used to effectively capture the long-range dependencies of pedestrian intentions and the coarse-to-fine-grained features of pedestrians, and reduce the complexity of the network. The multi-branch temporal convolution used in the multi-branch temporal convolution network structure consists of 6 branches: a 1×1 convolution branch, a maximum pooling branch, and 4 transient one-dimensional convolution branches. The convolution kernel size is 3 and the expansion coefficient is 1 to 4. The multi-branch temporal convolution first uses a 1×1 convolution to transform the features and divides them into 6 groups with equal channel widths. A single branch is used to process each feature group. The residual connection is when the convolution stride is 2, and a residual block is inserted in the residual path to match the input size with the main path output size. The six outputs are connected together to form the output of multi-branch temporal convolution. Multi-branch temporal convolution effectively improves the temporal modeling capability of pedestrian skeleton structure, and reduces the amount of calculation and parameters of the model by reducing the channel width of each branch;

[0025] (4) Improved ST-GCN model; the improved network is mainly composed of 9 basic spatiotemporal unit blocks. The number of output channels of each spatiotemporal unit block is 64, 64, 64, 128, 128, 128, 256, 256 and 256 respectively. A batch normalization layer is added at the beginning of the improved network to normalize the input data, so that the network can be trained under a better data distribution, which increases the convergence speed of the network. A global average pooling layer is executed at the end of the improved network to collect and unify the feature maps of different samples. The output of the global average pooling layer is sent to the Softmax classifier to calculate the probability of each category, and the category with the largest probability is used as the prediction result to obtain the pedestrian crossing intention prediction result.

[0026] The beneficial effects of the above technical solution of the present invention are as follows:

[0027] (1) The present invention can effectively identify pedestrians' intention to cross the street at different angles and in different traffic scenarios, and meet the needs of predicting and identifying pedestrians' intention to cross the street by autonomous driving vehicles;

[0028] (2) The method for pedestrian intention recognition of autonomous driving vehicles using the improved ST-GCN proposed in the present invention is a lightweight and powerful model framework. Deployment of it in an autonomous driving vehicle or an advanced driver assistance system can meet the requirements for rapidity and accuracy in pedestrian crossing intention recognition, and can avoid collisions between autonomous driving vehicles and pedestrians to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0030] Figure 1 is a flow chart of the method of the present invention.

[0031] Figure 2 It is a schematic diagram of the spatial-temporal relationship of the pedestrian skeleton in step 4 of the present invention.

[0032] Figure 3 It is a schematic diagram of the overall framework of the improved ST-GCN network in step 4 of the present invention.

[0033] Figure 4 It is a schematic diagram of the improved adaptive graph convolutional network in step four of the present invention.

[0034] Figure 5 It is a schematic diagram of multi-branch temporal convolution in step 4 of the present invention.

[0035] Figure 6 It is a visualization view of the experiment of the present invention, using a heat map to represent the dependency of joint actions.

[0036] Figure 7 It is an autonomous driving vehicle experimental platform used in the present invention. DETAILED DESCRIPTION

[0037] The present invention is described in detail below in conjunction with specific embodiments.

[0038] Improve the ST-GCN method for pedestrian recognition in autonomous driving vehicles. The method is implemented using the following steps:

[0039] Step 1: Capture the pedestrian video stream in the traffic scene through the vehicle-mounted monocular camera;

[0040] Step 2: Use the human posture estimation network to predict the key points of pedestrians, extract the coordinates of the key points of pedestrians, and generate a 2D skeleton structure diagram of pedestrians;

[0041] Step three, by integrating the adaptive graph convolutional network and multi-branch temporal convolution with the ST-GCN network, pedestrian intention recognition for autonomous driving vehicles is achieved.

[0042] In step 1, the specific steps include: the monocular USB camera carried by the autonomous driving vehicle collects the color video stream of the intelligent driving vehicle in the traffic scene with a resolution of up to 2K, transmits the video stream to the on-board computing platform, and obtains the video resolution to define the pixel coordinate system such as Figure 7 shown.

[0043] In the step 2, the specific steps include:

[0044] (1) A single-stage object detector is constructed by introducing a 5×5 depthwise separable convolution into the basic building block of CSPDarkNet to detect pedestrians and obtain their positions in the pixel coordinate system of the input image frame. The human body image is extracted based on the located pedestrian frame. The pedestrian dataset is the public JAAD dataset and a self-made dataset based on actual traffic scenarios.

[0045] (2) The pedestrian dataset is annotated in MSCOCO format to train the human posture estimation network. Through the dataset training, the human posture network learns the coordinates of the pedestrian's joints on the X-axis and Y-axis to generate a pedestrian skeleton graph; the generated pedestrian joints include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, such as Figure 6 shown.

[0046] In the step three, the specific steps include:

[0047] (1) Construct the temporal and spatial structure of the pedestrian skeleton graph; each frame of the pedestrian skeleton is represented by an undirected spatiotemporal graph G = (V, E), where V represents the joint point and E represents the bone edge. The joint point set in the pedestrian skeleton spatiotemporal graph is set to V = {v ti |t=1,...,T,i=1,...,N}, t represents the time sequence number of the skeleton frame, and i represents the spatial sequence number of the joint point in each frame. The feature vector of the pedestrian skeleton key point i in the tth frame is F(v ti ), which consists of the x, y coordinates and confidence of the point. Divide all edges E into two sets. In the spatial domain, it represents the skeleton information of the human skeleton, and E S = {v ti v tj |(i,j)∈H}, where H represents the connection of the key points of the pedestrian skeleton; in the time domain, it represents the skeleton time motion information, denoted by E F = {v ti v (t+1)i |t=1,...,T-1}, for joint point i, if E F The time edges in the graph convolution network are connected to form the time motion trajectory of the joint point i. ti The calculation formula for the graph convolution operation is as follows:

[0048]

[0049] Where v is the human joint point of the graph, and f is the feature graph; B ti Indicates v ti The convolution sampling area, v tj Corresponding to v ti The neighboring nodes with a distance of 1 in the sampling area; ω is the weight function. B ti The number of vertices in is variable; the mapping function m ti Make joints have unique weights.

[0050] The graph convolution formula can be summarized as follows:

[0051]

[0052] Among them, K is the number of subset categories; represents element-wise multiplication; M is an attention map representing the importance of each joint point; Represents the adjacency matrix of the undirected graph connected in the object; I is the identity matrix; W k is the trainable weight matrix. Adding the characteristics of its own node, it is expressed as D k is a normalized degree matrix, such as Figure 2 As shown;

[0053] (2) Improved adaptive graph convolutional network: In the spatial dimension, the characteristics of pedestrians are expressed as the coordinates of the key points of the pedestrian skeleton in a single frame. The spatial graph convolution of the original ST-GCN network is based on the natural connection relationship of the pedestrian skeleton. The topological structure of the pedestrian skeleton is fixed, and there is no direct connection between the arms and legs of the pedestrian. For pedestrian intention recognition, it is impossible to assign weights to them, causing the network to ignore the relationship between the arms and legs. It is impossible to learn well actions such as walking and running that require consideration of the interaction between the upper and lower limbs of pedestrians. Therefore, this paper improves the spatial graph convolution layer of the original ST-GCN network, uses the improved adaptive graph convolutional network, learns the connection between pedestrian nodes, and optimizes unnecessary redundant parameters, which are specifically expressed as follows:

[0054]

[0055] Among them, f in To input pedestrian skeleton features, W k is the convolution weight. K v is the convolution kernel size. Since the spatial partition strategy of dividing nodes into three groups, namely root, centripetal, and centrifugal, is adopted, the convolution kernel size is set to 3. B k It is a parameterized graph that can be learned from data during training. The initial parameters are the normalized adjacency matrix of the physical connection of the pedestrian skeleton structure. α represents a trainable coefficient, which represents Figure C k The importance in different blocks is initialized to 0. C k The calculation method is as follows:

[0056]

[0057] Among them, W θk Denotes the weight of the embedded convolution operation θ, and V represents the number of pedestrian joints. Use the convolution operation θ to embed the input feature f in The correlation between two pedestrian joints is measured using the embedded feature map and its deconvolution, resulting in the matrix And use tanh to normalize C k Therefore, C k The element in the i-th row and j-th column represents the pedestrian joint point v i and joint point v j The correlation between Figure 4 As shown;

[0058] (3) Multi-branch temporal convolution: In the temporal dimension, the original ST-GCN network uses a one-dimensional convolution with a convolution kernel of 9 to model temporal information. The large convolution kernel covers a wide temporal receptive field. This design lacks flexibility and will lead to redundancy in model parameters and computation. In order to more effectively capture the long-range dependencies of pedestrian intentions, multi-branch temporal convolution is used to effectively capture the long-range dependencies of pedestrian intentions and the coarse-to-fine-grained features of pedestrians, and reduce the complexity of the network. The multi-branch temporal convolution used in the multi-branch temporal convolution network structure consists of 6 branches: a 1×1 convolution branch, a maximum pooling branch, and 4 transient one-dimensional convolution branches. The convolution kernel size is 3 and the expansion coefficient is 1 to 4. The multi-branch temporal convolution first uses a 1×1 convolution to transform the features and divides them into 6 groups with equal channel widths. A single branch is used to process each feature group. The residual connection is when the convolution stride is 2, and a residual block is inserted in the residual path to match the input size with the main path output size. The six outputs are connected together to form the output of multi-branch temporal convolution. Multi-branch temporal convolution effectively improves the temporal modeling capability of pedestrian skeleton structure, and reduces the amount of calculation and parameters of the model by reducing the channel width of each branch. Figure 5 As shown;

[0059] (4) Improved ST-GCN network; the improved network is mainly composed of 9 basic spatiotemporal unit blocks. The number of output channels of each spatiotemporal unit block is 64, 64, 64, 128, 128, 128, 256, 256 and 256 respectively. A batch normalization layer is added at the beginning of the improved network to normalize the input data so that the network can be trained under better data distribution, increasing the convergence speed of the network. A global average pooling layer is executed at the end of the improved network to collect and unify the feature maps of different samples. The output of the global average pooling layer is sent to the Softmax classifier to calculate the probability of each category, and the category with the largest probability is used as the prediction result to obtain the pedestrian crossing intention prediction result, such as Figure 3 shown.

Claims

1. An improved ST-GCN method for pedestrian intention recognition in an autonomous driving vehicle, characterized in that: Includes steps: Step 1: Capture the pedestrian video stream in the traffic scene through the vehicle-mounted monocular camera; Step 2: Use the human posture estimation network to predict the key points of pedestrians, extract the coordinates of the key points of pedestrians, and generate a 2D skeleton structure diagram of pedestrians; Step three, by integrating the adaptive graph convolutional network and multi-branch temporal convolution with the ST-GCN network, pedestrian intention recognition for autonomous driving vehicles is achieved.

2. The method for pedestrian recognition of an autonomous driving vehicle based on an improved ST-GCN according to claim 1 is characterized by: In the step one, specifically: the monocular USB camera carried by the autonomous driving vehicle collects the color video stream of the intelligent driving vehicle in the traffic scene with a resolution of up to 2K, transmits the video stream to the on-board computing platform, and obtains the video resolution to define the pixel coordinate system.

3. The improved ST-GCN pedestrian intention recognition method for autonomous driving vehicles according to claim 1 is characterized by: In the step 2, the specific steps include: (1) A single-stage object detector is constructed by introducing a 5×5 depthwise separable convolution into the basic building block of CSPDarkNet to detect pedestrians and obtain their positions in the pixel coordinate system of the input image frame. The human body image is extracted based on the located pedestrian frame. The pedestrian dataset is the public JAAD dataset and a self-made dataset based on actual traffic scenarios. (2) The pedestrian dataset is annotated into the MSCOCO format to train the human posture estimation network. Through the dataset training, the human posture network learns the coordinates of the pedestrian's joints on the X-axis and Y-axis to generate a pedestrian skeleton graph; wherein the generated pedestrian joints include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

4. The method for pedestrian intention recognition of an autonomous driving vehicle based on the improved ST-GCN according to claim 1 is characterized in that: In the step three, the specific steps include: (1) Construct the temporal and spatial structure of the pedestrian skeleton graph; each frame of the pedestrian skeleton is represented by an undirected spatiotemporal graph G = (V, E), where V represents the joint point and E represents the bone edge. The joint point set in the pedestrian skeleton spatiotemporal graph is set to V = {v ti |t=1,...,T,i=1,...,N}, t represents the time sequence number of the skeleton frame, and i represents the spatial sequence number of the joint point in each frame. The feature vector of the pedestrian skeleton key point i in the tth frame is F(v ti ), which consists of the x, y coordinates and confidence of the point. Divide all the edges E into two sets. In the spatial domain, it represents the skeleton information of the human skeleton, and E S = {v ti v tj |(i,j)∈H}, where H represents the connection of the key points of the pedestrian skeleton; in the time domain, it represents the skeleton time motion information, denoted by E F = {v ti v (t+1)i |t=1,...,T-1}, for joint point i, if E F The time edges in the graph convolution network are connected to form the time motion trajectory of the joint point i. ti The calculation formula for the graph convolution operation is as follows: Where v is the human joint point of the graph, and f is the feature graph; B ti Indicates v ti The convolution sampling area, v tj Corresponding to v ti The neighboring nodes with a distance of 1 in the sampling area; ω is the weight function. B ti The number of vertices in is variable; the mapping function m ti Make joints have unique weights. The graph convolution formula can be summarized as follows: Among them, K is the number of subset categories; represents element-wise multiplication; M is an attention map representing the importance of each joint point; Represents the adjacency matrix of the undirected graph connected in the object; I is the identity matrix; W k is the trainable weight matrix. Adding the characteristics of its own node, it is expressed as D k is a normalized degree matrix; (2) Improved adaptive graph convolutional network: In the spatial dimension, the characteristics of pedestrians are expressed as the coordinates of the key points of the pedestrian skeleton in a single frame. The spatial graph convolution of the original ST-GCN network is based on the natural connection relationship of the pedestrian skeleton. The topological structure of the pedestrian skeleton is fixed, and there is no direct connection between the arms and legs of the pedestrian. For pedestrian intention recognition, it is impossible to assign weights to them, causing the network to ignore the relationship between the arms and legs. It is impossible to learn well actions such as walking and running that require consideration of the interaction between the upper and lower limbs of pedestrians. Therefore, this paper improves the spatial graph convolution layer of the original ST-GCN network, uses the improved adaptive graph convolutional network, learns the connection between pedestrian nodes, and optimizes unnecessary redundant parameters, which are specifically expressed as follows: Among them, f in To input pedestrian skeleton features, W k is the convolution weight. K v is the convolution kernel size. Since the spatial partition strategy of dividing nodes into three groups, namely root, centripetal, and centrifugal, is adopted, the convolution kernel size is set to 3. B k It is a parameterized graph that can be learned from data during training. The initial parameters are the normalized adjacency matrix of the physical connection of the pedestrian skeleton structure. α represents a trainable coefficient, which represents Figure C k The importance in different blocks is initialized to 0. C k The calculation method is as follows: Among them, W θk Denotes the weight of the embedded convolution operation θ, and V represents the number of pedestrian joints. Use the convolution operation θ to embed the input feature f in The correlation between two pedestrian joints is measured using the embedded feature map and its deconvolution, resulting in the matrix And use tanh to normalize C k Therefore, C k The element in the i-th row and j-th column represents the pedestrian joint point v i and joint point v j The correlation between (3) Multi-branch temporal convolution: In the temporal dimension, the original ST-GCN network uses a one-dimensional convolution with a convolution kernel of 9 to model temporal information. The large convolution kernel covers a wide temporal receptive field. This design lacks flexibility and will lead to redundancy in model parameters and computation. In order to more effectively capture the long-range dependencies of pedestrian intentions, multi-branch temporal convolution is used to effectively capture the long-range dependencies of pedestrian intentions and the coarse-to-fine-grained features of pedestrians, and reduce the complexity of the network. The multi-branch temporal convolution used in the multi-branch temporal convolution network structure consists of 6 branches: a 1×1 convolution branch, a maximum pooling branch, and 4 transient one-dimensional convolution branches. The convolution kernel size is 3 and the expansion coefficient is 1 to 4. The multi-branch temporal convolution first uses a 1×1 convolution to transform the features and divides them into 6 groups with equal channel widths. A single branch is used to process each feature group. The residual connection is when the convolution stride is 2, and a residual block is inserted in the residual path to match the input size with the main path output size. The six outputs are connected together to form the output of multi-branch temporal convolution. Multi-branch temporal convolution effectively improves the temporal modeling capability of pedestrian skeleton structure, and reduces the amount of calculation and parameters of the model by reducing the channel width of each branch; (4) Improved ST-GCN network; the improved network is mainly composed of 9 basic spatiotemporal unit blocks. The number of output channels of each spatiotemporal unit block is 64, 64, 64, 128, 128, 128, 256, 256 and 256 respectively. A batch normalization layer is added at the beginning of the improved network to normalize the input data, so that the network can be trained under a better data distribution, which increases the convergence speed of the network. A global average pooling layer is executed at the end of the improved network to collect and unify the feature maps of different samples. The output of the global average pooling layer is sent to the Softmax classifier to calculate the probability of each category, and the category with the largest probability is used as the prediction result to obtain the pedestrian crossing intention prediction result.

Images