A method for predicting pedestrian crossing intention

Abstract

The present invention discloses a method for predicting pedestrian crossing intention, which includes: obtaining a fully trained target prediction model, an RGB image to be processed, and a pedestrian optical flow map, wherein the target prediction model includes a color map prediction network structure and an optical flow map prediction network structure, the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module; the computational complexity of the color map and the pedestrian optical flow map is compressed by the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, and then the feature extraction and learning capabilities of important channels in complex spaces are improved by the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, thereby reducing the computational complexity and parameter quantity of the pedestrian crossing intention prediction model and improving the possibility of the prediction model being applied in practice.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for predicting pedestrian crossing intention. Background Art

[0002] Data from the WTO's 2018 Road Safety Report shows that approximately 1.35 million people die in traffic accidents worldwide each year, and more than half of these accidents involve pedestrians, cyclists, and the like. These vulnerable road users are unable to protect themselves when a traffic accident occurs and can only rely on the driver's judgment. With the continuous development of unmanned driving, autonomous driving systems have increasingly higher levels of autonomous judgment capabilities. Therefore, in order to reduce the occurrence of such traffic accidents, autonomous driving systems should have the ability to predict pedestrian crossing intentions and perceive possible dangerous situations in advance. Since the assembly of sensor equipment such as lidar is very expensive, it is not easy to put them into use on a large scale in vehicles. However, cameras, as relatively inexpensive environmental perception devices, can be installed on various vehicles. Relying on the development of computer vision, monocular cameras can also complete various visual perception tasks. Therefore, the present invention achieves fast and accurate prediction of pedestrian crossing intentions using only a monocular camera.

[0003] Most current pedestrian crossing intention prediction algorithms use pedestrian pose estimation algorithms to obtain key points of various pedestrian parts, such as the head, hands, and legs. Object detection algorithms can accurately determine pedestrian positions, but in densely populated scenes, there is a large overlap between pedestrians, and adverse factors such as lighting can significantly reduce the accuracy of pedestrian pose estimation algorithms. Furthermore, the detection results of pedestrian pose estimation algorithms often only contain 18 or even 25 feature points, making them more prone to missed detections and offsets. Other algorithms based on RGB images rely on traditional convolutional methods to extract features from pedestrian images and predict pedestrian crossing intentions. While the application of 3D convolution has achieved good prediction results, the increased computational complexity and parameters of the model make the application of such 3D convolutional neural networks impractical for edge devices. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above technical deficiencies and provide a method for predicting pedestrian crossing intention, so as to solve the technical problem in the prior art that the practical application of the prediction model is limited due to the increase in the calculation amount and parameters of the prediction model.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for predicting pedestrian crossing intention, comprising:

[0007] Obtain a fully trained target prediction model, wherein the target prediction model includes a color map prediction network structure and an optical flow map prediction network structure, wherein the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module;

[0008] Obtain the color image to be processed and the pedestrian optical flow map to be processed;

[0009] Based on the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, spatial feature extraction is performed on the color image to be processed and the pedestrian optical flow map to be processed, respectively, to obtain a color space feature map and an optical flow space feature map; based on the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, channel selection is performed on the color space features and the optical flow space features, respectively, to obtain color prediction channel information and optical flow prediction channel information;

[0010] Based on the color prediction channel information and the optical flow prediction channel information, a pedestrian crossing intention prediction result is determined.

[0011] In some embodiments, the color 3D depth separable convolution layer includes a first color 3D point convolution layer and a color 3D depth convolution layer; based on the color 3D depth separable convolution layer, performing spatial feature extraction on the color image to be processed to obtain color space features includes:

[0012] Performing channel dimension increase on the color image to be processed based on the first color 3D point convolution layer to obtain a first color high-dimensional spatial feature map;

[0013] The color 3D depth convolution layer is used to perform size transformation on the first color high-dimensional space feature map to obtain a color transformation space feature map.

[0014] In some embodiments, the color lightweight attention mechanism module includes a color global average pooling layer, a first color fully connected layer, a second color fully connected layer, and a color cross product operation layer;

[0015] The color global average pooling layer is used to perform a global average pooling operation on the color transformation space feature map to obtain a color global average pooling feature map;

[0016] The first color fully connected layer is used to extract channel features of the color global average pooling feature map to obtain a first color channel feature map;

[0017] The second color fully connected layer is used to extract channel features of the first color channel feature map to obtain a second color channel feature map;

[0018] The color cross multiplication operation layer is used to perform a cross multiplication operation on the first color transformation space feature map and the second color channel feature map to obtain the color prediction channel information.

[0019] In some embodiments, the color map prediction network structure further includes a second color 3D point convolution layer and a color addition operation layer;

[0020] The second color 3D point convolution layer is used to perform channel adjustment on the color prediction channel information to obtain a second color high-dimensional space feature map;

[0021] The color addition operation layer is used to perform an addition operation on the color transformation space feature map and the second color high-dimensional space feature map to obtain an addition operation result.

[0022] In some embodiments, the optical flow 3D depth separable convolution layer includes a first optical flow 3D point convolution layer and an optical flow 3D depth convolution layer; based on the optical flow 3D depth separable convolution layer, performing spatial feature extraction on the optical flow image to be processed to obtain an optical flow spatial feature map includes:

[0023] Performing channel dimension increase on the optical flow image to be processed based on the first optical flow 3D point convolution layer to obtain a first optical flow high-dimensional spatial feature map;

[0024] The optical flow 3D depth convolution layer is used to perform size transformation on the first optical flow high-dimensional spatial feature map to obtain an optical flow transformed spatial feature map.

[0025] In some embodiments, the optical flow lightweight attention mechanism module includes an optical flow global average pooling layer, a first optical flow fully connected layer, a second optical flow fully connected layer, and an optical flow cross product operation layer;

[0026] The optical flow global average pooling layer is used to perform a global average pooling operation on the optical flow transformation spatial feature map to obtain an optical flow global average pooling feature map;

[0027] The first optical flow fully connected layer is used to extract channel features of the optical flow global average pooling feature map to obtain a first optical flow channel feature map;

[0028] The second optical flow fully connected layer is used to extract channel features of the first optical flow channel feature map to obtain a second optical flow channel feature map;

[0029] The optical flow cross multiplication operation layer is used to perform a cross multiplication operation on the first optical flow transformation space feature map and the second optical flow channel feature map to obtain the optical flow prediction channel information.

[0030] In some embodiments, the color map prediction network structure further includes a second color 3D point convolution layer and a color addition operation layer;

[0031] The second color 3D point convolution layer is used to perform channel adjustment on the color prediction channel information to obtain a second color high-dimensional space feature map;

[0032] The color addition operation layer is used to perform an addition operation on the color transformation space feature map and the second color high-dimensional space feature map to obtain an addition operation result.

[0033] In some embodiments, the target prediction model further includes a first color 3D convolution layer, a plurality of the color map prediction network structures, a second color 3D convolution layer, a color pooling layer, a third color 3D convolution layer, a fourth color 3D convolution layer, a first optical flow 3D convolution layer, a plurality of the optical flow map prediction network structures, a second optical flow 3D convolution layer, an optical flow pooling layer, a third optical flow 3D convolution layer, a fourth optical flow 3D convolution layer, and a weighted addition layer;

[0034] The first color 3D convolution layer is used to perform spatial feature dimension processing on the color image to be processed to obtain a first color spatial dimension feature map;

[0035] The multiple color map prediction network structures are used to perform channel information processing on the first color space dimension feature map to obtain a color channel information feature map;

[0036] The second color 3D convolution layer is used to perform a spatial dimension enhancement operation on the color channel information feature map to obtain a color high-dimensional spatial feature map;

[0037] The color pooling layer is used to perform dimension selection on the color high-dimensional space feature map to obtain a color dimension feature map;

[0038] The third color 3D convolution layer is used to perform feature extraction on the color dimension feature map to obtain a color feature map;

[0039] The fourth color 3D convolution layer is used to perform feature scaling on the color feature map to obtain a color scaling feature map;

[0040] The first optical flow 3D convolution layer is used to perform spatial feature dimension processing on the optical flow image to be processed to obtain a first optical flow spatial dimension feature map;

[0041] The multiple optical flow map prediction network structures are used to perform channel information processing on the first optical flow spatial dimension feature map to obtain an optical flow channel information feature map;

[0042] The second optical flow 3D convolution layer is used to perform a spatial dimension enhancement operation on the optical flow channel information feature map to obtain an optical flow high-dimensional spatial feature map;

[0043] The optical flow pooling layer is used to perform dimension selection on the optical flow high-dimensional spatial feature map to obtain an optical flow dimensional feature map;

[0044] The third optical flow 3D convolution layer is used to extract features from the optical flow dimension feature map to obtain an optical flow feature map;

[0045] The fourth optical flow 3D convolution layer is used to perform feature scaling on the optical flow feature map to obtain an optical flow scaling feature map;

[0046] The weighted addition layer is used to perform a weighted addition operation on the color scaling feature map and the optical flow scaling feature map to obtain the probability of the pedestrian crossing intention.

[0047] In some embodiments, obtaining the RGB image to be processed includes:

[0048] Use the preset YOLOv5 algorithm to obtain the rough location of pedestrians in the image;

[0049] Based on the rough position, a preset DeepSORT network is used to obtain continuous frame trajectory images of the pedestrian;

[0050] The continuous frame images are enhanced to obtain enhanced images, and an RGB image to be processed is obtained based on the enhanced images.

[0051] In some embodiments, the pedestrian optical flow map to be processed includes:

[0052] The preset DenseFlow algorithm is used to obtain the pedestrian optical flow map to be processed corresponding to the RGB image based on the enhanced image.

[0053] Compared with the prior art, the pedestrian crossing intention prediction method provided by the present invention sets a target prediction model including a color map prediction network structure and an optical flow map prediction network structure, and the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module; the computational amount of the color map and the pedestrian optical flow map is compressed respectively through the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, and then the feature extraction and learning ability of important channels in complex space are improved through the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, thereby reducing the computational amount and parameter amount of the pedestrian crossing intention prediction model, and improving the possibility of the prediction model being applied in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of an embodiment of a method for predicting pedestrian crossing intention provided by the present invention;

[0055] Figure 2 2 is a schematic structural diagram of an embodiment of a color 3D depth separable convolutional layer in the pedestrian crossing intention prediction method provided by the present invention;

[0056] Figure 3 2 is a structural diagram of an embodiment of a color lightweight attention mechanism module in the pedestrian crossing intention prediction method provided by the present invention;

[0057] Figure 4 1 is a structural diagram of an embodiment of a color map prediction network structure in the pedestrian crossing intention prediction method provided by the present invention;

[0058] Figure 5 2 is a schematic structural diagram of an embodiment of an optical flow 3D depth-separable convolutional layer in the pedestrian crossing intention prediction method provided by the present invention;

[0059] Figure 6 2 is a schematic structural diagram of an embodiment of an optical flow lightweight attention mechanism module in the pedestrian crossing intention prediction method provided by the present invention;

[0060] Figure 7 1 is a schematic structural diagram of an embodiment of an optical flow map prediction network structure in a pedestrian crossing intention prediction method provided by the present invention;

[0061] Figure 8 1 is a schematic structural diagram of an embodiment of a target prediction model in the pedestrian crossing intention prediction method provided by the present invention;

[0062] Figure 9 This is a flow chart of an embodiment of step S102 in the pedestrian crossing intention prediction method provided by the present invention;

[0063] Figure 10 It is a schematic diagram of the operating environment of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] Currently, deep learning is developing rapidly in the field of computer vision. The powerful data feature extraction and scene generalization capabilities of convolutional neural networks have enabled deep learning to excel in visual tasks such as target detection and target tracking. Based on the results of the above algorithm, the present invention proposes a pedestrian crossing intention prediction method, device, electronic device and storage medium. Relying on a lightweight neural network model, it can quickly and accurately predict pedestrian intentions, which is more conducive to helping vehicles perceive pedestrian behavior and avoid traffic accidents. The pedestrian intention prediction method involved in the present invention can not only adapt to various complex traffic scenarios, but also adapt to changeable road and weather environments, and can quickly and accurately predict pedestrian crossing intentions.

[0066] An embodiment of the present invention provides a method for predicting pedestrian crossing intention, which is described below.

[0067] Figure 1 A flowchart of an embodiment of the method for predicting pedestrian crossing intention provided by the present invention is shown as follows: Figure 1 As shown in Figure 2, the pedestrian crossing intention prediction method includes:

[0068] S101. Acquire a fully trained target prediction model, wherein the target prediction model includes a color map prediction network structure and an optical flow map prediction network structure, wherein the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module;

[0069] S102, obtaining a color image to be processed and a pedestrian optical flow map to be processed;

[0070] S103, based on the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, respectively extracting spatial features of the color image to be processed and the pedestrian optical flow map to be processed to obtain a color space feature map and an optical flow space feature map; based on the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, respectively perform channel selection on the color space features and the optical flow space features to obtain color prediction channel information and optical flow prediction channel information;

[0071] S104: Determine a pedestrian crossing intention prediction result based on the color prediction channel information and the optical flow prediction channel information.

[0072] Compared with the prior art, the pedestrian crossing intention prediction method provided by the embodiment of the present invention sets the target prediction model to include a color map prediction network structure and an optical flow map prediction network structure, and the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module; the computational amount of the color map and the pedestrian optical flow map is compressed respectively through the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, and then the feature extraction and learning capabilities of important channels in complex space are improved through the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, thereby reducing the computational amount and parameter amount of the pedestrian crossing intention prediction model, and improving the possibility of the prediction model being applied in practice.

[0073] In a specific embodiment of the present invention, 16 consecutive frames of pedestrian RGB images and pedestrian optical flow images are input into the color map prediction network structure and the optical flow map prediction network structure respectively. The RGB image is a 3-channel image, and the pedestrian optical flow map is a two-channel image, which is the moving imaging of pixels in the horizontal direction and the vertical direction respectively. Therefore, the color map prediction network structure and the optical flow map prediction network structure will adjust the internal parameters of the color map prediction network structure and the optical flow map prediction network structure accordingly for input data of different channel numbers and dimensions. For feature extraction of continuous image frames, the traditional method of 2D convolution + LSTM cannot achieve good results, so the present invention can directly perform feature processing on video data through 3D depth-separable convolution.

[0074] It should be noted that, in a specific implementation of the present invention, the convolutional network structure for processing RGB images and the convolutional network structure for processing pedestrian optical flow images are trained separately. Among them, the behavior of pedestrians crossing the road is a positive sample, and the output of the color map prediction network structure and the optical flow map prediction network structure model should be as close to 1 as possible. The scenes such as pedestrians moving parallel to vehicles or pedestrians walking on the sidewalk far away from the road are defined as negative samples, so the output of the color map prediction network structure and the optical flow map prediction network structure model should be as close to 0 as possible. Since the output results have only two values, the loss function of the color map prediction network structure and the optical flow map prediction network structure model are both cross entropy loss. Specifically, the loss function is:

[0075]

[0076] Where N represents the number of samples, y i Indicates a label, Represents the model prediction value.

[0077] In some embodiments, see Figure 2The color 3D depth separable convolution layer includes a first color 3D point convolution layer and a color 3D depth convolution layer; based on the color 3D depth separable convolution layer, performing spatial feature extraction on the color image to be processed to obtain color space features, including:

[0078] Performing channel dimension increase on the color image to be processed based on the first color 3D point convolution layer to obtain a first color high-dimensional spatial feature map;

[0079] The color 3D depth convolution layer is used to perform size transformation on the first color high-dimensional space feature map to obtain a color transformation space feature map.

[0080] In this embodiment, the color 3D separable convolution includes two parts: the first color 3D point convolution (PWise) and the color 3D depth convolution (Dwise). The first color 3D point convolution is used to perform channel dimension upgrade on the input features to extract richer data features. The color 3D depth convolution is used to extract the spatial features of the image and change the input size of the image. In general, the computational cost of the color 3D separable convolution is: C in ·K·K·K·W out ·H out ·T out +C in ·C out W out ·H out ·T out , while the calculation amount of ordinary 3D convolution is: C in ·K·K·K·C out W out ·H out ·T out , where C in with C out Indicates the number of input and output channels, K indicates the size of the convolution kernel, and W out , H out With T out They represent the width, height and time dimensions of the output feature vector respectively. Therefore, the use of color 3D depth separable convolution can compress the amount of calculation to that of ordinary 3D convolution. This reduces the computational complexity of the prediction model.

[0081] In some embodiments, see Figure 3 The color lightweight attention mechanism module includes a color global average pooling layer, a first color fully connected layer, a second color fully connected layer and a color cross product operation layer;

[0082] The color global average pooling layer is used to perform a global average pooling operation on the color transformation space feature map to obtain a color global average pooling feature map;

[0083] The first color fully connected layer is used to extract channel features of the color global average pooling feature map to obtain a first color channel feature map;

[0084] The second color fully connected layer is used to extract channel features of the first color channel feature map to obtain a second color channel feature map;

[0085] The color cross multiplication operation layer is used to perform a cross multiplication operation on the first color transformation space feature map and the second color channel feature map to obtain the color prediction channel information.

[0086] In this embodiment, since the importance of different channels of the multi-dimensional features of the color transformation space feature map is different during the convolution pooling process, the color SE module is used to process the input features into a channel feature size of 1*1*c through a global average pooling operation, and then the channel importance features are learned through the first color fully connected FC layer and the second color fully connected layer, the useful channel features are amplified, and the useless channel features are suppressed, and finally multiplied with the original input features to obtain color prediction channel information.

[0087] In some embodiments, see Figure 4 , the color map prediction network structure also includes a second color 3D point convolution layer and a color addition operation layer;

[0088] The second color 3D point convolution layer is used to perform channel adjustment on the color prediction channel information to obtain a second color high-dimensional space feature map;

[0089] The color addition operation layer is used to perform an addition operation on the color transformation space feature map and the second color high-dimensional space feature map to obtain an addition operation result.

[0090] In this embodiment, the second color 3D Pointwise Convolution is used in the network structure to adjust the number of output channels of the addition operation result to obtain a second color high-dimensional space feature map, and the color transformation space feature map and the second color high-dimensional space feature map are directly added using the residual structure to obtain the color addition operation result.

[0091] In some embodiments, see Figure 5 The optical flow 3D depth separable convolution layer includes a first optical flow 3D point convolution layer and an optical flow 3D depth convolution layer; based on the optical flow 3D depth separable convolution layer, spatial feature extraction is performed on the optical flow image to be processed to obtain an optical flow spatial feature map, including:

[0092] Performing channel dimension increase on the optical flow image to be processed based on the first optical flow 3D point convolution layer to obtain a first optical flow high-dimensional spatial feature map;

[0093] The optical flow 3D depth convolution layer is used to perform size transformation on the first optical flow high-dimensional spatial feature map to obtain an optical flow transformed spatial feature map.

[0094] In this embodiment, the optical flow 3D separable convolution includes two parts: the first optical flow 3D point convolution (PWise) and the optical flow 3D depth convolution (Dwise). The first optical flow 3D point convolution is used to perform channel dimension upgrade on the input features to extract richer data features. The optical flow 3D depth convolution is used to extract the spatial features of the image and change the input size of the image. In general, the calculation amount of the optical flow 3D separable convolution is: C in ·K·K·K·W out ·H out ·T out +C in ·C out W out ·H out ·T out , while the calculation amount of ordinary 3D convolution is: C in ·K·K·K·C out W out ·H out ·T out , where C in with C out Indicates the number of input and output channels, K indicates the size of the convolution kernel, and W out , H out With T out They represent the width, height and time dimensions of the output feature vector respectively. Therefore, using optical flow 3D depth separable convolution can compress the computational complexity to that of ordinary 3D convolution. This reduces the computational complexity of the prediction model.

[0095] In some embodiments, see Figure 6 The optical flow lightweight attention mechanism module includes an optical flow global average pooling layer, a first optical flow fully connected layer, a second optical flow fully connected layer and an optical flow cross product operation layer;

[0096] The optical flow global average pooling layer is used to perform a global average pooling operation on the optical flow transformation spatial feature map to obtain an optical flow global average pooling feature map;

[0097] The first optical flow fully connected layer is used to extract channel features of the optical flow global average pooling feature map to obtain a first optical flow channel feature map;

[0098] The second optical flow fully connected layer is used to extract channel features of the first optical flow channel feature map to obtain a second optical flow channel feature map;

[0099] The optical flow cross multiplication operation layer is used to perform a cross multiplication operation on the first optical flow transformation space feature map and the second optical flow channel feature map to obtain the optical flow prediction channel information.

[0100] In this embodiment, since the importance of different channels of the multi-dimensional features of the optical flow transformation spatial feature map is different during the convolution pooling process, the optical flow SE module is used to process the input features into a channel feature size of 1*1*c through a global average pooling operation, and then the channel importance features are learned through the first optical flow fully connected FC layer and the second optical flow fully connected layer, the useful channel features are amplified, and the useless channel features are suppressed, and finally multiplied with the original input features to obtain the optical flow prediction channel information.

[0101] In some embodiments, see Figure 7 , the color map prediction network structure also includes a second optical flow 3D point convolution layer and an optical flow addition operation layer;

[0102] The second optical flow 3D point convolution layer is used to perform channel adjustment on the optical flow prediction channel information to obtain a second optical flow high-dimensional spatial feature map;

[0103] The optical flow addition operation layer is used to perform an addition operation on the optical flow transformation spatial feature map and the second optical flow high-dimensional spatial feature map to obtain an optical flow addition operation result.

[0104] In some embodiments, see Figure 8 , the target prediction model also includes a first color 3D convolution layer, a plurality of the color map prediction network structures, a second color 3D convolution layer, a color pooling layer, a third color 3D convolution layer, a fourth color 3D convolution layer, a first optical flow 3D convolution layer, a plurality of the optical flow map prediction network structures, a second optical flow 3D convolution layer, an optical flow pooling layer, a third optical flow 3D convolution layer, a fourth optical flow 3D convolution layer and a weighted addition layer;

[0105] The first color 3D convolution layer is used to perform spatial feature dimension processing on the color image to be processed to obtain a first color spatial dimension feature map;

[0106] The multiple color map prediction network structures are used to perform channel information processing on the first color space dimension feature map to obtain a color channel information feature map;

[0107] The second color 3D convolution layer is used to perform a spatial dimension enhancement operation on the color channel information feature map to obtain a color high-dimensional spatial feature map;

[0108] The color pooling layer is used to perform dimension selection on the color high-dimensional space feature map to obtain a color dimension feature map;

[0109] The third color 3D convolution layer is used to perform feature extraction on the color dimension feature map to obtain a color feature map;

[0110] The fourth color 3D convolution layer is used to perform feature scaling on the color feature map to obtain a color scaling feature map;

[0111] The first optical flow 3D convolution layer is used to perform spatial feature dimension processing on the optical flow image to be processed to obtain a first optical flow spatial dimension feature map;

[0112] The multiple optical flow map prediction network structures are used to perform channel information processing on the first optical flow spatial dimension feature map to obtain an optical flow channel information feature map;

[0113] The second optical flow 3D convolution layer is used to perform a spatial dimension enhancement operation on the optical flow channel information feature map to obtain an optical flow high-dimensional spatial feature map;

[0114] The optical flow pooling layer is used to perform dimension selection on the optical flow high-dimensional spatial feature map to obtain an optical flow dimensional feature map;

[0115] The third optical flow 3D convolution layer is used to extract features from the optical flow dimension feature map to obtain an optical flow feature map;

[0116] The fourth optical flow 3D convolution layer is used to perform feature scaling on the optical flow feature map to obtain an optical flow scaling feature map;

[0117] The weighted addition layer is used to perform a weighted addition operation on the color scaling feature map and the optical flow scaling feature map to obtain the probability of the pedestrian crossing intention.

[0118] In this embodiment, the RGB scaling feature map and the optical flow scaling feature map are weighted to obtain the final pedestrian crossing intention probability y pred ;

[0119] y pred =(y rgb +y flow ) / 2

[0120] where y rgb is the RGB convolution network output, y flow is the output of the optical flow convolutional network. If the weighted probability of a pedestrian crossing intention is greater than 50%, it is judged that the pedestrian is about to cross the road, and the vehicle should slow down appropriately to avoid a collision. If the probability of a pedestrian crossing intention is less than 50%, the pedestrian and the vehicle may be moving in the same direction or at a large distance. At this time, the vehicle can maintain its speed to avoid being rear-ended by the following vehicle and causing a traffic accident.

[0121] In some embodiments, the RGB image to be processed is obtained. Figure 9,include:

[0122] S901, using the preset YOLOv5 algorithm to obtain the rough position of the pedestrian in the image;

[0123] S902: Based on the rough position, a preset DeepSORT network is used to obtain continuous frame trajectory images of the pedestrian;

[0124] S903 : Perform enhancement processing on the continuous frame images to obtain an enhanced image, and obtain an RGB image to be processed based on the enhanced image.

[0125] In this embodiment, the vehicle continuously captures and saves images using its front-facing camera. The YOLOv5 algorithm is used to determine the pedestrian's bounding box, or the approximate location of the pedestrian within the image. Based on this bounding box, the DeepSORT network is then used to determine the pedestrian's trajectory across consecutive frames.

[0126] In a specific embodiment, the backbone network of the YOLOv5 algorithm mainly adopts the Focus structure and the CPS structure. The Focus structure is a slicing operation. With a 4*4*3 image distance, it slices into a 2*2*12 feature map at a specific position through 4 slice modules. As a special downsampling method, the Focus structure can better associate non-adjacent pixels in the image. The CSP structure divides the input into two branches. The first branch passes through multiple residual structures, and the second branch directly performs convolution. The number of channels output by both branches is half of the input. Finally, the concat function is used to splice the input and output size of the entire structure unchanged, allowing the model to learn more features. The network Neck and Head parts are similar to YOLOv4, and obtain richer graphic space features in a multi-scale manner. As a single-stage target detection algorithm, the YOLO algorithm is more conducive to deployment at the edge compared to the two-stage target detection algorithm, both in terms of network model size and detection speed.

[0127] In a specific embodiment of the present invention, based on the results of pedestrian target detection, the DeepSORT network is used to track pedestrian targets and match the trajectory of pedestrian bouding-box in consecutive frames. The DeepSORT algorithm can be divided into three stages. The first stage predicts the position of the pedestrian bounding-box in the next frame. The second stage is data association, which matches the two most correlated pedestrian bounding-boxes. The third stage is to obtain the latest pedestrian bounding-box. The Kalman filter algorithm is used in the prediction of the first stage, and the pedestrian bounding-box is represented as a state vector x, where x contains the following 8 attributes Where cx and cy are the horizontal and vertical coordinates of the target center, r is the aspect ratio, h is the height, and the remaining four are derivatives. Because a linear uniform velocity model is used, the target position at time t+1 can be directly derived using the state transition matrix F. During the data association stage, the Mahalanobis distance and the apparent cosine metric are used as cost metrics, and the Hungarian matching algorithm is used to find the globally optimal allocation as the data association result. The final output size is (N, 8), where N represents the N pedestrians detected in the scene and 8 is the coordinates of the upper left and lower right corners of each pedestrian's bounding box, which facilitates the application of subsequent pedestrian image enhancement algorithms.

[0128] It should be noted that in order to utilize additional environmental feature information, the pedestrian image is enhanced, the pedestrian bounding-box is expanded to (1+α) times the original size and scaled to a pedestrian image of 224*224 size. By enhancing the pedestrian image, the pedestrian can be kept in the center of the cropped image. While including appropriate environmental features, the scaled pedestrian maintains its original size information.

[0129] Specifically, first, the image frame captured by the vehicle camera is edge-filled in four directions in equal amounts according to the specified pixel values. img The padding operation is to avoid the expanded pedestrian image from being cropped out of bounds. By calculating the obtained pedestrian bouding-box coordinates and the edge padding size, the img The center coordinate P of the pedestrian bounding-box in the image c Determine the pedestrian image expansion factor α and calculate the width W of the expanded pedestrian bounding-box e With height H e Since the output image is a square, the longer side after expansion is taken as the crop size S c , the corresponding output edge is M s In order to ensure that the cropped image does not cross the border and contains the original pedestrian bounding-box, it is necessary to make S c Try to make it as small as possible so that the pedestrians in the expanded image will be clearer. The calculation formula is as follows:

[0130] S c =min(S c ,P c / T*M s *2-1,(E img -P c ) / T*M s *2-1)

[0131] Where T is the size of the target output image, and the optimal S is obtained by comparison.c Size, crop the RGB image to be processed corresponding to the pedestrian from the expanded image.

[0132] In some embodiments, the pedestrian optical flow map to be processed includes:

[0133] The preset DenseFlow algorithm is used to obtain the pedestrian optical flow map to be processed corresponding to the RGB image based on the enhanced image.

[0134] It should be noted that the embodiment of the present invention uses a pedestrian image enhancement algorithm to expand the pedestrian bounding-box obtained on the basis of its inclusion of environmental information within a certain range in the two-way pedestrian image. By extracting the characteristics of the pedestrian's environment, the network model can achieve higher prediction accuracy.

[0135] In this embodiment, the pedestrian optical flow map corresponding to the pedestrian RGB image is extracted through the DenseFlow algorithm, and the pixel-level movement features make up for the shortcomings of the RGB pedestrian image.

[0136] like Figure 10 As shown, based on the above pedestrian crossing intention prediction method, the present invention also provides an electronic device, which can be a computing device such as a mobile terminal, desktop computer, notebook, PDA, server, etc. The electronic device includes a processor 1010, a memory 1020, and a display 1030. Figure 10 Only some of the components of the electronic device are shown, but it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0137] In some embodiments, the memory 1020 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 1020 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 1020 may also include both an internal storage unit of the electronic device and an external storage device. The memory 1020 is used to store application software and various types of data installed in the electronic device, such as program codes installed in the electronic device. The memory 1020 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a pedestrian crossing intention prediction program 1040 is stored on the memory 1020, and the pedestrian crossing intention prediction program 1040 can be executed by the processor 1010, thereby realizing the pedestrian crossing intention prediction method of each embodiment of the present application.

[0138] In some embodiments, the processor 1010 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 1020, such as executing a pedestrian crossing intention prediction method.

[0139] In some embodiments, the display 1030 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 1030 is used to display information on the pedestrian crossing intention prediction device and to display a visual user interface. Components 1010-1030 of the electronic device communicate with each other via a system bus.

[0140] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0141] The specific embodiments of the present invention described above do not limit the scope of protection of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A pedestrian crossing intention prediction method, characterized in that: include: Obtain a fully trained target prediction model, wherein the target prediction model includes a color map prediction network structure and an optical flow map prediction network structure, wherein the color map prediction network structure includes a color 3D depth-separable convolution layer and a color lightweight attention mechanism module, and the optical flow map prediction network structure includes an optical flow 3D depth-separable convolution layer and an optical flow lightweight attention mechanism module; Obtain the color image to be processed and the pedestrian optical flow map to be processed; Based on the color 3D depth-separable convolution layer and the optical flow 3D depth-separable convolution layer, spatial feature extraction is performed on the color image to be processed and the pedestrian optical flow map to be processed, respectively, to obtain a color space feature map and an optical flow space feature map; based on the color lightweight attention mechanism module and the optical flow lightweight attention mechanism module, channel selection is performed on the color space features and the optical flow space features, respectively, to obtain color prediction channel information and optical flow prediction channel information; Determining a pedestrian crossing intention prediction result based on the color prediction channel information and the optical flow prediction channel information; The color lightweight attention mechanism module includes a color global average pooling layer, a first color fully connected layer, a second color fully connected layer and a color cross product operation layer; The color global average pooling layer is used to perform a global average pooling operation on the color transformation space feature map to obtain a color global average pooling feature map; The first color fully connected layer is used to extract channel features of the color global average pooling feature map to obtain a first color channel feature map; The second color fully connected layer is used to extract channel features of the first color channel feature map to obtain a second color channel feature map; The color cross multiplication operation layer is used to perform a cross multiplication operation on the first color transformation space feature map and the second color channel feature map to obtain the color prediction channel information; The optical flow lightweight attention mechanism module includes an optical flow global average pooling layer, a first optical flow fully connected layer, a second optical flow fully connected layer and an optical flow cross product operation layer; The optical flow global average pooling layer is used to perform a global average pooling operation on the optical flow transformation spatial feature map to obtain an optical flow global average pooling feature map; The first optical flow fully connected layer is used to extract channel features of the optical flow global average pooling feature map to obtain a first optical flow channel feature map; The second optical flow fully connected layer is used to extract channel features of the first optical flow channel feature map to obtain a second optical flow channel feature map; The optical flow cross multiplication operation layer is used to perform a cross multiplication operation on the first optical flow transformation space feature map and the second optical flow channel feature map to obtain the optical flow prediction channel information.

2. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The color 3D depth separable convolution layer includes a first color 3D point convolution layer and a color 3D depth convolution layer; Based on the color 3D depth separable convolution layer, spatial feature extraction is performed on the color image to be processed to obtain color space features, including: Performing channel dimension increase on the color image to be processed based on the first color 3D point convolution layer to obtain a first color high-dimensional spatial feature map; The color 3D depth convolution layer is used to perform size transformation on the first color high-dimensional space feature map to obtain a color transformation space feature map.

3. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The color map prediction network structure also includes a second color 3D point convolution layer and a color addition operation layer; The second color 3D point convolution layer is used to perform channel adjustment on the color prediction channel information to obtain a second color high-dimensional space feature map; The color addition operation layer is used to perform an addition operation on the color transformation space feature map and the second color high-dimensional space feature map to obtain a color addition operation result.

4. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The optical flow 3D depth separable convolution layer includes a first optical flow 3D point convolution layer and an optical flow 3D depth convolution layer; Based on the optical flow 3D depth separable convolution layer, spatial feature extraction is performed on the pedestrian optical flow map to be processed to obtain an optical flow spatial feature map, including: Performing channel dimension increase on the pedestrian optical flow map to be processed based on the first optical flow 3D point convolution layer to obtain a first optical flow high-dimensional spatial feature map; The optical flow 3D depth convolution layer is used to perform size transformation on the first optical flow high-dimensional spatial feature map to obtain an optical flow transformed spatial feature map.

5. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The optical flow map prediction network structure also includes a second optical flow 3D point convolution layer and an optical flow addition operation layer; The second optical flow 3D point convolution layer is used to perform channel adjustment on the optical flow prediction channel information to obtain a second optical flow high-dimensional spatial feature map; The optical flow addition operation layer is used to perform an addition operation on the optical flow transformation spatial feature map and the second optical flow high-dimensional spatial feature map to obtain an optical flow addition operation result.

6. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The target prediction model further includes a first color 3D convolution layer, a plurality of the color map prediction network structures, a second color 3D convolution layer, a color pooling layer, a third color 3D convolution layer, a fourth color 3D convolution layer, a first optical flow 3D convolution layer, a plurality of the optical flow map prediction network structures, a second optical flow 3D convolution layer, an optical flow pooling layer, a third optical flow 3D convolution layer, a fourth optical flow 3D convolution layer and a weighted addition layer; The first color 3D convolution layer is used to perform spatial feature dimension processing on the color image to be processed to obtain a first color spatial dimension feature map; The multiple color map prediction network structures are used to perform channel information processing on the first color space dimension feature map to obtain a color channel information feature map; The second color 3D convolution layer is used to perform a spatial dimension enhancement operation on the color channel information feature map to obtain a color high-dimensional spatial feature map; The color pooling layer is used to perform dimension selection on the color high-dimensional space feature map to obtain a color dimension feature map; The third color 3D convolution layer is used to perform feature extraction on the color dimension feature map to obtain a color feature map; The fourth color 3D convolution layer is used to perform feature scaling on the color feature map to obtain a color scaling feature map; The first optical flow 3D convolution layer is used to perform spatial feature dimension processing on the pedestrian optical flow map to be processed to obtain a first optical flow spatial dimension feature map; The multiple optical flow map prediction network structures are used to perform channel information processing on the first optical flow spatial dimension feature map to obtain an optical flow channel information feature map; The second optical flow 3D convolution layer is used to perform a spatial dimension enhancement operation on the optical flow channel information feature map to obtain an optical flow high-dimensional spatial feature map; The optical flow pooling layer is used to perform dimension selection on the optical flow high-dimensional spatial feature map to obtain an optical flow dimensional feature map; The third optical flow 3D convolution layer is used to extract features from the optical flow dimension feature map to obtain an optical flow feature map; The fourth optical flow 3D convolution layer is used to perform feature scaling on the optical flow feature map to obtain an optical flow scaling feature map; The weighted addition layer is used to perform a weighted addition operation on the color scaling feature map and the optical flow scaling feature map to obtain the probability of the pedestrian crossing intention.

7. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The step of obtaining a color image to be processed includes: Use the preset YOLOv5 algorithm to obtain the rough location of pedestrians in the image; Based on the rough position, a preset DeepSORT network is used to obtain continuous frame trajectory images of the pedestrian; The continuous frame trajectory image is enhanced to obtain an enhanced image, and a color image to be processed is obtained based on the enhanced image.

8. The pedestrian crossing intention prediction method according to claim 1, characterized in that: The pedestrian optical flow map to be processed includes: The preset DenseFlow algorithm is used to obtain the pedestrian optical flow map to be processed corresponding to the RGB image based on the enhanced image.

Images