Vehicle lane changing intention prediction method and system based on ResNet

Abstract

The invention discloses a vehicle lane changing intention prediction method and system based on ResNet, and relates to the field of intention prediction of automatic driving. According to the method, the motion state of a target vehicle, the motion state of surrounding vehicles and road information are stored through three channels of an RGB image respectively, when the motion state of the vehicle is represented, the size of the vehicle is reflected by setting the size of a rectangle, the sequence of observation moments is reflected by means of the depth of image colors, and when the road information is represented, a drivable area and a lane boundary are considered; and building a ResNet network embedded with the CBAM, and realizing classification of vehicle lane changing intentions. According to the method, the scene can be visually displayed, and the interaction relation of different types of information is reflected while the different types of information are distinguished. In addition, the method can fully consider the influence of the vehicle volume on vehicle-vehicle interaction.

Description

Technical Field

[0001] The present invention relates to the field of intention prediction of autonomous driving, and specifically to a vehicle lane change intention prediction method and system based on ResNet. Background Art

[0002] Vehicle driving is a highly interactive process, and the behavior of the vehicle is often affected by the intentions of surrounding traffic participants. The autonomous driving system consists of three major modules: perception, decision-making, and control. In the autonomous driving system, the interaction between the vehicle and surrounding vehicles requires the joint participation of the perception module and the decision-making module, and relies on the accurate prediction of the future intentions of surrounding traffic participants. Intention prediction is a high-level abstraction and further understanding of the perception information. The prediction results are input into the decision-making module to assist in making reasonable, safe, comfortable, and reliable trajectory planning. Therefore, reliable prediction of the intentions of surrounding vehicles is an important bridge between the perception module and the decision-making module. Vehicle intention prediction aims to infer the future behavior of the target vehicle through driver behavior or historical motion states such as vehicle position, speed change, and lane position. Its essence is a classification task.

[0003] Existing methods for predicting vehicle lane-changing intentions include generative models, discriminative models, and deep neural networks. Among them, deep neural networks represented by long short-term memory networks can learn the temporal and spatial characteristics of vehicles while driving, and have a high prediction accuracy, and have been widely studied. However, the vehicle intention prediction method based on recurrent neural networks does not take vehicle size into account, resulting in the inability to fully model the differences in interaction behaviors of different types of vehicles and the impact of vehicle size on interaction, which limits the improvement of the accuracy of lane-changing intention prediction. Summary of the invention

[0004] In view of this, the present invention provides a vehicle lane-changing intention prediction method based on ResNet, which considers the motion characteristics of the target vehicle, the interaction between the target vehicle and surrounding vehicles, the road structure and the restrictions of traffic rules on the target vehicle, and predicts the vehicle's lane-changing intention on a straight road. This method is suitable for multi-lane scenarios and takes into account the impact of vehicle size.

[0005] To achieve the above object, the technical solution of the present invention comprises the following steps:

[0006] Step 1: Obtain information about the target vehicle, surrounding vehicles, and road conditions.

[0007] Step 2: Generate an RGB image based on the scene information of the target vehicle, i.e., an image for intent prediction. Multiple images for intent prediction constitute a training set image:

[0008] Step 3: Embed CBAM into the ResNet network to build a vehicle intention prediction model; select ResNet-18 as the backbone network and embed CBAM before and after the residual block, that is, between the initial convolution layer and the first residual block and between the last residual block and the output processing layer.

[0009] Step 4: Use the training set images to train the vehicle intention prediction model, learn the classification parameters, and obtain the trained vehicle intention prediction model.

[0010] Step 5: Generate a corresponding RGB image based on the scene information to be predicted, and input the corresponding RGB image of the scene into the trained vehicle intention prediction model to obtain the predicted vehicle lane change intention.

[0011] Furthermore, in step 1, the information obtained includes:

[0012] Obtain information about the target vehicle and surrounding vehicles, including the length, width, center of mass position of the vehicle at current and historical times, and lane to which it belongs; surrounding vehicles include the vehicles in front of and behind the target vehicle and closest to it in the left adjacent lane, right adjacent lane, and current lane of the target vehicle.

[0013] Road information includes the drivable area, lane boundaries, left and right adjacent lanes, and the width of the current lane.

[0014] The drivable area is defined as a combination of a series of lanes. Vehicles change lanes or overtake between lanes within the drivable area. The boundary of the drivable area is the solid road line or shoulder.

[0015] Furthermore, in step 1, corresponding information is obtained from the NGSIM dataset, and the data in the NGSIM dataset is processed as follows:

[0016] 1) Correct the vehicle number, that is, sort, split and renumber the data according to the vehicle number, location and timestamp information.

[0017] 2) Filter the data within the ROI, determine whether there is a lane change behavior based on the change in the lane number of the vehicle, regard the part that does not include the ramp as the ROI area, and filter out the data to be processed based on the lane number in the area.

[0018] 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the road boundary is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point is found according to the lane number of the vehicle provided by the NGSIM dataset. Traverse the dataset according to the vehicle number, and then traverse the motion sequence of the target vehicle in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number.

[0019] 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information n seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly extract a timestamp and save the time, position, and lane information n seconds before the timestamp.

[0020] 5) Filter the surrounding vehicles of each target vehicle and record their motion status; traverse the motion sequence of each target vehicle to obtain the position and lane of the target vehicle in each timestamp; filter out all vehicles in the lane where the target vehicle is located and the adjacent lanes from the data set according to the timestamp and the lane where the target vehicle is located; sort the vehicles in each lane according to their positions, filter out the front and rear vehicles closest to the target vehicle, and save the corresponding time and position information.

[0021] Furthermore, step 2: generating an RGB image according to the scene information of the target vehicle, specifically:

[0022] The motion sequences of the target vehicle and surrounding vehicles are obtained respectively, and the image size is determined.

[0023] The target vehicle motion image, surrounding vehicle motion image, and road image are generated in the R, G, and B channels in sequence.

[0024] Finally, the three-channel information is fused to obtain an image for intent prediction.

[0025] Furthermore, in step 2, the motion sequences of the target vehicle and surrounding vehicles are obtained respectively, and the image size is determined, specifically: the image size is determined according to scene information such as the speed limit or average speed, lane width, etc.:

[0026] Image length = road speed limit or average traffic speed × advance prediction time × 2 / length represented by each pixel.

[0027] Image width = width of three lanes centered on the current lane / width represented by each pixel.

[0028] Furthermore, in step 2, the target vehicle motion image, the surrounding vehicle motion image, and the road image are generated in the R, G, and B channels in sequence, specifically including:

[0029] Draw the target vehicle motion image in the R channel, that is, store the current and historical motion positions of the target vehicle in the R channel, draw the vehicle with a rectangle, the rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0030] Draw the surrounding vehicle motion image in the G channel, that is, store the current and historical motion positions of the surrounding vehicles in the G channel, and use rectangles to represent the surrounding vehicles. The rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0031] Draw the drivable area image in the B channel, that is, store the road information in the B channel, and use different filling values ​​to distinguish the drivable area, non-drivable area and lane boundary.

[0032] Preferably, the target vehicle motion image is drawn in the R channel, specifically:

[0033] The target vehicle observed at the last moment is placed at the center of the image so that the geometric center of the vehicle coincides with the center of the image. The vehicle is represented by a rectangle. The upper left corner coordinate of the target vehicle at time t is and the lower right corner coordinates They are

[0034]

[0035] Among them, x, y are the vehicle position coordinates, w is the vehicle width, and l is the vehicle length.

[0036] The color depth of the pixel is used to indicate the distance between the observation time and the current time. h In the observation sequence of , the vehicle at the tth observation time is filled as

[0037]

[0038] The earlier the observation time, the smaller t is, and the lighter the color of the vehicle is. Conversely, the later the observation time is, the darker the color of the vehicle is.

[0039] Preferably, an image of the drivable area is drawn in the B channel: the pixel value of the drivable area is the default value 0, and the value of the non-drivable area is set to 255. Correspondingly, the drivable area is displayed in black in the image, and the non-drivable area is displayed in blue; on this basis, lane lines are drawn, and the B channel value of each lane boundary is set to 255.

[0040] Furthermore, in step three, the process of building a vehicle intention prediction model is as follows:

[0041] The input of the vehicle intention prediction model is a three-channel image, where each channel stores the motion state of the target vehicle, the motion state of surrounding vehicles, and the lane boundary information.

[0042] ResNet-18 is used as the backbone network, and the residual module is used to achieve deep feature learning.

[0043] Two CBAMs are integrated before and after the residual block, namely between the initial convolutional layer Conv1_x and the first residual block Conv2_x and between the last residual block Conv5_x and the output processing layer.

[0044] The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, the probability of each intent is obtained, and the intent with the highest probability is regarded as the prediction result.

[0045] Another embodiment of the present invention also provides a vehicle lane change intention prediction system based on ResNet, including an information acquisition module, an input image generation module, a vehicle intention prediction module, a training module and a real-time prediction module.

[0046] The information acquisition module is used to obtain information about the target vehicle, surrounding vehicles and road information.

[0047] The input image generation module is used to generate an RGB image based on the scene information of the target vehicle.

[0048] Vehicle intention prediction model, embedding CBAM into the ResNet network to obtain the vehicle intention prediction model; selecting ResNet-18 as the backbone network, and embedding CBAM before and after the residual block, that is, between the initial convolution layer and the first residual block and between the last residual block and the output processing layer.

[0049] The training module takes the existing data set, calls the information acquisition module and the input image generation module, generates RGB images as training images, and multiple training images constitute a training set. The training set is used to train the vehicle intention prediction model, learn the classification parameters, and obtain the trained vehicle intention prediction model.

[0050] The real-time prediction module calls the information acquisition module and the input image generation module according to the scene information to be predicted, generates a corresponding RGB image, and inputs the scene corresponding RGB image into the trained vehicle intention prediction model to obtain the predicted vehicle lane changing intention.

[0051] Furthermore, the information acquisition module acquires the following information:

[0052] Obtain information about the target vehicle and surrounding vehicles, including the length, width, center of mass position of the vehicle at current and historical times, and lane to which it belongs; surrounding vehicles include the vehicles in front of and behind the target vehicle and closest to it in the left adjacent lane, right adjacent lane, and current lane of the target vehicle.

[0053] Road information includes the drivable area, lane boundaries, left and right adjacent lanes, and the width of the current lane.

[0054] The drivable area is defined as a combination of a series of lanes. Vehicles change lanes or overtake between lanes within the drivable area. The boundary of the drivable area is the solid road line or shoulder.

[0055] Furthermore, the training module uses the existing data set as the NGSIM data set. The training module specifically includes: calling the information acquisition module to acquire corresponding information from the NGSIM data set, and performing the following processing on the data in the NGSIM data set:

[0056] 1) Correct the vehicle number, that is, sort, split and renumber the data according to the vehicle number, location and timestamp information.

[0057] 2) Filter the data within the ROI, determine whether there is a lane change behavior based on the change in the lane number of the vehicle, regard the part that does not include the ramp as the ROI area, and filter out the data to be processed based on the lane number in the area.

[0058] 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the boundary of the road is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point is found according to the lane number of the vehicle provided by the NGSIM dataset. The dataset is traversed according to the vehicle number, and then the motion sequence of the target vehicle is traversed in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number.

[0059] 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information n seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly extract a timestamp and save the time, position, and lane information n seconds before the timestamp.

[0060] 5) Filter the surrounding vehicles of each target vehicle and record their motion status; traverse the motion sequence of each target vehicle to obtain the position and lane of the target vehicle in each timestamp; filter out all vehicles in the lane where the target vehicle is located and the adjacent lanes from the data set according to the timestamp and the lane where the target vehicle is located; sort the vehicles in each lane according to their positions, filter out the front and rear vehicles closest to the target vehicle, and save the corresponding time and position information.

[0061] Furthermore, the input image generation module is used to generate an RGB image according to the scene information of the target vehicle, and specifically performs the following steps:

[0062] Get the target vehicle and surrounding vehicle motion sequences respectively, and determine the image size. Specifically, determine the image size based on scene information such as speed limit or average speed, lane width, etc.:

[0063] Image length = road speed limit or average traffic speed × advance prediction time × 2 / length represented by each pixel.

[0064] Image width = width of three lanes centered on the current lane / width represented by each pixel.

[0065] Generate target vehicle motion image, surrounding vehicle motion image, and road image in R, G, and B channels in sequence, including:

[0066] Draw the target vehicle motion image in the R channel, that is, store the current and historical motion positions of the target vehicle in the R channel, draw the vehicle with a rectangle, the rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0067] Draw the surrounding vehicle motion image in the G channel, that is, store the current and historical motion positions of the surrounding vehicles in the G channel, and use rectangles to represent the surrounding vehicles. The rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0068] Draw the drivable area image in the B channel, that is, store the road information in the B channel, and use different filling values ​​to distinguish the drivable area, non-drivable area and lane boundary.

[0069] Finally, the three-channel information is fused to obtain an image for intent prediction.

[0070] Furthermore, the vehicle intention prediction model is constructed as follows:

[0071] The input of the vehicle intention prediction model is a three-channel image, where each channel stores the motion state of the target vehicle, the motion state of surrounding vehicles, and the lane boundary information.

[0072] ResNet-18 is used as the backbone network, and the residual module is used to achieve deep feature learning.

[0073] Two CBAMs are integrated before and after the residual block, namely between the initial convolutional layer Conv1_x and the first residual block Conv2_x and between the last residual block Conv5_x and the output processing layer.

[0074] The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, the probability of each intent is obtained, and the intent with the highest probability is regarded as the prediction result.

[0075] Beneficial effects:

[0076] Compared with directly inputting sequence data, the present invention presents scenes more intuitively, distinguishes the visualization methods of different types of information, reflects their interactive relationships, and fully considers the impact of vehicle volume on vehicle-to-vehicle interaction. The present method shows high intention prediction accuracy and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of a scenario of an intention prediction method of the present invention, the blue vehicle is the target vehicle to be predicted, the gray vehicles are vehicles in six directions around the target vehicle, the black solid line is the boundary of the drivable area, and the black dotted line is the lane boundary;

[0078] Figure 2 A data processing flow chart of an intention prediction method of the present invention;

[0079] Figure 3 The present invention is a schematic diagram of a scene image generation process of an intention prediction method. First, the target vehicle and surrounding vehicle motion sequences are obtained from the target scene; R channel and G channel images are generated respectively; then, road information is obtained from the target scene to generate a B channel image; finally, the three channel images are combined to obtain a scene image for predicting intention;

[0080] Figure 4 A schematic diagram of scene image generation results of an intention prediction method of the present invention; Figure 4 (a) is the result of changing lanes to the left; (b) is the result of keeping straight; (c) is the result of changing lanes to the right;

[0081] Figure 5 This is a schematic diagram of the neural network structure of an intent prediction method of the present invention. The backbone network is ResNet-18. Two CBAMs are embedded between the Conv_1 layer and the Conv_2 layer, and between the Conv_5 layer and the average pooling layer, respectively. The number of fully connected layer outputs used for classification is 3. DETAILED DESCRIPTION

[0082] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0083] Embodiment 1:

[0084] This embodiment provides a vehicle lane change intention prediction method based on ResNet, comprising the following steps:

[0085] Step 1: Data acquisition and processing

[0086] The feasibility of the method is verified using the data of the I-80 section of the NGSIM dataset. Figure 1As shown, the target vehicle moves on a straight road, and the road boundary, lane boundary and positions of all vehicles at each moment in the scene are known.

[0087] Since the NGSIM dataset does not specifically process straight lane changing scenarios, it is necessary to filter lane changing vehicles, mark lane changing intentions, and obtain surrounding vehicle information. Figure 2 shown.

[0088] 1) Correct vehicle numbers. The data for the I-80 section can be divided into three time periods: 16:00-16:15, 17:00-17:15, and 17:15-17:30. Since the information extraction process for each time period is independent of each other, there are cases where vehicles in different time periods use the same number. In order to facilitate subsequent processing, the data needs to be sorted, split, and renumbered according to information such as vehicle number, location, and timestamp.

[0089] 2) Filter data within ROI. The I-80 section contains a ramp, and the road is assigned a separate number. Since the data set does not provide lane boundary information, it is necessary to determine whether there is a lane change based on the change in the lane number of the vehicle. To avoid misjudgment at the merging point of traffic, the part that does not contain the ramp is regarded as the ROI area, and the data to be processed is filtered out based on the lane numbers in the area.

[0090] 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the boundary of the road is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point can be found according to the lane number of the vehicle provided by the NGSIM dataset. Traverse the dataset according to the vehicle number, and then traverse the motion sequence of the target vehicle in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number.

[0091] 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information 5 seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly select a timestamp and save the time, position, and lane information 5 seconds before the timestamp.

[0092] 5) Filter the surrounding vehicles of each target vehicle and record their motion status. Traverse the motion sequence of each target vehicle and obtain the position and lane of the target vehicle at each timestamp. According to the timestamp and the lane of the target vehicle, filter out all vehicles in the lane of the target vehicle and the adjacent lanes from the data set. Sort the vehicles in each lane by position, filter out the front and rear vehicles closest to the target vehicle, and save the corresponding time and position information.

[0093] Step 2: Scene image generation

[0094] In order to combine the target vehicle motion state sequence, the surrounding vehicle motion state sequence and the road information, a processing method is adopted to convert the scene information near the target vehicle into an RGB image.

[0095] Since the three primary colors of the RGB image correspond to three independent channels, consider using the R, G, and B channels to store the image information of the target vehicle, surrounding vehicles, and drivable area. The scene image generation process is as follows: Figure 3 First, the target vehicle and surrounding vehicle motion sequences are obtained from the data set to determine the image size; then, the target vehicle motion image, surrounding vehicle motion image, and road image are generated in the R, G, and B channels respectively; finally, the three-channel information is fused to obtain the image used for intention prediction.

[0096] First, determine the image size based on scene information such as speed limit or average speed, lane width, etc. The average speed of traffic on the I-80 section is about 60 kilometers per hour. The distance traveled by the vehicle is calculated based on 2 seconds of movement, and twice the distance is used as the image length. Each pixel represents 0.1 meters, and the image length is 878 pixels. In the straight lane change scenario, it is necessary to consider the vehicle movement status of the lane to which the vehicle belongs and the adjacent lanes. The lane width of this section is 4 meters, and the three lanes are 12 meters in total. The representation accuracy of each pixel is 0.1 meters, so the image width is set to 120 pixels.

[0097] Draw the target vehicle motion image in the R channel. The vehicle motion image needs to reflect the size, position, and speed of the vehicle. Place the target vehicle observed at the last moment in the center of the image so that the vehicle geometric center coincides with the image center. Use a rectangle to represent the vehicle. To reduce the complexity of processing and increase the drawing speed, ignore the effect of the vehicle head direction on the vehicle posture. Then the upper left corner coordinate of the target vehicle at time t is and the lower right corner coordinates They are

[0098]

[0099] Among them, x, y are the vehicle position coordinates, w is the vehicle width, and l is the vehicle length.

[0100] The color depth of the pixel is used to indicate the distance between the observation time and the current time. h In the observation sequence of , the vehicle at the tth observation time is filled as

[0101]

[0102] The earlier the observation time, the smaller t is, and the lighter the color of the vehicle is. Conversely, the later the observation time is, the darker the color of the vehicle is.

[0103] Draw the peri-vehicle motion image in the G channel. The drawing process of the peri-vehicle motion image is basically the same as that of the ego-vehicle motion image.

[0104] Draw the drivable area image in the B channel. The pixel value of the drivable area is the default value 0, and the value of the non-drivable area is set to 255. Correspondingly, the drivable area is displayed in black in the image, and the non-drivable area is displayed in blue. Draw the lane lines on this basis, and set the B channel value of each lane boundary to 255.

[0105] By fusing the R, G, and B channel information, we get a vertically flipped image. Figure 4 shown. Figure 4 (a) is the result of changing lanes to the left; (b) is the result of keeping straight; (c) is the result of changing lanes to the right;

[0106] Step 3: Neural network construction and parameter learning

[0107] A deep learning model based on ResNet and CBAM is designed to predict the lane-changing behavior of the target vehicle by processing and analyzing the target vehicle's motion state, surrounding traffic conditions, and road conditions. The model uses ResNet-18 as the basic architecture and uses the residual module to achieve deep feature learning. On this basis, a lightweight attention module CBAM is introduced to enhance the model's ability to express input features.

[0108] The input of the model is a three-channel image, each channel stores the motion state of the target vehicle, the motion state of the surrounding vehicles and the lane boundary information. CBAM is integrated before and after the residual block, that is, between the initial convolution layer (Conv1_x) and the first residual block (Conv2_x) and between the last residual block (Conv5_x) and the output processing layer, as shown in Figure 5 As shown in the figure. CBAM first learns the importance of different channels through the channel attention mechanism, assigns weights to each channel element, and then uses the spatial attention mechanism to determine which positions in the image are the most critical, multiplying the weights with the elements of each position, emphasizing the key elements while weakening the non-critical elements. This setting allows the model to not only focus on the most informative features, but also optimize the spatial distribution of features, thereby more effectively capturing the interaction details related to lane changing behavior.

[0109] The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, it outputs the probability of each intent and regards the intent with the highest probability as the prediction result.

[0110] Step 4: Neural network parameter learning

[0111] Vehicle intention prediction is a classification task, so the loss function is set to the cross entropy loss function. The Adam optimizer is used to update the network parameters, and the step-wise decay strategy is used to adjust the learning rate. After every 15 rounds, the learning rate decays to 50% of the original. The batch size is set to 32, and the total iteration is 200 rounds.

[0112] Set the observation sequence length t h is 20, and the motion states of the target vehicle and surrounding vehicles are extracted from the preprocessed data in the order of 0.5 seconds, 1 second, 1.5 seconds, 2 seconds, and 2.5 seconds from the last observation value to the lane change point.

[0113] Since the straight driving scenarios are far more than the lane changing scenarios, the data set is balanced so that the number of straight driving trajectories is equal to the number of left lane changing trajectories. h = 20, 6375 straight-moving trajectories, 6375 left-lane-changing trajectories, and 1795 right-lane-changing trajectories are obtained. A scene image is generated based on the motion information of the target vehicle and surrounding vehicles in each scene and the known road information.

[0114] The training set and test set data are divided into 8:2 ratios, resulting in 11,636 training samples and 2,909 test samples.

[0115] The neural network was trained on an NVIDIA GeForce RTX 4090 and the network training took about 3 hours.

[0116] Step 5: Vehicle Intention Prediction

[0117] The trained neural network is used to predict the lane-changing intention of the test set samples, and the prediction accuracy, precision, recall and F1 score are calculated. The results are shown in Table 1.

[0118] Table 1

[0119]

[0120] Embodiment 2:

[0121] Another embodiment of the present invention also provides a vehicle lane change intention prediction system based on ResNet, including an information acquisition module, an input image generation module, a vehicle intention prediction module, a training module and a real-time prediction module.

[0122] The information acquisition module is used to obtain information about the target vehicle, surrounding vehicles and road information.

[0123] The input image generation module is used to generate an RGB image based on the scene information of the target vehicle.

[0124] Vehicle intention prediction model, embedding CBAM into the ResNet network to obtain the vehicle intention prediction model; selecting ResNet-18 as the backbone network, and embedding CBAM before and after the residual block, that is, between the initial convolution layer and the first residual block and between the last residual block and the output processing layer.

[0125] The training module takes the existing data set, calls the information acquisition module and the input image generation module, generates RGB images as training images, and multiple training images constitute a training set. The training set is used to train the vehicle intention prediction model, learn the classification parameters, and obtain the trained vehicle intention prediction model.

[0126] The real-time prediction module calls the information acquisition module and the input image generation module according to the scene information to be predicted, generates a corresponding RGB image, and inputs the scene corresponding RGB image into the trained vehicle intention prediction model to obtain the predicted vehicle lane changing intention.

[0127] Furthermore, the information acquisition module acquires the following information:

[0128] Obtain information about the target vehicle and surrounding vehicles, including the length, width, center of mass position of the vehicle at current and historical times, and lane to which it belongs; surrounding vehicles include the vehicles in front of and behind the target vehicle and closest to it in the left adjacent lane, right adjacent lane, and current lane of the target vehicle.

[0129] Road information includes the drivable area, lane boundaries, left and right adjacent lanes, and the width of the current lane.

[0130] The drivable area is defined as a combination of a series of lanes. Vehicles change lanes or overtake between lanes within the drivable area. The boundary of the drivable area is the solid road line or shoulder.

[0131] Furthermore, the training module uses the existing data set as the NGSIM data set. The training module specifically includes: calling the information acquisition module to acquire corresponding information from the NGSIM data set, and performing the following processing on the data in the NGSIM data set:

[0132] 1) Correct the vehicle number, that is, sort, split and renumber the data according to the vehicle number, location and timestamp information.

[0133] 2) Filter the data within the ROI, determine whether there is a lane change behavior based on the change in the lane number of the vehicle, regard the part that does not include the ramp as the ROI area, and filter out the data to be processed based on the lane number in the area.

[0134] 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the boundary of the road is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point is found according to the lane number of the vehicle provided by the NGSIM dataset. The dataset is traversed according to the vehicle number, and then the motion sequence of the target vehicle is traversed in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number.

[0135] 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information n seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly extract a timestamp and save the time, position, and lane information n seconds before the timestamp.

[0136] 5) Filter the surrounding vehicles of each target vehicle and record their motion status; traverse the motion sequence of each target vehicle to obtain the position and lane of the target vehicle in each timestamp; filter out all vehicles in the lane where the target vehicle is located and the adjacent lanes from the data set according to the timestamp and the lane where the target vehicle is located; sort the vehicles in each lane according to their positions, filter out the front and rear vehicles closest to the target vehicle, and save the corresponding time and position information.

[0137] Furthermore, the input image generation module is used to generate an RGB image according to the scene information of the target vehicle, and specifically performs the following steps:

[0138] Get the target vehicle and surrounding vehicle motion sequences respectively, and determine the image size. Specifically, determine the image size based on scene information such as speed limit or average speed, lane width, etc.:

[0139] Image length = road speed limit or average traffic speed × advance prediction time × 2 / length represented by each pixel.

[0140] Image width = width of three lanes centered on the current lane / width represented by each pixel.

[0141] Generate target vehicle motion image, surrounding vehicle motion image, and road image in R, G, and B channels in sequence, including:

[0142] Draw the target vehicle motion image in the R channel, that is, store the current and historical motion positions of the target vehicle in the R channel, draw the vehicle with a rectangle, the rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0143] Draw the surrounding vehicle motion image in the G channel, that is, store the current and historical motion positions of the surrounding vehicles in the G channel, and use rectangles to represent the surrounding vehicles. The rectangle position is calculated according to the vehicle position and size, and the color depth of the rectangle indicates the distance between the observation time and the last observation time.

[0144] Draw the drivable area image in the B channel, that is, store the road information in the B channel, and use different filling values ​​to distinguish the drivable area, non-drivable area and lane boundary.

[0145] Finally, the three-channel information is fused to obtain an image for intent prediction.

[0146] Furthermore, the vehicle intention prediction model is constructed as follows:

[0147] The input of the vehicle intention prediction model is a three-channel image, where each channel stores the motion state of the target vehicle, the motion state of surrounding vehicles, and the lane boundary information.

[0148] ResNet-18 is used as the backbone network, and the residual module is used to achieve deep feature learning.

[0149] Two CBAMs are integrated before and after the residual block, namely between the initial convolutional layer Conv1_x and the first residual block Conv2_x and between the last residual block Conv5_x and the output processing layer.

[0150] The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, the probability of each intent is obtained, and the intent with the highest probability is regarded as the prediction result.

[0151] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle lane-changing intention prediction method based on ResNet, characterized in that: The following steps are involved: Step 1: Obtain information about the target vehicle, surrounding vehicles, and road conditions; Step 2: Generate an RGB image based on the scene information of the target vehicle, i.e., an image for intent prediction. Multiple images for intent prediction constitute a training set image: Step 3: Embed CBAM into the ResNet network to build a vehicle intention prediction model; select ResNet-18 as the backbone network and embed CBAM before and after the residual block, that is, between the initial convolution layer and the first residual block and between the last residual block and the output processing layer; Step 4: Using the training set images to train the vehicle intention prediction model, learn classification parameters, and obtain a trained vehicle intention prediction model; Step 5: Generate a corresponding RGB image based on the scene information to be predicted, and input the corresponding RGB image of the scene into the trained vehicle intention prediction model to obtain the predicted vehicle lane changing intention.

2. A vehicle lane-changing intention prediction method based on ResNet as claimed in claim 1, characterized in that: In step 1, the information obtained includes: Obtain information about the target vehicle and surrounding vehicles, including the length and width of the vehicle, as well as the center of mass position of the vehicle at the current and historical times, and the lane to which it belongs; the surrounding vehicles include the vehicles that are located in front of and behind the target vehicle and are closest to the target vehicle in the left adjacent lane, right adjacent lane, and current lane of the target vehicle; The road information includes the drivable area, lane boundaries, left and right adjacent lanes, and the width of the current lane; The drivable area is defined as a combination of a series of lanes. Vehicles change lanes or overtake between lanes within the drivable area. The boundary of the drivable area is the solid line of the road or the shoulder.

3. The method for predicting vehicle lane-changing intention based on ResNet as claimed in claim 1, characterized in that: In the step 1, corresponding information is obtained from the NGSIM data set, and the data in the NGSIM data set is processed as follows: 1) Correct the vehicle number, that is, sort, split and renumber the data according to the vehicle number, location and timestamp information; 2) Filter the data in the ROI, determine whether there is a lane change behavior based on the change in the lane number of the vehicle, regard the part that does not include the ramp as the ROI area, and filter the data to be processed based on the lane numbers in the area; 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the road boundary is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point is found according to the lane number of the vehicle provided by the NGSIM dataset. The dataset is traversed according to the vehicle number, and then the motion sequence of the target vehicle is traversed in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number. 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information n seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly extract a timestamp and save the time, position, and lane information n seconds before the timestamp. 5) Screen the surrounding vehicles of each target vehicle and record their movement status; Traverse the motion sequences of each target vehicle and obtain the position and lane of the target vehicle at each timestamp; According to the timestamp and the lane where the target vehicle is located, all vehicles in the lane where the target vehicle is located and the adjacent lanes are filtered out from the data set; The vehicles on each lane are sorted by position, the front and rear vehicles closest to the target vehicle are selected, and the corresponding time and position information are saved.

4. A vehicle lane change intention prediction method based on ResNet as claimed in claim 1 or 2, characterized in that: The second step is to generate an RGB image according to the scene information of the target vehicle, specifically: Obtain the motion sequences of the target vehicle and surrounding vehicles respectively and determine the image size; Generate target vehicle motion image, surrounding vehicle motion image, and road image in R, G, and B channels in sequence; Finally, the three-channel information is fused to obtain an image for intent prediction.

5. A vehicle lane-changing intention prediction method based on ResNet as claimed in claim 4, characterized in that: In the step 2, the motion sequences of the target vehicle and the surrounding vehicles are obtained respectively, and the image size is determined, specifically: the image size is determined according to scene information such as the speed limit or average speed, lane width, etc.: Image length = road speed limit or average traffic speed × advance prediction time × 2 / length of each pixel; Image width = width of three lanes centered on the current lane / width represented by each pixel.

6. A vehicle lane-changing intention prediction method based on ResNet as claimed in claim 4, characterized in that: In the step 2, the target vehicle motion image, the surrounding vehicle motion image, and the road image are generated in the R, G, and B channels in sequence, specifically including: Draw the target vehicle motion image in the R channel, that is, store the current and historical motion positions of the target vehicle in the R channel, draw the vehicle with a rectangle, the position of the rectangle is calculated according to the position and size of the vehicle, and the color of the rectangle indicates the distance between the observation time and the last observation time; Draw the surrounding vehicle motion image in the G channel, that is, store the current and historical motion positions of the surrounding vehicles in the G channel, and use rectangles to represent the surrounding vehicles. The rectangle position is calculated according to the vehicle position and size, and the color of the rectangle indicates the distance between the observation time and the last observation time. Draw the drivable area image in the B channel, that is, store the road information in the B channel, and use different filling values ​​to distinguish the drivable area, non-drivable area and lane boundary.

7. A vehicle lane-changing intention prediction method based on ResNet as claimed in claim 6, characterized in that: The target vehicle motion image is drawn in the R channel, specifically: The target vehicle observed at the last moment is placed at the center of the image so that the geometric center of the vehicle coincides with the center of the image. The vehicle is represented by a rectangle. The upper left corner coordinate of the target vehicle at time t is and the lower right corner coordinates They are Among them, x, y are the vehicle position coordinates, w is the vehicle width, and l is the vehicle length; The color depth of the pixel is used to indicate the distance between the observation time and the current time. h In the observation sequence of , the vehicle at the tth observation time is filled as The earlier the observation time, the smaller t is, and the lighter the color of the vehicle is. Conversely, the later the observation time is, the darker the color of the vehicle is.

8. A vehicle lane change intention prediction method based on ResNet as claimed in claim 6 or 7, characterized in that: The drivable area image is drawn in the B channel: the pixel value of the drivable area is the default value 0, and the value of the non-drivable area is set to 255. Correspondingly, the drivable area is displayed in black in the image, and the non-drivable area is displayed in blue; on this basis, the lane lines are drawn, and the B channel value of each lane boundary is set to 255.

9. The method for predicting vehicle lane-changing intention based on ResNet as claimed in claim 1, characterized in that: The process of building a vehicle intention prediction model in step 3 is as follows: The input of the vehicle intention prediction model is a three-channel image, each channel stores the motion state of the target vehicle, the motion state of the surrounding vehicles and the lane boundary information; ResNet-18 is used as the backbone network, and the residual module is used to achieve deep feature learning; Two CBAMs are integrated before and after the residual block, namely between the initial convolutional layer Conv1_x and the first residual block Conv2_x and between the last residual block Conv5_x and the output processing layer; The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, the probability of each intent is obtained, and the intent with the highest probability is regarded as the prediction result.

10. A vehicle lane-changing intention prediction system based on ResNet, characterized in that: It includes information acquisition module, input image generation module, vehicle intention prediction module, training module and real-time prediction module: The information acquisition module is used to acquire information about the target vehicle and surrounding vehicles as well as road information; The input image generation module is used to generate an RGB image according to the scene information of the target vehicle; The vehicle intention prediction model embeds CBAM into the ResNet network to obtain the vehicle intention prediction model; ResNet-18 is selected as the backbone network, and CBAM is embedded before and after the residual block, that is, between the initial convolution layer and the first residual block and between the last residual block and the output processing layer; The training module takes an existing data set and calls the information acquisition module and the input image generation module to generate an RGB image as a training image. A plurality of training images constitute a training set. The vehicle intention prediction model is trained using the training set to learn classification parameters and obtain a trained vehicle intention prediction model. The real-time prediction module calls the information acquisition module and the input image generation module according to the scene information to be predicted, generates a corresponding RGB image, and inputs the scene corresponding RGB image into the trained vehicle intention prediction model to obtain the predicted vehicle lane change intention; Furthermore, the information acquisition module acquires the following information: Obtain information about the target vehicle and surrounding vehicles, including the length and width of the vehicle, as well as the center of mass position of the vehicle at the current and historical times, and the lane to which it belongs; the surrounding vehicles include the vehicles that are located in front of and behind the target vehicle and are closest to the target vehicle in the left adjacent lane, right adjacent lane, and current lane of the target vehicle; The road information includes the drivable area, lane boundaries, left and right adjacent lanes, and the width of the current lane; The drivable area is defined as a combination of a series of lanes. Vehicles change lanes or overtake between lanes within the drivable area. The boundary of the drivable area is the solid line of the road or the shoulder. Furthermore, the existing data set taken by the training module is the NGSIM data set. The training module specifically: calls the information acquisition module to acquire corresponding information from the NGSIM data set, and performs the following processing on the data in the NGSIM data set: 1) Correct the vehicle number, that is, sort, split and renumber the data according to the vehicle number, location and timestamp information; 2) Filter the data in the ROI, determine whether there is a lane change behavior based on the change in the lane number of the vehicle, regard the part that does not include the ramp as the ROI area, and filter the data to be processed based on the lane numbers in the area; 3) Obtain the lane-changing point of the target vehicle. The moment when the center of the target vehicle is at the road boundary is regarded as the lane-changing point. The moment when each vehicle is closest to the lane-changing point is found according to the lane number of the vehicle provided by the NGSIM dataset. The dataset is traversed according to the vehicle number, and then the motion sequence of the target vehicle is traversed in order from early to late. If there is a lane number jump, the last timestamp before the jump is regarded as the lane-changing point, and the lane-changing direction is marked according to the positive or negative change of the lane number. 4) Obtain the motion state of the target vehicle before it reaches the lane-changing point. For vehicles that change lanes, save the time, position, and lane information n seconds before it reaches the lane-changing point. For vehicles that do not change lanes throughout the entire process, randomly extract a timestamp and save the time, position, and lane information n seconds before the timestamp. 5) Filter the surrounding vehicles of each target vehicle and record their motion status; traverse the motion sequence of each target vehicle to obtain the position and lane of the target vehicle in each timestamp; filter out all vehicles in the lane where the target vehicle is located and the adjacent lanes from the data set according to the timestamp and the lane where the target vehicle is located; sort the vehicles in each lane according to their positions, filter out the front and rear vehicles closest to the target vehicle, and save the corresponding time and position information; Furthermore, the input image generation module is used to generate an RGB image according to the scene information of the target vehicle, and specifically performs the following steps: Get the target vehicle and surrounding vehicle motion sequences respectively, and determine the image size. Specifically, determine the image size based on scene information such as speed limit or average speed, lane width, etc.: Image length = road speed limit or average traffic speed × advance prediction time × 2 / length of each pixel; Image width = width of three lanes centered on the current lane / width represented by each pixel; Generate target vehicle motion image, surrounding vehicle motion image, and road image in R, G, and B channels in sequence, including: Draw the target vehicle motion image in the R channel, that is, store the current and historical motion positions of the target vehicle in the R channel, draw the vehicle with a rectangle, the position of the rectangle is calculated according to the position and size of the vehicle, and the color of the rectangle indicates the distance between the observation time and the last observation time; Draw the surrounding vehicle motion image in the G channel, that is, store the current and historical motion positions of the surrounding vehicles in the G channel, and use rectangles to represent the surrounding vehicles. The rectangle position is calculated according to the vehicle position and size, and the color of the rectangle indicates the distance between the observation time and the last observation time. Draw a drivable area image in the B channel, that is, store road information in the B channel, and use different filling values ​​to distinguish the drivable area, non-drivable area and lane boundary; Finally, the three-channel information is fused to obtain an image for intent prediction; Furthermore, the vehicle intention prediction model is constructed in the following steps: The input of the vehicle intention prediction model is a three-channel image, each channel stores the motion state of the target vehicle, the motion state of the surrounding vehicles and the lane boundary information; ResNet-18 is used as the backbone network, and the residual module is used to achieve deep feature learning; Two CBAMs are integrated before and after the residual block, namely between the initial convolutional layer Conv1_x and the first residual block Conv2_x and between the last residual block Conv5_x and the output processing layer; The output layer is a fully connected layer with a dimension of 3. After being processed by the softmax function, the probability of each intent is obtained, and the intent with the highest probability is regarded as the prediction result.

Images