End-to-end steering angle prediction method, device and system focusing on drivable area

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an end-to-end steering angle prediction method, device and system focusing on a drivable area. The method comprises the steps of collecting time sequence image data of a vehicle in real time; the time sequence image data are multi-frame driving images; inputting the time sequence image data of the vehicle into a pre-designed image segmentation network, and outputting a lane region image; the image segmentation network comprises an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module and a decoder fused with the convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category; performing binary mask processing on the lane area image, and extracting drivable area features of the vehicle; and inputting the travelable region characteristics of the vehicle into a pre-designed time sequence classification module, and outputting a steering angle prediction result of the vehicle. According to the invention, the steering angle result of the drivable area can be predicted more accurately.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and more specifically, relates to an end-to-end steering angle prediction method, device and system focusing on a drivable area. Background Art

[0002] Over the past decade, autonomous driving technology has made remarkable progress and attracted widespread attention from many industries and academic institutions. This remarkable achievement is largely due to the continuous research and development and progress of deep neural network technology and high-performance computing hardware. In particular, steering angle prediction, as a core element in the autonomous driving system, has received special attention because it is directly related to the safety and efficiency of vehicle driving.

[0003] In recent years, a variety of end-to-end vehicle steering prediction methods have emerged, among which vision-based vehicle steering has been widely studied, mainly because vehicle-mounted cameras have a practical advantage of low cost compared to radar. Therefore, how to obtain rich information from images to guide steering has become a research focus.

[0004] At present, there are two main types of end-to-end vehicle steering prediction methods based on camera images: one is based on convolutional neural networks (CNN), and the other is a method that combines convolutional neural networks (CNN) with long short-term memory networks (LSTM). The earliest method based on convolutional neural networks can be traced back to the ALVINN neural network developed by Carnegie Mellon University in 1989. It uses neural networks to process image data collected by on-board cameras and generate vehicle steering commands. ALVINN is an important milestone in the field of autonomous driving, which has influenced the future application of visual navigation and deep learning in autonomous driving. In 2016, Bojarski et al. proposed NVIDIAPilotNet to use a deep neural network with a convolutional layer to predict vehicle steering. In recent years, models based on convolutional neural networks such as ChauffeurNet and CILRS have been proposed for vehicle steering control prediction. Although the method based on convolutional neural networks can obtain rich information from images to guide vehicle steering, it may also learn information from images that is not conducive to steering, and this method does not take into account the temporal relationship between image frames. Although the method that combines convolutional neural networks (CNN) with long short-term memory networks (LSTM) takes into account the temporal relationship between image frames, LSTM is better at handling long-term dependencies, which may be harmful to autonomous driving tasks because drivers will not consider the scenes a few seconds ago when turning. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides an end-to-end steering angle prediction method, device and system focusing on the drivable area, which aims to learn information related to the steering height from the image and use NCP to obtain timing information to avoid long-term dependence problems.

[0006] To achieve the above object, the first aspect of the present invention proposes an end-to-end steering angle prediction method focusing on a drivable area, the method comprising:

[0007] Collecting time-series image data of the vehicle in real time; the time-series image data is a multi-frame driving image;

[0008] Inputting the time series image data of the vehicle into a pre-designed image segmentation network, and outputting a lane area image; the image segmentation network includes an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category;

[0009] Processing the lane area image using a binary mask to extract features of a drivable area for the vehicle;

[0010] The drivable area features of the vehicle are input into a pre-designed time series classification module, and the steering angle prediction result of the vehicle is output.

[0011] To achieve the above object, the second aspect of the present invention provides an end-to-end steering angle prediction device focusing on a drivable area, comprising:

[0012] A time-series image acquisition module is used to collect time-series image data of the vehicle in real time; the time-series image data is a plurality of frames of driving images;

[0013] An image region segmentation module, used for inputting the time series image data of the vehicle into a pre-designed image segmentation network, and outputting a lane region image; the image segmentation network comprises an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane region image is a lane image indicating a lane category and a non-lane category;

[0014] A region feature extraction module, used to process the lane region image using a binary mask to extract features of a drivable region of the vehicle;

[0015] The steering angle information prediction module is used to input the drivable area characteristics of the vehicle into a pre-designed time series classification module and output the steering angle prediction result of the vehicle.

[0016] To achieve the above object, the third aspect of the present invention provides an end-to-end steering angle prediction system focusing on a drivable area, comprising:

[0017] A steering angle camera is arranged on the front fenders of the left and right front wheels of the vehicle and is used to collect time-series image data of the vehicle in real time; the time-series image data is a multi-frame driving image;

[0018] A central processing unit is used to input the time series image data of the vehicle into a pre-designed image segmentation network and output a lane area image; the image segmentation network includes an encoder integrated with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module and a decoder integrated with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category; the lane area image is processed using a binary mask to extract the vehicle's drivable area features; the vehicle's drivable area features are input into a pre-designed time series classification module to output the vehicle's steering angle prediction result.

[0019] Furthermore, the steering angle camera is arranged on the front fenders of the left and right front wheels of the vehicle; the central processor is arranged inside the vehicle; and the interior of the vehicle includes the central console or instrument panel of the vehicle.

[0020] Furthermore, it also includes a display, which is used to display the steering angle prediction result of the vehicle.

[0021] Beneficial effects of the present invention:

[0022] The present invention provides an end-to-end steering angle prediction method, device and system focusing on the drivable area. The present invention integrates a convolutional long short-term memory network layer in the encoder and decoder in the image segmentation network, which can retain spatial information while supplementing the feature changes in temporal information to obtain a lane area image with more time-series consistency; compared with the prediction technology based on convolutional neural network, the present invention discards the features in the vehicle image that are irrelevant to the steering task, such as the sky, trees, buildings, etc., by performing binary mask processing on the lane area image, thereby reducing the computing cost and eliminating the interference of harmful factors to a certain extent to enhance the prediction performance; compared with the prediction technology combining convolutional neural network and long short-term memory network, the present invention adopts a temporal classification module that is more suitable for the autonomous driving steering prediction task, avoiding the problem that long-term dependencies are not suitable for autonomous driving steering scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of an end-to-end steering angle prediction method focusing on a drivable area according to an embodiment of the present invention;

[0024] Figure 2is a schematic diagram of an end-to-end steering angle prediction network structure focusing on a drivable area according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of the encoder structure of an image segmentation network according to an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of a decoder structure of an image segmentation network according to an embodiment of the present invention;

[0027] Figure 5 is a structural diagram of an end-to-end steering angle prediction device focusing on a drivable area according to an embodiment of the present invention;

[0028] Figure 6 It is a schematic diagram of an end-to-end steering angle prediction system focusing on a drivable area according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] According to an embodiment of the present invention, an embodiment of an end-to-end steering angle prediction method focusing on a drivable area is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0031] In this embodiment, an end-to-end steering angle prediction method focusing on a drivable area is provided, which can be used in electronic devices such as computers or mobile terminals. Figure 1 is a flow chart of an end-to-end steering angle prediction method focusing on a drivable area according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0032] 101. Collecting time-series image data of a vehicle in real time; the time-series image data is a plurality of frames of driving images;

[0033] In some embodiments, a camera can be added to a traditional vehicle to collect time-series image data of the vehicle in real time during driving, especially when the vehicle is approaching a turn; wherein the camera can be installed on the front fenders of the left and right front wheels, one on each side, facing the left and right sides of the front of the vehicle, to realize data collection on the left and right sides of the front of the vehicle. For example, when the vehicle turns right, the camera installed on the right front wheel is used to collect time-series image data; when the vehicle turns left, the camera installed on the left front wheel is used to collect time-series image data; this method of installing cameras on both sides can obtain more accurate time-series image data.

[0034] In some embodiments, the time-series image data of the vehicle may include RGB image data and corresponding steering angle data during the vehicle's transformation process; each frame of RGB image data must have corresponding steering angle data. If there is no corresponding steering angle data, the RGB image data may be removed to ensure the accuracy of the steering prediction result. The time-series image data of the vehicle may be expressed as:

[0035] [I t ,I t+1 ,I t+2 ...,I t+n ]

[0036] Among them, I t represents the image frame of the vehicle at time t, and n represents the length of the selected time series.

[0037] In some embodiments, the time series image data of the vehicle may be from public datasets such as the SullyChen dataset, the CARLA simulator dataset, and the Udacity simulator dataset. The experimental environment may be PyTorch on two NVIDIA RTX 3090ti GPUs, and the original time series image data is normalized, and the pixel values ​​are in the range of [-1,1], which can be used as RGB image input.

[0038] In an embodiment of the present invention, in order to solve the problems that traditional modular methods are difficult to promote and maintain, the present invention implements vehicle steering prediction in an end-to-end manner, reducing certain cascade errors caused by human operations, and the model autonomously learns steering from input data.

[0039] In an embodiment of the present invention, Figure 2 The schematic diagram of the end-to-end steering angle prediction network structure focusing on the drivable area is shown. The following will combine steps 102-104 with Figure 2 The prediction network shown illustrates the prediction method of the present invention.

[0040] 102. Input the time series image data of the vehicle into a pre-designed image segmentation network, and output a lane area image; the image segmentation network includes an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category;

[0041] In some embodiments, Figure 3 As shown, the encoder integrated with the convolutional long short-term memory network layer includes a feature extraction layer and a convolutional long short-term memory network layer; the feature extraction layer includes a plurality of feature extraction blocks at different levels; the plurality of feature extraction blocks at different levels are used to extract multi-scale features of the time series image data of the vehicle; the convolutional long short-term memory network layer is used to perform time series encoding on the extracted multi-scale features to generate a sequence of encoded feature maps.

[0042] In the embodiment of the present invention, the time-series image data of the vehicle is positioned from the data dimension as follows:

[0043] Input = (sample size, time series length, C, H, W)

[0044] Among them, C represents the number of channels of the RGB image, H represents the height of the RGB image, and W represents the width of the RGB image. The RGB image here refers to the driving image; the encoder in the image segmentation network receives such a dimensional input; the encoder part contains 3 main feature extraction layers: level1, level2_0, and level3_0 extract low-, medium-, and high-level features respectively.

[0045] The initial convolutional layer (level1) uses a convolutional block to process the input multi-frame driving images, increasing the number of channels from 3 to 16, and downsampling the spatial resolution by 2 times. The structure of the convolutional block is convolutional layer Conv2-> batch normalization layer BatchNorm2d-> parameterized rectified linear unit PReLU. The purpose of this convolutional block is to extract basic features while reducing the spatial resolution. Multi-scale downsampling sample1 and sample2 are used to provide richer information for subsequent feature fusion. Sample1 downsamples the input once (spatial size is halved), and sample2 downsamples the input twice (spatial size is reduced to 1 / 4 of the original).

[0046] The first downsampling and multi-scale feature extraction block (level2_0) connects the output of level1 and the result of sample1 after downsampling, and uses grouped convolution and dilated convolution to further downsample, thereby halving the spatial resolution again and increasing the number of channels from 19 to 64, so that the output feature map has higher feature expression capabilities. The multi-scale feature extraction block contains p residual blocks to further process the downsampled output. The dilated convolution in the residual block uses a 3×3 convolution kernel with different dilation rates. By cascading these 3×3 convolution kernels with different dilation rates, spatial information of different scales can be captured without increasing the number of parameters.

[0047] The second layer of downsampling and multi-scale feature extraction block (level3_0) concatenates the output of level2_0 and the feature map of sample2, and then further reduces the spatial size by half through downsampling, increasing the number of channels to 128. The multi-scale feature extraction part contains q residual blocks to continue to extract features of different receptive fields. The feature maps output by level3_0 are concatenated and input into the ConvLSTM module in chronological order, with 128 input channels, 64 hidden layers, and a 3×3 convolution kernel. The input is a series of frame sequences, which can be expressed as:

[0048] [I1,I2,...,I T ]Where T represents the total number of time steps, the input frame I of each time step t After the encoder layer is integrated with the convolutional long short-term memory network layer, the encoded feature map Enc is obtained. t , the encoded feature map of each layer is input into the convolutional long short-term memory network layer ConvLSTM, so as to generate an updated sequence of encoded feature maps in time sequence, which can be expressed as:

[0049] Enc′1,Enc′2,...,Enc′ T The number of output channels is set to be consistent with the number of level3_0 output channels, which facilitates direct processing of subsequent modules without increasing computational overhead. In the embodiment of the present invention, a convolutional long short-term memory network layer ConvLSTM module is added after the output of level3_0, which can enhance the temporal expression of the encoded global features and retain a large amount of temporal context information. This requires less computing resources and reduces the amount of computation for the processing of vehicle temporal image data.

[0050] Then, the PAM position attention module is used to capture the important regional information in the spatial dimension and the CAM channel attention module is used to capture the correlation in the channel dimension, helping the network to focus on the key channels. In this way, the timing information will be integrated into the global features first, so that the PAM position attention module can have better processing capabilities according to the data at different time steps.

[0051] The PAM position attention module process is as follows:

[0052] Given an input feature map X(B,T,C,H,W), where B is the batch size, S is the time series length, C is the number of channels, H is the height, and W is the width. First, it is mapped to three new feature spaces through a convolutional layer to generate three feature maps: query, key, and value. The query and key feature maps are matrix multiplied according to the channel dimension to calculate the attention weight of each pixel to all other pixels, forming a spatial attention map with a shape of H*W×H*W, where H*W represents the number of spatial positions. The attention weights of the spatial attention map are normalized by Softmax to ensure that the sum of the weights is 1. Use this attention map to multiply the value feature map to generate a weighted feature representation in the spatial dimension, and finally add the result to the original input feature to obtain the enhanced output feature map.

[0053] The CAM channel attention module process is as follows:

[0054] Given an input feature map X(B,T,C,H,W), first reduce the spatial dimension of the feature map and flatten it into C x N (where N = H*W). Calculate the relationship between each channel and other channels through matrix multiplication to generate a channel similarity matrix (of shape C×C) to represent the response relationship between different channels. Normalize the channel similarity matrix through Softmax to generate a channel attention map. Use this channel attention map to weight the original input feature map to generate an enhanced feature map. Add the weighted channel feature map to the input feature map to obtain the channel enhanced feature representation.

[0055] In this embodiment of the present invention, the outputs of the two attention modules are finally fused through the convolution of conv_pam and conv_cam.

[0056] In some embodiments, Figure 4 As shown, the decoder integrated with the convolutional long short-term memory network layer includes multiple decoding blocks, each decoding block includes an upsampling block, a convolutional long short-term memory network layer, a feature fusion layer, a batch normalization layer and a PReLU activation function layer; the upsampling block is used to upsample the input feature map to extract spatial information; the convolutional long short-term memory network layer is used to perform time series processing on the input feature map to extract time series information; the feature fusion layer is used to fuse the spatial information, time series information and encoded feature map of the corresponding level.

[0057] In some embodiments, the decoder of the present invention is composed of 3 blocks, each of which depends on the upsampling block, ConvLSTM layer and feature fusion. The processing order of each block is Upsampling block -> ConvLSTM layer -> feature fusion -> batch normalization and PReLU activation function. The processing steps of each Upsampling block are as follows:

[0058] The feature map is enlarged to twice its original spatial size using methods such as bilinear interpolation or deconvolution (transposed convolution). The upsampled feature map is concatenated with the feature map of the corresponding layer in the encoder to enrich the multi-scale information required in the upsampling process. The concatenated feature map is further processed by the convolution layer to extract and fuse cross-layer features, making the concatenated feature map more suitable for subsequent upsampling operations. The decoder gradually upsamples the high-level feature maps to the resolution of the original input image through layer-by-layer upsampling. Each upsampling operation not only increases the resolution of the feature map, but also combines the multi-scale features in the encoder through jump connections, gradually restoring the spatial structure of the image while maintaining the integrity of the semantic information.

[0059] Each upsampling is followed by a ConvLSTM layer to fuse the upsampled spatial information with the temporal information layer by layer, thereby gradually enhancing the temporal consistency during feature decoding. The corresponding feature map in the encoder is spliced ​​to the decoder feature map using skip connections to introduce multi-scale spatial information. The spliced ​​feature map contains multi-scale features of the encoder and decoder. In order to further fuse this information, convolution operations are used to integrate it. Then batch normalization and PReLU activation function are performed.

[0060] The process between each block of the decoder of the present invention is as follows:

[0061] The first block first upsamples the feature map of the last layer of the encoder by one time, and the output shape is: (B, T, C_L, H_L*2, W_L*2), where B is the batch size, T is the time series length, C_L is the number of channels, and H_L and W_L are the height and width of the feature map of the current layer. The upsampled feature map sequence is input into ConvLSTM to extract the timing information of the feature map. The ConvLSTM output feature map shape is (B, T, C_L, H_L*2, W_L*2), where the number of channels is usually set to the same as the upsampling layer so that the features can be directly transferred. It is then concatenated with the output of the penultimate layer of the encoder, and the concatenated shape is (B, C_L+C_E, H_L*2, W_L*2), where C_L+C_E is the total number of channels after concatenation. The concatenated feature map contains the multi-scale features of the encoder and decoder, and then a 3×3 convolution operation is used to capture more local spatial information, so that the spatial features of the encoder feature map and the decoder feature map are better integrated, and then batch normalization and PReLU activation function are performed.

[0062] The second block further upsamples and processes the upsampled features of the first layer through ConvLSTM and concatenates them with the feature map of an earlier layer of the encoder for further fusion processing.

[0063] The third block further amplifies the upsampling result of the second layer by one time and concatenates and fuses it with the encoder features of the bottom layer through ConvLSTM processing to generate a feature map with full spatial resolution.

[0064] 103. Process the lane area image using a binary mask to extract features of a drivable area for the vehicle;

[0065] In an embodiment of the present invention, the lane image is processed using a binary mask to extract the drivable area features of the vehicle, including: setting the pixel value representing the lane category to 0 and the pixel value representing the non-lane category to 1; performing a bitwise AND operation on the binary mask and the lane area image to retain only the lane area and generate a drivable area image; using a convolutional layer to extract the initial drivable area features from the drivable area image, using two fully connected layers to combine the initial drivable area features, and performing dimensionality reduction on the combined drivable area features to generate the final drivable area features.

[0066] It can be understood that the present invention introduces a binary mask to process the output of the decoder, which assigns a value of 0 to pixels representing lane categories and 1 to non-lane pixels. It is represented as a two-dimensional array, where each element corresponds to a pixel in the image and takes a value of 0 or 1. A value of 0 indicates that the pixel belongs to the lane area. The binary mask, denoted as B, has a size of M×N, where M is the image height and N is the image width. Each element in the binary mask B i,j represents the value of the pixel in the i-th row and j-th column; when the pixel in the i-th row and j-th column belongs to the lane area, then B i,j = 0, when the pixel in the i-th row and j-th column does not belong to the lane area, then B i,j =1.

[0067] In some embodiments, this embodiment uses convolution with a 3×3 convolution kernel to further extract features from the output of the binary mask, and then uses FC(64) and FC(32) to combine these features and reduce the dimensionality of the high-dimensional convolution output.

[0068] 104. Input the drivable area features of the vehicle into a pre-designed time series classification module, and output the steering angle prediction result of the vehicle.

[0069] In some embodiments, the time series classification module can be a common network that can be used for time series information classification, such as a one-dimensional convolutional neural network (1D-CNN), a temporal convolutional network (TCN), a gated recurrent unit (GRU), etc.

[0070] In some preferred embodiments, the time series classification module of the present invention is implemented using the NCP neural circuit strategy. The present invention further extracts the time series information features by introducing the NCP neural circuit strategy to ultimately guide steering. The 32 sensory neurons of the NCP receive the 32 feature maps finally outputted in the previous step, and the steering instructions are finally outputted by the motor neurons through the intermediate neurons and command neurons. The neural dynamics of the NCP are given by continuous-time ordinary differential equations (ODEs). In essence, the NCP has a nonlinear time-varying synaptic transmission mechanism, which improves its expressive power in time series modeling compared to its deep learning counterparts.

[0071] Based on the above embodiments, the present invention focuses on the end-to-end steering angle prediction method of the drivable area, alleviates the problem of convolutional neural network extracting features of irrelevant areas and neglecting time information, as well as the problem of LSTM's long-term dependence on the task. The present invention provides a more reliable and accurate optimization solution, improves the robustness of the prediction model, and brings important improvements to the steering prediction of autonomous driving vehicles. Reduce the cost and resources required for autonomous driving.

[0072] In this embodiment, an end-to-end steering angle prediction device focusing on the drivable area is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0073] This embodiment provides an end-to-end steering angle prediction device focusing on the drivable area, such as Figure 5 As shown, the device comprises:

[0074] The time-series image acquisition module 201 is used to collect the time-series image data of the vehicle in real time; the time-series image data is a plurality of frames of driving images;

[0075] The image region segmentation module 202 is used to input the time series image data of the vehicle into a pre-designed image segmentation network and output a lane region image; the image segmentation network includes an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module and a decoder fused with a convolutional long short-term memory network layer; the lane region image is a lane image indicating a lane category and a non-lane category;

[0076] The area feature extraction module 203 is used to process the lane area image using a binary mask to extract the features of the vehicle's drivable area;

[0077] The steering angle information prediction module 204 is used to input the drivable area characteristics of the vehicle into a pre-designed time series classification module and output the steering angle prediction result of the vehicle.

[0078] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0079] This embodiment provides an end-to-end steering angle prediction system focusing on the drivable area, such as Figure 6 As shown, the system includes:

[0080] The steering angle camera 301 is used to collect the time-series image data of the vehicle in real time; the time-series image data is a multi-frame driving image;

[0081] Specifically, the steering angle camera 301 collects time-series image data of the vehicle in real time during driving, especially during driving close to a turn; wherein, the camera can be installed on the front fenders of the left and right front wheels, one on each side, facing the left and right sides of the front of the vehicle, to realize data collection on the left and right sides of the front of the vehicle. For example, when the vehicle turns right, the camera installed on the right front wheel is used to collect time-series image data; when the vehicle turns left, the camera installed on the left front wheel is used to collect time-series image data; this method of installing cameras on both sides can obtain more accurate time-series image data.

[0082] When performing end-to-end steering angle prediction, the steering angle camera 301 is connected to the central processor 302 through interfaces such as MIPI, USB, Ethernet, etc.; after collecting and processing the time-series image, the steering angle camera 301 transmits the time-series image data to the central processor 302.

[0083] The central processing unit 302 is used to input the time series image data of the vehicle into a pre-designed image segmentation network to output a lane area image; the image segmentation network includes an encoder integrated with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module and a decoder integrated with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category; the lane area image is processed using a binary mask to extract the vehicle's drivable area features; the vehicle's drivable area features are input into a pre-designed time series classification module to output the vehicle's steering angle prediction result.

[0084] In some embodiments, the central processor 302 is disposed inside a vehicle; the vehicle interior includes a central console or a dashboard of the vehicle.

[0085] In some embodiments, the system further comprises a display, wherein the display is used to display the steering angle prediction result of the vehicle, so that the vehicle user can intuitively observe the steering angle prediction result.

[0086] Based on the above embodiments, the present invention alleviates the problems of convolutional neural networks extracting features of irrelevant areas and neglecting time information, as well as the long-term dependence of LSTM on the task in the end-to-end steering angle prediction task focusing on the drivable area, provides a more reliable and accurate optimization solution, improves the robustness of the algorithm, and brings important improvements to the steering prediction of autonomous driving vehicles.

[0087] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An end-to-end steering angle prediction method focusing on the drivable area, characterized in that: The method comprises: Collecting time-series image data of the vehicle in real time; the time-series image data is a multi-frame driving image; Inputting the time series image data of the vehicle into a pre-designed image segmentation network, and outputting a lane area image; the image segmentation network includes an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category; Processing the lane area image using a binary mask to extract features of a drivable area for the vehicle; The drivable area features of the vehicle are input into a pre-designed time series classification module, and the steering angle prediction result of the vehicle is output.

2. The end-to-end steering angle prediction method focusing on the drivable area according to claim 1 is characterized in that: The encoder integrated with the convolutional long short-term memory network layer includes a feature extraction layer and a convolutional long short-term memory network layer; the feature extraction layer includes multiple feature extraction blocks at different levels; the multiple feature extraction blocks at different levels are used to extract multi-scale features of the vehicle's time series image data; the convolutional long short-term memory network layer is used to perform time series encoding on the extracted multi-scale features to generate a sequence of encoded feature maps.

3. The end-to-end steering angle prediction method focusing on the drivable area according to claim 2 is characterized in that: The decoder integrated with the convolutional long short-term memory network layer includes multiple decoding blocks, each decoding block includes an upsampling block, a convolutional long short-term memory network layer, a feature fusion layer, a batch normalization layer and a PReLU activation function layer; the upsampling block is used to upsample the input feature map and extract spatial information; the convolutional long short-term memory network layer is used to perform time series processing on the input feature map and extract time series information; the feature fusion layer is used to fuse the spatial information, time series information and encoded feature map of the corresponding level.

4. The end-to-end steering angle prediction method focusing on the drivable area according to claim 1, characterized in that: The lane image is processed using a binary mask to extract the features of the vehicle's drivable area, including: setting the pixel value representing the lane category to 0 and the pixel value representing the non-lane category to 1; performing a bitwise AND operation on the binary mask and the lane area image to retain only the lane area and generate a drivable area image; and using a convolutional layer to extract the initial drivable area features from the drivable area image.

5. The end-to-end steering angle prediction method focusing on the drivable area according to claim 4 is characterized in that: The extracting of the drivable area features includes combining the initial drivable area features by using two fully connected layers, and performing dimensionality reduction on the combined drivable area features to generate the final drivable area features.

6. The end-to-end steering angle prediction method focusing on the drivable area according to claim 1, characterized in that: The temporal classification module is implemented using the NCP neural circuit strategy.

7. An end-to-end steering angle prediction device focusing on the drivable area, characterized in that: include: A time-series image acquisition module, used for collecting time-series image data of vehicles in real time; The time-series image data is a plurality of frames of driving images; An image region segmentation module, used for inputting the time series image data of the vehicle into a pre-designed image segmentation network, and outputting a lane region image; the image segmentation network comprises an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane region image is a lane image indicating a lane category and a non-lane category; A region feature extraction module, used to process the lane region image using a binary mask to extract features of a drivable region of the vehicle; The steering angle information prediction module is used to input the drivable area characteristics of the vehicle into a pre-designed time series classification module and output the steering angle prediction result of the vehicle.

8. An end-to-end steering angle prediction system focusing on the drivable area, characterized in that: include: Steering angle camera, used to collect time-series image data of the vehicle in real time; The time-series image data is a plurality of frames of driving images; A central processing unit, configured to input the time series image data of the vehicle into a pre-designed image segmentation network, and output a lane area image; the image segmentation network comprises an encoder fused with a convolutional long short-term memory network layer, a PAM position attention module, a CAM channel attention module, and a decoder fused with a convolutional long short-term memory network layer; the lane area image is a lane image indicating a lane category and a non-lane category; Processing the lane area image using a binary mask to extract features of a drivable area for the vehicle; The drivable area features of the vehicle are input into a pre-designed time series classification module, and the steering angle prediction result of the vehicle is output.

9. The end-to-end steering angle prediction system focusing on the drivable area according to claim 8, characterized in that: The steering angle camera is arranged on the front fenders of the left and right front wheels of the vehicle; the central processor is arranged inside the vehicle; and the interior of the vehicle includes the central console or instrument panel of the vehicle.

10. The end-to-end steering angle prediction system focusing on the drivable area according to claim 9, characterized in that: Also included is a display, which is used to display the steering angle prediction result of the vehicle.

Images