Improved lane line category recognition method of yolov5s network model

Abstract

The application relates to the technical field of assisted driving, in particular to a lane line type identification method of an improved YOLOv5s network model, which comprises the following steps: a vehicle-mounted camera is used to shoot a front road video containing lane lines, and lane line images are obtained; inverse perspective transformation is performed on the road images containing the lane lines, and a bird's eye view reflecting the real world is obtained; the data images are labeled, and the positions and categories of the targets to be detected in the images are obtained; a loss function in an original network is replaced, and a new activation function is introduced into a convolution layer; a network structure for model training is built; hyperparameters for training are set, and the model is optimized; inverse perspective transformation is performed on a front road video shot in real time by a vehicle, the trained model is placed into an assisted driving system, and lane line type identification is performed. The application reduces the parameter quantity of the model, accelerates the inference speed of the model, and guarantees the real-time performance and accuracy of the assisted driving system.

Description

Technical Field

[0001] This invention relates to the field of driver assistance technology, specifically to an improved method for lane line type recognition using a YOLOv5s network model. Background Technology

[0002] As people's living standards continue to improve, automobiles, as the most common means of transportation in modern society, have become an indispensable part of people's daily lives. The advent of automobiles has made our lives more agile, but traffic accidents have also occurred frequently, which not only greatly endangers people's lives and safety but also causes huge property losses and hinders social development.

[0003] A significant proportion of road traffic accidents are caused by driver error. To address these traffic problems, researchers have begun extensive studies on driver assistance systems (ADAS). ADAS utilizes advanced technologies to automatically perform some of the driver's tasks, significantly reducing driver workload and helping them avoid making incorrect operations and judgments.

[0004] Advanced driver assistance systems (ADAS) primarily utilize various types of onboard cameras to extract environmental information about the vehicle's vicinity. This information is uploaded to a cloud platform, where computer vision algorithms and technologies identify lane markings and potential obstacles to intelligently alert the driver and prompt appropriate actions. ADAS mainly consists of four modules: perception, decision-making, control, and execution. Lane marking recognition is a crucial perception module within ADAS, serving as a precursor to the subsequent three modules. This technology has significant application value in proactively reducing traffic accidents.

[0005] For example, a Chinese patent discloses a method for determining control parameters of a driving assistance system and / or an autonomous driving system (application number: 202211083416.5). This method includes the following steps: acquiring driving data of the vehicle and surrounding vehicles, wherein the driving data includes the attitude information of the vehicle and surrounding vehicles and the positional relationship between the vehicle and surrounding vehicles (S1); acquiring lane line information of the lane the vehicle is currently in (S2); and determining control parameters of the driving assistance system and / or autonomous driving system based on the driving data and the lane line information (S3). The disadvantage of this method is that, in terms of lane line recognition, most detection methods only detect the geometric model of the lane line, without identifying the type of lane line. These methods allow the car to track the detected lane lines and provide lane-keeping assistance, but if the driver wants to overtake or change lanes, these methods cannot identify the type of lane line, and the car will not be able to determine whether it is overtaking, changing lanes, or making a U-turn while adhering to traffic markings. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an improved method for lane line type identification using a YOLOv5s network model.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: an improved method for lane line type recognition of YOLOv5s network model, comprising the following steps: S01: using an on-board camera to capture a video of the road ahead containing lane lines, and performing frame segmentation on the video to obtain lane line images;

[0008] S02: Perform inverse perspective transformation on road images containing lane lines to obtain a bird's-eye view that reflects the real world, which serves as a dataset for lane line type recognition;

[0009] S03: Label the data images, obtain the location and category of the target to be detected in the image, and divide the dataset into training set, validation set and test set;

[0010] S04: Replace the original IOU loss function with the SIOU loss function and use the non-linear activation function FReLU in the convolutional layer to improve the robustness of recognition;

[0011] S05: Build a network structure for model training. Based on YOLOv5s, use the C3_Faster module based on the FasterNet lightweight model and introduce the ECA attention mechanism to improve detection accuracy.

[0012] S06: Set the hyperparameters for training, optimize the model, train the training set based on the improved YOLOv5s algorithm, and obtain the model file with the best convergence effect;

[0013] S07: Perform inverse perspective transformation on the real-time video of the road ahead captured by the vehicle, and use the trained model to input into the driver assistance system to identify lane line types.

[0014] Preferably, in step S01, a video of the lane containing road markings is acquired by the vehicle-mounted camera, the video is frame-trimmed, an average of one road image is captured every 20 frames, and the images are filtered to remove road images that do not contain lane markings, and road images with different weather conditions and different brightness levels are collected.

[0015] Preferably, in step S02, the captured road image is converted into a bird's-eye view road image that reflects the real world using the inverse perspective transformation method.

[0016] Preferably, in step S03, the dataset image for model training is obtained through inverse perspective transformation. The dataset image contains eight lane line types: white dashed lines, white solid lines, yellow dashed lines, yellow solid lines, white dashed and solid lines, double yellow solid lines, double yellow dashed lines, and yellow dashed and solid lines. Among them, white dashed and solid lines and yellow dashed and solid lines can be divided into left dashed and right solid lines and left solid and right dashed lines according to the driving position. The dataset is divided into training set, validation set, and test set in an 8:1:1 ratio. The training set samples are then subjected to data augmentation operations such as shifting, scaling, brightening, rotation, random cropping, and adding noise. The labelimg file is used to annotate the training set and validation set samples, and a .txt file is generated and stored in the data folder.

[0017] Preferably, in S04, the Cost functions of the SIOU loss function are defined as follows:

[0018]

[0019] in The coordinates of the center of the true bounding box. The coordinates of the center of the prediction box;

[0020] Distance cost: Δ = ∑ t＝x,y (1-e -γρt ) = 2 - e -γρx -e -γρy

[0021] in γ=2-Λ,C w C h The width and height of the minimum bounding rectangle between the ground truth bounding box and the predicted bounding box;

[0022] Shape cost:

[0023] in w and h are the width and height of the predicted bounding box, w gt h gt θ represents the width and height of the ground truth bounding box, and controls the degree of attention paid to shape loss. To avoid over-focusing on shape loss and reducing the movement of the predicted bounding box, the parameter range is set to [2,6].

[0024]

[0025] Finally, the SIOU loss function is defined as:

[0026]

[0027] Preferably, in S05, the C3 module in the original YOLOv5 network is replaced with the C3_Faster module based on the FasterNet lightweight model, and a FasterNet block stack is added to each C3 module. The FasterNet block contains a PConv layer, followed by two 1×1 convolutional layers, with a normalization function and an activation function placed between the two convolutional layers.

[0028] Preferably, the normalization function uses a BN layer; the activation function is the FRelU function.

[0029] Preferably, in S06, the loss rate is calculated for each iteration, and the loss rate is observed to reach its minimum until the model converges; the number of iterations is set to 300; and the batch size is set to 16.

[0030] The beneficial effects of this invention are: this invention can provide information on adjacent traffic markings for lane change assistance in assisted driving systems, and can identify the type of lane markings on the road ahead in real time, allowing the car to perform overtaking, lane changing and U-turn operations normally without violating traffic marking rules.

[0031] This invention utilizes an improved YOLOv5s algorithm for lane line type recognition training, achieving a final mAP value of 93.15% for the trained model. Compared to the original YOLOv5s algorithm, the mAP value is improved by 3.07%; it also reduces the number of model parameters, accelerates the model's inference speed, and ensures the real-time performance and accuracy of the assisted driving system. Attached Figure Description

[0032] Figure 1 This is a flowchart of the present invention;

[0033] Figure 2a This is the image before the inverse perspective transformation in this invention;

[0034] Figure 2b This is an image after inverse perspective transformation in this invention;

[0035] Figure 3 This is a data label diagram of lane line types in this invention;

[0036] Figure 4 This is a diagram illustrating the SIOU loss function calculation scheme in this invention;

[0037] Figure 5 This is a diagram of the improved YOLOv5 network structure in this invention;

[0038] Figure 6 This is a network structure diagram of the C3_Faster module in this invention;

[0039] Figure 7 This is a graph showing the test results of the trained model in this invention;

[0040] Figure 8 This is a graph showing the various parameters of the model in this invention. Detailed Implementation

[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0042] The following is combined Figure 1 The algorithm process and its principles are described.

[0043] An improved method for lane line type identification using a YOLOv5s network model includes the following steps:

[0044] S01: Use the vehicle-mounted camera to capture video of the road ahead containing lane lines, and perform frame-by-frame operations on the video to obtain lane line images.

[0045] The system uses an onboard camera to capture video of lane markings ahead. Frames are extracted from the video, averaging one image every 20 frames. These images are then filtered to remove those that do not contain lane markings. Road images with varying weather conditions and lighting levels are collected to enhance the model's generalization ability. Finally, 5804 images containing lane markings were selected after filtering.

[0046] S02: Perform inverse perspective transformation on road images containing lane lines to obtain a bird's-eye view that reflects the real world, which serves as a dataset for lane line type recognition.

[0047] To more effectively acquire the geometric features of lane lines and enhance recognition accuracy, this invention employs inverse perspective transformation to convert captured road images into bird's-eye view road images reflecting the real world. In perspective transformation, the transformation relationship between the image before and after perspective can be represented by a 3×3 transformation matrix. This perspective transformation matrix is ​​solved using four corresponding points in the data image, as shown in the following equation:

[0048]

[0049] Where map_matrix is ​​the 3×3 inverse perspective transformation matrix obtained from point pairs, (x i ,y i (x′) represents the coordinates of the input point. i ,y′ i The coordinates of the output points are shown below, where i takes values ​​of 0, 1, 2, and 3, representing the four corresponding points. This method is used to obtain a bird's-eye view of a road. It does not require a large amount of computation and is simple to operate. It only needs to process the ROI region containing lane lines, removing a large amount of interference information. The result of the inverse perspective transformation is shown below. Figure 2a and Figure 2b As shown.

[0050] S03: Label the data images to obtain the location and category of the target to be detected in the images, and divide the dataset into training set, validation set and test set.

[0051] Inverse perspective transformation yields the dataset images used for model training. The dataset images contain eight lane line types: white dashed lines, white solid lines, yellow dashed lines, yellow solid lines, white dashed and solid lines, double yellow solid lines, double yellow dashed lines, and yellow dashed and solid lines. White dashed and solid lines and yellow dashed and solid lines can be further categorized based on vehicle position: left dashed and right solid, and left solid and right dashed. The dataset types are labeled with names such as... Figure 3 As shown, s (single) represents a single line, d (double) represents a double line, w (white) represents white, y (yellow) represents yellow, f (full) represents a solid line, i (imaginary) represents a dashed line, l (left) represents left, and r (right) represents right.

[0052] The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. Data augmentation operations such as shifting, scaling, brightening, rotating, random cropping, and adding noise were performed on the training set samples. The labelimg tool was used to annotate the training and validation set samples, generating .txt files which were then saved in the data folder.

[0053] S04: Replace the original IOU loss function with the SIOU loss function and use the non-linear activation function FReLU in the convolutional layer to improve the robustness of recognition.

[0054] The YOLOv5 algorithm includes three loss functions: classification loss (cls_loss), localization loss (box_loss), and confidence loss (obj_loss). This invention defines the total loss rate as:

[0055] L total =0.3×cls_loss+0.3×box_loss+0.4×obj_loss

[0056] The localization loss of this invention replaces the original IOU loss function with the SIOU loss function. SIOU takes into account the vector angle between the desired regressions, redefines the penalty metric, and improves training speed and inference accuracy. The loss function consists of four cost functions: angle cost, distance cost, shape cost, and IOU cost.

[0057] A schematic diagram of the SIOU loss function calculation scheme is shown below. Figure 4 As shown. The definitions of each Cost function are as follows:

[0058]

[0059] in The coordinates of the center of the true bounding box. These are the coordinates of the center of the prediction box.

[0060] Distance cost: Δ = ∑ t＝x,y (1-e -γρt ) = 2 - e -γx -e -γρy

[0061] in γ=2-Λ,C w C h The width and height of the minimum bounding rectangle for the ground truth bounding box and the predicted bounding box.

[0062]

[0063] in w and h are the width and height of the predicted bounding box, w gt h gtθ represents the width and height of the ground truth bounding box. θ controls the degree of attention paid to shape loss. To avoid overemphasizing shape loss and reducing the movement of the predicted bounding box, the parameter range is set to [2,6].

[0064]

[0065] Finally, the SIOU loss function is defined as:

[0066]

[0067] This invention introduces the FReLU nonlinear activation function into the convolutional layer module. By using spatial conditions in the activation function, the FReLU function can extend ReLU to a visual parameter activation function with pixel-level modeling capabilities, extracting a fine spatial layout of an object and improving robustness in visual recognition tasks.

[0068] FReLU samples the same max(x) function as a simple nonlinear function. However, for the conditional part, FReLU extends it to a 2D condition that depends on the spatial context of each pixel. The function is as follows:

[0069] f(x c,i,j ) = max(x c,i,j ,Τ(x c,i,j ))

[0070] in This is represented as a funnel-shaped condition, which means that the network is allowed to generate spatial conditions for each pixel during nonlinear activation. This represents a window centered at position (i,j) in 2D, with c channels. This indicates shared parameters for the same channel within this window.

[0071] Leveraging the spatial condition T(x) and the properties of the max(x) function, spatial layout capabilities can be provided, making it easier to extract the spatial structure of the object. Previously, convolution and nonlinear activation functions were separate, with activation independent of spatial conditions; however, FReLU introduces spatial dependence while performing nonlinear activation, making activation dependent on spatial conditions, thus resolving the spatial insensitivity issue in activation functions. Furthermore, calculations show that the additional computational cost when using the FReLU function is infinitesimally small compared to the original computational cost, thus enhancing algorithm performance without causing a significant accumulation of computational overhead.

[0072] S05: Build the network structure for model training. Based on YOLOv5s, use the C3_Faster module based on the FasterNet lightweight model and introduce the ECA attention mechanism to improve detection accuracy.

[0073] The dataset is input into a deep learning neural network. This invention is based on an improved YOLOv5 network model, and the improved network model is as follows: Figure 5 As shown.

[0074] This invention replaces the C3 module in the original YOLOv5 network with the C3_Faster module based on the FasterNet lightweight model. For example... Figure 6 As shown, a FasterNet block stack is added to each C3 module. The FasterNet block contains a PConv layer, followed by two 1×1 convolutional layers. To maintain feature diversity and achieve lower latency, a normalization function and an activation function are placed between these two convolutional layers.

[0075] The PConv layer effectively reduces computational redundancy and memory access, applying only regular convolutions to a portion of the input channels for spatial feature extraction while keeping the remaining channels unchanged. For continuous or regular memory access, the first or last consecutive c... p Channels are calculated as a representation of the entire feature map. Without loss of generality, the input and output feature maps are assumed to have the same number of channels. The number of floating-point operations per second of PConv is calculated as follows. It is 1 / 16th the size of a regular convolutional layer; furthermore, PConv has a smaller memory access requirement, i.e. It is 1 / 4 the size of a regular convolutional layer.

[0076] The normalization function uses a Batch Normalization (BN) layer, which can be merged into its adjacent convolutional layers for faster inference; the activation function uses the FRelU function mentioned above, taking into account both runtime and recognition effectiveness.

[0077] Avoiding dimensionality reduction is crucial for learning channel attention, and achieving cross-channel interaction can significantly reduce model complexity while maintaining performance. This invention uses the ECA attention mechanism to implement a local cross-channel interaction strategy without dimensionality reduction through one-dimensional convolution, requiring only a small number of parameters to achieve performance improvement. Given the aggregated features [C,1,1] obtained through average pooling layers, the ECA attention mechanism generates channel weights by performing a one-dimensional convolution with a kernel size of k, as shown in the following equation:

[0078]

[0079] Here, C represents the number of channels. Through ablation experiments, γ was set to 2 and b was set to 1. These two parameters are used to change the ratio between the number of channels C and the kernel size k. When the number of channels is large, the kernel k will increase through C; conversely, when the number of channels is small, the kernel k will decrease through C. This can fully realize the interaction between the channels in the fusion part.

[0080] S06: Set the hyperparameters for training, optimize the model, train the training set based on the improved YOLOv5s algorithm, and obtain the model file with the best convergence effect.

[0081] Set hyperparameters before training. Since the lane lines are all long and thin, to avoid the influence of the pre-trained model on the training results, the pre-trained model will not be used; calculate the loss rate for each iteration and observe the loss rate until it reaches its minimum until the model converges; to prevent overfitting, set the number of iterations to 300; set the batch size to 16.

[0082] By improving the algorithm and training the training set, the model file with the best convergence performance was obtained. The model was then tested, and the results were as follows: Figure 7 As shown. The loss, recall, precision, and mAP values ​​for the training and validation sets are as follows: Figure 8 As shown.

[0083] S07: Perform inverse perspective transformation on the real-time video of the road ahead captured by the vehicle, and use the trained model to input into the driver assistance system to identify lane line types.

[0084] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An improved method for lane line type recognition using a YOLOv5s network model, characterized in that: Includes the following steps: S01: Use the vehicle-mounted camera to capture a video of the road ahead containing lane lines, and perform frame-by-frame operations on the video to obtain lane line images; S02: Perform inverse perspective transformation on road images containing lane lines to obtain a bird's-eye view that reflects the real world, which serves as a dataset for lane line type recognition; S03: Label the data images, obtain the location and category of the target to be detected in the image, and divide the dataset into training set, validation set and test set; S04: Replace the original IOU loss function with the SIOU loss function and use the non-linear activation function FReLU in the convolutional layer to improve the robustness of recognition; S05: Build a network structure for model training. Based on YOLOv5s, use the C3_Faster module based on the FasterNet lightweight model and introduce the ECA attention mechanism to improve detection accuracy. S06: Set the hyperparameters for training, optimize the model, train the training set based on the improved YOLOv5s algorithm, and obtain the model file with the best convergence effect; S07: Perform inverse perspective transformation on the real-time video of the road ahead captured by the vehicle, and use the trained model to input into the driver assistance system to identify lane line types. In S05, the C3 module in the original YOLOv5 network is replaced with the C3_Faster module based on the FasterNet lightweight model. A FasterNet block stack is added to each C3 module. The FasterNet block contains a PConv layer, followed by two 1×1 convolutional layers. The normalization function and activation function are placed between these two convolutional layers.

2. The lane line type recognition method of the improved YOLOv5s network model according to claim 1, characterized in that: In S01, a video of the lane containing road markings is acquired through an onboard camera. The video is then framed, with an average of one road image captured every 20 frames. The images are then filtered to remove those that do not contain lane markings, and road images with different weather conditions and different levels of brightness are collected.

3. The lane line type identification method of the improved YOLOv5s network model according to claim 1, characterized in that: In S02, the inverse perspective transformation method is used to convert the captured road images into bird's-eye view road images that reflect the real world.

4. The lane line type identification method of the improved YOLOv5s network model according to claim 1, characterized in that: In S03, inverse perspective transformation is used to obtain the dataset image for model training. The dataset image contains eight types of lane lines: white dashed lines, white solid lines, yellow dashed lines, yellow solid lines, white dashed and solid lines, double yellow solid lines, double yellow dashed lines, and yellow dashed and solid lines. Among them, white dashed and solid lines and yellow dashed and solid lines can be divided into left dashed and right solid lines and left solid and right dashed lines according to the driving position. The dataset is divided into training set, validation set and test set in an 8:1:1 ratio. Data augmentation operations such as shifting, scaling, brightening, rotating, random cropping and adding noise are performed on the training set samples. The labelimg is used to label the training set and validation set samples, and the generated .txt file is saved in the data folder.

5. The lane line type identification method of the improved YOLOv5s network model according to claim 1, characterized in that: In S04, the Cost functions of the SIOU loss function are defined as follows: Angle cost: ; in , , The coordinates of the center of the true bounding box. The coordinates of the center of the prediction box; Distance cost: ; in , , The width and height of the minimum bounding rectangle between the ground truth bounding box and the predicted bounding box; Shape cost: ; in , w , h To predict the width and height of the bounding box, , The width and height of the actual bounding box. To control the degree of attention given to shape loss, and to avoid reducing the movement of the prediction box due to excessive focus on shape loss, the parameter range is set to [2,6]. IOU cost: ; Finally, the SIOU loss function is defined as: 。 6. The lane line type identification method of the improved YOLOv5s network model according to claim 1, characterized in that: The normalization function uses a BN layer; the activation function is the FRelU function.

7. The improved lane line type recognition method for the YOLOv5s network model according to claim 1, characterized in that: In S06, the loss rate is calculated for each iteration, and the loss rate is observed to reach its minimum until the model converges; the number of iterations is set to 300; the batch size is set to 16.

Images