Lane line category detection method and device based on feature fusion in intelligent driving scene, electronic equipment and readable storage medium

Abstract

The invention discloses a lane line category detection method and device based on feature fusion in an intelligent driving scene, electronic equipment and a readable storage medium, and the method comprises the following steps: extracting the advanced features of an image, fusing the multi-scale features of the image, detecting the spatial position of a lane line, and judging the type of the lane line; and finally, outputting the position and category information of the lane line. According to the feature fusion module CAASPP provided by the invention, the characterization capability of the network on the lane lines is enhanced by fusing features of different scales, the judgment capability of the classification network is optimized, and the identification capability on different lane line types is effectively enhanced, especially the virtual and real attributes of the lane lines are distinguished. In an actual driving scene, the proposed feature fusion module is helpful for an intelligent automobile to accurately identify various lane lines such as virtual lines and solid lines, and accurate identification can still be realized under complicated road surface conditions such as abrasion, roadblocks and shielding, so that reasonable lane change under a safe condition is ensured.

Description

Technical Field

[0001] The present invention relates to the field of lane line detection technology, and in particular to a lane line category detection method, device, electronic device and readable storage medium based on feature fusion in an intelligent driving scenario. Background Art

[0002] With the rapid development of intelligent driving technology, autonomous driving systems are gradually becoming a core component of future transportation systems. In autonomous driving systems, lane detection and classification are important components for vehicles to maintain correct driving trajectories and are of vital importance. The development and application of lane detection and classification technology can not only improve the driving safety and stability of vehicles, but also effectively reduce the occurrence of traffic accidents, improve traffic flow, and enhance traffic management efficiency.

[0003] At present, lane detection methods are mainly divided into traditional image processing methods and deep learning-based methods. Traditional image processing methods rely on image edge detection, color threshold segmentation, Hough transform and other technologies. These methods can achieve lane detection to a certain extent, but they are easily disturbed in complex road scenes (such as lighting changes, shadows, road wear, etc.), and the detection accuracy and robustness are poor.

[0004] Compared with traditional methods, lane detection methods based on deep learning show higher accuracy and robustness in complex environments. Such methods usually use convolutional neural networks (CNNs) to extract and classify features from images. Through a large amount of data training, the model can automatically learn and extract features from images. However, there are still some challenges in existing deep learning methods. For example, in multi-lane scenarios, how to accurately distinguish different categories of lane lines (such as solid lines, dotted lines, white lane lines, yellow lane lines, etc.) and how to maintain high detection rates and low false alarm rates under different weather and lighting conditions are still the difficulties and hot spots of current research.

[0005] In patent CN118366129A, the extracted high-resolution feature map is input into the path aggregation network through horizontal connection to obtain lane line features of different levels, and compared with the real lane line coordinates, and the various losses calculated are used to update the network parameters, and the model is continuously updated and iterated until convergence. Although this invention improves the detection accuracy and generalization ability of the model for lane lines, the use of non-maximum suppression to remove highly overlapping lane lines does not take into account the lane line discrimination in complex scenes. The detection rate is not high enough under bad weather and lighting conditions, and it is easy to make mistakes. Patent CN118397586A uses the current frame point cloud collected by the radar, the image data collected by the camera and the current frame vehicle posture data of the navigation system to determine the ground point cloud map. The combination of the camera detection of the lane line and the radar detection of the lane line improves the accuracy of the lane line detection, but the point cloud data is complex. In order to match it with the image collected by the camera, a large number of irrelevant points need to be eliminated. In addition, the multi-modal combination increases the difficulty and time of data processing, and real-time lane line detection cannot be achieved. In complex road scenes such as lighting changes, shadows, road wear, etc., a large number of irrelevant point clouds will seriously disrupt the recognition of the lane line point cloud, and the detection accuracy and robustness are poor. Summary of the invention

[0006] The purpose of the present invention is to provide a lane line category detection method, device, electronic device and readable storage medium based on feature fusion in an intelligent driving scenario, so as to solve the problems of low accuracy and poor robustness of lane line recognition and classification in the prior art under worn road surfaces, roadblocks, vehicle interference and shadow occlusion.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A lane line category detection method based on feature fusion in an intelligent driving scenario includes the following steps:

[0009] Obtain a road image containing lane lines, input the image into the ResNet feature extraction network, and extract high-level features of the road image;

[0010] Inputting the high-level features of the road image into a feature fusion module CA_ASPP that combines dilated convolution and coordinate attention mechanism, the feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales;

[0011] The lane line position is predicted through the anchor detection method, the spatial position of the lane line is accurately identified, and the coordinate position of the lane line is output. The lane line features are extracted and classified through a fully connected, batch normalized and ReLU activation function sequence, and a fully connected and ReLU activation function sequence. The softmax activation function is then used to generate the final classification probability. Different digital identifiers are output according to the obtained probability to represent the lane line category.

[0012] The lane line detection result is output based on the coordinate information and category information of the lane line to complete the lane line detection task.

[0013] Furthermore, the dilated convolution extends the ordinary convolution by introducing a parameter dilation rate to set each element in the convolution kernel.

[0014] Furthermore, when the high-level features of the road image are input into the feature fusion module CA_ASPP, the road image is first input into three main branches respectively: a 1×1 convolution branch, three atrous convolution branches with different expansion rates, and an average pooling branch.

[0015] Furthermore, the coordinate attention mechanism effectively integrates the spatial position information into the channel attention framework, splits the channel attention mechanism into two independent one-dimensional feature encoding processes, aggregates the features along different spatial dimensions, and realizes fine-grained processing of features.

[0016] Furthermore, the lane anchor detection method predicts the lane line position by dividing the original H×W image into grids to obtain a plurality of cells with position information, including the following steps:

[0017] First, according to the position of the lane line in the input image, it can be known that there is no lane line in the upper area, reducing the detection range;

[0018] Then, a background cell is added in the row direction to indicate that there is no lane line, resulting in a total of h×(w+1) cells;

[0019] Finally, after the row-anchor grid segmentation, each cell in the image contains a probability of a predicted lane line, which is calculated as follows:

[0020] P i,j =f ij (X) i∈[1,n],j∈[1,h]

[0021] Among them, P i,j represents the probability that each cell in the image contains a lane line, n represents the number of lane lines in the image, and h represents the number of row anchors.

[0022] A lane line category detection device based on feature fusion in an intelligent driving scenario, comprising:

[0023] The extraction module obtains the road image containing the lane lines, inputs the image into the ResNet feature extraction network, and extracts the high-level features of the road image;

[0024] A fusion module, which inputs the high-level features of the road image into a feature fusion module CA_ASPP that combines a dilated convolution and a coordinate attention mechanism. The feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales;

[0025] The lane detection module obtains image features through the anchor detection method, accurately identifies the spatial position of the lane, outputs the coordinate position of the lane, extracts lane features and classifies them through a fully connected, batch normalized and ReLU activation function sequence, and a fully connected and ReLU activation function sequence, and then uses the softmax activation function to generate the final classification probability, and outputs different digital identifiers to represent the lane category according to the obtained probability;

[0026] The output module outputs the lane line detection results based on the coordinate information and category information of the lane line to complete the lane line detection task.

[0027] An electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the nighttime target tracking method based on multi-dimensional spatial attention in the intelligent driving scenario as described above.

[0028] A computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the nighttime target tracking method based on multi-dimensional spatial attention in the intelligent driving scenario as described above.

[0029] Beneficial effects:

[0030] 1) The feature fusion module CA_ASPP proposed in the present invention enhances the network's ability to represent lane lines by fusing features of different scales, optimizes the judgment ability of the classification network, and effectively enhances the ability to recognize different types of lane lines, especially to distinguish the virtual and real attributes of lane lines. In actual driving scenarios, the proposed feature fusion module helps smart cars accurately identify virtual and real lines to ensure that they can change lanes reasonably under safe conditions.

[0031] 2) The parallel detection and classification dual-branch design proposed in the present invention not only ensures high detection accuracy, but also realizes efficient parallel processing of position and category information. This enables the output of lane line detection results containing rich category information, providing more detailed and accurate road information for intelligent driving vehicles. Compared with the baseline network, the improved network has stronger adaptability and robustness to complex environments, and has better detection effects in identifying and classifying lane lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0033] Figure 1 It is a flow chart of a lane line category detection method based on feature fusion in an intelligent driving scenario according to an embodiment of the present application;

[0034] Figure 2 This is an overall network structure diagram of a lane line category detection method based on feature fusion in an intelligent driving scenario provided according to an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of a feature fusion module CA_ASPP provided according to an embodiment of the present application;

[0036] Figure 4 This is a visualization diagram of the lane line category detection model provided according to an embodiment of the present application on the TuSimple dataset;

[0037] Figure 5 is an example diagram of a lane line category detection device based on feature fusion in an intelligent driving scenario according to an embodiment of the present application;

[0038] Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0040] Example

[0041] A lane line category detection method based on feature fusion in intelligent driving scenarios, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0042] S101, obtain a road image containing lane lines, input the image into the ResNet feature extraction network, and extract high-level features of the road image. Specifically, by introducing skip connections, the problem of difficult deep network training is solved, ensuring that the network can better learn complex features, improve the accuracy of lane line detection, and maintain high detection performance under different road scenes, lighting conditions and weather changes, and extract high-level features.

[0043] S102, inputting the high-level features of the road image into a feature fusion module CA_ASPP that combines dilated convolution and coordinate attention mechanism, wherein the feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales.

[0044] S103-1, predict the lane line position through the anchor detection method, accurately identify the spatial position of the lane line, and output the coordinate position of the lane line. S103-2, extract the lane line features and classify them through a full connection, batch normalization and ReLU activation function sequence, and a full connection and ReLU activation function sequence, and then use the softmax activation function to generate the final classification probability, and output different digital identifiers to represent the lane line category according to the obtained probability. The function sequence can extract more complex and abstract feature representations, and the introduction of batch normalization has a regularization effect to avoid model overfitting.

[0045] S104, outputting a lane line detection result based on the coordinate information and category information of the lane line, thereby completing the lane line detection task.

[0046] Among them, the dilated convolution in step S102 expands the ordinary convolution by introducing a parameter dilation rate to set each element in the convolution kernel. The feature fusion module CA_ASPP first uses dilated convolution to expand the receptive field of the convolution kernel to meet the needs of multi-scale spatial features in complex tasks. For example, in a feature map with an initial receptive field size of 1×1, after a 3×3 convolution operation, the receptive field is expanded to 3×3. If a dilated convolution with a dilation rate of 2 is applied next, the effect is equivalent to a 5×5 convolution kernel operation, which quickly expands the receptive field of the feature map to 7×7.

[0047] The coordinate attention mechanism effectively integrates the spatial position information into the channel attention framework, splits the channel attention mechanism into two independent one-dimensional feature encoding processes, aggregates the features along different spatial dimensions, and realizes fine-grained processing of features.

[0048] Specifically, the coordinate attention mechanism strengthens the network's ability to focus on lane line features by assigning higher weights to key pixels in classification decisions. For a given input feature map C×H×W, two spatial range pooling kernels (H, 1) and (1, W) are used to encode each channel along the horizontal coordinate and vertical coordinate, respectively.

[0049] The output expression of the nth channel at height h in the horizontal direction is as follows:

[0050]

[0051] The output expression of the nth channel in the vertical direction at a width of w is as follows:

[0052]

[0053] The feature maps are averaged in the X and Y directions to obtain C×H×1 and C×1×W feature maps, respectively. The two feature maps are then concatenated and reduced in dimension through a 1×1 convolution to obtain a new feature map f, which is expressed as follows:

[0054] f=δ(F1([z h ,z w ]))

[0055] Among them, δ represents the nonlinear activation function, F1 represents the 1×1 convolution operation, and [,] represents the splicing operation along the horizontal direction.

[0056] The feature map f is then processed by the batch normalization (BN) layer and the ReLU activation function. This process retains the nonlinear processing capability of the module and improves the stability of the training process. Then, the processed feature map f is split into feature vectors in two directions and each is dimensionally increased by a 1×1 convolution. Finally, the horizontal and vertical attention vectors g are obtained by the Sigmod function. h and g w , these two vectors are element-wise multiplied with the original feature map to obtain the final output as follows:

[0057] y(i,j)=x(i,j)×g h (i)×g w (j)

[0058] The high-level features of the road image are input into the feature fusion module CA_ASPP, such as Figure 3 As shown,

[0059] First, the road image is input into three main branches respectively: a 1×1 convolution branch, three hole convolution branches with different dilation rates, and an average pooling branch.

[0060] Among them, the 1×1 convolution branch is designed to retain the original information of the input feature map to ensure that the detailed features are not lost. The three parallel deployed hole convolution branches enable the module to capture image features at different scales. Each hole convolution adopts a different expansion rate to cover a wider range of contexts, thereby enriching the receptive field of the model and extracting richer spatial information. The average pooling branch focuses on extracting the global features of the image, including the overall layout and background information, which helps the model to parse the global content of the image.

[0061] Then, the feature maps output by the above branches are concatenated to obtain a comprehensive feature map containing multi-scale information.

[0062] Finally, the feature map is sent to the coordinate attention module, which adjusts the key areas in the feature map by weight to enhance the extraction of key information in the image.

[0063] Preferably, the lane line position prediction method of the line anchor detection method is to divide the original H×W image into grids to obtain a plurality of cells with position information, including the following steps:

[0064] First, according to the position of the lane lines in the input image, it can be known that there will be no lane lines in the upper area, reducing the detection range.

[0065] Then, a background cell is added in the row direction to indicate that there is no lane line, resulting in a total of h×(w+1) cells.

[0066] Finally, after the row-anchor grid segmentation, each cell in the image contains a probability of a predicted lane line, which is calculated as follows:

[0067] P i,j =f ij (X) i∈[1,n],j∈[1,h]

[0068] Among them, P i,j represents the probability that each cell in the image contains a lane line, n represents the number of lane lines in the image, and h represents the number of row anchors.

[0069] This embodiment also provides a lane line category detection device 10 based on feature fusion in an intelligent driving scenario. Figure 5 As shown, including:

[0070] The extraction module 100 obtains a road image including lane lines, inputs the image into a ResNet feature extraction network, and extracts high-level features of the road image.

[0071] The fusion module 200 inputs the high-level features of the road image into a feature fusion module CA_ASPP that combines the hole convolution and the coordinate attention mechanism. The feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales.

[0072] The lane line detection module 300 obtains image features through the line anchor detection method, accurately identifies the spatial position of the lane line, outputs the coordinate position of the lane line, and extracts and classifies the lane line features through a fully connected, batch normalized and ReLU activation function sequence, and a fully connected and ReLU activation function sequence, and then uses the softmax activation function to generate the final classification probability, and outputs different digital identifiers to represent the lane line category according to the obtained probability.

[0073] The output module 400 outputs the lane line detection result based on the coordinate information and category information of the lane line to complete the lane line detection task.

[0074] The present invention not only ensures high detection accuracy through the design of dual branches of detection and classification, but also realizes efficient parallel processing of position and category information, thereby outputting lane line detection results containing rich category information, providing more detailed and accurate road information for intelligent driving vehicles. The present invention also combines the feature fusion module CA_ASPP of void convolution and coordinate attention, which not only optimizes feature representation, but also improves the network's recognition ability for small targets such as lane lines.

[0075] The present invention is tested in six different road scenarios, including ordinary road surface, worn road surface, road barrier interference, vehicle occlusion, shadow section and multi-occluded road surface, such as Figure 4 As shown in the figure, the results show that the model can maintain accurate recognition and classification of lane lines when facing complex environmental interference, and is more adaptable and robust to complex environments.

[0076] An electronic device, such as Figure 6 As shown, it includes: a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor. The processor 602 executes the program to implement the lane line category detection method based on feature fusion in the intelligent driving scenario as provided in the above embodiment.

[0077] The electronic device further includes a communication interface 603 for communication between the memory 601 and the processor 602. The memory 601 is used to store a computer program that can be run on the processor 602. The memory 601 may include a high-speed RAM memory, and may also include a nonvolatile memory, such as at least one disk memory.

[0078] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0079] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a nighttime target tracking method based on multidimensional spatial attention in an intelligent driving scenario as provided in the above-mentioned embodiment. Wherein, the computer-readable medium can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) with one or N wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or processing in other suitable ways as necessary, and then stored in a computer memory.

[0080] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the accompanying drawings and above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A lane line category detection method based on feature fusion in an intelligent driving scenario, characterized in that: The following steps are involved: Obtain a road image containing lane lines, input the image into the ResNet feature extraction network, and extract high-level features of the road image; Inputting the high-level features of the road image into a feature fusion module CA_ASPP that combines dilated convolution and coordinate attention mechanism, the feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales; The lane line position is predicted through the anchor detection method, the spatial position of the lane line is accurately identified, and the coordinate position of the lane line is output. The lane line features are extracted and classified through a fully connected, batch normalized and ReLU activation function sequence, and a fully connected and ReLU activation function sequence. The softmax activation function is then used to generate the final classification probability. Different digital identifiers are output according to the obtained probability to represent the lane line category. The lane line detection result is output based on the coordinate information and category information of the lane line to complete the lane line detection task.

2. The lane line category detection method based on feature fusion in an intelligent driving scenario according to claim 1 is characterized in that: The dilated convolution expands the ordinary convolution by introducing a parameter dilation rate interval to set each element in the convolution kernel.

3. The lane line category detection method based on feature fusion in the intelligent driving scenario according to claim 2 is characterized in that: When the high-level features of the road image are input into the feature fusion module CA_ASPP, the road image is firstly input into three main branches respectively: a 1×1 convolution branch, three atrous convolution branches with different dilation rates, and an average pooling branch.

4. The lane line category detection method based on feature fusion in an intelligent driving scenario according to claim 1, characterized in that: The coordinate attention mechanism effectively integrates the spatial position information into the channel attention framework, splits the channel attention mechanism into two independent one-dimensional feature encoding processes, aggregates features along different spatial dimensions, and realizes fine-grained processing of features.

5. The lane line category detection method based on feature fusion in an intelligent driving scenario according to claim 1, characterized in that: The lane anchor detection method predicts the lane line position by dividing the original H×W image into grids to obtain multiple cells with position information, including the following steps: First, according to the position of the lane line in the input image, it can be known that there is no lane line in the upper area, reducing the detection range; Then, a background cell is added in the row direction to indicate that there is no lane line, resulting in a total of h×(w+1) cells; Finally, after the row-anchor grid segmentation, each cell in the image contains a probability of a predicted lane line, which is calculated as follows: P i,j =f ij (X) i∈[1,n],j∈[1,h] Among them, P i,j represents the probability that each cell in the image contains a lane line, n represents the number of lane lines in the image, and h represents the number of row anchors.

6. A lane line category detection device based on feature fusion in an intelligent driving scenario, characterized in that: include: The extraction module obtains the road image containing the lane lines, inputs the image into the ResNet feature extraction network, and extracts the high-level features of the road image; A fusion module, which inputs the high-level features of the road image into a feature fusion module CA_ASPP that combines a dilated convolution and a coordinate attention mechanism. The feature fusion module CA_ASPP enhances the network's ability to represent lane lines by fusing features of different scales; The lane detection module obtains image features through the anchor detection method, accurately identifies the spatial position of the lane, outputs the coordinate position of the lane, extracts lane features and classifies them through a fully connected, batch normalized and ReLU activation function sequence, and a fully connected and ReLU activation function sequence, and then uses the softmax activation function to generate the final classification probability, and outputs different digital identifiers to represent the lane category according to the obtained probability; The output module outputs the lane line detection results based on the coordinate information and category information of the lane line to complete the lane line detection task.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lane line category detection method based on feature fusion in an intelligent driving scenario as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the lane line category detection method based on feature fusion in an intelligent driving scenario as described in any one of claims 1 to 5.

Images