A lane line detection method, system, device and medium for a foggy scene

Abstract

The application relates to the technical field of automatic driving, and discloses a lane line detection method, system, device and medium for a foggy scene. The application constructs a foggy scene dataset based on a public dataset and actual foggy scene collection data, uses bilinear interpolation to perform down-sampling processing on image data in the foggy scene dataset, enhances the down-sampled image data, obtains data enhancement images, uses a backbone network and a global feature fusion module to extract features of the data enhancement images, fuses information between images with different resolutions based on a feature pyramid, creates a lane line detection task based on the data after feature extraction, calculates the total loss of the lane line detection task, and predicts the position of lane lines in an image based on the lane line detection task and a given input image. The application improves the accuracy of feature extraction, reduces the amount of calculation, improves the real-time performance of detection, and can be effectively deployed on a detection device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a lane line detection method, system, device and medium for foggy scenes. BACKGROUND

[0002] Lane line detection is a crucial task in road safety and autonomous driving technology. It helps vehicles stay within the correct lane, ensures traffic order, and reduces the occurrence of traffic accidents by identifying and tracking lane lines on the road, providing necessary positioning and navigation information for autonomous driving systems.

[0003] The previous lane line detection methods can be divided into two categories: one is the traditional method based on computer vision, and the other is the method based on deep learning. The traditional computer vision method usually uses hand-set operators to extract features first, and then uses post-processing methods such as Hough transform and random sample consensus to fit straight lines. However, the robustness of this traditional method is insufficient when dealing with the diversity of lane lines in different scenes, while the lane line detection method based on deep learning can make up for this deficiency and has good generalization, but its recognition ability is still unstable when facing complex spatial scenes such as fog, therefore, it is necessary to design a more accurate lane line detection method for foggy scenes. SUMMARY

[0004] The purpose of the present application is to solve the problem of insufficient recognition accuracy of road information when the existing lane line detection method faces complex foggy scenes.

[0005] To achieve the above purpose, in a first aspect, the present application provides a lane line detection method for foggy scenes, comprising:

[0006] Constructing a foggy scene dataset based on public datasets and actual foggy data and preprocessing the foggy scene dataset;

[0007] Constructing a lane line detection model based on image data in the foggy scene dataset; the lane line detection model includes a backbone network, a neck network, an instance detection structure, an instance segmentation structure, and an edge enhancement module;

[0008] Extracting features of images in the foggy scene dataset through the backbone network to obtain image features; fusing image features of different resolutions through the neck network to obtain fused features;

[0009] The instance detection structure is used to predict a set of dynamic convolution kernels for each position through the fusion feature, to obtain a predicted heat map of lane line starting points, and to calculate a first prediction loss between the predicted heat map and a true value heat map; the road region is predicted through the dynamic convolution kernel, the road region is divided into a row-based segmentation map using the instance segmentation structure, the horizontal coordinate value of the lane line in each row is predicted, a lane line prediction map based on row segmentation is obtained, and a second prediction loss of row-based segmentation prediction is calculated; the vertical range of the lane line is predicted according to whether the current row is passed through through the lane line prediction map based on row segmentation, and a third prediction loss of lane line vertical range prediction is calculated; the offset of the coordinates predicted by the row-based segmentation map is predicted, and a fourth prediction loss of the offset prediction is calculated; the edge of the lane line is predicted through the edge enhancement module by the edge segmentation binary classification task, and a fifth prediction loss of the lane line edge prediction is calculated;

[0010] A total loss function of the lane line detection model is calculated according to the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss and the fifth prediction loss, and the lane line detection model is trained according to the total loss function;

[0011] Using the trained lane line detection model, the lane line position in the image is identified through a given input image.

[0012] Further, the foggy scene dataset is constructed based on the public dataset and the actual foggy data collected, and the foggy scene dataset is preprocessed, including:

[0013] The foggy scene dataset includes a modeling foggy dataset and an actual foggy dataset;

[0014] The modeling foggy dataset is generated by using the public dataset CULane and the following data modeling formula:

[0015]

[0016] Wherein, is a foggy image, is a clear image without fog, is an atmospheric light value at infinity, is a transmittance, which is inversely proportional to the distance, is an attenuation coefficient, which depends on the concentration, size, type and distribution of particulate matter in the air, is a distance variable;

[0017] The lane line image data of the actual foggy scene is collected to construct an actual foggy dataset;

[0018] The image data in the foggy scene data set is down-sampled by bilinear interpolation, and the image size in the foggy scene data set is converted into a specific pixel;

[0019] The converted image is rotated, horizontally flipped and brightness transformed to obtain a data enhanced image.

[0020] Further, the first prediction loss is:

[0021]

[0022] wherein, is a position on a data image, GT is a true value heat map, is a coordinate of a target point in a data image, control the decay rate of the Gaussian function is the value of the predicted heat map at the position, is the number of pixel points in the heat map, and alpha and beta are exponential weights;

[0023] The second prediction loss is:

[0024]

[0025] wherein, is the number of effective rows, is the vertical range of the lane marking, is the true horizontal coordinate, is the x coordinate of each row of lane lines;

[0026] The third prediction loss is:

[0027]

[0028] wherein, represents the first row, is the probability of the predicted first behavior being positive, is the ground truth.

[0029] The fourth prediction loss is:

[0030]

[0031] wherein, is the area near the lane line with a fixed width, is the number of pixels in , is the number of pixels in , and are the predicted offset and the label offset on the coordinate, respectively; ​

[0032] The fifth prediction loss is:

[0033]

[0034] wherein, is a label of a pixel, is a probability of being positive.

[0035] Further, the instance detection structure comprises a first branch and a second branch;

[0036] The first branch is used for predicting a lane line starting point;

[0037] The second branch is used for predicting a set of dynamic convolution kernels for each position, and taking the dynamic convolution kernel of the predicted position of the lane line starting point to do soft labeling on the lane line starting point true value.

[0038] Further, the total loss function is obtained by weighting and fusing the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss and the fifth prediction loss, and the total loss function is expressed as:

[0039]

[0040] wherein, is a weighting coefficient corresponding to each lane line detection task.

[0041] Further, the information between images of different resolutions is fused through the neck network to obtain fusion features, comprising:

[0042] Through the multi-scale fusion network based on the feature pyramid, the feature maps of multiple stages output by the backbone network are fused through a multi-resolution module to obtain the fusion features.

[0043] Further, the lane line detection model is used to predict the lane line position in the image by giving an input image, comprising:

[0044] Using a given input image , the coordinates on the lane line are predicted as:

[0045]

[0046] wherein, is the i-th lane line, is the i-th lane line, is the i-th lane line, is the i-th lane line,

[0047] The coordinates on the lane line are summarized to obtain the predicted position of the lane line in the image: ​

[0048] .

[0049] In a second aspect, the present application provides a lane line detection system for a foggy scene, which is used to implement the lane line detection method for a foggy scene described above, and the system comprises:

[0050] a data processing module, configured to construct a foggy scene dataset based on a public dataset and actual foggy scene collection data, and to preprocess the foggy scene dataset.

[0051] a model construction module, configured to construct a lane line detection model based on image data in the foggy scene dataset; the lane line detection model comprises a backbone network, a neck network, an instance detection structure, an instance segmentation structure, and an edge enhancement module.

[0052] a feature extraction module, configured to perform feature extraction on images in the foggy scene dataset by the backbone network to obtain image features, and to fuse image features of different resolutions by the neck network to obtain fused features.

[0053] a loss calculation module, configured to use the instance detection structure to predict a set of dynamic convolution kernels for each position through the fused features to obtain a predicted heat map of lane line starting points, and to calculate a first prediction loss between the predicted heat map and a true value heat map; to predict a road region through the dynamic convolution kernels, to divide the road region into a row-based segmentation map using the instance segmentation structure, to predict horizontal coordinate values of lane lines in each row to obtain a lane line prediction map based on row segmentation, and to calculate a second prediction loss of row-based segmentation prediction; to predict a vertical range of the lane line according to whether the current row is passed through based on the lane line prediction map based on row segmentation, and to calculate a third prediction loss of lane line vertical range prediction; to make a prediction of an offset amount for the coordinates predicted based on the row-based segmentation map, and to calculate a fourth prediction loss of the offset prediction; to use the edge enhancement module to predict edges of the lane line through a binary classification task of edge segmentation, and to calculate a fifth prediction loss of lane line edge prediction.

[0054] a model training module, configured to calculate a total loss function of the lane line detection model according to the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss, and the fifth prediction loss, and to train the lane line detection model according to the total loss function.

[0055] a lane line detection module, configured to use the trained lane line detection model to identify lane line positions in an image through a given input image.

[0056] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the lane line detection method for foggy scenes as described above when executing the computer program.

[0057] In a fourth aspect, the present application provides a computer readable storage medium, comprising a stored computer program; wherein the computer program, when executed, controls a device in which the computer readable storage medium is located to perform the lane line detection method for foggy scenes as described above.

[0058] The lane line detection method for foggy scenes, system, device and medium of the present application have the beneficial effects that, compared with the prior art, the global feature fusion network is designed to learn the correlation between different inputs, improving the accuracy of feature extraction; the input image is down-sampled in data processing, effectively reducing the amount of calculation and improving the real-time performance of detection; and the detection uses fewer calculation parameters, which can be effectively deployed on a detection device. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 A step flowchart of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0060] Figure 2 A network overall architecture diagram of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0061] Figure 3 A main network diagram of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0062] Figure 4 A neck network structure diagram of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0063] Figure 5 A detection head structure diagram based on dynamic convolution of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0064] Figure 6 A convolution kernel feature fusion module schematic diagram of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0065] Figure 7 A low-layer edge enhancement module schematic diagram of the lane line detection method for foggy scenes provided by the embodiment of the present application;

[0066] Figure 8A structural block diagram of a lane line detection system for a foggy scene provided by an embodiment of the present application is shown in the figure.

[0067] Figure 9 A structural diagram of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0068] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0069] It should be noted that the step numbers in the text are only for the convenience of explaining the specific embodiments, and do not serve as a limitation on the execution sequence of the steps. The method provided by the present embodiment can be executed by a related server, and the server is taken as an example for execution in the following description.

[0070] As shown in the figure, Figure 1 The lane line detection method for a foggy scene provided by an embodiment of the present application includes steps S11 to S16:

[0071] Step S11, constructing a foggy scene dataset based on a public dataset and actual foggy collection data and preprocessing the foggy scene dataset.

[0072] Using the public dataset CULane, the modeling foggy dataset SimFoggyCULane is generated by the following data modeling formula:

[0073]

[0074] Among them, is a foggy image, is a clear image without fog, is the atmospheric light value at infinity, is the transmittance, is the attenuation coefficient, which is closely related to the concentration, size, type and distribution of particulate matter in the air, represents the distance, and the transmittance is smaller when the distance is larger.

[0075] Collect lane line image data of actual foggy scenes to construct an actual foggy dataset.

[0076] Specifically, the foggy lane data collected on October 26, 2023, and December 31, 2023 in Xiangcheng District, Suzhou, and on December 30 and 31, 2023 in Shawan City, Xinjiang Uygur Autonomous Region, is used to construct the actual foggy dataset FoggyLane.

[0077] It should be noted that the present application can also use other time and place collected actual fog day scene lane line image data for constructing the actual fog day data set, and the above data collection scene does not limit the data collection range of the present application.

[0078] Under complex weather conditions, the image clarity and contrast of the camera are reduced, color distortion and image detail feature loss occur, and the like, thereby affecting the accuracy of lane line detection, and for data-driven deep learning, the number of training samples and the diversity of the environment directly affect the detection performance of the trained model on the lane line, therefore, the present application uses the new lane line data set SimFoggyCULane and FoggyLane containing foggy images to improve the lane line recognition precision and accuracy under complex weather conditions, and the lane line recognition rate of the SimFoggyCULane data set under thin fog, moderate fog and heavy fog weather conditions is significantly improved compared with the CULane data set, and the lane line recognition effect on the FoggyLane data set is still significantly better than that of the current advanced lane line detection method.

[0079] The image data in the foggy scene data set is down-sampled using bilinear interpolation, and the down-sampled image data is enhanced to obtain data enhanced images.

[0080] Since the image size of the SimFoggyCULane and FoggyLane data sets is 1640x590, if direct processing is performed, the data calculation amount is too large, the present application uses bilinear interpolation to down-sample all image data in the foggy scene data set SimFoggyCULane and FoggyLane, and converts the data in SimFoggyCULane and FoggyLane to 800x320 pixels.

[0081] After down-sampling, the converted image is rotated, horizontally flipped and brightness transformed to obtain data enhanced images.

[0082] Within the range of image clarity, the image is down-sampled, which can effectively reduce the calculation amount and the calculation cost, the smaller image data can be calculated faster, and the real-time performance of lane line recognition is improved.

[0083] Step S12, constructing a lane line detection model based on the image data in the foggy scene data set, the lane line detection model comprising a backbone network, a neck network, an instance detection structure, an instance segmentation structure and an edge enhancement module.

[0084] As Figure 2As shown, the embodiment designs a backbone network containing a global feature fusion module, specifically, the window attention mechanism of the last stage of the Swin Transformer is replaced by a general multi-head attention mechanism to learn the correlation between all inputs.

[0085] At the same time, in order to capture feature information of different scales, a multi-scale fusion neck network is introduced to fuse feature information of different scales.

[0086] An instance detection structure and an instance segmentation structure are designed, and a convolution kernel feature fusion module is designed in the instance detection structure to enable the network to automatically learn the relationship between the predicted lane line instances.

[0087] At the same time, in order to make up for the lack of edge information, a low-level edge enhancement module is introduced to enhance the learning of lane line edge features.

[0088] Step S13, the image in the fog scene data set is extracted by the backbone network to obtain the image feature; the image features of different resolutions are fused by the neck network to obtain the fused features.

[0089] As Figure 3 shown, in order to learn the correlation between the features of all input images, the embodiment of the application designs a backbone network containing a global feature fusion module (Global Feature Fusion Module, GFFM) to extract features from the data enhanced images obtained in step S11, specifically, the window attention mechanism of the last stage of the backbone network Swin Transformer is replaced by a general multi-head attention mechanism to better learn the correlation between different inputs.

[0090] Based on the feature pyramid as Figure 4 shown, the information between the features extracted from different data images is fused.

[0091] The feature pyramid has the characteristics of different resolutions at different scales, and different size targets can have appropriate feature representations at corresponding scales. By fusing multi-scale information, the multi-scale change problem in object detection can be well handled with very small amount of calculation.

[0092] Step S14, a set of dynamic convolution kernels is predicted for each position by fusing features using an instance detection structure, a predicted heat map of lane line starting points is obtained, a first prediction loss between the predicted heat map and a true value heat map is calculated, a road region is predicted by the dynamic convolution kernel, the road region is divided into a row-based segmentation map using an instance segmentation structure, a horizontal coordinate value of the lane line in each row is predicted, a lane line prediction map based on row segmentation is obtained, and a second prediction loss of the row-based segmentation prediction is calculated, a vertical range of the lane line is predicted according to whether the current row is passed through based on the lane line prediction map based on row segmentation, and a third prediction loss of the lane line vertical range prediction is calculated, a prediction of an offset amount is made for the coordinates predicted based on the row-based segmentation map, and a fourth prediction loss of the offset prediction is calculated, an edge of the lane line is predicted by an edge enhancement module through a binary classification task of edge segmentation, and a fifth prediction loss of the lane line edge prediction is calculated.

[0093] Specifically, based on the data after feature extraction, a lane line detection task is created, and a total loss of the lane line detection task is calculated, including the following steps:

[0094] Step S141, a lane line heat map prediction is performed using a dynamic convolution-based instance detection structure, and a prediction loss is calculated.

[0095] As shown in Figure 5 , the embodiment of the present application introduces a multi-scale fusion neck network, specifically, first, the feature maps of the four stages output by the backbone network are fused through a multi-resolution module to capture feature information of different scales. Then, the low-resolution feature map is upsampled and spliced with the highest-resolution feature map. The high-resolution feature map that fuses all scale information is downsampled by a pooling layer to obtain feature maps of different scales. Finally, each feature map is output by a convolution layer.

[0096] The instance detection structure has two branches: one branch performs lane line starting point prediction to obtain a heat map of lane line starting points; the other branch predicts a set of dynamic convolution kernels for each position, and the dynamic convolution kernel corresponding to the starting position of the lane line is responsible for subsequent segmentation.

[0097] In addition, the lane line instances in a graph have certain relevance in shape, structure and position, that is, they are usually parallel lines in the top view, similar in shape and fixed in spacing. In order to make full use of these special structural information, as shown in Figure 6 , the embodiment of the present application designs a kernel feature fusion module (KFFM) to let the network automatically learn the relationship between the predicted lane line instances.

[0098] At the same time, as shown in The function "softly labels" the true value, which makes the network converge more easily.

[0099] The true value heat map GT is The value of a position is represented as:

[0100]

[0101] in, is the position on the low-resolution image, is the coordinate of the target point in the data image, Controls the Gaussian sum function decay speed.

[0102] Use FocalLoss to calculate the loss of the predicted heat map and the true value heat map:

[0103]

[0104] in, To predict the heat map The value of the position, is the number of pixels in the heat map, α and β are exponential weights. Specifically, in this embodiment, α=2 and β=4.

[0105] In step S142 , a row-based segmentation map is predicted using conditional convolution through the instance segmentation structure and its prediction loss is calculated.

[0106] Specifically, we first use the dynamic convolution kernel predicted by the instance detection structure to obtain a feature map with the maximum resolution output by the neck network. For each row, we can get the probability of the lane line appearing on each grid:

[0107]

[0108] in Representative OK, It is the location map The eigenvectors of the rows, It is Rows correspond to probability vectors.

[0109] The lane line coordinates (x coordinates) of each row can be calculated using a weighted summation method:

[0110]

[0111] in, For all possible values ​​of the horizontal coordinate of this row, is the probability value corresponding to each coordinate.

[0112] Therefore, the prediction loss function of the row-based segmentation map is:

[0113]

[0114] in, is the number of valid rows, is the vertical extent of the lane marking, is the true horizontal axis.

[0115] In step S143 , conditional convolution is used to predict the vertical vector of a lane line range through the instance segmentation structure and calculate its prediction loss.

[0116] The basis for determining the vertical range of the lane line is whether the lane line passes through the current line. In this embodiment of the present invention, a The vector represents it. If the lane line passes through the If the line is passed, its value is 1, if not passed, it is 0.

[0117] Use the cross entropy function to optimize the vertical range of the lane line and calculate its prediction loss:

[0118]

[0119] Among them, the prediction loss of the vertical range of the lane line is:

[0120]

[0121] in, Representative OK, For the predicted The probability of a positive behavior, is the ground truth.

[0122] Step S144 : predicting the lane line offset loss based on the calculation of the row-based segmentation map.

[0123] Since the predicted lane line position in step S142 is a grid point rather than an exact coordinate value, its offset loss needs to be calculated. The offset loss is expressed as:

[0124]

[0125] in, is the area near the lane line of fixed width, yes The number of pixels in and The predicted offset and coordinates are The label offset.

[0126] Step S145 , using a cross entropy loss function to calculate the computational loss of the lane edge position.

[0127] For lane line detection tasks, lane line edge detection is crucial, such as Figure 7 As shown in FIG, for the edge enhancement task, an embodiment of the present invention introduces a low-level edge enhancement module (LEEM) to add an additional auxiliary path to the network to enable it to learn the edge features of the input.

[0128] The input image is edge detected using the Canny operator, and missing lane edges are added using lane label information. Lane edge segmentation is treated as a binary classification task, where edges are labeled as 1 and non-edges are labeled as 0.

[0129] At the same time, in order not to affect the extraction of high-dimensional semantic information and subsequent detection and segmentation, the low-level network is enhanced to learn low-dimensional features such as edge texture.

[0130] The cross entropy loss function is used to optimize the edge detection of the lane line, and the prediction loss of the lane line edge position is obtained as:

[0131]

[0132] in, Pixels The label at The probability of predicting positive for this label.

[0133] Step S15: Calculate the total loss function of the lane detection model based on the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss, and the fifth prediction loss, and train the lane detection model based on the total loss function.

[0134] In order to balance the impact of different tasks on the overall network training, it is necessary to perform weighted fusion based on the different effects of different detection tasks on the lane line detection task. The total loss function of the lane line detection task is:

[0135]

[0136] in, is the weighting coefficient corresponding to each lane line detection task.

[0137] Step S16: Based on the lane line detection task, the lane line position in the image is predicted by a given input image.

[0138] After all the lane line detection tasks are completed, the lane line prediction position in the image can be obtained through the given input image under the foggy scene , the coordinates on the lane line are predicted :

[0139]

[0140] wherein, is the th lane line, is the th lane line maximum sample point number.

[0141] Then, the set of all predicted lane line coordinates is summarized, and the lane line prediction position in the image can be obtained:

[0142] .

[0143] The lane line detection method for the foggy scene provided by the present application learns the correlation between different inputs through the design of the global feature fusion network, improves the accuracy of feature extraction, performs down-sampling processing on the input image in data processing, effectively reduces the calculation amount, improves the real-time performance of detection, uses fewer calculation parameters in detection, can be effectively deployed on detection equipment, and introduces the bottom and edge enhancement module to make up for the lack of edge information in the foggy day.

[0144] The embodiment of the present application also provides a lane line detection system for the foggy scene, which is used for executing the lane line detection method for the foggy scene as described above, Figure 8 is the system structure block diagram of the embodiment of the present application, and the system comprises:

[0145] The data processing module 21 is used for constructing a foggy scene data set based on a public data set and actual foggy day collection data and pre-processing the foggy scene data set;

[0146] The model construction module 22 is used for constructing a lane line detection model based on image data in the foggy scene data set; the lane line detection model comprises a backbone network, a neck network, an instance detection structure, an instance segmentation structure and an edge enhancement module;

[0147] The feature extraction module 23 is used for extracting features of the image in the foggy scene data set through the backbone network to obtain image features; and the image features of different resolutions are fused through the neck network to obtain fused features;

[0148] The loss calculation module 24 is configured to use the instance detection structure to predict a set of dynamic convolution kernels for each position through the fusion feature, obtain a predicted heat map of lane line starting points, and calculate a first prediction loss between the predicted heat map and a true value heat map; predict a road region through the dynamic convolution kernel, divide the road region into a row-based segmentation map using the instance segmentation structure, predict a horizontal coordinate value of the lane line in each row to obtain a lane line prediction map based on row segmentation, and calculate a second prediction loss of row-based segmentation prediction; predict a vertical range of the lane line according to whether the current row is passed through based on the lane line prediction map based on row segmentation, and calculate a third prediction loss of lane line vertical range prediction; make a prediction of an offset amount for the coordinates predicted based on the row-based segmentation map, and calculate a fourth prediction loss of the offset prediction; use the edge enhancement module to predict the edge of the lane line through an edge segmentation binary classification task, and calculate a fifth prediction loss of lane line edge prediction.

[0149] The model training module 25 is configured to calculate a total loss function of the lane line detection model according to the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss and the fifth prediction loss, and train the lane line detection model according to the total loss function.

[0150] The lane line detection module 26 is configured to use the trained lane line detection model to identify lane line positions in an image through a given input image.

[0151] The technical features and technical effects of the system provided in the embodiments of the present application are the same as those of the method provided in the embodiments of the present application, and are not repeated here. Each module in the above system can be realized by software, hardware and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0152] The embodiments of the present application also provide a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the lane line detection method for foggy scenes as described above when running.

[0153] The embodiments of the present application also provide a computer device, Figure 9 A preferred embodiment of a computer device provided by the present application is shown in a structural block diagram, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the lane line detection method for foggy scenes as described above when executing the computer program.

[0154] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, …), which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0155] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor. The processor is the control center of the computer device, and connects various parts of the computer device through various interfaces and lines.

[0156] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, and can also be a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., or the memory can also be other volatile solid-state storage devices.

[0157] It should be noted that the above computer device can include, but is not limited to, the processor, the memory, and those skilled in the art can understand that, Figure 9 The structural block diagram is only an example of the computer device, and does not constitute a limitation on the computer device, and can include more or fewer components than the diagram, or combine certain components, or different components.

[0158] To sum up, the lane line detection method, system, device and medium for a foggy scene provided by the embodiment of the application learn the correlation between different inputs by designing a global feature fusion network, improve the accuracy of feature extraction, perform down-sampling processing on the input image in data processing, effectively reduce the amount of calculation, and improve the real-time performance of detection, use fewer calculation parameters in detection, can be effectively deployed on a detection device, and introduce a bottom and edge enhancement module to make up for the lack of edge information in a foggy day.

[0159] The above merely describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and replacements without departing from the counting principle of the present application, and these improvements and replacements should also be considered as the protection scope of the present application.

Claims

1. A lane line detection method for foggy scenes, characterized in that: The method comprises: Constructing a foggy scene dataset based on a public dataset and actual foggy day collected data and preprocessing the foggy scene dataset; Constructing a lane detection model based on the image data in the foggy scene dataset; the lane detection model includes a backbone network, a neck network, an instance detection structure, an instance segmentation structure and an edge enhancement module; The backbone network includes a global feature fusion module. The backbone network replaces the window attention mechanism of the last stage of SwinTransformer with a common multi-head attention mechanism. The backbone network extracts features of images in the foggy scene dataset to obtain image features. The neck network fuses image features of different resolutions to obtain fused features. The instance detection structure includes a convolution kernel feature fusion module for automatically learning the relationship between predicted lane line instances; using the instance detection structure to predict a set of dynamic convolution kernels for each position through the fusion features, obtain a predicted heat map of the lane line starting point, and calculate the first prediction loss between the predicted heat map and the true value heat map; using the dynamic convolution kernel to predict the road area, using the instance segmentation structure to divide the road area into a row-based segmentation map, predict the horizontal coordinate value of the lane line in each row, obtain a row-based segmentation lane line prediction map, and calculate the second prediction loss of the row-based segmentation prediction; using the row-based segmentation lane line prediction map, predict the vertical range of the lane line according to whether it passes through the current row, and calculate the third prediction loss of the lane line vertical range prediction; make an offset prediction for the coordinates predicted by the row-based segmentation map, and calculate the fourth prediction loss of the offset prediction; using the edge enhancement module through the binary classification task of edge segmentation, predict the edge of the lane line, and calculate the fifth prediction loss of the lane line edge prediction; Calculating a total loss function of a lane detection model according to the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss, and the fifth prediction loss, and training the lane detection model according to the total loss function; The trained lane line detection model is used to identify the lane line positions in a given input image.

2. The lane line detection method for foggy scenes according to claim 1, characterized in that: The step of constructing a foggy scene dataset based on a public dataset and actual foggy day collected data and preprocessing the foggy scene dataset includes: The foggy scene dataset includes a modeled foggy dataset and an actual foggy dataset; Using the public dataset CULane, the modeled foggy dataset is generated using the following data modeling formula: in, For foggy images, For clear, fog-free images, is the atmospheric light value at infinity, is the transmittance, which is inversely proportional to the distance, is the attenuation coefficient, which depends on the concentration, size, type and distribution of particles in the air. is the distance variable; Collect lane line image data of actual foggy scenes and build an actual foggy dataset; Downsampling the image data in the foggy scene dataset by bilinear interpolation to convert the image size in the foggy scene dataset into specific pixels; The converted image is rotated, horizontally flipped, and brightness transformed to obtain a data enhanced image.

3. The lane line detection method for foggy scenes according to claim 1, characterized in that: The first prediction loss is: in, is the position on the data image, To predict the heat map The value of the position, is the number of pixels in the heat map, α and β are exponential weights, The true value heat map GT is The value of the position; The second prediction loss is: in, is the number of valid rows, is the vertical extent of the lane marking, is the true horizontal coordinate, is the x-coordinate of each lane line; The third prediction loss is: in, Representative OK, For the predicted The probability of a positive behavior, for ground truth; The fourth prediction loss is: in, is the area near the lane line of fixed width, yes The number of pixels in and The predicted offset and coordinates are The label offset; The fifth prediction loss is: in, Pixels The label at The probability of predicting it to be positive.

4. The lane line detection method for foggy scenes according to claim 1, characterized in that: The instance detection structure includes a first branch and a second branch; The first branch is used to predict the starting point of the lane line; The second branch is used to predict a set of dynamic convolution kernels for each position, and take the dynamic convolution kernel of the predicted position of the lane line starting point to perform soft labeling on the true value of the lane line starting point.

5. The lane line detection method for foggy scenes according to claim 3, characterized in that: The total loss function is obtained by weighted fusion of the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss and the fifth prediction loss. The total loss function is expressed as: in is the weighting coefficient corresponding to each lane line detection task.

6. The lane line detection method for foggy scenes according to claim 1, characterized in that: The neck network is used to fuse information between images of different resolutions to obtain fusion features, including: Through a multi-scale fusion network based on feature pyramid, the feature maps of multiple stages output by the backbone network are fused through a multi-resolution module to obtain fused features.

7. The lane line detection method for foggy scenes according to claim 1, characterized in that: The method of using the trained lane line detection model to predict the lane line position in the image by giving an input image includes: Using a given input image , the coordinates on the predicted lane line are: in, For the Lane lines, For the Maximum number of sample points per lane line; Summarize the coordinates of the lane lines to obtain the predicted position of the lane lines in the image: 。 8. A lane line detection system for foggy scenes, characterized by: The system comprises: A data processing module is used to construct a foggy scene dataset based on a public dataset and actual foggy day collected data and to preprocess the foggy scene dataset; A model construction module is used to construct a lane detection model based on the image data in the foggy scene dataset; the lane detection model includes a backbone network, a neck network, an instance detection structure, an instance segmentation structure and an edge enhancement module; A feature extraction module, comprising a global feature fusion module for the backbone network, wherein the backbone network replaces the window attention mechanism in the last stage of SwinTransformer with a common multi-head attention mechanism; the backbone network is used to extract features from images in the foggy scene dataset to obtain image features; and the neck network is used to fuse image features of different resolutions to obtain fused features; A loss calculation module, which includes a convolution kernel feature fusion module in the instance detection structure, and is used to automatically learn the relationship between the predicted lane line instances; uses the instance detection structure to predict a set of dynamic convolution kernels for each position through the fusion features, obtains a predicted heat map of the lane line starting point, and calculates a first prediction loss between the predicted heat map and the true value heat map; predicts the road area through the dynamic convolution kernel, divides the road area into a row-based segmentation map using the instance segmentation structure, predicts the horizontal coordinate value of the lane line in each row, obtains a row-based segmentation lane line prediction map, and calculates a second prediction loss for the row-based segmentation prediction; predicts the vertical range of the lane line based on whether it passes through the current row through the row-based segmentation lane line prediction map, and calculates a third prediction loss for the lane line vertical range prediction; predicts the offset of the coordinates predicted by the row-based segmentation map, and calculates a fourth prediction loss for the offset prediction; uses the edge enhancement module to predict the edge of the lane line through the binary classification task of edge segmentation, and calculates a fifth prediction loss for the lane line edge prediction; a model training module, configured to calculate a total loss function of a lane detection model based on the first prediction loss, the second prediction loss, the third prediction loss, the fourth prediction loss, and the fifth prediction loss, and train the lane detection model based on the total loss function; The lane line detection module is used to use the trained lane line detection model to identify the position of the lane lines in the image given an input image.

9. A computer device, characterized in that: The system comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the lane line detection method for foggy scenes according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the lane line detection method for foggy scenes according to any one of claims 1 to 7.

Images