Lane line detection method, system, device and storage medium in image

Abstract

The embodiment of the application discloses a lane line detection method, system, device and storage medium in an image, and belongs to the technical field of image detection; the method comprises the following steps: acquiring a road image; using a trained lane line detection model to identify the road image, and obtaining pixel point information corresponding to a lane line in the road image. The pixel point information of the previous lane line detection result can represent the information of the lane line in the road image, and when the pixel point information does not meet the fitting condition, the pixel point information is combined with the previous lane line detection result, instead of only using the information in the currently processed road image to detect the lane line, which is helpful to improve the detection accuracy of the lane line.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and more particularly to a method, system, apparatus, and storage medium for detecting lane lines in an image. Background Technology

[0002] Autonomous vehicles have gradually become a hot topic in society. Autonomous driving technology usually relies on sensors to detect lane lines in the current lane to ensure that the vehicle's driving complies with traffic regulations and improve traffic safety.

[0003] Current technologies utilize images acquired by sensors and rely on traditional edge detection algorithms to extract lane line information by obtaining color and boundary information from the images. However, interference from factors such as changes in lighting, road pollution, and object occlusion significantly reduces the noise in the images, severely impacting the accuracy of lane line detection. Summary of the Invention

[0004] In view of this, the present invention provides a method, system, apparatus, and storage medium for lane line detection in images, to solve the problem of poor lane line detection accuracy in the prior art. To achieve one, some, or all of the above objectives, or other objectives, the present invention proposes a method, system, apparatus, and storage medium for lane line detection in images, in a first aspect:

[0005] A method for detecting lane lines in an image, comprising:

[0006] Acquire road images;

[0007] The trained lane line detection model is used to identify the road image to obtain the pixel information corresponding to the lane lines in the road image;

[0008] When the pixel information does not meet the fitting conditions, the pixel information is processed according to the previous lane detection results of the previous road image to obtain the lane detection results.

[0009] Preferably, the step of processing the pixel information based on the prior lane detection result of the prior road image to obtain the lane detection result includes:

[0010] When the previous lane line detection result is empty, the pixel information is stored, deleted, or ignored, and the lane line detection result that does not contain lane lines is obtained.

[0011] When the prior lane detection result is not empty, distance information is calculated using the pixel information and the prior lane detection result;

[0012] When the distance information matches a preset distance threshold, the pixel information is fitted to obtain the corresponding lane line detection result.

[0013] Preferably, fitting the pixel information to obtain the corresponding lane detection result includes:

[0014] Obtain the image height and image width of the road image;

[0015] Based on the operational relationship between the image height, the image width, and the pixel information, a lane line curve function with height as the horizontal axis and width as the vertical axis is obtained;

[0016] The lane line curve function is fitted using a quadratic function to obtain the fitting parameters;

[0017] The pixel information and the fitting parameters are fitted using the least squares method to obtain a fitting formula;

[0018] The fitting parameters that minimize the residual function are calculated according to the fitting formula, and the lane line detection result is obtained as the lane line fitting curve.

[0019] Preferably, before recognizing the road image using the trained lane detection model, the method further includes:

[0020] Obtain labeled training images; the labeled training images are labeled with lane line orientation and type information;

[0021] The lane detection model to be trained, after parameter initialization, is used to identify the labeled training image to obtain lane segmentation results; the lane segmentation results include the orientation and type of the lane lines.

[0022] The model parameters of the lane detection model to be trained are updated according to the lane line segmentation results, and the training of the lane detection model to be trained is determined to be completed when the lane detection model to be trained meets the preset cutoff condition.

[0023] Preferably, updating the model parameters of the lane detection model to be trained based on the lane segmentation results includes:

[0024] Based on the lane line segmentation results, the loss value is calculated using the cross-entropy loss function;

[0025] The model parameters of the lane detection model trained using the adaptive momentum stochastic optimization model are updated based on the loss value.

[0026] Preferably, the lane detection model includes a convolutional layer, a pooling layer, and an upsampling layer;

[0027] The lane detection model is divided into an image sampling unit, a global feature extraction and fusion unit, and a classification unit.

[0028] Preferably, before recognizing the road image using the trained lane detection model, the method further includes:

[0029] The road image is cropped to obtain a region of interest image for the lane lines;

[0030] The resolution of the region of interest image is adjusted to obtain the updated road image.

[0031] The second aspect:

[0032] A lane line detection system in an image includes an acquisition module for acquiring road images;

[0033] The model module is used to identify the road image using the trained lane line detection model to obtain the pixel information corresponding to the lane lines in the road image;

[0034] The detection module is used to process the pixel information according to the previous lane line detection result of the previous road image when the pixel information does not meet the fitting conditions, so as to obtain the lane line detection result.

[0035] Preferably, the detection module includes a first detection unit, used to store, delete or ignore the pixel information and obtain the lane line detection result that does not contain lane lines when the previous lane line detection result is empty;

[0036] The second detection unit is used to calculate distance information using the pixel information and the prior lane detection result when the prior lane detection result is not empty.

[0037] The third detection unit is used to fit the pixel information when the distance information matches a preset distance threshold to obtain the corresponding lane line detection result.

[0038] Preferably, the third detection unit includes an acquisition subunit for acquiring the image height and image width of the road image;

[0039] The operation subunit is used to obtain a lane line curve function with height as the horizontal axis and width as the vertical axis based on the operation relationship between the image height, the image width and the pixel information.

[0040] The fitting subunit is used to fit the lane line curve function using a quadratic function to obtain fitting parameters;

[0041] The least squares subunit is used to fit the pixel information and the fitting parameters using the least squares method to obtain a fitting formula;

[0042] The result subunit is used to calculate the fitting parameters that minimize the residual function according to the fitting formula, and obtain the lane line detection result as the lane line fitting curve.

[0043] Preferably, the system further includes a training module for acquiring labeled training images; the labeled training images are labeled with lane line orientation and type information;

[0044] The recognition module is used to recognize the labeled training image using the lane detection model to be trained after parameter initialization, and obtain lane segmentation results; the lane segmentation results include the orientation and type of the lane lines;

[0045] The cutoff module is used to update the model parameters of the lane detection model to be trained based on the lane segmentation results, and to determine that the lane detection model to be trained has completed training when the lane detection model to be trained meets the preset cutoff conditions.

[0046] Preferably, the cutoff module includes a loss unit, used to calculate the loss value using the cross-entropy loss function based on the lane line segmentation result;

[0047] An automatic update unit is used to update the model parameters of the lane detection model trained using an adaptive momentum stochastic optimization model based on the loss value.

[0048] Preferably, the lane detection model includes a convolutional layer, a pooling layer, and an upsampling layer;

[0049] The lane detection model is divided into an image sampling unit, a global feature extraction and fusion unit, and a classification unit.

[0050] Preferably, the system further includes a cropping module, used to crop the road image before recognizing the road image using the trained lane line detection model, to obtain a region of interest image for the lane lines;

[0051] The resolution module is used to adjust the resolution of the region of interest image to obtain the updated road image.

[0052] Third aspect:

[0053] An image lane line detection device includes a memory and a processor. The memory stores an image lane line detection method, and the processor is used to employ the image lane line detection method when executing the image lane line detection method.

[0054] Fourth aspect:

[0055] A storage medium storing a computer program that can be loaded by a processor and execute the methods described above.

[0056] Implementing the embodiments of the present invention will have the following beneficial effects:

[0057] Pixel information can characterize lane line information in road images, such as location, length, and coverage. When pixel information does not meet the fitting conditions, combining the pixel information with previous lane line detection results, rather than using only information from the currently processed road image to detect lane lines, helps improve the detection accuracy of lane lines. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] in:

[0060] Figure 1 This is a flowchart of a lane line detection method in an image, as shown in one embodiment.

[0061] Figure 2 This is a flowchart illustrating the lane line detection results of a lane line detection method in an image, as shown in one embodiment.

[0062] Figure 3 This is a flowchart illustrating the computation of the adaptive momentum stochastic optimization model in a lane line detection method for images in one embodiment.

[0063] Figure 4 This is a block diagram of a lane line detection system in an image, as shown in one embodiment.

[0064] Figure 5 This is a schematic diagram of the structure of a lane line detection device in an image in one embodiment. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0068] This application discloses a method for detecting lane lines in images. In existing technologies, lane line information is typically extracted from images acquired by sensors using traditional edge detection algorithms to obtain color and boundary information. However, interference from factors such as lighting variations, road pollution, and object occlusion significantly affects the accuracy of lane line detection due to noise in the image.

[0069] To address the aforementioned shortcomings, this application discloses a method for detecting lane lines in images, such as... Figure 1 As shown, it includes:

[0070] 101. Obtain road images.

[0071] In one embodiment, a road image refers to an image containing a road. For ease of understanding, in one application scenario, a road image is obtained by frame decomposition of road video captured by a camera device; that is, the road image is a continuously captured image containing a road. In another application scenario, the road image is a pre-stored set of multiple images, which are obtained by continuously capturing the images.

[0072] In one embodiment, the road image is acquired in real time by the current executing entity using a camera device; that is, the road image captured by the camera device is transmitted to the current executing entity in real time, and the current executing entity passively receives the road image. In another embodiment, the road image is retrieved by the current executing entity from storage space; that is, the road image is actively acquired by the current executing entity. This embodiment does not specifically limit the method of acquiring the road image, but it should be noted that in this embodiment, one or more road images can be acquired at a time.

[0073] 102. Use the trained lane line detection model to identify the road image and obtain the pixel information corresponding to the lane lines in the road image.

[0074] In one embodiment, the lane detection model is a neural network model used to identify lane lines in a road image. Specifically, in one application scenario, a road image contains multiple pixels, each with corresponding coordinates. The lane detection model can identify the pixels in the road image that belong to lane lines, and output the set of pixels belonging to lane lines to obtain pixel information.

[0075] In one embodiment, the pixel information can be the position information of pixels other than lane lines, or the position information of pixels corresponding to lane lines. This embodiment does not specifically limit this, as long as the basic information of lane lines in the road image can be determined based on the pixel information. The basic information of lane lines can be at least one of the following: the position, area, length, and solid / dark type of the lane line in the road image. This embodiment does not specifically limit this either.

[0076] 103. When the pixel information does not meet the fitting conditions, the pixel information is processed according to the previous lane detection results of the previous road image to obtain the lane detection results.

[0077] In one embodiment, the fitting condition is preset and aims to determine whether the pixel information can sufficiently represent the lane line information. For example, in an application scenario, the fitting condition is whether the number of pixels exceeds a preset threshold. The pixel information includes the total number of pixels used to represent the lane line. If the total number is greater than the threshold, the pixel information is determined to meet the fitting condition; otherwise, the pixel information is determined not to meet the fitting condition. This embodiment does not specifically limit the content of the fitting condition.

[0078] It should be noted that when pixel information does not meet the fitting conditions, it indicates that the relevant information about lane lines obtained from the currently processed road image is insufficient. To improve the recognition accuracy of lane lines, this embodiment combines the previous lane line detection results of the previous road image to process the pixel information and obtain the lane line detection results. Here, the previous road image refers to the road image preceding the currently processed road image; that is, the currently processed road image and the previous road image are adjacent images in consecutive frames, and the previous road image is the image from the previous frame.

[0079] When pixel information does not meet the fitting conditions, it means that lane lines cannot be accurately detected based solely on information from the currently processed road image. Therefore, the lane line detection results are combined with those from the previous frame of the road image to obtain the lane line detection results. Compared to existing technologies, this makes the lane line detection results less susceptible to noise interference, which helps improve the anti-interference ability of the lane line detection process and improves the detection accuracy.

[0080] In another embodiment of this application, the step of processing the pixel information based on the prior lane detection result of the prior road image to obtain the lane detection result includes:

[0081] 201. When the prior lane line detection result is empty, store, delete or ignore the pixel information and obtain the lane line detection result that does not contain lane lines.

[0082] It should be noted that when the road image being processed is the first frame, there are no preceding road images, so the preceding lane line detection results for preceding road images are empty.

[0083] In one embodiment, when the lane line detection result is empty, the pixel information can be stored, deleted, or no processing can be performed on the pixel information, and the process can directly jump to the next step or obtain the next frame of the road image. In this case, the lane line detection result is determined to be that no lane line is included; in other embodiments, the lane line detection result can also be determined to be a detection failure. This embodiment does not specifically limit this.

[0084] 202. When the detection result of the preceding lane line is not empty, distance information is calculated using the pixel information and the detection result of the preceding lane line.

[0085] If the previous lane detection result is not empty, it means that the road image being processed is not the first frame image, and the previous frame image, i.e., the earlier road image, has a corresponding previous lane detection result. This previous lane detection result contains information about the lane lines detected in the earlier road image. Specifically, in one application scenario, the previous lane detection result is the previous fitting result (the previous lane line function curve) corresponding to the lane lines in the online road image. Through the previous fitting result, we can know the position, area, whether it is a left or right lane line, and whether it is a solid or dashed line in the earlier road image, etc. This embodiment does not make specific limitations on this.

[0086] In one embodiment, the distance information is the distance between each pixel in the pixel information and the preceding lane line detection result. In one application scenario, the preceding lane line detection result is a preceding lane line function curve, and the pixel information contains the position information of at least one pixel. The average distance from each pixel in the pixel information to the preceding lane line function curve is calculated using the position information of the pixels, and the average distance is determined as the distance information.

[0087] 203. When the distance information matches a preset distance threshold, the pixel information is fitted to obtain the corresponding lane line detection result.

[0088] Specifically, if the distance information is less than the distance threshold, it is determined that the distance information matches the distance threshold; otherwise, it does not match. Furthermore, when the distance information does not match the distance threshold, it is proven that the pixels in the pixel information are interference pixels, i.e., noise pixels of the lane lines.

[0089] In one embodiment, the pixel information is fitted to obtain a lane line detection result, which is a lane line fitting curve. The lane line fitting curve can characterize the information of lane lines in the road image, such as position, area, dashed and solid lines, and left and right sides, etc., which are not specifically limited in this embodiment.

[0090] By utilizing the lane line detection results, it is determined whether all or some pixels in the pixel information are interference pixels. If they are not interference pixels, the pixel information is fitted to obtain the lane line detection result. In other words, by using the lane line detection results to help identify interference pixels, it helps improve the accuracy of lane line detection.

[0091] In another embodiment of this application, such as Figure 2 As shown, fitting the pixel information to obtain the corresponding lane detection result includes:

[0092] 301. Obtain the image height and image width of the road image.

[0093] In one embodiment, the image height and image width of the road image are known quantities. The current executing entity can receive the corresponding values ​​through the interface or retrieve the corresponding values ​​from the storage space. This embodiment does not make specific limitations on this.

[0094] It should be noted that images from the same camera or from the same batch of road images have the same height and width.

[0095] 302. Based on the operational relationship between the image height, the image width, and the pixel information, a lane line curve function with height as the horizontal axis and width as the vertical axis is obtained.

[0096] In one embodiment, let (x, y) be a pair of pixel coordinates, H be the image height, and W be the image width, then:

[0097] x = [x1, x2, ... x n ] T ∈H, y∈W;

[0098] Where x is the x-coordinate of a pixel, y is the y-coordinate of a pixel, H is the image height, and W is the image width.

[0099] The lane line curve, with height as the x-axis and width as the y-axis, satisfies the following function, i.e., the lane line curve function:

[0100] y = f(x, ω);

[0101] Where ω is the fitting parameter.

[0102] 303. Fit the lane line curve function using a quadratic function to obtain the fitting parameters.

[0103] In one embodiment, a quadratic function is used to fit the lane line curve function, and the fitting parameters are as follows:

[0104] ω = [ω1ω2ω3];

[0105] Where ω is the fitting parameter.

[0106] 304. The pixel information and the fitting parameters are fitted using the least squares method to obtain the fitting formula.

[0107] In one embodiment, the least squares method is used for fitting, resulting in the following fitting formula:

[0108]

[0109] 305. The fitting parameters that minimize the residual function are calculated according to the fitting formula, and the lane line detection result is obtained as the lane line fitting curve.

[0110] In one embodiment, the lane line detection results are subjected to coordinate system scaling transformation to maintain correspondence with the road image.

[0111] By fitting calculations, it is helpful to obtain curves that characterize the basic information of lane lines in road images, which helps to improve the detection efficiency of lane lines.

[0112] In another embodiment of this application, before recognizing the road image using the trained lane detection model, the method further includes:

[0113] 401. Obtain labeled training images.

[0114] In one embodiment, a preset annotation rule is provided. This rule uses a two-digit decimal system to label the lanes to which the current vehicle belongs, where the left / right lane attribute is represented by the first digit of the two-digit decimal system, and the solid / virtual lane attribute is represented by the second digit. Annotating the original image yields an annotated training image. This annotated training image is used to train an untrained lane detection model; therefore, the lane line information in the annotated training image is known.

[0115] In one embodiment, the labeled training images are divided into a training dataset and a test dataset in a 4:1 ratio. The labeled training images are then cropped according to the following rule: the lower left corner of the image is used as the reference point, and the height is equal to the original height. Figure 2 / 3, width unchanged. This yields a labeled training image with useless sky information cropped out.

[0116] In one embodiment, the labeled training image is labeled with lane line orientation information and type information. The orientation information refers to either the left or right lane line, and the type information refers to either a solid line or a dashed line.

[0117] 402. The lane detection model to be trained, after parameter initialization, is used to identify the labeled training image to obtain lane segmentation results.

[0118] The lane detection model to be trained is used to recognize and process the labeled training images to obtain predicted segmentation results, i.e., lane segmentation results. Specifically, in one embodiment, a batch of labeled training images is randomly selected from the training dataset for training. Using the PyTorch deep learning framework, a batch of labeled training images is input into the lane detection model to be trained to obtain a batch of lane segmentation results.

[0119] In one embodiment, the lane line segmentation result includes the orientation and type of the lane lines.

[0120] 403. Update the model parameters of the lane detection model to be trained according to the lane line segmentation results, and determine that the training of the lane detection model to be trained is complete when the lane detection model to be trained meets the preset cutoff condition.

[0121] In one embodiment, the average loss value of all labeled training images in a batch is calculated using the lane line segmentation results. Since the lane line annotation information and pixel segmentation information of the labeled training images are known quantities, the obtained lane line segmentation results are compared with the known quantities, and the model parameters are updated based on the deviation.

[0122] In one embodiment, the cutoff condition is that the batch size of the input labeled training images reaches a preset batch threshold; in another embodiment, the cutoff condition is that the deviation between the obtained lane segmentation result and the known quantity is less than a preset deviation threshold; in another embodiment, the cutoff condition is that the calculated average loss value meets a preset loss threshold; in another embodiment, the cutoff condition is that all batches of labeled training images have been processed by the lane detection model to be trained; in other embodiments, the cutoff condition is that after inputting the test dataset into the lane detection model to be trained, the deviation between the calculated result and the known result of the test dataset is less than a preset similarity threshold. This embodiment does not impose specific limitations on these aspects.

[0123] By training the lane detection model and setting training cutoff conditions, the computational accuracy of the trained lane detection model was ensured.

[0124] In another embodiment of this application, updating the model parameters of the lane detection model to be trained based on the lane segmentation results includes:

[0125] 501. Based on the lane line segmentation results, the loss value is calculated using the cross-entropy loss function.

[0126] In one embodiment, the cross-entropy loss function is:

[0127]

[0128] Among them, y i,k p represents the true label of the i-th sample. There are K label values ​​and N samples in total. i,k This represents the probability that the i-th sample is predicted to be the k-th label value.

[0129] In one embodiment, adding a branch module to the loss function in the lane detection model can improve the regression performance of the lane detection model. The branch module is added to the loss function using a weighted approach, as shown in the following expression:

[0130]

[0131] Among them, L cls L represents the loss value that distinguishes the network's output feature map from the true label. aux The loss value representing the classification result of the branching structure. This represents the weighting of the model's loss function.

[0132] 502. Update the model parameters of the lane detection model trained using the adaptive momentum stochastic optimization model based on the loss value.

[0133] like Figure 3 The diagram shows the process by which the adaptive momentum stochastic optimization model adjusts the model parameters of the lane detection model.

[0134] By utilizing an adaptive momentum stochastic optimization model to automatically update model parameters based on the loss value, the training efficiency of the lane detection model is improved.

[0135] In another embodiment of this application, the lane detection model includes a convolutional layer, a pooling layer, and an upsampling layer;

[0136] The lane detection model is divided into an image sampling unit, a global feature extraction and fusion unit, and a classification unit.

[0137] In one embodiment, the image sampling unit is used to sample shallow information; the global feature extraction and fusion unit is used to extract image features of the road image and perform feature fusion; and the classification unit is used to classify the extracted feature points to obtain classified lane information.

[0138] In another embodiment of this application, before recognizing the road image using the trained lane detection model, the method further includes:

[0139] 701. The road image is cropped to obtain a region of interest image for the lane lines.

[0140] In one embodiment, the road image is cropped using the lower left corner of the lane image as a reference point, with a height of 2 / 3 of the original image and the width unchanged, to remove useless sky area information from the road image, thus obtaining the region of interest image.

[0141] 702. Adjust the resolution of the region of interest image to obtain the updated road image.

[0142] In one embodiment, the road image has a resolution of 1080*1920, and the region of interest image has a resolution of 720*1920. After resolution adjustment, the updated road image has a resolution of 192*448.

[0143] By removing useless information and adjusting the resolution of road images, the efficiency and quality of subsequent processing can be improved, thus obtaining more accurate detection results.

[0144] In addition, to facilitate understanding of the lane detection model, the model processing steps are explained as follows:

[0145] The lane detection model can be divided into three parts: a downsampling learning module, a global feature extraction module, and a feature fusion module. When constructing the lane detection model, convolutional layers, pooling layers, and upsampling layers are used to build a neural network. The kernel size and stride of the convolutional sliding in the lane detection model, only the number of output feature maps, need to be customized according to specifications. A pyramid parsing module and an Encoder-Decoder structure are used to build the model, classifying the feature maps according to the depth of the network architecture, and fusing shallow and deep information through addition.

[0146] In the application of the lane line detection model, the input region of interest image is denoted as T1, and its resolution is 3*192*448.

[0147] Pass T1 through a convolution kernel of size 3*3, with 32 output channels, a stride of 2, and padding of 1; after this convolution operation, the resulting data size is 32*96*224, denoted as T2;

[0148] Pass T2 through a convolution kernel of size 3*3, with 64 output channels, a stride of 2, and padding of 1; after this convolution operation, the resulting data size is 64*48*l12, denoted as T3;

[0149] T3 is passed through a basic block with 128 output channels (m), a step size (n) of 2, and a padding value of 1. The resulting data size after passing through this basic block is 128*24*56, denoted as feat1.

[0150] Pass feat1 through a basic block. This block has 192 output channels (m), a step size (n), and a padding value of 1. After passing through this basic block, the resulting data size is 192 * 12 * 28, which is denoted as feat2.

[0151] Pass feat2 through a basic block. This block has 256 output channels (m), a step size (n), and a padding value of 1. After passing through this basic block, the resulting data size is 256 * 6 * 14, which is denoted as feat3.

[0152] Pass feat3 through a convolution kernel of size 3*3, with 256 output channels, a stride of 2, and padding of 1; the resulting data size after this convolution operation is 256*3*7, denoted as feat4;

[0153] The data size of feat4 is 256*6*14 after passing through an upsampling layer and is denoted as Re_feat1. feat3 is added to Re_feat1 to obtain feat5.

[0154] Pass feat5 through a convolution kernel of size 3*3, with 192 output channels, a stride of 1, and padding of 1; the resulting data size after this convolution operation is 192*6*14, denoted as feat6;

[0155] After passing feat6 through an upsampling layer, the resulting data size is 192*12*28, denoted as Re_feat2. Adding feat2 to Re_feat2 yields feat7.

[0156] Pass feat7 through a convolutional kernel of size 3*3, with 128 output channels, a stride of 1, and padding of 1; the resulting data size after this convolution operation is 128*24*56, denoted as feat8;

[0157] After passing feat8 through an upsampling layer, the resulting data size is 128*24*56, denoted as Re_feat3.

[0158] Add feat1 and Re_feat3 together to get feat9;

[0159] After passing feat9 through an upsampling layer, the resulting data size is 128*48*112, denoted as Re_feat4;

[0160] T3 is passed through a convolution kernel of size 3*3, with 128 output channels, a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 128*48*112, denoted as feat10.

[0161] Add feat10 to Re_feat4 to get feat11;

[0162] After passing feat11 through an upsampling layer, the resulting data size is 128*96*224, denoted as Re_feat5;

[0163] The Re_feat5 is passed through a 3*3 convolution kernel with the number of output channels equal to the number of output classes (5 in this example), a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 5*96*224, denoted as feat_final.

[0164] Re_feat5 is passed through a convolutional kernel of size 3*3, with 64 output channels, a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 64*96*224, denoted as feat_aux1.

[0165] Pass feat_aux1 through a Dropout layer. The Dropout layer randomly removes some neurons during the training of a batch. The Dropout parameter is set to 0.1, and it is denoted as feat_aux2.

[0166] The feat_aux2 is passed through a convolutional kernel of size 3*3, with the number of output channels equal to the number of output classes (5 in this example), a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 5*96*224, denoted as feat_aux_final.

[0167] Re_feat5 is passed through a convolutional kernel of size 3*3, with 64 output channels, a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 64*96*224, denoted as feat_aux1.

[0168] feat_aux1 is passed through a Dropout layer. The Dropout layer randomly removes some neurons during the training of a batch. The Dropout parameter of this layer is set to 0.1, and it is denoted as feat_aux2.

[0169] The feat_aux2 is passed through a convolutional kernel of size 3*3, with the number of output channels equal to the number of output classes (5 in this example), a stride of 1, and padding of 1. After this convolution operation, the resulting data size is 5*96*224, denoted as feat_aux_final.

[0170] In summary, feat_final is the output layer result of the model, and feat_aux_final is the output result of the model branch. It is worth noting that in order to improve the speed of least squares fitting of lane lines in post-processing, the data size of feat_final is half the length and width of the input image. During training, the label size needs to be set to 96*224.

[0171] It should be noted that in the above model structure description, T2 and T3 are the downsampling learning modules of the model, the process from feat1 to feat4 is the global feature extraction module, the process from feat4 to feat11 is the feature fusion part, and finally the fusion result is obtained by convolution. The layer with feat_aux is the branch structure part during training, which only plays a role during training and does not exist during testing and deployment.

[0172] Pixel information can characterize lane line information in road images, such as location, length, and coverage. When pixel information does not meet the fitting conditions, combining the pixel information with previous lane line detection results, rather than using only information from the currently processed road image to detect lane lines, helps improve the detection accuracy of lane lines.

[0173] This application also discloses a lane line detection system in an image, such as... Figure 4 As shown, it includes an acquisition module 1, used to acquire road images;

[0174] Model module 2 is used to identify the road image using the trained lane line detection model to obtain the pixel information corresponding to the lane lines in the road image;

[0175] The detection module 3 is used to process the pixel information according to the previous lane line detection result of the previous road image when the pixel information does not meet the fitting conditions, so as to obtain the lane line detection result.

[0176] Preferably, the detection module 3 includes a first detection unit, used to store, delete or ignore the pixel information and obtain the lane line detection result that does not contain lane lines when the previous lane line detection result is empty;

[0177] The second detection unit is used to calculate distance information using the pixel information and the prior lane detection result when the prior lane detection result is not empty.

[0178] The third detection unit is used to fit the pixel information when the distance information matches a preset distance threshold to obtain the corresponding lane line detection result.

[0179] Preferably, the third detection unit includes an acquisition subunit for acquiring the image height and image width of the road image;

[0180] The operation subunit is used to obtain a lane line curve function with height as the horizontal axis and width as the vertical axis based on the operation relationship between the image height, the image width and the pixel information.

[0181] The fitting subunit is used to fit the lane line curve function using a quadratic function to obtain fitting parameters;

[0182] The least squares subunit is used to fit the pixel information and the fitting parameters using the least squares method to obtain a fitting formula;

[0183] The result subunit is used to calculate the fitting parameters that minimize the residual function according to the fitting formula, and obtain the lane line detection result as the lane line fitting curve.

[0184] Preferably, the system further includes a training module for acquiring labeled training images; the labeled training images are labeled with lane line orientation and type information;

[0185] The recognition module is used to recognize the labeled training image using the lane detection model to be trained after parameter initialization, and obtain lane segmentation results; the lane segmentation results include the orientation and type of the lane lines;

[0186] The cutoff module is used to update the model parameters of the lane detection model to be trained based on the lane segmentation results, and to determine that the lane detection model to be trained has completed training when the lane detection model to be trained meets the preset cutoff conditions.

[0187] Preferably, the cutoff module includes a loss unit, used to calculate the loss value using the cross-entropy loss function based on the lane line segmentation result;

[0188] An automatic update unit is used to update the model parameters of the lane detection model trained using an adaptive momentum stochastic optimization model based on the loss value.

[0189] Preferably, the lane detection model includes a convolutional layer, a pooling layer, and an upsampling layer;

[0190] The lane detection model is divided into an image sampling unit, a global feature extraction and fusion unit, and a classification unit.

[0191] Preferably, the system further includes a cropping module, used to crop the road image before recognizing the road image using the trained lane line detection model, to obtain a region of interest image for the lane lines;

[0192] The resolution module is used to adjust the resolution of the region of interest image to obtain the updated road image.

[0193] It should be noted that the above description of the lane line detection system embodiments in images is similar to the method description above and has the same beneficial effects as the method embodiments. For technical details not disclosed in the lane line detection system embodiments of the present invention, those skilled in the art should refer to the description of the method embodiments of the present invention for understanding.

[0194] It should be noted that, in the embodiments of the present invention, if the above methods are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0195] Accordingly, embodiments of this application also disclose a storage medium storing a computer program that can be loaded by a processor and execute the above-described methods.

[0196] This application also discloses a lane line detection device in an image, such as... Figure 5 As shown, the system includes a processor 100, at least one communication bus 200, a user interface 300, at least one external communication interface 400, and a memory 500. The communication bus 200 is configured to enable communication between these components. The user interface 300 may include a display screen, and the external communication interface 400 may include standard wired and wireless interfaces. The memory 500 stores a lane line detection method in an image. The processor 100 is used to employ the lane line detection method stored in the memory 500 when executing the method.

[0197] The above description of the lane line detection device and storage medium embodiments applied to images is similar to the description of the method embodiments described above, and has similar beneficial effects. For technical details not disclosed in the embodiments of the lane line detection device and storage medium of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0198] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the invention, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the invention. The sequence numbers of the above-described embodiments of the invention are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0199] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0200] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0201] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0202] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0203] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0204] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0205] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting lane lines in an image, characterized in that, include: Acquire road images; The trained lane line detection model is used to identify the road image to obtain the pixel information corresponding to the lane lines in the road image; When the pixel information does not meet the fitting conditions, the pixel information is processed according to the previous lane detection results of the previous road image to obtain the lane detection results; The step of processing the pixel information based on the prior lane detection results of the prior road image to obtain the lane detection results includes: When the previous lane line detection result is empty, the pixel information is stored, deleted, or ignored, and the lane line detection result that does not contain lane lines is obtained. When the prior lane detection result is not empty, distance information is calculated using the pixel information and the prior lane detection result; When the distance information matches a preset distance threshold, the pixel information is fitted to obtain the corresponding lane line detection result.

2. The lane line detection method in an image as described in claim 1, characterized in that, Fitting the pixel information to obtain the corresponding lane detection result includes: Obtain the image height and image width of the road image; Based on the operational relationship between the image height, the image width, and the pixel information, a lane line curve function with height as the horizontal axis and width as the vertical axis is obtained; The lane line curve function is fitted using a quadratic function to obtain the fitting parameters; The pixel information and the fitting parameters are fitted using the least squares method to obtain a fitting formula; The fitting parameters that minimize the residual function are calculated according to the fitting formula, and the lane line detection result is obtained as the lane line fitting curve.

3. The lane line detection method in an image as described in claim 1, characterized in that, Before using the trained lane detection model to identify the road image, the method further includes: Obtain labeled training images; the labeled training images are labeled with lane line orientation and type information; The lane detection model to be trained, after parameter initialization, is used to identify the labeled training image to obtain lane segmentation results; the lane segmentation results include the orientation and type of the lane lines. The model parameters of the lane detection model to be trained are updated according to the lane line segmentation results, and the training of the lane detection model to be trained is determined to be completed when the lane detection model to be trained meets the preset cutoff condition.

4. The lane line detection method in an image as described in claim 3, characterized in that, The step of updating the model parameters of the lane detection model to be trained based on the lane segmentation results includes: Based on the lane line segmentation results, the loss value is calculated using the cross-entropy loss function; The model parameters of the lane detection model trained using the adaptive momentum stochastic optimization model are updated based on the loss value.

5. The lane line detection method in an image as described in claim 3 or 4, characterized in that, The lane detection model includes convolutional layers, pooling layers, and upsampling layers; The lane detection model is divided into an image sampling unit, a global feature extraction and fusion unit, and a classification unit.

6. The lane line detection method in an image as described in claim 1, characterized in that, Before using the trained lane detection model to identify the road image, the method further includes: The road image is cropped to obtain a region of interest image for the lane lines; The resolution of the region of interest image is adjusted to obtain the updated road image.

7. A lane line detection system in an image, characterized in that, Includes an acquisition module for acquiring road images; The model module is used to identify the road image using the trained lane line detection model to obtain the pixel information corresponding to the lane lines in the road image; The detection module is used to process the pixel information according to the previous lane line detection result of the previous road image when the pixel information does not meet the fitting conditions, so as to obtain the lane line detection result. When the detection module processes the pixel information based on the prior lane detection results of the prior road image to obtain the lane detection results, it includes: When the previous lane line detection result is empty, the pixel information is stored, deleted, or ignored, and the lane line detection result that does not contain lane lines is obtained. When the prior lane detection result is not empty, distance information is calculated using the pixel information and the prior lane detection result; When the distance information matches a preset distance threshold, the pixel information is fitted to obtain the corresponding lane line detection result.

8. A lane line detection device in an image, comprising a memory and a processor, characterized in that, The memory stores a method for detecting lane lines in an image, and the processor is used to employ the method for detecting lane lines in an image as described in any one of claims 1-6 when executing the method for detecting lane lines in an image.

9. A storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1-6.

Images