Lane line detection method and device of vehicle, vehicle and storage medium

Abstract

The application relates to a lane line detection method and device of a vehicle, the vehicle and a storage medium, wherein the method comprises the following steps: acquiring a training image of the vehicle and a lane line point set true value label; converting the lane line point set true value label into the size of a preset anchor frame to obtain an anchor frame label; inputting the training image and the anchor frame label into a lane line detection model to be trained to determine optimal neural network weights learned in advance, generate a lane line detection model based on a neural network which is pre-trained, wherein the lane line detection model comprises a classification network for distinguishing whether an anchor frame has a lane line feature, and the lane line detection result is obtained by using the lane line detection model based on the neural network which is pre-trained. According to the application, the lane line segmentation is converted into a lane line binary classification problem, the requirements of algorithm processing and calculation amount are reduced, the detection problem of a transverse lane line is well solved, the real-time performance of lane line detection is improved, the reliability of the vehicle is improved, and the use requirement of a user is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a lane line detection method and device for a vehicle, a vehicle and a storage medium. BACKGROUND

[0002] Lane line detection has always been an important issue and basic application technology required to be solved by an automatic driving and an advanced driving assistance system (ADAS). A traditional image processing method is usually based on visual information, for example, an HSI (Hue-Saturation-Intensity, hue-saturation-intensity) color mode and edge detection in image processing, such as various carefully designed edge detection operator filters, including a Sobel operator, a Prewitt operator, a Laplacian operator, or a Hough transformation. With the development of statistical learning or machine learning (ML), pattern matching and a support vector machine (SVM) can also be used for image processing. However, with the increasing diversification of application scenarios and the demand for lane line prior positions, the above methods have many problems, for example, they cannot well adapt to complex scenes, are poor in noise resistance, are helpless for regions with high light effects and completely blocked regions, and need to set many rules manually. In summary, a method based entirely on traditional image processing has been difficult to meet the current needs of ADAS and automatic driving for lane line detection.

[0003] With the development of neural network technology, a method for processing lane line detection according to image segmentation by a neural network can be used in the market and academia. However, a neural network based on segmentation is difficult to meet the real-time requirements of automatic driving and ADAS for real-time detection during vehicle driving. At the same time, since ADAS and automatic driving systems need to involve multiple camera inputs, the required computing amount and computing power of the target detection model are greatly increased. In addition, when lane line detection is performed based on image segmentation, the receptive field of convolution is limited, which affects the detection performance.

[0004] Currently, the related art can obtain a sample image of a target domain, input the sample image into an initial lane line detection network, determine an initial feature map of the sample image, and further generate a probability feature map of the sample image, determine a pseudo label corresponding to the sample image and a lane line detection result corresponding to the sample image, train the initial lane line detection network, and obtain a target lane line detection network corresponding to the target domain. In addition, the related art can input a training image into a lane line detection model to obtain a lane line starting point detection result and a lane line detection result, calculate a loss function of the lane line starting point detection result and a lane line starting point label, and calculate a loss function of the lane line detection result and a lane line point set label to determine subsequent operations.

[0005] However, the related art cannot guarantee the real-time performance of lane line detection, and it is difficult to detect horizontal lane lines (such as parking lines and parking space lines). Therefore, it is urgent to solve the problem. SUMMARY

[0006] The present application provides a lane line detection method and device for a vehicle, a vehicle and a storage medium to solve the problem that the related art cannot guarantee the real-time performance of lane line detection and it is difficult to detect horizontal lane lines.

[0007] The first aspect of the present application provides a lane line detection method for a vehicle, which is applied to an offline training stage. The method comprises the following steps: obtaining a training image of a vehicle and a lane line point set true value label; converting the lane line point set true value label into a size of a preset anchor box to obtain an anchor box label; inputting the training image and the anchor box label into a lane line detection model to be trained to determine optimal neural network weights learned in advance, and generating a lane line detection model based on a neural network trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether the anchor box has a lane line feature, and the lane line detection result is obtained by using the lane line detection model based on the neural network trained in advance.

[0008] According to the above technical means, the present application embodiment processes the lane line feature based on the anchor box, adjusts the size of the anchor box, adopts the combination of the encoder and the decoder to complete the detection of the horizontal line, trains the lane line detection model including only the classification network, greatly reduces the requirements of the processing algorithm and the calculation amount, solves the problem of the detection of the horizontal line, guarantees the learning ability and the performance of the model, improves the generalization performance of the model, and provides reliable technical support for the realization of the real-time detection of the lane line.

[0009] Optionally, in an embodiment of the present application, the inputting the training image and the anchor box label into the lane line detection model to be trained, determining the pre-learned optimal neural network weight, and generating the pre-trained neural network-based lane line detection model comprises: based on the training image and the anchor box label, a feature map is obtained through multiple extractions by using a convolution module and a CSP module of a classification network; based on the feature map, a complete feature map is extracted by using an SElayer module of the classification network, the complete feature map is input into a full connection layer of the classification network through a decoder, and dimension conversion is performed to obtain a final feature map meeting a preset condition; loss function calculation and back propagation are performed based on the final feature map until training is completed, and the pre-trained neural network-based lane line detection model is generated.

[0010] According to the above technical means, in the embodiment of the present application, the loss function value of the lane line result and the lane line true value label is calculated by using only the detection model of the classification network, the optimal neural network weight is obtained and saved, thereby greatly reducing the requirement of calculation amount, and the number of predicted line segments is not limited, which provides reliable technical support for real-time detection of lane lines.

[0011] Optionally, in an embodiment of the present application, the converting the lane line point set true value label into the size of the preset anchor box to obtain the anchor box label comprises: determining the anchor box height according to the image height of the lane line point set true value label, and determining the anchor box width according to the image width of the lane line point set true value label.

[0012] According to the above technical means, in the embodiment of the present application, the size of the anchor box can be fine-tuned and unified according to the image of the lane line point set true value label, thereby providing reliable input data for subsequent model training and effectively guaranteeing the performance of model training.

[0013] The second aspect embodiment of the present application provides a lane line detection method of a vehicle, applied to an online detection stage, wherein the method comprises the following steps: acquiring a lane line image of a vehicle; extracting a plurality of anchor boxes from the lane line image, wherein each anchor box contains or does not contain lane line features; and inputting the lane line image into a pre-trained neural network-based lane line detection model to perform inference based on a pre-learned optimal neural network weight, and outputting a lane line detection result, wherein the lane line detection model comprises a classification network that distinguishes whether an anchor box has the lane line features.

[0014] According to the above technical means, in the embodiment of the present application, the lane line detection can be performed by using a lightweight and low-computing amount detection model, which improves the real-time performance of detection, effectively guarantees the accuracy and detection effect of model detection, and greatly meets the use demand and use experience of users.

[0015] Optionally, in an embodiment of the present application, the lane line features include straight line features, curved lane line features, and transverse line features.

[0016] According to the above technical means, the embodiments of the present application improve the accuracy and comprehensiveness of lane line detection by setting appropriate lane line features, effectively solving the problems of transverse line detection and line quantity limitation.

[0017] The third aspect of the present application provides a lane line detection device of a vehicle, applied to an offline training stage, wherein the device comprises: a first acquisition module configured to acquire a training image of a vehicle and a lane line point set true value label; a conversion module configured to convert the lane line point set true value label into a size of a preset anchor box to obtain an anchor box label; and a training module configured to input the training image and the anchor box label into a lane line detection model to be trained, determine optimal neural network weights learned in advance, and generate a lane line detection model based on a neural network trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether the anchor box has the lane line feature, so as to obtain a lane line detection result by using the lane line detection model based on the neural network trained in advance.

[0018] Optionally, in an embodiment of the present application, the training module comprises: a processing unit configured to obtain a feature map by using a convolution module and a CSP module of the classification network through multiple extractions based on the training image and the anchor box label; a dimension conversion unit configured to extract a complete feature map by using an SElayer module of the classification network based on the feature map, input the complete feature map to a fully connected layer of the classification network through a decoder, and perform dimension conversion to obtain a final feature map satisfying a preset condition; and a generation unit configured to perform loss function calculation and back propagation based on the final feature map until training is completed, and generate the lane line detection model based on the neural network trained in advance.

[0019] Optionally, in an embodiment of the present application, the conversion module comprises: a determination unit configured to determine an anchor box height according to an image height of the lane line point set true value label, and determine an anchor box width according to an image width of the lane line point set true value label.

[0020] The fourth aspect of the present application provides a lane line detection device of a vehicle, which is applied to an online detection stage, wherein the device comprises: a second acquisition module configured to acquire a lane line image of the vehicle; an extraction module configured to extract a plurality of anchor boxes from the lane line image, wherein each anchor box contains or does not contain a lane line feature; and a detection module configured to input the lane line image into a lane line detection model based on a neural network which is pre-trained, to perform inference based on pre-learned optimal neural network weights, and to output a lane line detection result, wherein the lane line detection model comprises a classification network configured to distinguish whether an anchor box contains the lane line feature.

[0021] Optionally, in an embodiment of the present application, the lane line feature comprises a straight line feature, a curved lane line feature, and a transverse line feature.

[0022] The fifth aspect of the present application provides a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lane line detection method of the vehicle as described in the above embodiments.

[0023] The sixth aspect of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the lane line detection method of the vehicle as described above.

[0024] Therefore, the embodiments of the present application have the following beneficial effects:

[0025] (1) The embodiments of the present application process the lane line feature based on the anchor box, adjust the size of the anchor box, complete the detection of the transverse line by combining the encoder and the decoder, train the lane line detection model comprising only the classification network, thereby greatly reducing the requirements of the processing algorithm and the calculation amount, solving the detection problem of the transverse line, guaranteeing the learning ability and the performance of the model, improving the generalization performance of the model, and providing reliable technical support for the realization of the real-time detection of the lane line.

[0026] (2) The embodiments of the present application obtain and save the optimal neural network weights by calculating the loss function value of the lane line result and the lane line true value label by using the detection model comprising only the classification network, thereby greatly reducing the requirements of the calculation amount and not limiting the number of predicted line segments, and providing reliable technical support for the real-time detection of the lane line.

[0027] (3) The embodiments of the present application can fine-tune and unify the size of the anchor box according to the image of the lane line point set true value label, thereby providing reliable input data for subsequent model training and effectively guaranteeing the performance of the model training.

[0028] (4) The lane line detection method provided by the embodiment of the present application can detect lane lines by using a lightweight and low-computing model, thereby improving the real-time performance of detection, effectively ensuring the accuracy and detection effect of model detection, and greatly meeting the use demand and use experience of users.

[0029] (5) The embodiment of the present application sets appropriate lane line features, thereby improving the accuracy and comprehensiveness of lane line detection, and effectively solving the problems of horizontal line detection and line quantity limitation.

[0030] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A flowchart of an offline training stage of a lane line detection method for a vehicle according to an embodiment of the present application is shown in FIG. 1.

[0033] Figure 2 An effect diagram of a label conversion corresponding to an original image according to an embodiment of the present application is shown in FIG. 2.

[0034] Figure 3 A logic architecture diagram of a lane line detection model training process according to an embodiment of the present application is shown in FIG. 3.

[0035] Figure 4 An execution logic diagram of a lane line detection model training process according to an embodiment of the present application is shown in FIG. 4.

[0036] Figure 5 A lane line detection model structure diagram according to an embodiment of the present application is shown in FIG. 5.

[0037] Figure 6 An example diagram of an offline training stage of a lane line detection device for a vehicle according to an embodiment of the present application is shown in FIG. 6.

[0038] Figure 7 A flowchart of an online detection stage of a lane line detection method for a vehicle according to an embodiment of the present application is shown in FIG. 7.

[0039] Figure 8 An example diagram of an online detection stage of a lane line detection device for a vehicle according to an embodiment of the present application is shown in FIG. 8.

[0040] Figure 9 A structure diagram of a vehicle according to an embodiment of the present application is shown in FIG. 9.

[0041] 10, a lane line detection device of a vehicle, 101, a first obtaining module, 102, a conversion module, 103, a training module; 20, a lane line detection device of a vehicle, 201, a second obtaining module, 202, an extraction module, 203, a detection module, 901, a memory, 902, a processor, 903, a communication interface. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0043] The lane line detection method, device, vehicle and storage medium of the vehicle of the embodiments of the present application are described below with reference to the accompanying drawings. In view of the problems mentioned in the above background art, the present application provides a lane line detection method of a vehicle, in which the training image of the vehicle and the lane line point set true value label are obtained; the lane line point set true value label is converted into the size of a preset anchor box to obtain an anchor box label; the training image and the anchor box label are input into a lane line detection model to be trained to determine the optimal neural network weight learned in advance, and generate a lane line detection model based on a neural network trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether the anchor box has a lane line feature, so as to obtain a lane line detection result by using the lane line detection model based on the neural network trained in advance. The present application processes the lane line feature based on the anchor box, changes from pixel point matching to classification after clustering in the anchor box, adjusts the size of the anchor box, and uses the combination of the encoder and the decoder to complete the detection of the horizontal line, so as to convert the lane line segmentation into the binary classification of the lane line, greatly reduces the requirements and the amount of calculation of the algorithm processing, effectively improves the real-time performance of the lane line detection, improves the reliability of the vehicle, and meets the use requirements of the user. Thus, the problems that the related art cannot guarantee the real-time performance of the lane line detection and is difficult to detect the horizontal lane line are solved.

[0044] Specifically, Figure 1 A flowchart of a lane line detection method of a vehicle provided by the embodiments of the present application.

[0045] As Figure 1 shown, the lane line detection method of the vehicle is applied to an offline training stage, wherein the method comprises the following steps:

[0046] In step S101, the training image of the vehicle and the lane line point set true value label are obtained.

[0047] In actual implementation, the embodiment of the present application can obtain lane line images by using a front monocular camera in a frame extraction manner and use the lane line images as training data of a lane line detection network model, for example, one frame of image data is collected every fixed time length or fixed frame number, or a key frame extraction method is used to obtain lane line images, so as to avoid collecting a large amount of invalid redundant image data and causing waste of computing and storage resources.

[0048] Secondly, after obtaining the training image data, the embodiment of the present application can also perform a preprocessing operation on the training image data, for example, adjusting the size of the image and introducing a letterbox operation, so that the aspect ratio of the original image is maintained during the image size adjustment process and the image is scaled by the same ratio, so as to obtain image data of a square size required by the network, thereby avoiding the problem of image distortion caused by direct scaling.

[0049] In addition, the embodiment of the present application can also obtain lane line point set ground truth labels and perform the same size scaling operation on the lane line point set ground truth labels to match the size of the input RGB (Red-Green-Blue) image. At the same time, the input image and the lane line point set ground truth label both need to be subjected to a data enhancement operation. During the data enhancement process, all parameters of the input image and the lane line point set ground truth label are the same.

[0050] Therefore, the embodiment of the present application provides reliable data support for subsequent model training by obtaining the training image of the vehicle and the lane line point set ground truth label and performing corresponding preprocessing operations, thereby effectively guaranteeing the generalization performance of the model.

[0051] In step S102, the lane line point set ground truth label is converted into the size of a preset anchor box to obtain an anchor box label.

[0052] After obtaining the training image of the vehicle and the lane line point set ground truth label and scaling the size of the lane line point set ground truth label, further, the embodiment of the present application also needs to perform a conversion operation on the scaled lane line point set ground truth label to obtain an anchor box label, thereby guaranteeing the construction of a lightweight and low-computing lane line detection model.

[0053] Optionally, in an embodiment of the present application, converting the lane line point set ground truth label into the size of a preset anchor box to obtain an anchor box label includes: determining an anchor box height according to the image height of the lane line point set ground truth label and determining an anchor box width according to the image width of the lane line point set ground truth label.

[0054] It should be noted that the lane line point set true value label is composed of a point set and a pixel point. Embodiments of the present application can select the height of the anchor box according to the height of the lane line point set true value label, and at the same time, select a certain range of pixel numbers as the width of the anchor box according to the width of the lane line point set true value label. For some parts of the lane line point set true value label, if the pixel points contained in the anchor box do not meet the above range value, embodiments of the present application can set all the positions corresponding to the original lane line point set true value label to 0 or convert them into noise, as shown in Figure 2

[0055] Therefore, embodiments of the present application fine-tune and unify the size of the anchor box according to the image of the lane line point set true value label, thereby providing reliable input data for subsequent model training and effectively guaranteeing the performance of model training.

[0056] In step S103, the training image and the anchor box label are input into the lane line detection model to be trained to determine the optimal neural network weight learned in advance, and a lane line detection model based on a neural network trained in advance is generated, wherein the lane line detection model includes a classification network for distinguishing whether the anchor box has a lane line feature, so as to obtain a lane line detection result by using the lane line detection model based on the neural network trained in advance.

[0057] Further, embodiments of the present application can input the above training image and anchor box label into the lane line detection model to be trained, obtain and save the optimal weight of the lane line detection model, as shown in Figure 3

[0058] It should be noted that the lane line detection model in embodiments of the present application only includes a classification network for detecting lane lines, so the detection result only includes the detection result of the lane line feature.

[0059] The above classification network mainly includes a plurality of convolution combinations, an activation function, a full connection, and a loss function. Since the network model in embodiments of the present application is a binary classification model, a FocalLoss cross-entropy loss function is used to solve this problem to avoid the existence of serious imbalance between positive and negative samples and guarantee the performance of model training. The expression of the loss function is as follows:

[0060]

[0061]

[0062] In addition, in addition to using the FocalLoss loss function, embodiments of the present application can further avoid the occurrence of imbalance between positive and negative samples by weighting the loss result of the positive sample when the feature map processed by the convolution module passes through the loss function.​​

[0063] The embodiment of the present application can also artificially set the positive sample anchor box and the negative sample anchor box when data preprocessing is performed on the true value point set label, and combine the classification neural network encoder and the decoder to obtain a classification result.

[0064] Therefore, the embodiment of the present application greatly reduces the requirements for processing algorithms and calculation amount, does not limit the number of predicted line segments, effectively solves the detection problem of the horizontal line, and provides reliable technical support for real-time detection of the lane line.

[0065] Optionally, in an embodiment of the present application, the training image and the anchor box label are input into a lane line detection model to be trained, the optimal neural network weight learned in advance is determined, and a lane line detection model based on the neural network trained in advance is generated, including: based on the training image and the anchor box label, a feature map is obtained through multiple extractions by using a convolution module and a CSP (Cross Stage Partial Network) module of a classification network; based on the feature map, a complete feature map is extracted by using an SElayer module of the classification network, the complete feature map is input into a full connection layer of the classification network through a decoder, and dimension conversion is performed to obtain a final feature map meeting a preset condition; loss function calculation and back propagation are performed based on the final feature map until the training is completed, and the lane line detection model based on the neural network trained in advance is generated.

[0066] After the lane line detection model is trained and the optimal weight is saved, further, the embodiment of the present application can perform inference on related tasks through the optimal weight.

[0067] As understood by those skilled in the art, since commonly used backbone networks such as ResNet and VGG have high requirements for calculation amount and equipment when performing network inference, the CSP module combines the feature maps of each stage in the network through the channel splitting and fusion strategy, avoids a large amount of repeated gradient information, greatly reduces the calculation amount, and improves the inference calculation speed and the model training performance.

[0068] Therefore, the embodiment of the present application can use the encoder as the backbone network and use the CSP module as the main component of the entire backbone network to extract the main features of the input image data.

[0069] It should be noted that the above-mentioned specific components of the backbone network include four groups of convolution modules, four groups of CSP modules, one group of SPP (Spatial Pyramid Pooling) modules, and one group of SElayer (Squeeze-Excitation) modules.

[0070] Figure 4 An execution logic schematic diagram for a lane line detection model training process. As shown in Figure 4 , the specific process of lane line detection model training is described as follows:

[0071] S401, acquire an image, a lane line point set ground truth label, and perform data preprocessing;

[0072] S402, convert the point set ground truth label into an anchor box, and fine-tune the anchor box size;

[0073] S403, enter the first convolution module, and deliver the obtained feature map to the CSP module. This process needs to be repeated three times, as shown in Figure 5 , to perform feature extraction; the extracted feature map is input into the SPP module, and then the feature is further delivered to a CSP module, a convolution module, and an SElayer module, to complete the extraction of the entire feature;

[0074] S404, perform dimension conversion on the complete feature map, and input it into a fully connected layer. The result processed by the fully connected layer is subjected to dimension conversion twice again, so that the size of the feature map is equal to the size of the label generated by data preprocessing;

[0075] S405, activate the FocalLoss loss function, and perform back propagation to update the parameter values. The process is repeated until the training is completed;

[0076] S406, save the optimal model weight to perform subsequent online real-time lane line detection of vehicles and other related tasks.

[0077] Each convolution module is composed of convolution, batch normalization processing, and LeakyReLu activation function; the CSP module mainly includes two groups of convolution modules, a residual module, two groups of general convolution layers, batch normalization processing, and LeakyReLu activation function.

[0078] It should be noted that, in actual execution process, due to the lane line picture under the camera photo because of perspective, the feature of the lane line in the distance will become smaller. In the encoder processing process, the feature map may become smaller and even disappear after down-sampling by the convolution layer. Therefore, the model can also adopt the feature pyramid method to extract and retain small target features.

[0079] It can be understood that the embodiments of the present application adopt the modified SPP module to complete the above-mentioned step of extracting and retaining small target features, so that the local features and global features are fused at the feature map level, thereby greatly enriching the expression ability of the final output feature map, improving the performance of the model, and to a certain extent, reducing the loss of small features at the feature level after convolution.

[0080] The lane line detection method of the vehicle according to the embodiment of the present application comprises the following steps: obtaining a training image of the vehicle and a lane line point set true value label; converting the lane line point set true value label into a size of a preset anchor box to obtain an anchor box label; inputting the training image and the anchor box label into a lane line detection model to be trained to determine an optimal neural network weight learned in advance, and generating a lane line detection model based on the neural network trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether the anchor box has a lane line feature, and the lane line detection result is obtained by using the lane line detection model based on the neural network trained in advance. The lane line feature is processed based on the anchor box, the size of the anchor box is adjusted at the same time, the encoder and the decoder are combined to complete the detection of the horizontal line, the lane line detection model comprising only the classification network is trained, and therefore the requirements of the processing algorithm and the calculation amount are greatly reduced, the detection problem of the horizontal line is solved, the learning ability and the performance of the model are ensured, the generalization performance of the model is improved, and reliable technical support is provided for the realization of the real-time detection of the lane line.

[0081] Secondly, the lane line detection device of the vehicle according to the embodiment of the present application is described with reference to the accompanying drawings.

[0082] Figure 7 is a block schematic diagram of the lane line detection device of the vehicle according to the embodiment of the present application.

[0083] As shown in Figure 7 , the lane line detection device 10 of the vehicle is applied to an offline training stage, wherein the device comprises a first acquisition module 101, a conversion module 102 and a training module 103.

[0084] The first acquisition module 101 is configured to obtain a training image of the vehicle and a lane line point set true value label.

[0085] The conversion module 102 is configured to convert the lane line point set true value label into a size of a preset anchor box to obtain an anchor box label.

[0086] The training module 103 is configured to input the training image and the anchor box label into a lane line detection model to be trained, determine an optimal neural network weight learned in advance, and generate a lane line detection model based on the neural network trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether the anchor box has a lane line feature, and the lane line detection result is obtained by using the lane line detection model based on the neural network trained in advance.

[0087] Optionally, in an embodiment of the present application, the training module 103 comprises a processing unit, a dimension conversion unit and a generating unit.

[0088] The processing unit is configured to obtain a feature map by using a convolution module and a CSP module of the classification network for multiple times of extraction based on the training image and the anchor box label.

[0089] a dimension conversion unit configured to extract a complete feature map by using an SElayer module of the classification network based on the feature map, input the complete feature map to a fully connected layer of the classification network through the decoder, and perform dimension conversion to obtain a final feature map satisfying a preset condition.

[0090] a generation unit configured to perform loss function calculation and back propagation based on the final feature map until training is completed, and generate a pre-trained lane line detection model based on the neural network.

[0091] Optionally, in an embodiment of the present application, the conversion module 102 comprises a determination unit configured to determine an anchor box height according to an image height of the lane line point set ground truth label, and determine an anchor box width according to an image width of the lane line point set ground truth label.

[0092] It should be noted that the foregoing explanation and description of the embodiment of the vehicle lane line detection method also applies to the vehicle lane line detection device of this embodiment, which will not be described here again.

[0093] The vehicle lane line detection device according to the embodiment of the present application obtains a training image of a vehicle and a lane line point set ground truth label; converts the lane line point set ground truth label into a size of a preset anchor box to obtain an anchor box label; inputs the training image and the anchor box label into a lane line detection model to be trained to determine optimal neural network weights learned in advance, and generate a pre-trained lane line detection model based on the neural network, wherein the lane line detection model comprises a classification network for distinguishing whether an anchor box has a lane line feature, so as to obtain a lane line detection result by using the pre-trained lane line detection model based on the neural network. The present application processes the lane line feature based on the anchor box, adjusts the size of the anchor box, uses an encoder combined with a decoder to complete the detection of a horizontal line, trains the lane line detection model comprising only the classification network, thereby greatly reducing the requirements for processing algorithms and calculation amount, solving the problem of horizontal line detection, guaranteeing the learning ability and performance of the model, improving the generalization performance of the model, and providing reliable technical support for the realization of real-time lane line detection.

[0094] The above embodiment describes the offline training phase, and the following describes an embodiment of the online detection phase.

[0095] Figure 7 A flowchart of a vehicle lane line detection method provided by the embodiment of the present application is shown.

[0096] As shown in Figure 7 The vehicle lane line detection method is applied to the online detection phase, and the method comprises the following steps:

[0097] In step S701, a lane line image of the vehicle is acquired.

[0098] Embodiments of the present application can actively control the vehicle-mounted camera and other devices to acquire lane line image data after the user starts the lane line detection mode, and can analyze the image data acquired within a certain time period (e.g., 3 seconds) to obtain key frames with high image quality, or acquire image data by frame extraction and perform scaling and other preprocessing operations on the acquired images, thereby reducing the computational load of lane line detection and improving detection speed and effect.

[0099] In step S702, a plurality of anchor boxes are extracted from the lane line image, wherein each anchor box contains or does not contain lane line features.

[0100] After acquiring the lane line image data of the vehicle, further, embodiments of the present application can process the lane line image data based on anchor boxes, extract a plurality of anchor boxes, thereby clustering in the anchor boxes first, and then extract lane line features to detect lane lines in real time. Each anchor box does not necessarily contain lane line features, and all anchor boxes containing lane line features are 1, and those not containing lane line features are 0.

[0101] Thus, embodiments of the present application change the detection of lane lines from a task of dividing dense pixels to a classification task, greatly improving the real-time performance of model detection.

[0102] Optionally, in an embodiment of the present application, the lane line features include straight line features, curved lane line features, and transverse line features.

[0103] It should be noted that in embodiments of the present application, the lane line features include straight line features, curved lane line features, and transverse line features, thereby improving the accuracy and comprehensiveness of lane line detection, effectively solving the problems of transverse line detection and line quantity limitation.

[0104] In step S703, the lane line image is input into a pre-trained lane line detection model based on a neural network to perform inference based on the optimal neural network weight learned in advance, and output a lane line detection result, wherein the lane line detection model includes a classification network that distinguishes whether an anchor box has lane line features.

[0105] After extracting a plurality of anchor boxes from the lane line image, further, embodiments of the present application can input the above-mentioned lane line image into the trained lane line detection model, use the optimal weight parameter learned by the detection model in the training stage to perform inference, and thereby obtain the detection result output by the lane line detection model.

[0106] It can be understood that, since the lane line detection model only contains a classification module for distinguishing whether the anchor box has lane line features, the embodiment of the present application completely converts lane line segmentation into lane line binary classification, greatly reducing the calculation amount and algorithm requirements of the detection model.

[0107] According to the lane line detection method of the vehicle, the lane line image of the vehicle is obtained; a plurality of anchor boxes are extracted from the lane line image, wherein each anchor box contains or does not contain lane line features; the lane line image is input into a lane line detection model based on a neural network which is trained in advance, to perform inference based on the optimal neural network weight learned in advance, and output a lane line detection result, wherein the lane line detection model includes a classification network for distinguishing whether the anchor box has lane line features. The lane line detection is performed by using a lightweight and low-computing detection model, which improves the real-time performance of detection, effectively guarantees the accuracy and detection effect of the model detection, and greatly meets the use demand and use experience of users.

[0108] Secondly, the lane line detection device of the vehicle according to the embodiment of the present application is described with reference to the accompanying drawings.

[0109] Figure 8 is a block schematic diagram of the lane line detection device of the vehicle according to the embodiment of the present application.

[0110] As shown in Figure 8 , the lane line detection device 20 of the vehicle is applied to an online detection stage, wherein the device includes a second acquisition module 201, an extraction module 202 and a detection module 203.

[0111] The second acquisition module 201 is configured to acquire the lane line image of the vehicle.

[0112] The extraction module 202 is configured to extract a plurality of anchor boxes from the lane line image, wherein each anchor box contains or does not contain lane line features.

[0113] The detection module 203 is configured to input the lane line image into a lane line detection model based on a neural network which is trained in advance, to perform inference based on the optimal neural network weight learned in advance, and output a lane line detection result, wherein the lane line detection model includes a classification network for distinguishing whether the anchor box has lane line features.

[0114] Optionally, in an embodiment of the present application, the lane line features include straight line features, curved lane line features and transverse line features.

[0115] It should be noted that the above explanation and description of the lane line detection method of the vehicle also apply to the lane line detection device of the vehicle, which will not be described here.

[0116] According to the lane line detection device of the vehicle provided by the embodiment of the application, the lane line image of the vehicle is acquired; a plurality of anchor boxes are extracted from the lane line image, wherein each anchor box contains or does not contain lane line features; the lane line image is input into a lane line detection model based on a neural network which is pre-trained, to perform inference based on pre-learned optimal neural network weights, and output a lane line detection result, wherein the lane line detection model comprises a classification network for distinguishing whether an anchor box contains lane line features. The lane line detection is performed by using a light-weight and low-computing detection model, which improves the real-time performance of detection, effectively guarantees the accuracy and detection effect of model detection, and greatly meets the use demand and use experience of users.

[0117] Figure 9 The vehicle provided by the embodiment of the application is shown in the structural schematic diagram. The vehicle can include:

[0118] The memory 901, the processor 902 and the computer program stored in the memory 901 and executable on the processor 902.

[0119] The processor 902 implements the lane line detection method of the vehicle provided in the above embodiment when executing the program.

[0120] Further, the vehicle further includes:

[0121] The communication interface 903 is used for communication between the memory 901 and the processor 902.

[0122] The memory 901 is used for storing the computer program executable on the processor 902.

[0123] The memory 901 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0124] If the memory 901, the processor 902 and the communication interface 903 are independently implemented, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 9 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0125] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can complete mutual communication through an internal interface.

[0126] The processor 902 can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0127] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the lane line detection method of the vehicle.

[0128] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0129] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one feature. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0130] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and that the various embodiments of the preferred implementation include the use of hardware and software configured to implement the functions or steps described in the illustrated or discussed order, including the use of modules or segments of code that are executed in substantially simultaneous or in reverse order, as will be appreciated by those skilled in the art. The descriptions of the various embodiments of the present application have been presented for the purpose of illustration and description and are not intended to be exhaustive or to limit the application to the precise forms disclosed. Persons skilled in the relevant art will recognize that many modifications, changes, substitutions, and alterations can be made without departing from the scope of the application. It is therefore intended that the application not be limited, except by the appended claims.

[0131] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy, flexible or other), a machine-readable storage card (e.g., ROM, EEPROM, flash memory or other), a machine- readable storage tape (e.g., magnetic, optical or other), a machine-readable storage medium (e.g., a portable electronic device, a computer diskette, a computer memory, a programmable logic device, an application specific integrated circuit, a silicon memory, or other), or a machine-readable interface device (e.g., a communications link, a modem, a wired or wireless network card, a wireless communication link, or other). The computer-readable medium can also be, or be included in, a transitory signal, such as an electrical, electromagnetic or optical signal, a sound or audio signal, or other. The computer-readable medium can be a paper or other physical medium that can have the program printed thereon, as the program can be electronically captured, via optical scanning for example, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and stored in a computer memory.

[0132] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, any of the following technologies, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0133] Those of skill in the art could readily implement the above described example methods with all or a subset of the recited steps carried out with a program of instructions executed by associated hardware, the program of instructions being stored in a computer readable storage medium that, when executed, includes one or a combination of the steps of the example methods.

[0134] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0135] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A lane line detection method of a vehicle, characterized by, The method is applied to an offline training phase, and the method comprises the following steps: obtaining a training image of a vehicle and a lane line point set true value label; converting the lane line point set true value label into a size of a preset anchor box to obtain an anchor box label; inputting the training image and the anchor box label into a lane line detection model to be trained to determine optimal neural network weights learned in advance, and generate a lane line detection model based on a neural network and trained in advance, wherein the lane line detection model comprises a classification network for distinguishing whether an anchor box has a lane line feature, and the lane line detection model based on the neural network and trained in advance is used to obtain a lane line detection result; the inputting the training image and the anchor box label into the lane line detection model to be trained to determine the optimal neural network weights learned in advance, and generate the lane line detection model based on the neural network and trained in advance comprises: based on the training image and the anchor box label, a feature map is obtained through multiple extractions by using a convolution module and a CSP module of the classification network; based on the feature map, a complete feature map is extracted by using an SElayer module of the classification network, the complete feature map is input into a full connection layer of the classification network through a decoder, and dimension conversion is performed to obtain a final feature map meeting a preset condition; loss function calculation and back propagation are performed based on the final feature map until training is completed, and the lane line detection model based on the neural network and trained in advance is generated.

2. The method of claim 1, wherein, the converting the lane line point set true value label into the size of the preset anchor box to obtain the anchor box label comprises: an anchor box height is determined according to an image height of the lane line point set true value label, and an anchor box width is determined according to an image width of the lane line point set true value label.

3. A lane line detection method of a vehicle, characterized by, The method is applied to an online detection phase, and the method comprises the following steps: obtaining a lane line image of a vehicle; extracting a plurality of anchor boxes from the lane line image, wherein each anchor box contains or does not contain a lane line feature; and inputting the lane line image into a lane line detection model based on a neural network and trained in advance to perform reasoning based on optimal neural network weights learned in advance, and output a lane line detection result, wherein the lane line detection model comprises a classification network for distinguishing whether an anchor box has the lane line feature; based on the training image and the anchor box label, a feature map is obtained through multiple extractions by using a convolution module and a CSP module of the classification network; based on the feature map, a complete feature map is extracted by using an SElayer module of the classification network, the complete feature map is input into a full connection layer of the classification network through a decoder, and dimension conversion is performed to obtain a final feature map meeting a preset condition; loss function calculation and back propagation are performed based on the final feature map until training is completed, and the lane line detection model based on the neural network and trained in advance is generated.

4. The method of claim 3, wherein, The lane line feature comprises a straight line feature, a curved lane line feature and a transverse line feature.

5. A lane line detection device for a vehicle, characterized by The device is applied to an offline training phase, and the device comprises: a first obtaining module configured to obtain a training image of a vehicle and a lane line point set true value label; The conversion module is configured to convert the lane line point set ground truth label into a size of a preset anchor box to obtain an anchor box label. The training module is configured to input the training image and the anchor box label into a lane line detection model to be trained, determine pre-learned optimal neural network weights, and generate a pre-trained neural network-based lane line detection model. The training module includes: The processing unit is configured to use a convolution module and a CSP module of the classification network to extract a feature map through multiple times of extraction based on the training image and the anchor box label. The dimension conversion unit is configured to use an SElayer module of the classification network to extract a complete feature map based on the feature map, input the complete feature map to a fully connected layer of the classification network through a decoder, and perform dimension conversion to obtain a final feature map that meets a preset condition. The generation unit is configured to perform loss function calculation and back propagation based on the final feature map until training is completed to generate the pre-trained neural network-based lane line detection model.

6. The apparatus of claim 5, wherein, The conversion module includes: The determination unit is configured to determine an anchor box height according to an image height of the lane line point set ground truth label, and determine an anchor box width according to an image width of the lane line point set ground truth label.

7. A lane line detection device for a vehicle, characterized by comprising: The device is applied to an online detection stage, and includes: The second acquisition module is configured to acquire a lane line image of a vehicle. The extraction module is configured to extract a plurality of anchor boxes from the lane line image, wherein each anchor box contains or does not contain a lane line feature. The detection module is configured to input the lane line image into a pre-trained neural network-based lane line detection model to perform inference based on pre-learned optimal neural network weights, and output a lane line detection result, wherein the lane line detection model includes a classification network that distinguishes whether an anchor box has the lane line feature. The processing unit is configured to use a convolution module and a CSP module of the classification network to extract a feature map through multiple times of extraction based on the training image and the anchor box label. The dimension conversion unit is configured to use an SElayer module of the classification network to extract a complete feature map based on the feature map, input the complete feature map to a fully connected layer of the classification network through a decoder, and perform dimension conversion to obtain a final feature map that meets a preset condition. The generation unit is configured to perform loss function calculation and back propagation based on the final feature map until training is completed to generate the pre-trained neural network-based lane line detection model.

8. The apparatus of claim 7, wherein, The lane line feature includes a straight line feature, a curved lane line feature, and a transverse line feature.

9. A vehicle characterized by comprising: The computer program is stored in the memory and executable on the processor, and the processor executes the program to implement the lane line detection method of the vehicle according to any one of claims 1-4. The program is executed by the processor to implement the lane line detection method of the vehicle according to any one of claims 1-4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images