Lane line detection method and device and related equipment

Abstract

The invention provides a lane line detection method and device and related equipment, and the method comprises the steps: carrying out the feature extraction of an obtained road condition image, and obtaining a feature extraction result; based on a feature extraction result, using a semantic segmentation model to perform semantic segmentation on the road condition image to obtain a semantic segmentation result, using a key point feature extraction network model to extract lane line key points, and obtaining a lane line key point extraction result; based on the semantic segmentation result, lane line identification is carried out, and a fitting lane line is obtained; and based on the lane line key point extraction result, correcting the fitting lane line to obtain a detection result of the target lane line. And when the semantic segmentation result has an error or a large error, the lane line is corrected according to the key point extraction result. And since the neural network models for semantic segmentation and key extraction share one feature extraction layer, the lane line detection precision is improved while the lane line extraction efficiency is ensured.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a lane line detection method, device and related equipment. Background Art

[0002] With the development of autonomous driving technology, people have higher and higher requirements for the accuracy of vehicle road detection. In vehicle road detection, lane line detection is one of the most important detection indicators in the field of autonomous driving technology. Existing lane line detection methods are mostly based on traditional image processing and deep learning neural network models to perform semantic segmentation on the collected images, perform target recognition based on the results of semantic segmentation, identify the pixels about the lane line and the key points of the lane line, and then determine the target lane line based on the pixels and key points of the lane line. This traditional lane line detection method does not necessarily have good accuracy because the neural network model for semantic segmentation performs semantic segmentation on different targets to obtain the results of different recognition objects. If any semantic recognition result in the lane line pixel extraction or key point recognition results is wrong, it is difficult to ensure the accuracy of the target lane line finally obtained. For this reason, how to provide a lane line detection method with high accuracy has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] In view of this, embodiments of the present application provide a lane line detection method, device and storage medium to at least partially solve the above-mentioned problems.

[0004] In a first aspect, an embodiment of the present application provides a lane line detection method, comprising:

[0005] Performing feature extraction on the acquired road condition image to obtain feature extraction results;

[0006] Based on the feature extraction results, the semantic segmentation model is used to perform semantic segmentation on the road condition image to obtain the semantic segmentation results, and the key points of the lane line are extracted using the key point feature extraction network model to obtain the lane line key point extraction results;

[0007] Based on the semantic segmentation result, lane line recognition is performed to obtain fitted lane lines;

[0008] Based on the lane line key point extraction result, the fitted lane line is corrected to obtain the detection result of the target lane line.

[0009] Optionally, in an embodiment of the present application, the step of performing feature extraction on the acquired road condition image to obtain a feature extraction result includes:

[0010] Cropping the road condition image to remove the non-lane line portion of the road condition image, and obtaining a cropped image that is at most 1 / 8 of the size of the road condition image;

[0011] According to the spatial relationship between the rows and columns of pixels in the cropped image, pixel superposition is performed through spatial feature aggregation to enhance the features in the cropped image and obtain a feature-enhanced image;

[0012] Perform feature extraction on the feature-enhanced image to obtain a feature extraction result.

[0013] Optionally, in an embodiment of the present application, performing feature extraction on the feature-enhanced image to obtain a feature extraction result includes:

[0014] Based on the feature extraction neural network model, the feature enhanced image is downsampled at least three times by using maximum pooling and 3*3 convolution with a step size of 2 to determine the sampling result;

[0015] Connecting the sampling results to obtain a feature-connected image;

[0016] Using the dilated convolution of the feature extraction neural network model, the receptive field of the feature connection image is enlarged to obtain a feature enlarged image;

[0017] Based on the feature extraction neural network model, a convolution kernel with a preset step size is used to perform feature extraction on the enlarged image to obtain a feature extraction result.

[0018] Optionally, in an embodiment of the present application, the method further includes: based on the size of the cropped image, upsampling the feature enhanced image to a preset output size.

[0019] Optionally, in one embodiment of the present application, upsampling the feature-enhanced image to a preset output size based on the size of the cropped image includes: using a single-branch transposed convolutional neural network model to upsample the feature-enhanced image to a preset output size.

[0020] Optionally, in an embodiment of the present application, the step of extracting lane key points using a key point feature extraction network model and obtaining lane key point extraction results includes:

[0021] Based on the feature extraction result, the trained key point feature extraction network is used to extract lane key points from three dimensions: key point confidence, offset information, and embedded features, so as to obtain a lane key point extraction result;

[0022] The confidence and offset information are used to locate and detect the lane key points, and the embedded features are used to classify the lane key points in post-processing.

[0023] Optionally, in an embodiment of the present application, the step of correcting the fitted lane line based on the lane line key point extraction result to obtain the target lane line detection result includes:

[0024] Performing logical judgment on the fitted lane line;

[0025] When the result of the detection of the logic judgment indicates that there is a break in the fitted lane line, using a virtual lane line determined according to the lane line key point extraction result to correct the fitted lane line;

[0026] According to the correction result, the detection result of the target lane line is obtained.

[0027] Optionally, in one embodiment of the present application, the semantic segmentation model and the key point feature extraction model use the same or the same feature extraction layer.

[0028] In a second aspect, based on the lane line detection method described in the first aspect of the present application, the present application further provides a lane line detection device, including:

[0029] An extraction module is used to extract features from the acquired road condition image and obtain feature extraction results;

[0030] A recognition module, configured to perform semantic segmentation on the road condition image based on the feature extraction result using a semantic segmentation model to obtain a semantic segmentation result, extract lane line key points using a key point feature extraction network model to obtain a lane line key point extraction result, wherein the semantic segmentation model and the key point feature extraction model use the same or the same feature extraction layer;

[0031] A fitting module, used to perform lane line recognition based on the semantic segmentation result and obtain a fitted lane line;

[0032] The determination module is used to correct the fitted lane line based on the lane line key point extraction result to obtain the detection result of the target lane line.

[0033] In a third aspect, an embodiment of the present application further provides a computer storage medium having computer executable instructions stored thereon, and when the computer executable instructions are executed, any lane line detection method as described in the first aspect of the present application is performed.

[0034] In a fourth aspect, an embodiment of the present application further provides an electronic device, including:

[0035] one or more processors;

[0036] a memory for storing one or more programs,

[0037] When the one or more programs are executed by the one or more processors, the one or more processors implement any lane line detection method described in the first aspect of the present application.

[0038] The present application provides a lane line detection method, device and related equipment, which extract features from the acquired road condition image to obtain feature extraction results; based on the feature extraction results, use a semantic segmentation model to semantically segment the road condition image to obtain semantic segmentation results, use a key point feature extraction network model to extract lane line key points, and obtain lane line key point extraction results; based on the semantic segmentation results, perform lane line recognition to obtain fitted lane lines; based on the lane line key point extraction results, correct the fitted lane lines to obtain the detection results of the target lane lines. In this lane construction detection method, the final lane line is determined based on the results of semantic recognition and key point detection. When the result of semantic segmentation is wrong or the error is large, the key point detection result is used as another layer of guarantee to correct the wrong lane line. And because the neural network model for semantic segmentation and key extraction shares a feature extraction layer, the lane line extraction efficiency is guaranteed while the lane line detection accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A schematic diagram of a workflow of a lane line detection method provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of a lane detection device provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0044] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0045] Embodiment 1

[0046] The present application embodiment provides a lane line detection method, which is applied to manage the data of a distributed blockchain application system, such as Figure 1 As shown, Figure 1 A schematic diagram of a lane detection method provided in an embodiment of the present application includes:

[0047] Step S101, extract features from the acquired road condition image to obtain feature extraction results. In the embodiment of the present application, feature extraction is performed on the road condition image, specifically, the acquired road condition image is input into an ENCODER for feature extraction to obtain features about lane lines in the road condition image, thereby improving the recognition efficiency and basic accuracy of lane lines.

[0048] Optionally, in an optional implementation of the embodiment of the present application, the feature extraction of the acquired road condition image and the acquisition of the feature extraction result include: cropping the road condition image, removing the image of the non-lane line part of the road condition image, and acquiring a cropped image of at most 1 / 8 of the size of the road condition image; performing pixel superposition through spatial feature aggregation according to the spatial relationship between pixel rows and columns in the cropped image to enhance the features in the cropped image and acquire a feature enhanced image; and performing feature extraction on the feature enhanced image to acquire a feature extraction result. In the actual application scenario of the present application, the road condition image acquired when performing lane line detection is generally acquired through the front camera mounted on the vehicle. The image size acquired by such a front camera is generally large, and most of the images also include a lot of image data unrelated to the lane line. If feature extraction is performed directly on it, it will undoubtedly take a long time and hardware resources. And when feature extraction is performed, it is also necessary to build a model that can adapt to a larger input size, which is easy to lose the information contained in the image, and it is difficult to guarantee the accuracy of the feature extraction results, and the extraction efficiency is poor. Lane line detection is generally used in real-time scenarios, and its actual business scenarios require higher detection efficiency and accuracy. Therefore, in order to improve the efficiency of lane line detection, the embodiment of the present application first crops the acquired road condition image, such as removing the upper half of the image, such as the sky. Of course, it is not limited to this, and useless information images of the vehicle body and other parts can also be removed. It can reduce the efficiency of feature extraction and improve the accuracy of feature extraction. Furthermore, in the embodiment of the present application, when performing feature extraction, since the lane line is an image target with a continuous and elongated strong shape, it is easily blocked by vehicles on the road or insufficient light. Therefore, in order to ensure the accuracy of the lane line, the embodiment of the present application adopts a spatial information transmission method based on the strong shape prior of the lane line to capture the relationship between different rows and columns in the cropped image, such as the gray value relationship. Specifically, the cropped image is used as a feature map, and information is transmitted in parallel in four directions: left, right, up, and down, based on the feature pixel rows and columns in the feature map. In this way, more comprehensive lane line features are improved, and this process of image processing does not consume too much hardware resources for image processing. Through the process of spatial information transmission, one pixel can be superimposed on another pixel multiple times to determine the blocked lane line. Thereby reducing the probability of the lane line being blocked and undetectable.

[0049] Optionally, in an embodiment of the present application, the feature extraction of the feature enhanced image to obtain the feature extraction result includes: based on the feature extraction neural network model, using maximum pooling and 3*3 convolution with a step size of 2 to perform at least three downsampling on the feature enhanced image, and determining the sampling result; connecting the sampling results to obtain a feature connection image; using the hole convolution of the feature extraction neural network model to increase the receptive field of the feature connection image and obtain a feature enlarged image; based on the feature extraction neural network model, using a convolution kernel with a preset step size, extracting features from the enlarged image to obtain a feature extraction result. In the actual application scenario of the embodiment of the present application, when performing feature extraction, three downsamplings are performed based on the feature extraction neural network model, while ensuring that the feature extraction accuracy meets the recognition requirements, with a relatively low data processing amount. Due to the strong shape prior features of the lane line, the maximum pooling and 3*3 convolution with a step size of 2 are used to perform downsampling respectively, and then they are connected together to ensure that the features in the feature map have more comprehensive and clear details. Using hole convolution to increase the receptive field makes the information in the feature map richer. Finally, based on the size (resolution) of the cropped image, a convolution kernel with a preset step size is set for feature extraction to better extract the detailed features of the lane lines.

[0050] Optionally, in the implementation of the embodiment of the present application, the method further includes: based on the size of the cropped image, upsampling the feature-enhanced image to a preset output size, so that the features in the image are more delicate, and it is also beneficial to the efficiency of later projecting the lane line detection results into the acquired road condition image.

[0051] Optionally, in an implementation of an embodiment of the present application, based on the size of the cropped image, the feature-enhanced image is upsampled to a preset output size, including: based on the size ratio between the cropped image and the road condition image, a single transposed convolutional neural network model is used to upsample the feature-enhanced image to a preset output size. If the cropped image is 1 / 8 of the road condition image, then the feature-enhanced image can be upsampled to 1 / 2 of the scale of the input road condition image. Of course, the embodiment of the present application is only used here to illustrate this process as an example, and does not mean that the present application is limited to this.

[0052] Step S102: Based on the result of feature extraction, the semantic segmentation model is used to perform semantic segmentation on the road condition image to obtain the semantic segmentation result, and the key point feature extraction network model is used to extract the lane line key points to obtain the lane line key point extraction result. In the embodiment of the present application, the lane lines in the extracted features are identified by semantic segmentation, and the key points of the lane lines are extracted by the key point feature extraction network model. The output results of the two models do not affect each other, so that the final lane lines determined according to the semantic recognition and key points have better accuracy, and the possibility of the final detected lane line results being inaccurate due to errors in the detection of any neural network model is reduced.

[0053] Optionally, in an implementation of the embodiment of the present application, the key point feature extraction network model is used to extract lane key points and obtain lane key point extraction results, including: based on the feature extraction results, the trained key point feature extraction network is used to extract lane key points from three dimensions: key point confidence, offset information, and embedded features, to obtain lane key point extraction results; wherein the confidence and offset information are used to locate and detect the lane key points, and the embedded features are used to classify lane key points in post-processing, and the classification includes such as solid white line, dashed white line, solid yellow line, dashed yellow line, etc. Through this key point detection, the target lane line finally determined is more accurate.

[0054] Optionally, in one implementation of the embodiment of the present application, the semantic segmentation model and the key point feature extraction model use the same or the same feature extraction layer. By sharing the same or the same feature extraction layer, that is, the backbone network of the semantic segmentation model and the key point feature extraction model used in the embodiment of the present application is the same, thereby reducing the time and resource costs required for the construction of the two neural network models while ensuring that their respective output results do not affect each other, and their respective data processing processes will not consume too much time.

[0055] Step S103: Based on the semantic segmentation result, lane line recognition is performed to obtain fitted lane lines. In the embodiment of the present application, target fitting of the semantically recognized lane lines is performed by extracting pixel points about the lane lines in the semantic segmentation result for target fitting. The process is easy to implement and has a low probability of error.

[0056] In an optional implementation of the present application, when the clarity of the acquired road condition image is greater than the preset clarity, such as when there is no obstruction in the current road condition image, the fitted lane line acquired in the above step S103 can be directly projected into the acquired road condition image, so that when the data processing volume is small and the lane line is easy to identify, the user can quickly know the lane line detection result, thereby improving the feedback efficiency of lane line detection in actual application scenarios, so as to improve the user's experience.

[0057] Step S104: Based on the lane line key point extraction result, the fitted lane line is corrected to obtain the detection result of the target lane line. Specifically, based on the positioning information, confidence, etc. of the lane line key point extraction result, the fitted lane line determined based on the semantic recognition result is corrected to obtain a high-precision detection result of the target lane line. Specifically, in this process, the concept of weight can be introduced, based on the positioning information in the lane line key point extraction result, and different weights are assigned based on the confidence, combined with the classification of lane line key points with different position information, to correct the fitted lane line to obtain a high-precision lane line detection result. Make the determined target lane line information more accurate and complete.

[0058] Optionally, in an implementation of the embodiment of the present application, based on the lane line key point extraction result, the fitting lane line is corrected to obtain the detection result of the target lane line, including: performing a logical judgment on the fitting lane line; when the result of the logical judgment indicates that there is a break in the fitting lane line, using a virtual lane line determined according to the lane line key point extraction result, correcting the result of the fitting lane line; and obtaining the detection result of the target lane line according to the correction result. Thereby improving the accuracy of the lane line detection result when the lane line is blocked during lane line detection.

[0059] Optionally, in an embodiment of the present application, the lane line detection method further includes: back-projecting the detection result of the target lane line into the road condition image to generate a lane line detection image for feedback to the user, so that the user can obtain the current road information in time and improve the user's driving experience.

[0060] The present application provides a lane line detection method, which performs feature extraction on the acquired road condition image to obtain feature extraction results; based on the feature extraction results, uses a semantic segmentation model to perform semantic segmentation on the road condition image to obtain semantic segmentation results, uses a key point feature extraction network model to extract lane line key points, and obtains lane line key point extraction results; based on the semantic segmentation results, performs lane line recognition to obtain fitted lane lines; based on the lane line key point extraction results, corrects the fitted lane lines to obtain the detection results of the target lane lines. In this lane construction detection method, the final lane line is determined based on the results of semantic recognition and key point detection. When the result of semantic segmentation is wrong or the error is large, the key point detection result is used as another layer of guarantee to correct the wrong lane line. And because the neural network model for semantic segmentation and key extraction shares a feature extraction layer, the lane line extraction efficiency is guaranteed while the lane line detection accuracy is improved.

[0061] Embodiment 2

[0062] Based on the lane line detection method described in the first embodiment of the present application, the present embodiment also provides a lane line detection method 20, such as Figure 2 As shown, Figure 2 A schematic diagram of the structure of a lane line detection device 20 provided in an embodiment of the present application, the lane line detection device 20 includes:

[0063] The extraction module 201 is used to extract features from the acquired road condition image and obtain feature extraction results;

[0064] The recognition module 202 is used to perform semantic segmentation on the road condition image based on the feature extraction result using the semantic segmentation model to obtain the semantic segmentation result, and to extract the lane line key points using the key point feature extraction network model to obtain the lane line key point extraction result;

[0065] Optionally, in an implementation of the embodiment of the present application, the semantic segmentation model and the key point feature extraction model use the same or the same feature extraction layer;

[0066] A fitting module 203, used to perform lane line recognition based on the semantic segmentation result and obtain a fitted lane line;

[0067] The determination module 204 is used to correct the fitted lane line based on the lane line key point extraction result to obtain the detection result of the target lane line.

[0068] Optionally, in one implementation of the embodiment of the present application, the extraction module 201 is also used to crop the road condition image, remove the image of the non-lane line portion of the road condition image, and obtain a cropped image that is at most 1 / 8 the size of the road condition image; based on the spatial relationship between pixel rows and columns in the cropped image, perform pixel superposition through spatial feature aggregation to enhance the features in the cropped image and obtain a feature-enhanced image; perform feature extraction on the feature-enhanced image to obtain a feature extraction result.

[0069] Optionally, in one implementation of the embodiment of the present application, the extraction module 201 is also used to: based on the feature extraction neural network model, use maximum pooling and 3*3 convolution with a step size of 2 to downsample the feature enhanced image at least three times respectively to determine the sampling results; connect the sampling results to obtain a feature connected image; use the hole convolution of the feature extraction neural network model to increase the receptive field of the feature connected image to obtain a feature enlarged image; based on the feature extraction neural network model, use a convolution kernel with a preset step size to perform feature extraction on the enlarged image to obtain a feature extraction result.

[0070] Optionally, in an embodiment of the present application, the lane line extraction device further includes a restoration module (not shown in the drawings), and the restoration module is used to: based on the size of the cropped image, upsample the feature enhanced image to a preset output size.

[0071] Optionally, in one embodiment of the present application, the restoration module is used to upsample the feature-enhanced image to a preset output size using a single transposed convolutional neural network model based on the size of the cropped image.

[0072] Optionally, in an implementation of the embodiment of the present application, the extraction module 201 is also used to: based on the feature extraction result, use the trained key point feature extraction network to perform lane line key point extraction from three dimensions of key point confidence, offset information and embedded features to obtain lane line key point extraction results; wherein the confidence and offset information are used to locate and detect the lane line key points, and the embedded features are used to classify the lane line key points in post-processing.

[0073] Optionally, in an implementation of the embodiment of the present application, the determination module 204 is also used to perform logical judgment on the fitted lane line; when the result of the detection of the logical judgment indicates that there is a break in the fitted lane line, the result of the fitted lane line is corrected using a virtual lane line determined according to the result of the lane line key point extraction; and based on the result of the correction, the detection result of the target lane line is obtained.

[0074] Optionally, in one embodiment of the present application, the lane line detection device 20 also includes a projection module (not shown in the drawings), which is used to back-project the detection result of the target lane line into the road condition image to generate a lane line detection image for feedback to the user.

[0075] The present application provides a lane line detection device, which is provided with an extraction module to perform feature extraction on the acquired road condition image, obtain the feature extraction result, and use the semantic segmentation model to perform semantic segmentation on the road condition image based on the feature extraction result of the recognition module to obtain the semantic segmentation result, and use the key point feature extraction network model to extract the lane line key points to obtain the lane line key point extraction result; the fitting module is provided to perform lane line recognition based on the semantic segmentation result to obtain the fitted lane line; the correction module is provided to correct the fitted lane line based on the lane line key point extraction result to obtain the detection result of the target lane line. In this lane construction detection device, since the final lane line is determined based on the results of semantic recognition and key point detection, when the result of semantic segmentation is wrong or the error is large, the key point detection result is used as another layer of guarantee to correct the wrong lane line. And since the neural network model for semantic segmentation and key extraction shares a feature extraction layer, the lane line extraction efficiency is guaranteed while the lane line detection accuracy is improved.

[0076] Embodiment 3

[0077] The present application also provides a storage medium on which a computer program is stored. When the program is executed by a processor, any lane line detection method as described in the first embodiment of the present application is implemented. The lane line detection method includes but is not limited to:

[0078] Performing feature extraction on the acquired road condition image to obtain feature extraction results;

[0079] Based on the feature extraction results, the semantic segmentation model is used to perform semantic segmentation on the road condition image to obtain the semantic segmentation results, and the key points of the lane line are extracted using the key point feature extraction network model to obtain the lane line key point extraction results;

[0080] Based on the semantic segmentation result, lane line recognition is performed to obtain fitted lane lines;

[0081] Based on the lane line key point extraction result, the fitted lane line is corrected to obtain the detection result of the target lane line.

[0082] Embodiment 4:

[0083] The present application also provides an electronic device, such as Figure 3 As shown, the electronic device includes:

[0084] One or more processors 301;

[0085] The memory 303 is used to store one or more programs.

[0086] When the one or more programs are instructed to be executed by the one or more processors 301 through the communication bus 304 based on the operation command received through the communication interface 302, the one or more processors 301 implement the lane line detection method as described in any one of the above embodiments. The lane line detection method includes but is not limited to:

[0087] Performing feature extraction on the acquired road condition image to obtain feature extraction results;

[0088] Based on the feature extraction results, the semantic segmentation model is used to perform semantic segmentation on the road condition image to obtain the semantic segmentation results, and the key points of the lane line are extracted using the key point feature extraction network model to obtain the lane line key point extraction results;

[0089] Based on the semantic segmentation result, lane line recognition is performed to obtain fitted lane lines;

[0090] Based on the lane line key point extraction result, the fitted lane line is corrected to obtain the detection result of the target lane line. So far, this application has described specific embodiments of the present subject matter. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0091] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system layer on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0092] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0093] The system layers, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0094] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, system layers, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0097] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0098] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system layer embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0099] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A lane line detection method, characterized in that: include: Performing feature extraction on the acquired road condition image to obtain feature extraction results; Based on the feature extraction results, the semantic segmentation model is used to perform semantic segmentation on the road condition image to obtain the semantic segmentation results, and the key points of the lane line are extracted using the key point feature extraction network model to obtain the lane line key point extraction results; Based on the semantic segmentation result, lane line recognition is performed to obtain fitted lane lines; Based on the lane line key point extraction result, the fitted lane line is corrected to obtain the detection result of the target lane line.

2. The lane line detection method according to claim 1, characterized in that: The step of extracting features from the acquired road condition image and obtaining feature extraction results includes: Cropping the road condition image to remove the non-lane line portion of the road condition image, and obtaining a cropped image that is at most 1 / 8 of the size of the road condition image; According to the spatial relationship between the rows and columns of pixels in the cropped image, pixel superposition is performed through spatial feature aggregation to enhance the features in the cropped image and obtain a feature-enhanced image; Perform feature extraction on the feature-enhanced image to obtain a feature extraction result.

3. The lane line detection method according to claim 2, characterized in that: The step of performing feature extraction on the feature enhanced image to obtain a feature extraction result includes: Based on the feature extraction neural network model, the feature enhanced image is downsampled at least three times by using maximum pooling and 3*3 convolution with a step size of 2 to determine the sampling result; Connecting the sampling results to obtain a feature-connected image; Using the dilated convolution of the feature extraction neural network model, the receptive field of the feature connection image is increased to obtain a feature enlarged image; Based on the feature extraction neural network model, a convolution kernel with a preset step size is used to perform feature extraction on the enlarged image to obtain a feature extraction result.

4. The lane line detection method according to claim 2, characterized in that: The method further includes upsampling the feature enhanced image to a preset output size based on the size of the cropped image.

5. The lane line detection method according to claim 4, characterized in that: The method of upsampling the feature-enhanced image to a preset output size based on the size of the cropped image includes: using a single transposed convolutional neural network model to upsampling the feature-enhanced image to a preset output size.

6. The lane line detection method according to claim 4, characterized in that: The method of extracting lane key points using a key point feature extraction network model and obtaining lane key point extraction results includes: Based on the feature extraction result, the trained key point feature extraction network is used to extract lane key points from three dimensions: key point confidence, offset information, and embedded features, so as to obtain a lane key point extraction result; The confidence and offset information are used to locate and detect the lane key points, and the embedded features are used to classify the lane key points in post-processing.

7. The lane line detection method according to claim 1, characterized in that: The step of correcting the fitted lane line based on the lane line key point extraction result to obtain the target lane line detection result includes: Performing logical judgment on the fitted lane line; When the result of the detection of the logic judgment indicates that there is a break in the fitted lane line, using a virtual lane line determined according to the lane line key point extraction result to correct the fitted lane line; According to the correction result, the detection result of the target lane line is obtained.

8. The lane line detection method according to claim 1, characterized in that: The semantic segmentation model and the key point feature extraction model use the same or the same feature extraction layer.

9. A lane line detection device, characterized in that: include: An extraction module, used to perform feature extraction on the acquired road condition image and obtain feature extraction results; The recognition module is used to perform semantic segmentation on the road condition image based on the feature extraction result using the semantic segmentation model to obtain the semantic segmentation result, and to extract the lane line key points using the key point feature extraction network model to obtain the lane line key point extraction result; A fitting module, used to perform lane line recognition based on the semantic segmentation result and obtain a fitted lane line; The determination module is used to correct the fitted lane line based on the lane line key point extraction result to obtain the detection result of the target lane line.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the lane line detection method as described in any one of claims 1-8 is performed.

Images