Lane line detection method, device, electronic device and storage medium

Abstract

The present invention discloses a lane line detection method, device, electronic device and storage medium, including: obtaining network features extracted by a neural network under a specified sampling multiple of an image to be detected, and manual features extracted by a feature extraction operator; fusing the network features and manual features obtained under the specified sampling multiple to obtain fused features; and obtaining lane lines contained in the image to be detected based on the fused features. The network features containing core information in the image to be detected are extracted by a neural network based on a small amount of calculation, and the manual features extracted by the feature extraction operator are supplemented, and the lane line is detected according to the obtained fused features, thereby reducing the amount of calculation while ensuring the accuracy of the detection result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a lane line detection method, device, electronic device and storage medium. Background Art

[0002] Lane lines are an important basis for local lateral positioning of vehicles on the road, and play an important role in the positioning of intelligent driving vehicles, especially when the global positioning information is unstable or even lost. Therefore, a stable and efficient lane line detection algorithm is crucial for intelligent driving vehicles or advanced driver assistance systems. The lane line detection algorithms currently used are mainly divided into detection algorithms based on traditional artificial feature extraction and detection algorithms based on deep learning.

[0003] However, the detection algorithm that only uses artificial feature extraction is often difficult to adapt to complex environmental change scenarios due to the relatively simple hand-designed feature extractors and rule generation algorithms, resulting in low detection accuracy. The deep learning algorithm that only uses the algorithm has a huge number of algorithm parameters and complex algorithm calculations, resulting in low calculation speed. Therefore, the existing lane line detection method cannot meet user needs. Summary of the invention

[0004] The embodiments of the present invention provide a lane line detection method, device, electronic device and storage medium to achieve lane line detection.

[0005] In a first aspect, an embodiment of the present invention provides a lane line detection method, including:

[0006] Obtain the network features extracted by the neural network under the specified sampling multiples of the image to be detected, as well as the manual features extracted by the feature extraction operator;

[0007] The network features and manual features obtained under the specified sampling multiples are fused to obtain fused features;

[0008] The lane lines contained in the image to be detected are obtained based on the fusion features.

[0009] In a second aspect, an embodiment of the present invention provides a lane line detection device, including:

[0010] The network feature and manual feature acquisition module is used to obtain the network features extracted by the neural network under the specified sampling multiples of the image to be detected, and the manual features extracted by the feature extraction operator;

[0011] A fusion feature acquisition module is used to fuse the network features and manual features acquired under a specified sampling multiple to obtain fusion features;

[0012] The lane line acquisition module is used to acquire the lane lines contained in the image to be detected based on the fusion features.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] A storage device for storing one or more programs;

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the method of any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method of any embodiment of the present invention.

[0018] In an embodiment of the present invention, a neural network is used to extract network features containing core information from an image to be detected based on a small amount of calculation, and the manual features extracted by a feature extraction operator are supplemented. Lane lines are detected based on the acquired fusion features, thereby reducing the amount of calculation while ensuring the accuracy of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A is a flow chart of a lane line detection method provided in Embodiment 1 of the present invention;

[0020] Figure 1B This is a schematic diagram of an application scenario of lane line detection provided by the first embodiment of the present invention;

[0021] Figure 1C is a curve diagram of a horizontal edge feature operator provided in the first embodiment of the present invention;

[0022] Figure 1D is a curve schematic diagram of a horizontal feature smoothing operator provided in the first embodiment of the present invention;

[0023] Figure 1E is a curve diagram of a feature truncation operator provided in the first embodiment of the present invention;

[0024] Figure 1F is a schematic diagram of a vertical edge feature operator curve provided in Embodiment 1 of the present invention;

[0025] Figure 1G is a curve schematic diagram of a vertical feature smoothing operator provided in Embodiment 1 of the present invention;

[0026] Figure 1H is a schematic diagram of the horizontal filter kernel size provided in the first embodiment of the present invention;

[0027] Fig. 1I is a schematic diagram of the vertical filter kernel size provided by the first embodiment of the present invention;

[0028] Figure 2 is a flow chart of a lane line detection method provided in Embodiment 2 of the present invention;

[0029] Figure 3 is a schematic diagram of the structure of a lane line detection device provided in Embodiment 3 of the present invention;

[0030] Figure 4 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0032] Embodiment 1

[0033] Figure 1A : is a flow chart of a lane line detection method provided by an embodiment of the present invention. This embodiment is applicable to the case of detecting lane lines. The method can be executed by a lane line detection device in an embodiment of the present invention. The device can be implemented by software and / or hardware. The method in the embodiment of the present invention specifically includes the following steps:

[0034] Step S101, obtaining network features extracted by a neural network under a specified sampling multiple of an image to be detected, and manual features extracted by a feature extraction operator.

[0035] Among them, Figure 1B The schematic diagram of the application scenario of lane line detection provided by this embodiment is shown. The neural network for network feature extraction can be a deep convolutional neural network, which includes four filter layers. The sampling multiples corresponding to each filter layer are 4 times downsampling, 8 times downsampling, 16 times downsampling and 32 times downsampling, respectively, and also includes a fully connected layer. For example, the deep convolutional neural network of this embodiment can specifically use ShuffleNetV2, and its network parameters are shown in Table 1 below:

[0036] Table 1

[0037]

[0038]

[0039] Optionally, obtaining network features extracted by a neural network at a specified sampling multiple of the image to be detected, and manual features extracted by a feature extraction operator, may include: obtaining network features extracted by a neural network at a specified sampling multiple of the image to be detected; obtaining lane horizontal manual features extracted by a feature extraction operator at a specified sampling multiple of the image to be detected; obtaining lane vertical manual features extracted by a feature extraction operator at a specified sampling multiple of the image to be detected.

[0040] Specifically, the specified sampling multiple in this embodiment can be the downsampling of any filter layer contained in the deep convolutional neural network. Since each filter layer is composed of a convolutional layer, batch normalization, nonlinear activation and pooling, it is possible to obtain features of different scales. And each filter layer can only contain a small number of filter kernels, so the deep convolutional neural network can extract network features containing key information in the image to be detected based on a small amount of calculation. Although the amount of calculation for network extraction through neural networks can be greatly reduced, the extracted network features only contain key information at this time, and the information may not be comprehensive enough for accurate lane line detection. Therefore, manual feature extraction can be performed on the image to be detected at the same sampling multiple to obtain manual features, so as to realize the information supplement of manual features to network features, and when performing manual feature extraction, the extracted manual features specifically include lane horizontal manual features and lane vertical manual features.

[0041] Optionally, obtaining lane level manual features extracted by a feature extraction operator from the image to be detected at a specified sampling multiple may include: obtaining initial horizontal features extracted by a horizontal edge feature operator from the image to be detected at a specified sampling multiple; smoothing the initial horizontal features by a feature smoothing operator to obtain horizontal smooth features; and truncating the horizontal smooth features by a feature truncation operator to obtain lane level manual features.

[0042] Among them, the lane lines in the image to be detected are mostly bright stripes of white or yellow colors, the ground is usually a dark texture, and the edge area between the lane lines and the ground often has relatively obvious and stable edge features, so the feature extraction operator can be used to extract manual features. Since the manual features include lane horizontal manual features and lane vertical manual features, the feature extraction operators used include: horizontal edge feature operator, horizontal feature smoothing operator, vertical edge feature operator, vertical feature smoothing operator and feature truncation operator. Therefore, when extracting the lane horizontal manual features of the image to be detected at a specified sampling multiple, the horizontal edge feature operator in the following formula (1) can be used to extract the initial horizontal features, and as Figure 1C The following is a curve diagram of the horizontal edge feature operator:

[0043]

[0044] Among them, F in (x, y) represents the pixel value corresponding to the pixel point with coordinates (x, y) in the image to be detected, i is the step size, w is the horizontal window size, Represents the initial horizontal feature corresponding to the pixel point with coordinates (x, y).

[0045] The initial horizontal feature corresponding to each pixel in the image to be detected is smoothed using the horizontal feature smoothing operator in the following formula (2) to obtain the horizontal smoothing feature, and Figure 1D Shown is a curve diagram of the horizontal feature smoothing operator:

[0046]

[0047] in, Represents the initial horizontal feature corresponding to the pixel point with coordinates (x, y), i is the step size, w is the horizontal window size, Represents the horizontal smoothing feature corresponding to the pixel point with coordinates (x, y).

[0048] For the horizontal smooth feature corresponding to each pixel in the image to be detected, the feature truncation operator in the following formula (3) is used to perform stage processing of the maximum eigenvalue to obtain the lane horizontal manual feature, and Figure 1E The following is a curve diagram of the characteristic truncation operator:

[0049]

[0050] Among them, thresh is the preset threshold, F 水平 (x, y) is the lane level manual feature corresponding to the pixel point with coordinates (x, y).

[0051] Optionally, obtaining the vertical manual features of the lane extracted by the feature extraction operator at the specified sampling multiple of the image to be detected may include: obtaining the initial vertical features extracted by the vertical edge feature operator at the specified sampling multiple of the image to be detected; smoothing the initial vertical features by the feature smoothing operator to obtain the vertical smooth features; and truncating the vertical smooth features by the feature truncation operator to obtain the vertical manual features of the lane.

[0052] Specifically, the method of obtaining the vertical manual features of the lane of the image to be detected in this embodiment is roughly the same as the method of obtaining the horizontal manual features of the lane. When extracting the vertical manual features of the lane of the image to be detected under a specified sampling multiple, the vertical edge feature operator in the following formula (4) can be used to extract the initial vertical features, and Figure 1F The following is a curve diagram of the vertical edge feature operator:

[0053]

[0054] Among them, F in (x, y) represents the pixel value corresponding to the pixel point with coordinates (x, y) in the image to be detected, i is the step size, h is the vertical window size, Represents the initial vertical feature corresponding to the pixel point with coordinates (x, y).

[0055] The vertical feature smoothing operator in the following formula (5) is used to smooth the initial vertical feature corresponding to each pixel in the image to be detected to obtain the vertical smoothing feature, and Figure 1G The following is a curve diagram of the vertical feature smoothing operator:

[0056]

[0057] in, Represents the initial vertical feature corresponding to the pixel point with coordinates (x, y), i is the step size, h is the vertical window size, Represents the vertical smoothing feature corresponding to the pixel point with coordinates (x, y).

[0058] For the vertical smooth features corresponding to each pixel in the image to be detected, the feature truncation operator in the following formula (6) is used to perform stage processing of the maximum eigenvalue to obtain the lane vertical manual features, and Figure 1E The following is a curve diagram of the characteristic truncation operator:

[0059]

[0060] Among them, thresh is the preset threshold, F 垂直 (x, y) is the lane vertical manual feature corresponding to the pixel point with coordinates (x, y). In this implementation, the curve diagram of the feature truncation operator used to determine the lane horizontal manual feature and lane vertical manual feature corresponding to each pixel point is the same. Of course, the curve diagram of the feature truncation operator used in actual applications may also be different, but in this case, the preset thresholds in formula (3) and formula (6) are different.

[0061] Among them, when extracting manual features through feature extraction operators, the horizontal window size w and the vertical window size h represent the sizes of the filter kernel in the horizontal and vertical directions, respectively, such as Figure 1H is a schematic diagram of the horizontal filter kernel size, such as Fig. 1Iis a schematic diagram of the vertical filter kernel size, and the above two figures show in detail the width of any inner lane line, and take the inner lane line as an example to explain the selection of the filter kernel size in the horizontal and vertical directions. H0 is the intersection of the two inner lane lines, and HOHmax is the ordinate axis corresponding to H0 as the origin of the coordinate system. The linear difference method can be used to determine the w and h of any height lane line, where w min ≤w≤w max ,h min ≤h≤h max .

[0062] For example, when the size of the image to be detected is 640×480, when the neural network model is used to extract features of the image to be detected by downsampling by 4 times, the network features obtained are FM net , and the corresponding output feature channel number is 46. When the feature is extracted manually, the lane level manual feature F corresponding to each pixel point 水平 (x, y) and the lane vertical manual features are known, so the lane horizontal manual features f of the entire image to be detected can be obtained 水平 , and the vertical manual features f of the lane of the image to be detected 垂直 , after 4 times downsampling, the extracted f 水平 and f 垂直 are multidimensional vectors, and the size can be 160×120, and f is extracted according to the image to be detected. 水平 and f 垂直 Manual feature FM can be directly obtained hand , and the corresponding number of output feature channels is 2. Of course, in this embodiment, 4-fold downsampling is used as an example to illustrate the extraction of network features and manual features for the detection image. The extraction methods of network features and manual features under other specified sampling multiples are roughly the same. For example, f 水平 and f 垂直 The corresponding size is 80×60 when down-sampling is 8 times, and the corresponding size is 40×30 when down-sampling is 16 times.

[0063] It should be noted that the deep convolutional neural network in this embodiment causes the scale of the image to be detected to change through the convolution pooling contained in each filter layer to obtain network features under a specified sampling multiple. When performing manual feature extraction, sampling is first performed to reach the specified sampling multiple, and then feature extraction is performed. Therefore, the principles of network feature extraction and manual extraction using the above two methods are different.

[0064] Step S102, the network features and manual features obtained under the specified sampling multiple are fused to obtain fused features.

[0065] Optionally, the network features and manual features obtained at the specified sampling multiples are fused to obtain fused features, which may include: splicing the network features, lane horizontal manual features, and lane vertical manual features obtained at the specified sampling multiples according to channel dimensions to obtain spliced ​​features; determining the weight of each channel according to the attention mechanism; and multiplying the weight of each channel by the spliced ​​features to obtain fused features.

[0066] Specifically, in this embodiment, the network features and manual features obtained at the specified sampling multiples are fused to obtain fused features, so that the fused features contain both the network features of the key information in the image to be detected and the manual features containing supplementary information in the image to be detected. When a deep convolutional neural network contains multiple filter layers, the fused features obtained by fusion of the previous filter layer can be used as the input of the next adjacent filter layer, and the network features and manual features corresponding to the specified sampling multiple of the next filter layer are extracted.

[0067] Step S103, obtaining the lane lines contained in the image to be detected based on the fusion features.

[0068] Optionally, before obtaining the lane lines contained in the image to be detected based on the fused features, the method may also include: inputting the fused features into the last filter layer in the neural network, wherein the sampling multiple corresponding to the last filter layer is greater than the specified sampling multiple; and performing feature extraction on the fused features through the filter layer to obtain the detection features.

[0069] Optionally, acquiring lane lines contained in the image to be detected based on fusion features may include: processing the detection features through a fully connected layer in a neural network to acquire curve parameters corresponding to each lane line; and determining the lane lines contained in the image to be detected based on the curve parameters corresponding to each lane line.

[0070] Specifically, in this embodiment, before obtaining the lane line contained in the image to be detected based on the fusion feature, the finally obtained fusion feature needs to be input into the last filter layer in the deep convolutional neural network, for example, Figure 1B As shown, the fused features obtained by 4 times downsampling the image to be detected are input into the filter layer corresponding to 32 times downsampling in the deep convolutional neural network, and the obtained fused features are extracted through the last filter layer and the fully connected layer to obtain the detection features. The ShuffleNetV2 deep neural network used in this embodiment contains two fully connected layers. It can be seen from Table 1 that the number of output feature channels corresponding to the first fully connected layer is 128, and the number of output feature channels corresponding to the second fully connected layer is 16. Since the actual lane usually contains four lane lines, each lane line satisfies the univariate cubic function y=a0+a1x+a2x 2 +a3x3 , the output of the second connection layer is a one-dimensional vector (A0, A1, ... A15), so the one-dimensional vector corresponds to 4 lane lines × 4 cubic curve parameters, that is, A1 to A4 in the one-dimensional vector correspond to a0 to a3 in the first lane line, and so on, the curve parameters a0 to a3 corresponding to each lane line are obtained in sequence, so as to determine the lane line contained in the image to be detected according to the curve parameters corresponding to each lane line.

[0071] In an embodiment of the present invention, a neural network is used to extract network features containing core information from an image to be detected based on a small amount of calculation, and the manual features extracted by a feature extraction operator are supplemented. Lane lines are detected based on the acquired fusion features, thereby reducing the amount of calculation while ensuring the accuracy of the detection results.

[0072] Embodiment 2

[0073] Figure 2 It is a flow chart of the lane detection method provided in an embodiment of the present invention. This embodiment is based on the above embodiment and specifically describes how to fuse the network features and manual features obtained under a specified sampling multiple to obtain fused features.

[0074] like Figure 2 As shown, the method of the embodiment of the present disclosure specifically includes:

[0075] Step S201, obtaining network features extracted by a neural network under a specified sampling multiple of the image to be detected, and manual features extracted by a feature extraction operator, wherein the manual features include lane horizontal manual features and lane vertical manual features.

[0076] Step S202 , the network features, lane horizontal manual features, and lane vertical manual features obtained under the specified sampling multiples are spliced ​​according to the channel dimension to obtain spliced ​​features.

[0077] Specifically, the network feature obtained under the specified sampling multiple in this embodiment is FM net , including lane level manual features f 水平 and lane vertical manual features f 垂直 The hand-crafted features are FM hand The following formula (7) is used for feature concatenation:

[0078]

[0079] Among them, F is the splicing feature, FM net is the network feature, FM hand For manual features, A scale matcher.

[0080] For example, based on the network feature FM obtained by 4 times downsampling net The output feature channel number is 46, based on the manual feature FM obtained by 4 times downsampling. hand It is a multi-dimensional matrix with 2 output channels. By combining the above formula (7), it can be determined that the obtained splicing features are a multi-dimensional matrix with 48 output channels.

[0081] Step S203, determine the weight of each channel according to the attention mechanism.

[0082] Specifically, after determining the number of output channels of the splicing feature, this embodiment can use the attention mechanism to determine the weight of each channel. Since the working principle of the attention mechanism is not the focus of this application, it will not be described in detail in this embodiment.

[0083] Step S204, multiplying the weight of each channel by the concatenated feature to obtain a fused feature.

[0084] After obtaining the weight of each channel, the following formula (8) is used to multiply the weight of each channel with the splicing feature to obtain the fusion feature FM: fused :

[0085]

[0086] Get the fused feature FM corresponding to 4 times downsampling fused After that, you can use FM fused As the input of the next adjacent filtering layer, this embodiment only takes 4 times downsampling to obtain fusion features as an example for explanation. The method of obtaining fusion features at other sampling multiples is roughly the same and will not be repeated in this embodiment.

[0087] Step S205: acquiring the lane lines contained in the image to be detected based on the fusion features.

[0088] In the embodiment of the present invention, the network features containing core information in the image to be detected are extracted by a neural network based on a small amount of calculation, and the manual features extracted by the feature extraction operator are supplemented, and the lane line is detected according to the obtained fusion features, thereby reducing the amount of calculation while ensuring the accuracy of the detection result. And by multiplying the weight of each channel with the splicing features obtained under the specified sampling multiple, the obtained fusion features are made more accurate.

[0089] Embodiment 3

[0090] Figure 3It is a structural diagram of a lane line detection device provided by an embodiment of the present invention, which specifically includes: a network feature and manual feature acquisition module 310, a fusion feature acquisition module 320 and a lane line acquisition module 330.

[0091] The network feature and manual feature acquisition module 310 is used to acquire the network features extracted by the neural network under the specified sampling multiple of the image to be detected, and the manual features extracted by the feature extraction operator;

[0092] The fusion feature acquisition module 320 is used to fuse the network features and manual features acquired under the specified sampling multiple to obtain fusion features;

[0093] The lane line acquisition module 330 is used to acquire the lane lines contained in the image to be detected based on the fusion features.

[0094] Optionally, the network feature and manual feature acquisition module includes: a network feature acquisition subunit, which is used to obtain the network features extracted by the neural network under the specified sampling multiple of the image to be detected;

[0095] A lane level manual feature acquisition subunit is used to acquire lane level manual features extracted by a feature extraction operator under a specified sampling multiple of the image to be detected;

[0096] The lane vertical manual feature acquisition subunit is used to obtain the lane vertical manual features extracted by the feature extraction operator under the specified sampling multiple of the image to be detected.

[0097] Optionally, a lane level manual feature acquisition subunit is used to acquire initial horizontal features of the image to be detected extracted by a horizontal edge feature operator at a specified sampling multiple;

[0098] The initial horizontal feature is smoothed by a horizontal feature smoothing operator to obtain a horizontal smoothing feature;

[0099] The horizontal smooth features are truncated by the feature truncation operator to obtain the lane horizontal manual features.

[0100] Optionally, a lane vertical manual feature acquisition subunit is used to obtain the initial vertical features extracted by the vertical edge feature operator under a specified sampling multiple of the image to be detected;

[0101] The vertical smoothing feature is obtained by smoothing the initial vertical feature through a vertical feature smoothing operator;

[0102] The vertical smooth features are truncated by the feature truncation operator to obtain the vertical manual features of the lane.

[0103] Optionally, a fusion feature acquisition module is used to splice the network features, lane horizontal manual features and lane vertical manual features acquired at a specified sampling multiple according to the channel dimension to obtain splicing features;

[0104] Determine the weight of each channel according to the attention mechanism;

[0105] Multiply the weight of each channel by the concatenated feature to obtain the fused feature.

[0106] Optionally, the device further includes a detection feature acquisition module, which is used to input the fusion feature into the last filter layer in the neural network, wherein the sampling multiple corresponding to the last filter layer is greater than the specified sampling multiple;

[0107] The fused features are extracted through the filter layer to obtain the detection features.

[0108] Optionally, the lane line acquisition module is used to: process the detection features through a fully connected layer in a neural network to obtain curve parameters corresponding to each lane line;

[0109] The lane lines included in the image to be detected are determined according to the curve parameters corresponding to each lane line.

[0110] The above device can execute the lane line detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by any embodiment of the present invention.

[0111] Embodiment 4

[0112] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 4 A block diagram of an exemplary electronic device 412 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The electronic device 412 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 4 As shown, the electronic device 412 is in the form of a general purpose computing device. The components of the electronic device 412 may include, but are not limited to: one or more processors 416, a memory 428, and a bus 418 connecting different system components (including the memory 428 and the processor 416).

[0114] Bus 418 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0115] The electronic device 412 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 412, including volatile and non-volatile media, removable and non-removable media.

[0116] The memory 428 is used to store instructions. The memory 428 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. The electronic device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 434 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 418 via one or more data medium interfaces. The memory 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0117] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 442 generally perform the functions and / or methods of the embodiments described herein.

[0118] The electronic device 412 may also communicate with one or more external devices 414 (e.g., keyboards, pointing devices, displays 424, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 412, and / or may communicate with any device that enables the electronic device 412 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 422. Furthermore, the electronic device 412 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 420. As shown, the network adapter 420 communicates with other modules of the electronic device 412 via a bus 418. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 412, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The processor 416 executes the lane line detection method by running the instructions stored in the memory 428: obtaining the network features extracted by the neural network at the specified sampling multiple of the image to be detected, and the manual features extracted by the feature extraction operator; fusing the network features and manual features obtained at the specified sampling multiple to obtain the fused features; and obtaining the lane lines contained in the image to be detected based on the fused features.

[0120] Embodiment 5

[0121] Embodiment 5 of the present invention further provides a storage medium including computer executable instructions. When the computer executable instructions are executed by a computer processor, the method for lane line detection is performed. The method includes:

[0122] Obtain the network features extracted by the neural network under the specified sampling multiples of the image to be detected, and the manual features extracted by the feature extraction operator; fuse the network features and manual features obtained under the specified sampling multiples to obtain fused features; obtain the lane lines contained in the image to be detected based on the fused features.

[0123] Through the above description of the implementation methods, technicians in the relevant field can clearly understand that the present invention can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform the lane line detection method described in each embodiment of the present invention.

[0124] It is worth noting that the various units and modules included in the above embodiments are only divided according to functional logic, but are not limited to the above divisions, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0125] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A lane line detection method, characterized in that: include: Obtain network features extracted by a neural network under a specified sampling multiple of the image to be detected, and manual features extracted by a feature extraction operator; wherein the neural network is a deep convolutional neural network, comprising four filter layers and one fully connected layer, and the sampling multiples corresponding to each of the filter layers are 4 times downsampling, 8 times downsampling, 16 times downsampling and 32 times downsampling in sequence; the manual features include lane horizontal manual features and lane vertical manual features; The network features and manual features obtained under the specified sampling multiples are fused to obtain fused features; Inputting the fused features into the last filter layer in the neural network; wherein the sampling multiple corresponding to the last filter layer is greater than the specified sampling multiple; Performing feature extraction on the fusion feature through the filter layer to obtain detection features; Processing the detection features through a fully connected layer in the neural network to obtain curve parameters corresponding to each lane line; The lane lines included in the image to be detected are determined according to the curve parameters corresponding to each lane line.

2. The method according to claim 1, characterized in that The method of obtaining network features extracted by a neural network under a specified sampling multiple of the image to be detected, and manual features extracted by a feature extraction operator, includes: Obtain the network features extracted by the neural network under the specified sampling multiples of the image to be detected; Obtain lane-level manual features extracted by a feature extraction operator at a specified sampling multiple of the image to be detected; Obtain the lane vertical manual features extracted by the feature extraction operator under the specified sampling multiple of the image to be detected.

3. The method according to claim 2, characterized in that The method of obtaining lane level manual features extracted by a feature extraction operator from an image to be detected at a specified sampling multiple includes: Obtaining the initial horizontal features of the image to be detected extracted by the horizontal edge feature operator at a specified sampling multiple; Smoothing the initial horizontal feature by a horizontal feature smoothing operator to obtain a horizontal smoothing feature; The lane horizontal manual feature is obtained by truncating the horizontal smooth feature using a feature truncation operator.

4. The method according to claim 3, characterized in that The step of obtaining the lane vertical manual features extracted by the feature extraction operator under the specified sampling multiple of the image to be detected includes: Obtaining the initial vertical features of the image to be detected extracted by the vertical edge feature operator at a specified sampling multiple; Smoothing the initial vertical feature by a vertical feature smoothing operator to obtain a vertical smoothing feature; The vertical smooth feature is truncated by a feature truncation operator to obtain the lane vertical manual feature.

5. The method according to claim 4, characterized in that The step of fusing the network features and manual features obtained under the specified sampling multiple to obtain fused features includes: The network features, lane horizontal manual features, and lane vertical manual features obtained under the specified sampling multiples are spliced ​​according to the channel dimension to obtain spliced ​​features; Determine the weight of each channel according to the attention mechanism; The weight of each channel is multiplied by the concatenated feature to obtain the fused feature.

6. A lane line detection device, characterized in that: The device comprises: The network feature and manual feature acquisition module is used to obtain the network features extracted by the neural network under the specified sampling multiples of the image to be detected, and the manual features extracted by the feature extraction operator; wherein the neural network is a deep convolutional neural network, including four filter layers and one fully connected layer, and the sampling multiples corresponding to each of the filter layers are 4 times downsampling, 8 times downsampling, 16 times downsampling and 32 times downsampling in sequence; the manual features include lane horizontal manual features and lane vertical manual features; A fusion feature acquisition module is used to fuse the network features and manual features acquired under a specified sampling multiple to obtain fusion features; A detection feature acquisition module, used for inputting the fusion feature into the last filter layer in the neural network; wherein the sampling multiple corresponding to the last filter layer is greater than the specified sampling multiple; and extracting the fusion feature through the filter layer to obtain the detection feature; A lane line acquisition module is used to process the detection features through the fully connected layer in the neural network to obtain the curve parameters corresponding to each lane line; and determine the lane lines contained in the image to be detected according to the curve parameters corresponding to each lane line.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images