A lane line detection method, electronic device and storage medium

Abstract

The application discloses a lane line detection method, an electronic device and a storage medium. The method comprises the following steps: acquiring a to-be-detected image, wherein the image has lane lines; performing feature information extraction on the to-be-detected image to obtain first to third feature information; processing the first to third feature information by using an attention mechanism to obtain first to third attention feature information; performing high-latitude semantic information extraction on the third feature information to obtain semantic feature information; determining first to-be-recognized feature information by using the second and third attention feature information and the semantic feature information; in the case that a target lane line is recognized in the first to-be-recognized feature information, determining second to-be-recognized feature information by using the first to-be-recognized feature information, the semantic feature information and the first attention feature information; and determining position information of the target lane line in the to-be-detected image by using the second to-be-recognized feature information. The application improves the traditional network structure, and enhances the feature extraction capability and the lane line judgment and detection capability.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a lane line detection method, electronic device, and storage medium. Background Technology

[0002] Lane detection is a key factor in autonomous driving, ensuring that vehicles stay within their lanes and comply with relevant traffic rules, as well as assisting in lane departure and trajectory planning decisions. Lane detection typically involves three processing steps: first, preprocessing the image to obtain the region of interest; second, predicting the areas where lane markings will appear; and third, fitting a lane model to the high-frequency areas.

[0003] In convolutional neural networks, the network extracts features through layer-by-layer abstraction. Higher-level networks have larger receptive fields, and their feature maps contain more semantic information, but their feature maps have lower resolution and lack spatial geometric details, making them suitable for classification and detection of large objects. Lower-level networks, on the other hand, have smaller receptive fields, and their feature maps contain more geometric details, offering high resolution but lacking semantic information, making them suitable for lane line localization.

[0004] The state of lane markings in a scene is uncertain. For example, lane markings may be occluded, worn, discontinuous due to road changes, or subject to glare. In these scenarios, lane markings are unpredictable. When lane markings may interfere with or be ambiguous with road markings and curbs, the network needs strong reasoning and discrimination capabilities for different situations. However, traditional convolutional neural networks often extract image features indiscriminately, thus failing to focus on key features, and convolution limits the size of the receptive field. Summary of the Invention

[0005] This application provides a lane line detection method, electronic device, and storage medium. It uses an improved traditional network structure to extract feature information from the image to be detected, which not only enhances the feature extraction capability but also enhances the ability to judge and detect lane lines.

[0006] On one hand, embodiments of this application provide a lane line detection method, the method comprising:

[0007] Acquire the image to be detected; the image to be detected includes lane lines;

[0008] Feature information is extracted from the image to be detected to obtain first feature information, second feature information, and third feature information;

[0009] The first feature information, the second feature information, and the third feature information are processed based on the attention mechanism to obtain the first attention feature information, the second attention feature information, and the third attention feature information;

[0010] High-dimensional semantic information is extracted from the third feature information to obtain semantic feature information;

[0011] The first feature information to be identified is determined based on the second attention feature information, the third attention feature information, and the semantic feature information;

[0012] If the target lane line is identified from the first feature information to be identified, the second feature information to be identified is determined based on the first feature information to be identified, semantic feature information, and first attention feature information.

[0013] The location information of the target lane line in the image to be detected is determined based on the second feature information to be identified.

[0014] In some possible embodiments, the first feature information, the second feature information, and the third feature information are determined based on an information extraction model;

[0015] The information extraction model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first information extraction layer, and a second information extraction layer.

[0016] In some possible embodiments, feature information is extracted from the image to be detected to obtain first feature information, second feature information, and third feature information, including:

[0017] Based on the first convolutional layer, feature information is extracted from the image to be detected to obtain the fourth feature information;

[0018] Based on the second convolutional layer, feature information is extracted from the fourth feature information to obtain the fifth feature information;

[0019] Based on the third convolutional layer, feature information is extracted from the fifth feature information to obtain the first feature information;

[0020] Based on the fourth convolutional layer and the first information extraction layer, feature information is extracted from the first feature information to obtain the first sub-feature information and the first semantic feature information, respectively.

[0021] The first sub-feature information and the first semantic feature information are concatenated to obtain the second feature information;

[0022] Based on the fifth convolutional layer and the second information extraction layer, feature information is extracted from the second feature information to obtain the second sub-feature information and the second semantic feature information.

[0023] The second sub-feature information and the second semantic feature information are concatenated to obtain the third feature information.

[0024] In some possible embodiments, the first and second features to be identified are determined based on an information restoration model;

[0025] The information restoration model includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and a fourth deconvolution layer.

[0026] In some possible embodiments, the first feature information to be identified is determined based on the second attention feature information, the third attention feature information, and the semantic feature information, including:

[0027] The third attention feature information and the semantic feature information are concatenated to obtain the first input feature information;

[0028] Based on the first deconvolutional layer, the first input feature information is restored to obtain the first sub-input feature information;

[0029] The first sub-input feature information and the second attention feature information are concatenated to obtain the first feature information to be identified.

[0030] In some possible embodiments, determining the second feature information to be identified based on the first feature information to be identified, semantic feature information, and first attention feature information includes:

[0031] Based on the second deconvolution layer, the first feature information to be identified is restored to obtain the second sub-input feature information;

[0032] The first attention feature information, the second sub-input feature information, and the semantic feature information are concatenated to obtain the second input feature information;

[0033] The third input feature information is obtained by restoring the second input feature information based on the third deconvolution layer.

[0034] The third input feature information is restored based on the fourth deconvolution layer to obtain the second feature information to be identified.

[0035] On the other hand, embodiments of this application provide a lane line detection device, which includes:

[0036] The image acquisition module is used to acquire the image to be detected, which includes lane lines.

[0037] The feature extraction module is used to extract feature information from the image to be detected, and obtain first feature information, second feature information and third feature information;

[0038] The attention mechanism module is used to process the first feature information, the second feature information, and the third feature information to obtain the first attention feature information, the second attention feature information, and the third attention feature information;

[0039] The semantic information extraction module is used to extract high-dimensional semantic information from the third feature information to obtain semantic feature information.

[0040] The first feature information determination module is used to determine the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information.

[0041] The second feature information determination module, when the target lane line is identified in the first feature information to be identified, is used to determine the second feature information to be identified based on the first feature information to be identified, semantic feature information and first attention feature information.

[0042] The location information determination module is used to determine the location information of the target lane line in the image to be detected based on the second feature information to be identified.

[0043] In some possible embodiments, the first feature information, the second feature information, and the third feature information are determined based on an information extraction model;

[0044] The information extraction model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first information extraction layer, and a second information extraction layer.

[0045] In some possible embodiments, the feature extraction module is used for:

[0046] Based on the first convolutional layer, feature information is extracted from the image to be detected to obtain the fourth feature information;

[0047] Based on the second convolutional layer, feature information is extracted from the fourth feature information to obtain the fifth feature information;

[0048] Based on the third convolutional layer, feature information is extracted from the fifth feature information to obtain the first feature information;

[0049] Based on the fourth convolutional layer and the first information extraction layer, feature information is extracted from the first feature information to obtain the first sub-feature information and the first semantic feature information, respectively.

[0050] The first sub-feature information and the first semantic feature information are concatenated to obtain the second feature information;

[0051] Based on the fifth convolutional layer and the second information extraction layer, feature information is extracted from the second feature information to obtain the second sub-feature information and the second semantic feature information.

[0052] The second sub-feature information and the second semantic feature information are concatenated to obtain the third feature information.

[0053] In some possible embodiments, the first and second features to be identified are determined based on an information restoration model;

[0054] The information restoration model includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and a fourth deconvolution layer.

[0055] In some possible embodiments, the first feature information determination module is used for:

[0056] The third attention feature information and the semantic feature information are concatenated to obtain the first input feature information;

[0057] Based on the first deconvolutional layer, the first input feature information is restored to obtain the first sub-input feature information;

[0058] The first sub-input feature information and the second attention feature information are concatenated to obtain the first feature information to be identified.

[0059] In some possible embodiments, the second feature information determination module is used for:

[0060] Based on the second deconvolution layer, the first feature information to be identified is restored to obtain the second sub-input feature information;

[0061] The first attention feature information, the second sub-input feature information, and the semantic feature information are concatenated to obtain the second input feature information;

[0062] The third input feature information is obtained by restoring the second input feature information based on the third deconvolution layer.

[0063] The third input feature information is restored based on the fourth deconvolution layer to obtain the second feature information to be identified.

[0064] On the other hand, embodiments of the present invention provide an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executed by any of the above-described lane detection methods.

[0065] On the other hand, embodiments of the present invention provide a computer storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement any of the above-described lane detection methods.

[0066] On the other hand, this embodiment of the invention provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform an implementation of any of the lane detection methods described above.

[0067] The lane line detection method, electronic device, and storage medium provided in this application have the following technical effects:

[0068] An image to be detected, including lane lines, is acquired. Feature information is extracted from the image to obtain first feature information, second feature information, and third feature information. The first feature information, second feature information, and third feature information are processed based on an attention mechanism to obtain first attention feature information, second attention feature information, and third attention feature information. High-dimensional semantic information is extracted from the third feature information to obtain semantic feature information. Based on the second attention feature information, third attention feature information, and semantic feature information, the first feature information to be identified is determined. If the target lane line is identified in the first feature information to be identified, the second feature information to be identified is determined based on the first feature information to be identified, semantic feature information, and first attention feature information. Based on the second feature information to be identified, the position information of the target lane line in the image to be detected is determined. This application embodiment extracts features from the image to be detected to obtain feature information at different scales. Based on an attention mechanism, it processes the feature information at different scales to obtain high-attention feature information at different scales. Based on high-dimensional semantic information extraction, it processes the feature information to obtain high-semantic feature information. Then, it fuses the high-attention feature information and high-semantic feature information at different scales with the feature information extracted from the image to be detected at different scales. This fusion of features at different scales enhances the features and allows for the identification and localization of target lane lines. The above process not only fuses high- and low-level features but also simultaneously acquires global spatial structural relationships and detailed localization accuracy, making the feature map possess rich geometric and semantic information, thereby improving the localization accuracy of lane lines. Attached Figure Description

[0069] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0071] Figure 2 This is a schematic diagram of a lane line detection method provided in an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of an information extraction model provided in an embodiment of this application;

[0073] Figure 4 This is a network structure diagram of an attention module provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram illustrating the determination of a first feature information to be identified, provided in an embodiment of this application.

[0075] Figure 6 This is a schematic diagram illustrating the determination of a second feature information to be identified, provided in an embodiment of this application.

[0076] Figure 7 This is a schematic diagram of a lane line detection device provided in an embodiment of this application;

[0077] Figure 8 This is a hardware structure block diagram of a server for a lane line detection method provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0079] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0080] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application, including a vehicle and a vehicle-side processor 10. The vehicle-side processor 10 can be a processor installed in the vehicle, or it can be a manufacturer's processor connected to and providing services to the processor in the vehicle.

[0081] In some possible embodiments, the vehicle-side processor 10 may include a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud audio recognition model training, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The operating system running on the server may include, but is not limited to, Android, iOS, Linux, Windows, Unix, etc.

[0082] The following describes a specific embodiment of a lane line detection method according to this application. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a lane line detection method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0083] S201: Acquire the image to be detected, which includes lane lines.

[0084] In this embodiment of the application, a camera installed on the vehicle can be used to acquire the image to be detected. The image to be detected can be an image taken while the vehicle is in motion, which is directly in front of the vehicle's field of vision, or an image taken diagonally in front of the vehicle. The image may contain lane lines and other objects.

[0085] In some possible embodiments, the lane lines contained in the image to be detected may be obscured by other objects, worn down, or discontinuous due to changes in the road itself.

[0086] S203: Extract feature information from the image to be detected to obtain first feature information, second feature information and third feature information.

[0087] In one alternative embodiment, Figure 3 This is a schematic diagram of an information extraction model provided in an embodiment of this application, such as... Figure 3As shown, the information extraction model may include a first convolutional layer 301, a second convolutional layer 302, a third convolutional layer 303, a fourth convolutional layer 304, a fifth convolutional layer 305, a first information extraction layer 306, and a second information extraction layer 307. Optionally, the number of convolutional layers, the size of the convolutional kernel, the stride of the convolution, and the connection relationship of the convolutional layers in the convolutional model can be adjusted according to the actual application scenario. The basic architecture of the information extraction model in this embodiment can adopt the standard ResNet50. The first feature information, second feature information, and third feature information in the figure can be determined based on the information extraction model. The first feature information can be the feature information obtained after the image to be detected has undergone convolution through the first, second, and third convolutional layers to extract the feature information of the lane lines. At this time, the geometric structure of the first feature information is relatively clear, but the semantic features gradually decrease, and the resolution of the feature map increases. The second feature information is obtained by fusing the first sub-feature information obtained by further convolving the first feature information with the first semantic feature information obtained by further extracting the required feature information from the first feature information by the first information extraction layer. At this point, the geometric feature details of the second feature information are clearer than the geometric structure of the first feature information, and the resolution of the feature map is further increased. However, compared with the first feature information, the second feature information contains more content and more information. The third feature information is obtained by fusing the second sub-feature information obtained by convolving the second feature information with the second semantic feature information obtained by extracting the required feature information from the second feature information by the second information extraction layer. The third feature information has a higher resolution and contains more feature information, with a clearer geometric structure. For example, if the image to be detected contains lane lines, people, and vehicles, after convolution processing, the shapes of people, facial expressions, and the colors of vehicles and lane lines will be diluted. This information can be collectively referred to as semantic information. In other words, after convolution, the semantic information in the image will be reduced, but the outlines of people, lane lines, and vehicles will gradually become clearer, meaning the geometric structure will become clearer, and the resolution of the feature map will be higher. Such an image is beneficial for judging objects with thin, elongated structures and low resolution, such as lane lines. After the vehicle acquires the image to be detected, it is input into the information extraction model, which can be an encoder, i.e., the network backbone, used to initially extract the feature information of the image. Specifically, the first information extraction layer 306 is located between the third convolutional layer 303 and the fourth convolutional layer 304 of the backbone network, and the second information extraction layer 307 is located between the fourth convolutional layer 304 and the fifth convolutional layer 305, used to enhance the feature extraction capability.The first information extraction layer 306 and the second information extraction layer 307 can be locally improved encoder transformers. These encoder transformers can be improvements on the vision transformer (ViT). Specifically, the encoder transformer in the first information extraction layer 306 first performs dilated convolution on the input first feature information. The kernel size can be set to 3×3, and the dilation rate can be set to 2. After convolution, the obtained feature information is sliced ​​to obtain one-dimensional feature information, and the sliced ​​information is then extracted using conventional vision transformer feature information extraction. Similarly, the encoder transformer in the second information extraction layer 307 performs dilated convolution on the input second feature information. The kernel size can be set to 3×3, and the dilation rate can be set to 4. After convolution, the obtained feature information is sliced ​​to obtain one-dimensional feature information, and the sliced ​​information is then extracted using conventional vision transformer feature information extraction. Performing a dilated convolution before extracting the first and second feature information increases the background information in different receptive fields.

[0088] The number of convolutional layers and information extraction layers in the information extraction model described above is only one optional embodiment of the model. The number of convolutional layers, the number of information extraction layers, and the connection relationships can be changed for different processing scenarios and requirements. Figure 3 The information extraction model shown is not intended to limit this application.

[0089] In some possible embodiments, combined Figure 3Continuing the explanation, feature extraction of the image to be detected, yielding first, second, and third feature information, can include: First convolutional layer 301 can first extract feature information from the image to be detected, obtaining fourth feature information. This feature information can be lane lines or other features, such as other vehicles or traffic signs in the image. Next, second convolutional layer 302 can extract feature information from the fourth feature information, obtaining fifth feature information. Then, third convolutional layer 303 can extract feature information from the fifth feature information, obtaining first feature information. For example, when extracting feature information from the image to be detected, which may contain lane lines, people, and vehicles, if lane lines need to be detected, feature information can first be extracted from the lane lines in the image, obtaining fourth feature information. However, this feature information has only undergone one convolution, resulting in insufficient information extraction and a low-resolution feature map. Therefore, further feature extraction is needed on the obtained fourth feature information to extract the detailed structural features of the lane lines, obtaining fifth feature information. Further lane line information is extracted from the fifth feature information to obtain first feature information with higher resolution and more complete details. Then, the fourth convolutional layer 304 and the first information extraction layer 306 extract feature information from the first feature information, obtaining first sub-feature information and first semantic feature information respectively. The first sub-feature information contains some detailed information about the lane lines, such as the geometric shape of the lane lines. The first semantic feature information can contain other detailed information about the lane lines, such as the color of the lane lines and their position in the image. The first sub-feature information and the first semantic feature information are then concatenated to obtain the second feature information. This concatenation can be a simple addition of numbers in a matrix or a concatenation between matrix channels. The concatenated second feature information contains richer lane line information and simultaneously includes the lane line geometric shape contained in the first sub-feature information and the lane line color and position in the image contained in the first semantic feature information.Finally, based on the fifth convolutional layer 305 and the second information extraction layer 307, feature information is extracted from the second feature information to obtain the second sub-feature information and the second semantic feature information. The second sub-feature information and the second semantic feature information are further extracted based on the second feature information. The second sub-feature information can contain more accurate lane curvature information or other information, and the second semantic feature information can contain more accurate lane length information or other information. Then, the second sub-feature information and the second semantic feature information are concatenated to obtain the third feature information. The concatenation here can be a simple addition of numbers in the matrix or a concatenation between matrix channels. The lane information in the obtained third feature is relatively rich, especially the spatial geometric feature details of the lane lines are more numerous, and the resolution of the feature map is also very high, which further improves the accuracy of lane line recognition.

[0090] By convolving the input image to be detected and performing feature extraction, the original low-resolution feature map, rich in semantic information, is gradually transformed into a high-resolution feature map with less semantic information but clear spatial geometric details of objects in the image. This type of map is more suitable for locating objects with slender structures and low resolution, such as lane lines. Before slicing the input feature map, the first and second information extraction layers perform dilated convolutions on the image, which increases the receptive field of the input feature map. A larger receptive field captures more comprehensive information and can fuse information at more scales. Embedding the encoder transformers of the first and second information extraction layers into the network backbone, i.e., the first to fifth convolutional layers, enhances the feature extraction effect.

[0091] S205: Process the first feature information, the second feature information, and the third feature information based on the attention mechanism to obtain the first attention feature information, the second attention feature information, and the third attention feature information.

[0092] In an optional embodiment, Figure 4 This is a network structure diagram of an attention module provided in an embodiment of this application, such as... Figure 4 As shown, the attention mechanism can be the Convolutional Block Attention Module (CBAM) shown in the figure. The attention module can replace the traditional convolution operation. The module will infer along the channel attention module 401 and the spatial attention module 402 in sequence, and integrate the attention information from both the channel attention module 401 and the spatial attention module 402 to make the information obtained more comprehensive.

[0093] Optionally, the input feature map (H×W×C) 403 can first be fed into the channel attention module 401. This module first uses global average pooling to convert the feature map (H×W×C) into a (1×1×C) feature map 404 and outputs it. Then, the obtained feature map 404 can be fed into a two-layer fully connected layer (FC) 405 to generate inter-channel attention weight features. The inter-channel attention weight features can be multiplied with each element of the input feature map 403, i.e., perform element-wise multiplication, to generate the input features 406 required by the spatial attention module 402. Then, the input features 406 are reduced to (H×W×1) features through spatial convolution, and then the spatial attention weights are calculated by the sigmoid function. Finally, the input of this module, i.e., the input features 406 required by the spatial attention module 402, is multiplied with the spatial attention weights to obtain the final feature 407.

[0094] Optionally, the first, second, and third feature information can be processed based on an attention mechanism to obtain first attention feature information, second attention feature information, and third attention feature information, respectively. Optionally, the obtained first, second, and third attention feature information can be obtained by focusing on information containing lane lines while reducing attention to other information during the extraction of the first, second, and third feature information. In other words, the attention mechanism can correct the features, retaining valuable features and discarding worthless features.

[0095] S207: Extract high-dimensional semantic information from the third feature information to obtain semantic feature information.

[0096] In an optional embodiment, high-dimensional semantic information extraction of the third feature information can be performed by improving the encoder transformer module of the highest layer connection of the backbone network, referencing the idea of ​​Atrous Spatial Pyramid Pooling (ASPP), to obtain the ASPP transformer module. This module can perform dilated convolution operations on the third feature information contained in the last layer feature map of the input using different dilation factors. The convolution kernel can be set to a size of 3×3, and the dilation factor r can be set to 1, 3, 5, and 7 respectively. The settings of the convolution kernel and dilation factor can be adjusted according to different models. Then, adaptive average pooling is used to obtain feature information of the same resolution, the same number of channels, but different scales. By combining the feature information of each scale, high-dimensional semantic information can be extracted from the third feature information to obtain semantic feature information.

[0097] The ASPP transformer module performs hollow spatial pyramid pooling on the input feature information before slicing, which expands the receptive field and enhances semantic information, thereby improving the ability to judge lane lines and infer lane line distribution.

[0098] S209: Determine the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information.

[0099] In some possible embodiments, Figure 5 This is a schematic diagram illustrating the determination of a first feature information to be identified, provided in an embodiment of this application. Figure 6 This is a schematic diagram illustrating how to determine a second feature information to be identified, as provided in an embodiment of this application. Figures 5-6 As shown, the first and second features to be identified can be determined based on an information restoration model. This information restoration model may include a first deconvolution layer 501, a second deconvolution layer 601, a third deconvolution layer 602, and a fourth deconvolution layer 603.

[0100] In some possible embodiments, combined Figure 5 Continuing the explanation, determining the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information can include: First, concatenating the third attention feature information and the semantic feature information to obtain the first input feature information. This concatenation can be a simple addition of numbers in a matrix or a concatenation between matrix channels. Then, based on the first deconvolution layer 501, information restoration can be performed on the first input feature information to obtain the first sub-input feature information. Finally, concatenating the first sub-input feature information and the second attention feature information to obtain the first feature information to be identified. This entire information restoration process can be equivalent to a feature pyramid network, which can be used to handle object detection problems at different scales and can fuse features at multiple scales. Optionally, the first feature information to be identified can be a fusion of high-level attention feature information (i.e., the second attention feature information) and the semantic feature information output by the ASPP transformer module. The resulting first feature information to be identified has a clear geometric structure and relatively high resolution, and can be used for lane line identification and lane line distribution inference.

[0101] S2011: If the target lane line is identified in the first feature information to be identified, the second feature information to be identified is determined based on the first feature information to be identified, the semantic feature information, and the first attention feature information.

[0102] In some possible embodiments, combined Figure 6 Continuing the explanation, determining the second feature information based on the first feature information to be identified, semantic feature information, and first attention feature information can include: First, restoring the first feature information to be identified using the second deconvolution layer 601 to obtain the second sub-input feature information. Second, concatenating the first attention feature information, the second sub-input feature information, and the semantic feature information to obtain the second input feature information. This concatenation can be a simple addition of numbers in a matrix, or a concatenation between matrix channels. Optionally, the second input feature information can be a concatenation of feature information extracted by the high-level ASPP transformer module (i.e., semantic feature information) and low-level attention feature information (i.e., the first attention feature information). This allows for the fusion of high and low-level features, enhancing the semantic features of the low-level information. The feature information obtained after deconvolution has a clearer geometric structure, higher resolution, and contains rich semantic information, making it suitable for lane line localization. Finally, restoring the second input feature information using the third deconvolution layer 602 to obtain the third input feature information. Finally, the third input feature information can be restored based on the fourth deconvolution layer 603 to obtain the second feature information to be identified.

[0103] S2013: Determine the position information of the target lane line in the image to be detected based on the second feature information to be identified.

[0104] Optionally, the restored second feature information has rich semantic features, high resolution, and clear geometric structure, so it can be used to locate lane lines, such as determining the position, length, or curvature of lane lines.

[0105] The model in this application is an improved fusion of convolutional neural networks, feature pyramid networks, and attention mechanisms. In convolutional neural networks, the network extracts features through layer-by-layer abstraction. Higher-level networks have larger receptive fields and feature maps containing more semantic information, but their resolution is lower and they lack spatial geometric details, making them suitable for classification and detection of large targets. Lower-level networks have smaller receptive fields and feature maps containing more geometric details and higher resolution, but they lack semantic information, making them suitable for lane line localization. The feature pyramid is constructed by upsampling features from higher levels and fusing them with the features of that layer before passing them to the lower layer. This supplements the semantic information features of the lower layers, resulting in high-resolution and strong semantic features. During the network model's processing of input data, the attention mechanism calculates the weights of the feature maps by calculating the network layers, thereby obtaining more detailed information related to the target features, increasing the focus on key information, and improving the efficiency of the neural network. This application improves the structure of the above-mentioned networks and splices and fuses features from high and low levels to simultaneously obtain global spatial structural relationships and detailed localization accuracy, making the feature information rich in both geometric and semantic features, thus improving the lane line localization accuracy.

[0106] This application also provides a lane line detection device. Figure 7 This is a schematic diagram of a lane line detection device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes an image acquisition module 701, a feature extraction module 702, an attention mechanism module 703, a semantic information extraction module 704, a first feature information determination module 705, a second feature information determination module 706, and a location information determination module 707.

[0107] Image acquisition module 701 is used to acquire an image to be detected, which includes lane lines. Optionally, the acquired lane lines may be obscured by other objects, worn, or discontinuous due to road variations.

[0108] The feature extraction module 702 is used to extract feature information from the image to be detected, and obtain first feature information, second feature information and third feature information.

[0109] The attention mechanism module 703 is used to process the first feature information, the second feature information and the third feature information to obtain the first attention feature information, the second attention feature information and the third attention feature information.

[0110] The semantic information extraction module 704 is used to extract high-dimensional semantic information from the third feature information to obtain semantic feature information.

[0111] The first feature information determination module 705 is used to determine the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information.

[0112] The second feature information determination module 706, when a target lane line is identified in the first feature information to be identified, is used to determine the second feature information to be identified based on the first feature information to be identified, semantic feature information, and first attention feature information.

[0113] The location information determination module 707 is used to determine the location information of the target lane line in the image to be detected based on the second feature information to be identified.

[0114] In some possible embodiments, the first feature information, the second feature information, and the third feature information can be determined based on an information extraction model, which may include a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first information extraction layer, and a second information extraction layer.

[0115] In some possible embodiments, the feature extraction module is used to extract feature information from the image to be detected based on a first convolutional layer to obtain fourth feature information. Then, it extracts feature information from the fourth feature information based on a second convolutional layer to obtain fifth feature information. Next, it extracts feature information from the fifth feature information based on a third convolutional layer to obtain first feature information. Finally, it extracts feature information from the first feature information based on the fourth convolutional layer and the first information extraction layer to obtain first sub-feature information and first semantic feature information, respectively. The first sub-feature information and the first semantic feature information are then concatenated to obtain second feature information. Finally, it extracts feature information from the second feature information based on the fifth convolutional layer and the second information extraction layer to obtain second sub-feature information and second semantic feature information, respectively. The second sub-feature information and the second semantic feature information are then concatenated to obtain third feature information.

[0116] In some possible embodiments, the first feature information to be identified and the second feature information to be identified can be determined based on an information restoration model, which may include a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and a fourth deconvolution layer.

[0117] In some possible embodiments, the first feature information determination module is used to concatenate the third attention feature information and semantic feature information to obtain the first input feature information. Information restoration is performed on the first input feature information based on the first deconvolution layer to obtain the first sub-input feature information. The first sub-input feature information and the second attention feature information are then concatenated to obtain the first feature information to be identified.

[0118] In some possible embodiments, the second feature information determination module is used to restore the first feature information to be identified based on a second deconvolution layer to obtain second sub-input feature information, and to concatenate the first attention feature information, the second sub-input feature information, and the semantic feature information to obtain second input feature information. The second input feature information is then restored based on a third deconvolution layer to obtain third input feature information, and finally, the third input feature information is restored based on a fourth deconvolution layer to obtain the second feature information to be identified.

[0119] The apparatus and method embodiments in this application are based on the same application concept.

[0120] The methods and embodiments provided in this application can be executed on a computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 8 This is a hardware structure block diagram of a server for a lane line detection method provided in an embodiment of this application. For example... Figure 8 As shown, the server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 810 (CPUs 810 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 830 for storing data, and one or more storage media 820 (e.g., one or more mass storage devices) for storing application programs 823 or data 822. The memory 830 and storage media 820 may be temporary or persistent storage. The program stored in the storage media 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 810 may be configured to communicate with the storage media 820 and execute the series of instruction operations stored in the storage media 820 on the server 800. Server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0121] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 800. In one example, the input / output interface 840 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 840 may be a radio frequency (RF) module for wireless communication with the Internet.

[0122] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 800 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0123] Embodiments of this application also provide a computer storage medium, which can be disposed in a server to store at least one instruction, at least one program, code set, or instruction set related to implementing a signal processing method in the method embodiment. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the lane detection method described above.

[0124] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] On the other hand, an electronic device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the processor loads and executes any lane detection method.

[0126] On the other hand, a computer program product is provided, which includes a computer program stored in a readable storage medium. At least one processor of the computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform any of the lane detection methods.

[0127] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0129] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0130] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A lane line detection method characterized by, The method includes: Acquire an image to be detected; the image to be detected includes lane lines; Based on the information extraction model, feature information is extracted from the image to be detected to obtain first feature information, second feature information, and third feature information; the information extraction model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first information extraction layer, and a second information extraction layer; The first feature information, the second feature information, and the third feature information are processed based on the attention mechanism to obtain the first attention feature information, the second attention feature information, and the third attention feature information; High-dimensional semantic information is extracted from the third feature information to obtain semantic feature information; The first feature information to be identified is determined based on the second attention feature information, the third attention feature information, and the semantic feature information; If a target lane line is identified from the first feature information to be identified, a second feature information to be identified is determined based on the first feature information to be identified, the semantic feature information, and the first attention feature information; The position information of the target lane line in the image to be detected is determined based on the second feature information to be identified. The step of extracting feature information from the image to be detected based on the information extraction model to obtain first feature information, second feature information, and third feature information includes: Based on the first convolutional layer, feature information is extracted from the image to be detected to obtain fourth feature information; based on the second convolutional layer, feature information is extracted from the fourth feature information to obtain fifth feature information; based on the third convolutional layer, feature information is extracted from the fifth feature information to obtain the first feature information; Based on the fourth convolutional layer and the first information extraction layer, feature information is extracted from the first feature information to obtain the first sub-feature information and the first semantic feature information; the first sub-feature information and the first semantic feature information are concatenated to obtain the second feature information. Based on the fifth convolutional layer and the second information extraction layer, feature information is extracted from the second feature information to obtain the second sub-feature information and the second semantic feature information; the second sub-feature information and the second semantic feature information are concatenated to obtain the third feature information.

2. The lane line detection method according to claim 1, characterized by, The first and second features to be identified are determined based on an information restoration model; The information restoration model includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and a fourth deconvolution layer.

3. The lane line detection method according to claim 2, characterized by, The step of determining the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information includes: The third attention feature information and the semantic feature information are concatenated to obtain the first input feature information; Based on the first deconvolutional layer, the first input feature information is restored to obtain the first sub-input feature information; The first sub-input feature information and the second attention feature information are concatenated to obtain the first feature information to be identified.

4. The lane line detection method according to claim 3, characterized in that, The step of determining the second feature information to be identified based on the first feature information to be identified, the semantic feature information, and the first attention feature information includes: Based on the second deconvolution layer, the first feature information to be identified is restored to obtain the second sub-input feature information; The first attention feature information, the second sub-input feature information, and the semantic feature information are concatenated to obtain the second input feature information; Based on the third deconvolution layer, the second input feature information is restored to obtain the third input feature information; The third input feature information is restored based on the fourth deconvolution layer to obtain the second feature information to be identified.

5. A lane line detection device characterized by, The device includes: An image acquisition module is used to acquire an image to be detected, wherein the image to be detected includes lane lines; The feature extraction module is used to extract feature information from the image to be detected based on the information extraction model, to obtain first feature information, second feature information and third feature information; the information extraction model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first information extraction layer and a second information extraction layer; The attention mechanism module is used to process the first feature information, the second feature information, and the third feature information to obtain first attention feature information, second attention feature information, and third attention feature information; The semantic information extraction module is used to extract high-dimensional semantic information from the third feature information to obtain semantic feature information; The first feature information determination module is used to determine the first feature information to be identified based on the second attention feature information, the third attention feature information, and the semantic feature information. The second feature information determination module is used to determine the second feature information to be identified based on the first feature information to be identified, the semantic feature information, and the first attention feature information when a target lane line is identified in the first feature information to be identified. The location information determination module is used to determine the location information of the target lane line in the image to be detected based on the second feature information to be identified; The feature extraction module is used for: Based on the first convolutional layer, feature information is extracted from the image to be detected to obtain fourth feature information; based on the second convolutional layer, feature information is extracted from the fourth feature information to obtain fifth feature information; based on the third convolutional layer, feature information is extracted from the fifth feature information to obtain the first feature information; Based on the fourth convolutional layer and the first information extraction layer, feature information is extracted from the first feature information to obtain the first sub-feature information and the first semantic feature information; the first sub-feature information and the first semantic feature information are concatenated to obtain the second feature information. Based on the fifth convolutional layer and the second information extraction layer, feature information is extracted from the second feature information to obtain the second sub-feature information and the second semantic feature information; the second sub-feature information and the second semantic feature information are concatenated to obtain the third feature information.

6. An electronic device, comprising: The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded by the processor and executed as described in any one of claims 1-4.

7. A computer storage medium, characterized in that The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the method as described in any one of claims 1-4.

8. A computer program product, characterised in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform the method as described in any one of claims 1-4.

Images