Method and system for detecting drivable area and lane line

Abstract

The invention relates to a drivable area and lane line detection method and system, and relates to the field of semantic segmentation. A drivable area and lane line detection model is constructed and trained, wherein the drivable area and lane line detection model comprises an encoder with a space and channel reconstruction convolution-state space model module, a lane line decoder, a drivable area decoder, a patch embedding layer and a final projection layer; a lane environment image is obtained and input into a drivable area and lane line detection model, multilayer depth coding features of the lane environment image are extracted through a patch embedding layer and an encoder, and a lane line decoder and a drivable area decoder cooperate with a final projection layer to obtain a corresponding lane line mask and a drivable area mask. And respectively extracting a lane line and a drivable area by using the mask. The lane line decoder and the drivable area decoder which are independently arranged can better learn and adjust features specific to drivable areas and lanes, and a more accurate segmentation result is provided.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a drivable area and lane line detection method and system. Background Art

[0002] For driverless cars, obtaining the drivable area and lane detection results in the front space is very important for subsequent planning and decision-making tasks. The current drivable area segmentation methods and lane detection can be divided into two types: traditional and deep learning-based. The traditional method generally divides the pixels of the image based on elementary image features. In essence, it is a process of merging a cluster of pixels. Among them, the threshold method divides different methods according to different pixel value levels. The representative method is the histogram method. With the continuous development of deep learning, drivable area and lane detection are usually regarded as semantic driving scene segmentation problems, among which models using convolutional neural networks (CNN) or Transformer have been widely studied. However, CNN-based methods are limited by their local receptive field and have limitations in long-range modeling capabilities, resulting in suboptimal segmentation results. Although the Transformer-based model performs well in global modeling, the self-attention mechanism needs to grow quadratically with the image size, which brings a huge computational burden. Therefore, how to further utilize the potential of deep learning models to achieve high accuracy in exploring drivable area detection and lane detection and reduce computational complexity is an important issue. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a drivable area and lane line detection method and system.

[0004] The present invention provides a drivable area and lane line detection method, comprising: constructing and training a drivable area and lane line detection model, wherein the drivable area and lane line detection model comprises: an encoder having a space and channel reconstruction convolution-state space model module, a lane line decoder and a drivable area decoder, a patch embedding layer and a final projection layer; the encoder part comprises: 4 groups of cascaded space and channel reconstruction convolution-state space model modules arranged in pairs, wherein the first three groups of space and channel reconstruction convolution-state space model modules and a patch merging layer are combined; a patch embedding layer is arranged before the first group of space and channel reconstruction convolution-state space model modules; the lane line decoder and the drivable area decoder have the same structure, both comprising: 4 groups of cascaded space and channel reconstruction convolution-state space model modules arranged in pairs, wherein the last three groups of space and channel reconstruction convolution-state space model modules and a patch expansion layer are combined; the last group of space and channel reconstruction convolution-state space model modules are connected to the final projection layer;

[0005] Acquire lane environment images through the vehicle-mounted camera;

[0006] The lane environment image is input into the drivable area and lane line detection model. The multi-layer deep coding features of the lane environment image are extracted through the encoder. The lane line decoder and the drivable area decoder obtain the corresponding lane line mask and drivable area mask respectively based on the common multi-layer deep coding features. The lane line mask and the drivable area mask are used to extract the lane line and the drivable area respectively.

[0007] Furthermore, the lane environment image with a dimension of 3×H×W is divided into non-overlapping patch images of size 4×4 through the patch embedding layer, and the patch images are mapped into channels, heights, and widths by 2D convolution processing. The patch embedding layer is implemented by 2D convolution with 3 input channels, C output channels, 4×4 convolution kernel size, 4 moving steps and layer normalization.

[0008] Furthermore, the spatial and channel reconstruction convolution-state space model module decomposes the feature X input to the spatial and channel reconstruction convolution-state space model module into sub-features X with the same channel size through feature segmentation split operation 1 and sub-feature X 2 ; Sub-feature X 1 Input to the spatial and channel reconstruction convolution branch, where the spatial and channel reconstruction convolution branch includes batch normalization and spatial and channel reconstruction convolution; sub-feature X 2 Input to the state space model branch, which includes: layer normalization, sub-feature X 2 After layer normalization, it is divided into two branches. One branch extracts the gating weight through the linear layer and SiLU activation function, and the other branch obtains the feature X after multiplying it with the gating weight through the linear layer, spatial and channel reconstruction convolution layer, SiLU activation function, 2D selective scanning module and layer normalization. 2 ', feature X 2 'Input to the linear layer, the output of the linear layer is consistent with the feature X 1 'Concatenate, shuffle the channel order of the concatenated features and then add them to feature X.

[0009] Furthermore, the 2D selective scanning module expands the input feature Z into four different sequences by pixels in four directions, namely, scanning from top to bottom along the width direction and its reverse direction, and scanning from left to right along the height direction and its reverse direction; these four different sequences are processed in parallel using a separate state-space model with a selection mechanism to obtain the result sequence, and the result sequence is reshaped and merged to form a complete feature map.

[0010] Furthermore, the patch merging layer divides the input feature Y into sub-features Y according to the odd and even rows. 1 , sub-feature Y 2 , sub-feature Y 3 and sub-feature Y 4 :

[0011] Sub-feature Y 1 Take all batches of even-numbered rows and columns of data from the input features,

[0012] Sub-feature Y 2 Take all batches of odd-numbered rows and even-numbered columns of the input features.

[0013] Sub-feature Y 3 Take all batches of even-numbered rows and odd-numbered columns of data from the input features,

[0014] Sub-feature Y 4 Take all batches of odd-numbered rows and columns of data from the input features;

[0015] Sub-feature Y 1 , sub-feature Y 2 , sub-feature Y 3 and sub-feature Y 4 Concatenate in sequence in the channel dimension to obtain the concatenated feature Y';

[0016] The feature Y' is standardized by layer normalization, and then the channel dimension is reduced by a linear layer to half the number of channels of feature Y'.

[0017] Furthermore, the patch expansion layer doubles the number of channels of the input features through a linear layer, and then rearranges the features after the expansion of the channel number. After the rearrangement, the feature width and height become twice, and the number of channels becomes one quarter of the features after the expansion of the channel number; the rearranged features are layer normalized.

[0018] Furthermore, the final projection layer expands the number of channels of the input features by four times through a linear layer, and then rearranges the features after the expansion of the number of channels. After the rearrangement, the width and height become four times, and the number of channels becomes one sixteenth of the features after the expansion of the number of channels; the rearranged features are layer normalized.

[0019] Furthermore, the first set of spatial and channel reconstruction convolution-state space model modules of any decoder receives the output of the last set of spatial and channel reconstruction convolution-state space model modules of the encoder;

[0020] The output of the first set of spatial and channel reconstruction convolution-state space model modules of any decoder and the output of the third patch merging layer of the encoder are added and input into the first patch expansion layer of the decoder;

[0021] The result of processing the first patch expansion layer by the second set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the second patch merging layer of the encoder and then input into the second patch expansion layer of the decoder;

[0022] The result of processing the second patch expansion layer by the third group of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the first patch merging layer of the encoder and then input into the third patch expansion layer of the decoder;

[0023] The fourth group of spatial and channel reconstruction convolution-state space model modules of any decoder processes the result of the third patch expansion layer and adds it to the output of the patch embedding layer and then inputs it into the final projection layer, which is mapped.

[0024] In a second aspect, the present invention provides a drivable area and lane line detection device, comprising: at least one processing unit, the processing unit, a storage unit and a collection unit are interconnected through a bus unit, the collection unit collects lane environment images, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the drivable area and lane line detection method is implemented.

[0025] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the drivable area and lane line detection method is implemented.

[0026] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0027] The present application constructs and trains a drivable area and lane detection model, which includes: an encoder with a spatial and channel reconstruction convolution-state space model module, a lane decoder and a drivable area decoder; the lane environment image is obtained and input into the drivable area and lane detection model, and the multi-layer deep coding features of the lane environment image are extracted by the encoder. The lane decoder and the drivable area decoder obtain the corresponding lane mask and drivable area mask based on the common multi-layer deep coding features, and the lane mask and the drivable area mask are used to extract the lane and drivable area respectively. The present application introduces a spatial and channel reconstruction convolution-state space model module with stronger feature extraction capability in the encoder and any decoder. The spatial and channel reconstruction convolution-state space model module is used as an image feature extraction unit, which not only establishes long-range correlation, but also has a complexity linear to the input size, effectively solving the problems of limited receptive field and high computational complexity. In the decoder part, a multi-output design is adopted, and two decoders are used to process feature mapping and obtain the final result of each task. Two dedicated decoders are used to support independent optimization of segmentation performance for each class. This allows for subsequent fine-tuning and improvement of the segmentation results for the drivable area and lanes without being affected by other categories. At the same time, using two independent decoders also helps improve segmentation accuracy. Each decoder can better learn and adjust features specific to the drivable area and lanes, providing more accurate segmentation results for each category. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0030] Figure 1 The architecture diagram of the drivable area and lane line detection model provided in the embodiment of the present invention;

[0031] Figure 2 An architecture diagram of a spatial and channel reconstruction convolution-state space model module provided in an embodiment of the present invention;

[0032] Figure 3 A flowchart of a method for detecting a drivable area and lane lines provided in an embodiment of the present invention;

[0033] Figure 4A schematic diagram of a drivable area and lane line detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0036] Example 1

[0037] like Figure 3 As shown, the present application provides a drivable area and lane line detection method, comprising:

[0038] Build and train a drivable area and lane detection model.

[0039] Based on the Unet framework of semantic segmentation, the present invention introduces a state space model and combines spatial and channel reconstruction convolution to build a drivable area and lane line detection model including an encoder, a lane line decoder and a drivable area decoder. The lane line decoder and the drivable area decoder have the same structure and share a decoder.

[0040] The structure of the drivable area and lane line detection model is explained in detail below. For the drivable area and lane line detection network, its input is an image with a channel, height, and width of 3, H, and W respectively, that is, the lane environment image in this application.

[0041] like Figure 1 As shown, the encoder part includes: a patch embedding layer, 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, among which the first 3 groups of spatial and channel reconstruction convolution-state space model modules and a combination of a patch merging layer.

[0042] The lane environment image is first divided into 4×4 non-overlapping patch images through the patch embedding layer, and the patch images are mapped into channels, heights, and widths through 2D convolution. Specifically, the patch embedding layer is implemented by 2D convolution with 3 input channels, C output channels, 4×4 convolution kernel size, and 4 moving steps, and layer normalization. Layer normalization normalizes the features of the 2D convolution output to obtain patch embedding, which is then sent to the first set of paired spatial and channel reconstruction convolution-state space model modules to continue feature extraction.

[0043] like Figure 2 As shown, the spatial and channel reconstruction convolution-state space model module decomposes the feature X input to the spatial and channel reconstruction convolution-state space model module into sub-features X with the same channel size by feature segmentation split operation. 1 and sub-feature X 2 .

[0044] Sub-feature X 1 Input to the spatial and channel reconstruction convolution branch, where the spatial and channel reconstruction convolution branch includes batch normalization, and spatial and channel reconstruction convolution; the spatial and channel reconstruction convolution consists of two units: the spatial reconstruction unit (SRU) and the channel reconstruction unit (CRU). The spatial reconstruction unit separates redundant features according to the weights and reconstructs them to suppress the redundancy in the spatial dimension and enhance the representation of features. The channel reconstruction unit adopts split transformation and fusion strategies to reduce the redundancy in the channel dimension as well as the computational cost and storage. The spatial and channel reconstruction convolution is a plug-and-play architecture unit. The spatial and channel reconstruction convolution can achieve better performance by reducing redundant features, and significantly reduces the complexity and computational cost. Sub-feature X 1 After batch normalization, it is input into the spatial and channel reconstruction convolution layer, and finally the feature X is obtained. 1 '.

[0045] Sub-feature X 2 Input to the state space model branch, which includes: layer normalization, sub-feature X 2 After layer normalization, it is divided into two branches. One branch extracts the gating weight through the linear layer and SiLU activation function, and the other branch obtains the feature X after multiplying it with the gating weight through the linear layer, spatial and channel reconstruction convolution layer, SiLU activation function, 2D selective scanning module and layer normalization. 2 ', feature X 2 'Input to the linear layer, the output of the linear layer is consistent with the feature X 1'Splicing, the splicing features are shuffled and added to the feature X. Among them, the 2D selective scanning module expands the input feature Z into four different sequences in four directions by pixel, and the four directions are scanning from top to bottom along the width direction and its reverse direction, and scanning from left to right along the height direction and its reverse direction; these four different sequences are processed in parallel using a separate state space model with a selection mechanism to obtain the result sequence, and the result sequence is reshaped and merged to form a complete feature map.

[0046] Spatial and Channel Reconstruction Convolution-State Space Model module does not change the dimension of features.

[0047] The patch merging layer divides the input features into four sub-features according to the odd and even rows. 1 , sub-feature Y 2 , sub-feature Y 3 and sub-feature Y 4 :Specifically, sub-feature Y 1 Take all batches of even-numbered rows and columns of data from the input features,

[0048] Sub-feature Y 2 Take all batches of odd-numbered rows and even-numbered columns of the input features.

[0049] Sub-feature Y 3 Take all batches of even-numbered rows and odd-numbered columns of data from the input features,

[0050] Sub-feature Y 4 Take all batches of odd-numbered rows and columns of data from the input features;

[0051] Sub-feature Y 1 , sub-feature Y 2 , sub-feature Y 3 and sub-feature Y 4 The concatenated feature Y' is obtained by concatenating in the channel dimension in order; in this way, the width and height of the feature of the input patch merging layer become half of the original, and the number of channels is four times the original. Feature Y' is standardized by layer normalization, and then the channel is reduced in dimension by a linear layer to half the number of channels of feature Y'.

[0052] The lane decoder and the drivable area decoder have the same structure, both of which contain: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, among which the last 3 groups of spatial and channel reconstruction convolution-state space model modules are a combination of patch expansion layers, and the last group of spatial and channel reconstruction convolution-state space model modules is connected to the final projection layer.

[0053] The patch expansion layer doubles the number of channels of the input feature through a linear layer, and then rearranges the features after the expansion of the channel number. After the rearrangement, the feature width and height become twice, and the number of channels becomes one quarter of the features after the expansion of the channel number; the rearranged features are layer normalized.

[0054] The final projection layer expands the number of channels of the input features to four times through a linear layer, and then rearranges the features after the expansion of the channel number. After rearrangement, the width and height become four times, and the number of channels becomes one sixteenth of the features after the expansion of the channel number; the rearranged features are layer normalized.

[0055] The first set of spatial and channel reconstruction convolution-state space model modules of any decoder receives the output of the last set of spatial and channel reconstruction convolution-state space model modules of the encoder; the output of the first set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the third patch merging layer of the encoder and then input into the first patch extension layer of the decoder; the result of the first patch extension layer processed by the second set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the second patch merging layer of the encoder and then input into the second patch extension layer of the decoder; the result of the second patch extension layer processed by the third set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the first patch merging layer of the encoder and then input into the third patch extension layer of the decoder; the result of the third patch extension layer processed by the fourth set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the patch embedding layer and then input into the final projection layer, and the final projection layer is mapped. The final projection layer mapping obtains the corresponding lane line mask and drivable area mask, and the lane line mask and drivable area mask are respectively extracted using the lane line mask and the drivable area mask.

[0056] During training, the lane environment image and its lane line mask and drivable area mask are used for training. The loss function used in training is the sum of corner loss and Dice loss:

[0057] L total =L Focal +L Dice ;

[0058] L Focal The focal loss shifts the model's attention more to samples that are difficult to classify, effectively improving the detection accuracy of difficult classification tasks such as lane detection.

[0059]

[0060] The Dice loss function can learn the category distribution and thus alleviate the category imbalance problem.

[0061]

[0062] Where C represents the number of categories. In this paper, C is 2, which represents the background and lane lines or drivable areas.

[0063] We use lane environment images of size 640×640 to train our model. The optimizer uses AdamW, the initial learning rate is set to 0.001, the total number of training epochs is 300, and the batch size is set to 16. During the inference process, the image size remains unchanged, and a merge strategy is adopted to speed up the inference speed. Merge specifically refers to merging convolution and batch normalization into one layer.

[0064] The lane environment image is acquired through the vehicle-mounted camera.

[0065] The obtained lane environment image is used as input and input into the drivable area and lane line detection model. After passing through the encoder of the drivable area and lane line detection model, the multi-layer deep coding features of the lane environment image are obtained.

[0066] The multi-layer deep coding features are input into the corresponding layers of the lane line decoder and the drivable area decoder to provide pre-information for the next step of segmentation detection. The lane line decoder and the drivable area decoder obtain the corresponding lane line masks and drivable area masks respectively based on the common multi-layer deep coding features, and use the lane line masks and drivable area masks to extract the lane lines and drivable areas respectively.

[0067] Based on Unet, this paper proposes an efficient detection method SC-Mamba for typical scenarios of unmanned vehicles, which can efficiently handle two key tasks in autonomous driving: drivable area segmentation and lane line segmentation, and achieve excellent detection and segmentation effects.

[0068] Example 2

[0069] See also Figure 4 As shown, an embodiment of the present invention provides a drivable area and lane line detection device, including: at least one processing unit, the processing unit, the storage unit and the acquisition unit are interconnected through a bus unit, the acquisition unit acquires lane environment images, and the storage unit is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as a software program, computer executable program and module corresponding to a drivable area and lane line detection method in an embodiment of the present invention. The processing unit implements the drivable area and lane line detection method by running the software program, computer executable program and module stored in the storage unit, including:

[0070] A drivable area and lane line detection model is constructed and trained, wherein the drivable area and lane line detection model comprises: an encoder having a spatial and channel reconstruction convolution-state space model module, a lane line decoder and a drivable area decoder, a patch embedding layer and a final projection layer; the encoder part comprises: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the first 3 groups of spatial and channel reconstruction convolution-state space model modules are a combination of a patch merging layer, and a patch embedding layer is set before the first group of spatial and channel reconstruction convolution-state space model modules; the lane line decoder and the drivable area decoder have the same structure, both comprising: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the last 3 groups of spatial and channel reconstruction convolution-state space model modules are a combination of a patch expansion layer, and the last group of spatial and channel reconstruction convolution-state space model modules are connected to the final projection layer;

[0071] Acquire lane environment images through the vehicle-mounted camera;

[0072] The lane environment image is input into the drivable area and lane line detection model. The multi-layer deep coding features of the lane environment image are extracted through the encoder. The lane line decoder and the drivable area decoder obtain the corresponding lane line mask and drivable area mask respectively based on the common multi-layer deep coding features. The lane line mask and the drivable area mask are used to extract the lane line and the drivable area respectively.

[0073] Of course, the computer program stored in the storage unit of the drivable area and lane line detection device provided in an embodiment of the present invention is not limited to the method operation described above, and can also execute related operations in the drivable area and lane line detection method provided in any embodiment of the present invention.

[0074] Example 3

[0075] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the drivable area and lane line detection method is implemented, including:

[0076] A drivable area and lane line detection model is constructed and trained, wherein the drivable area and lane line detection model comprises: an encoder having a spatial and channel reconstruction convolution-state space model module, a lane line decoder and a drivable area decoder, a patch embedding layer and a final projection layer; the encoder part comprises: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the first 3 groups of spatial and channel reconstruction convolution-state space model modules are a combination of a patch merging layer, and a patch embedding layer is set before the first group of spatial and channel reconstruction convolution-state space model modules; the lane line decoder and the drivable area decoder have the same structure, both comprising: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the last 3 groups of spatial and channel reconstruction convolution-state space model modules are a combination of a patch expansion layer, and the last group of spatial and channel reconstruction convolution-state space model modules are connected to the final projection layer;

[0077] Acquire lane environment images through the vehicle-mounted camera;

[0078] The lane environment image is input into the drivable area and lane line detection model. The multi-layer deep coding features of the lane environment image are extracted through the encoder. The lane line decoder and the drivable area decoder obtain the corresponding lane line mask and drivable area mask respectively based on the common multi-layer deep coding features. The lane line mask and the drivable area mask are used to extract the lane line and the drivable area respectively.

[0079] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in a drivable area and lane line detection method provided in any embodiment of the present invention.

[0080] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting a drivable area and lane lines, characterized in that: include: A drivable area and lane line detection model is constructed and trained, and the drivable area and lane line detection model includes: an encoder with a spatial and channel reconstruction convolution-state space model module, a lane line decoder and a drivable area decoder, a patch embedding layer and a final projection layer; the encoder part includes: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the first 3 groups of spatial and channel reconstruction convolution-state space model modules and a combination of a patch merging layer; a patch embedding layer is set before the first group of spatial and channel reconstruction convolution-state space model modules; the lane line decoder and the drivable area decoder have the same structure, both including: 4 groups of cascaded paired spatial and channel reconstruction convolution-state space model modules, wherein the last 3 groups of spatial and channel reconstruction convolution-state space model modules and a combination of a patch expansion layer; the last group of spatial and channel reconstruction convolution-state space model modules is connected to the final projection layer; Acquire lane environment images through the vehicle-mounted camera; The lane environment image is input into the drivable area and lane line detection model. The multi-layer deep coding features of the lane environment image are extracted through the encoder. The lane line decoder and the drivable area decoder obtain the corresponding lane line mask and drivable area mask respectively based on the common multi-layer deep coding features. The lane line mask and the drivable area mask are used to extract the lane line and the drivable area respectively.

2. The drivable area and lane line detection method according to claim 1, characterized in that: The lane environment image with a dimension of 3×H×W is divided into non-overlapping patch images of size 4×4 through the patch embedding layer, and the patch images are mapped into channels, heights, and widths through 2D convolution processing. The patch embedding layer is implemented by 2D convolution with 3 input channels, C output channels, 4×4 convolution kernel size, 4 moving steps and layer normalization.

3. The drivable area and lane line detection method according to claim 1, characterized in that: The spatial and channel reconstruction convolution-state space model module decomposes the feature X input to the spatial and channel reconstruction convolution-state space model module into sub-features X1 and sub-features X2 with the same channel size through feature segmentation split operation; the sub-feature X1 is input to the spatial and channel reconstruction convolution branch, wherein the spatial and channel reconstruction convolution branch includes batch normalization and spatial and channel reconstruction convolution; the sub-feature X2 is input to the state space model branch, and the state space model branch includes: layer normalization, the sub-feature X2 is divided into two branches after being processed by layer normalization, one branch extracts the gating weight through the linear layer and the SiLU activation function, and the other branch is processed by the linear layer, the spatial and channel reconstruction convolution layer, the SiLU activation function, the 2D selective scanning module and the layer normalization, and then multiplied with the gating weight to obtain the feature X2', the feature X2' is input to the linear layer, the output of the linear layer is spliced ​​with the feature X1', and the channel order of the spliced ​​features is disrupted by the shuffle operation and then added to the feature X.

4. The method for detecting drivable area and lane lines according to claim 3, characterized in that: The 2D selective scanning module expands the input feature Z into four different sequences by pixels in four directions. The four directions are scanning from top to bottom along the width direction and its reverse direction, and scanning from left to right along the height direction and its reverse direction. These four different sequences are processed in parallel using a separate state space model with a selection mechanism to obtain the result sequence, and the result sequence is reshaped and merged to form a complete feature map.

5. The drivable area and lane line detection method according to claim 1, characterized in that: The patch merging layer divides the input feature Y into sub-features Y1, Y2, Y3, and Y4 according to odd and even rows: Sub-feature Y1 takes all batches of even-numbered rows and even-numbered columns of the input feature. Sub-feature Y2 takes all batches of odd-numbered rows and even-numbered columns of the input feature. Sub-feature Y3 takes all batches of even-numbered rows and odd-numbered columns of the input feature. Sub-feature Y4 takes all batches of odd-numbered rows and columns of the input feature; The sub-feature Y1, the sub-feature Y2, the sub-feature Y3 and the sub-feature Y4 are sequentially concatenated in the channel dimension to obtain the concatenated feature Y'; The feature Y' is standardized by layer normalization, and then the channel dimension is reduced by a linear layer to half the number of channels of feature Y'.

6. The drivable area and lane line detection method according to claim 1, characterized in that: The patch expansion layer doubles the number of channels of the input feature through a linear layer, and then rearranges the features after the expansion of the channel number. After the rearrangement, the feature width and height become twice, and the number of channels becomes one quarter of the features after the expansion of the channel number; the rearranged features are layer normalized.

7. The drivable area and lane line detection method according to claim 1, characterized in that: The final projection layer expands the number of channels of the input features to four times through a linear layer, and then rearranges the features after the expansion of the channel number. After rearrangement, the width and height become four times, and the number of channels becomes one sixteenth of the features after the expansion of the channel number; the rearranged features are layer normalized.

8. The drivable area and lane line detection method according to claim 1, characterized in that: The first set of spatial and channel reconstruction convolution-state space model modules of any decoder receives the output of the last set of spatial and channel reconstruction convolution-state space model modules of the encoder; The output of the first set of spatial and channel reconstruction convolution-state space model modules of any decoder and the output of the third patch merging layer of the encoder are added and input into the first patch expansion layer of the decoder; The result of processing the first patch expansion layer by the second set of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the second patch merging layer of the encoder and then input into the second patch expansion layer of the decoder; The result of processing the second patch expansion layer by the third group of spatial and channel reconstruction convolution-state space model modules of any decoder is added to the output of the first patch merging layer of the encoder and then input into the third patch expansion layer of the decoder; The fourth group of spatial and channel reconstruction convolution-state space model modules of any decoder processes the result of the third patch expansion layer and adds it to the output of the patch embedding layer and then inputs it into the final projection layer, which is mapped.

9. A drivable area and lane line detection device, characterized in that: include: At least one processing unit, the processing unit, the storage unit and the acquisition unit are interconnected through a bus unit, the acquisition unit acquires lane environment images, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the drivable area and lane line detection method as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the drivable area and lane line detection method as described in any one of claims 1 to 8 is implemented.

Images