Lane segmentation method, device, electronic device and storage medium

Abstract

The present application provides a lane line segmentation method, device, electronic device and storage medium, which obtains image information of a road, wherein the image information includes: lane lines; inputs the image information into a pre-established lane line segmentation network to obtain the lane lines in the image information, and the lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and frequency features to obtain initial fused features, and output the initial fused features to the neck network. The neck network is used to perform feature fusion on the initial fused features to obtain fused features, and output the fused features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fused features, which can improve the lane line segmentation accuracy.

Description

Technical Field

[0001] The present application belongs to the technical field of lane line segmentation, and in particular relates to a lane line segmentation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of intelligent driving and advanced driver assistance systems, lane segmentation, as a crucial component of road scene understanding, has garnered widespread attention. The accuracy and robustness of lane segmentation directly impact vehicle path planning and control decisions. However, due to the complex shape characteristics of lanes, dynamic lighting changes, and the influence of scene interference (such as shadows, worn lane markings, and obstructions), traditional methods have limitations in handling diverse scenarios, resulting in low segmentation accuracy. Summary of the Invention

[0003] In response to the above problems, embodiments of the present application provide a lane line segmentation method, device, electronic device and storage medium, which can improve the lane line segmentation accuracy.

[0004] In a first aspect, an embodiment of the present application provides a lane segmentation method, the method comprising:

[0005] Acquiring image information of a road, wherein the image information includes: lane lines;

[0006] The image information is input into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is the image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features with the frequency features to obtain initial fusion features, and output the initial fusion features to the neck network. The neck network is used to perform feature fusion on the initial fusion features to obtain fusion features, and output the fusion features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fusion features.

[0007] In some embodiments, the backbone network includes: five convolution modules, a first cross-stage partial connection layer and three multi-scale entanglement learning modules, the convolution module is used to extract basic features, the first cross-stage partial connection layer is used to enhance feature fusion, and the multi-scale entanglement learning module is used to extract multi-scale local features and frequency features, and fuse them to obtain initial fusion features, wherein the five convolution modules include: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a fifth convolution module, and the three multi-scale entanglement learning modules include: a first multi-scale entanglement learning module, a second multi-scale entanglement learning module and a third multi-scale entanglement learning module, the input of the first convolution module is image information, and the input of the second convolution module is The output of the first convolution module and the input of the first cross-stage partially connected layer are the input of the second convolution module, the output of the first cross-stage partially connected layer is the input of the third convolution module, the input of the first multi-scale entangled learning module is the output of the third convolution module, the output of the first multi-scale entangled learning module is the input of the neck network and the fourth convolution module, the output of the fourth convolution module is the input of the second multi-scale entangled learning module, the output of the second multi-scale entangled learning module is the input of the neck network and the fifth convolution module, the output of the fifth convolution module is the input of the third multi-scale entangled learning module, and the output of the third multi-scale entangled learning module is the input of the neck network.

[0008] In some embodiments, the multi-scale entanglement learning module includes: a multi-scale local feature extraction unit, a frequency operation unit and an output unit. The input of the multi-scale local feature extraction unit is feature information. The multi-scale local feature extraction unit is used to extract multi-scale features based on parallel convolution layers, and splice and fuse the multi-scale features, and input the spliced ​​multi-scale features into the frequency operation unit and the output unit. The frequency operation unit is used to perform Fourier transform on the spliced ​​multi-scale features to obtain frequency domain features, and perform point convolution processing on the frequency domain features and then perform inverse Fourier transform to obtain frequency domain enhanced features, and output the frequency domain enhanced features to the output unit. The output unit is used to perform residual connection fusion on the frequency domain enhanced features and the spliced ​​multi-scale features, and adjust the channel weights to generate initial fusion features, and output the initial fusion features to the neck network.

[0009] In some embodiments, the neck network includes: multiple upsampling modules, multiple connection modules, multiple cross-stage partial connection layers, multiple convolution modules and multiple global adaptive attention mechanism modules, the upsampling module is used to increase the size of the feature map, the connection module is used to perform a splicing operation and merge multiple feature maps along the channel dimension, the cross-stage partial connection layer is used to enhance feature fusion, the global adaptive attention mechanism module is used to fuse the global information of the channel and space, and output the fused features to the detection head network, the multiple upsampling modules include: a first upsampling module and a second upsampling module, the multiple connection modules include: a first connection module, a second connection module, a third connection module and the fourth connection module, multiple cross-stage partial connection layers include: the second cross-stage partial connection layer, the third cross-stage partial connection layer, the fourth cross-stage partial connection layer and the fifth cross-stage partial connection layer, multiple convolution modules include: the sixth convolution module and the seventh convolution module, multiple global adaptive attention mechanism modules include: the first global adaptive attention mechanism module, the second global adaptive attention mechanism module and the third global adaptive attention mechanism module, wherein the input of the first connection module is the output of the second multi-scale entangled learning module and the first upsampling module, the input of the first upsampling module is the output of the third multi-scale entangled learning module, The input of the second cross-stage partially connected layer is the output of the first connection module, the output of the second cross-stage partially connected layer is the input of the second upsampling module and the fourth connection module, the input of the second connection module is the output of the second upsampling module and the first multi-scale entangled learning module, the output of the second connection module is the input of the third cross-stage partially connected layer, the output of the third cross-stage partially connected layer is the input of the first global adaptive attention mechanism module and the seventh convolution module, the output of the first global adaptive attention mechanism module is the input of the detection head network, the output of the seventh convolution module is the input of the fourth connection module, the output of the fourth connection module is the input of the fourth cross-stage partially connected layer, the output of the fourth cross-stage partially connected layer is the input of the second global adaptive attention mechanism module and the sixth convolution module, the output of the second global adaptive attention mechanism module is the input of the detection head network, the output of the sixth convolution module is the input of the third connection module, the output of the third connection module is the input of the fifth cross-stage partially connected layer, the output of the fifth cross-stage partially connected layer is the input of the third global adaptive attention mechanism module, and the output of the third global adaptive attention mechanism module is the input of the detection head network.

[0010] In some embodiments, the global adaptive attention mechanism module includes: a multi-scale channel attention mechanism unit and a multi-headed axial mixed self-attention mechanism unit. The multi-scale channel attention mechanism unit is used to input a feature map and output a channel-weighted feature map. The input of the multi-headed axial mixed self-attention mechanism unit is a channel-weighted feature map. The output of the multi-headed axial mixed self-attention mechanism unit is a fused feature after spatial attention weighting, and the fused feature is output to the detection head network.

[0011] In some embodiments, the multi-scale channel attention mechanism unit is used to extract features through multiple parallel convolutions, and to splice the extracted features to obtain spliced ​​feature information, and to perform global average pooling and maximum pooling on the spliced ​​feature information, and to generate channel weights through a multi-layer perceptron, and to perform point multiplication based on the channel weights and the feature information input into the multi-scale channel attention mechanism unit to obtain channel-weighted feature information, and to output the channel-weighted feature information to the multi-headed axial hybrid self-attention mechanism unit, and the multi-headed axial hybrid self-attention mechanism unit is used to perform axial feature decomposition on the channel-weighted feature information to obtain each axial feature, generate a query matrix, a key matrix and a value matrix based on each axial feature, and calculate an attention score based on the query matrix and the key matrix, and multiply the attention features of each axis based on the attention score and the value matrix, and multiply the attention features of each axis, and generate spatial weights based on the multiplied attention features, and perform point multiplication of the spatial weights and the channel-weighted feature information to obtain the fused feature.

[0012] In some embodiments, the regression loss function of the lane segmentation network includes:

[0013]

[0014] in, , is a fixed constant, , , , is the square of the Euclidean distance between the center point of the real box and the predicted box, is the square of the diagonal length of the rectangle surrounding the real box and the predicted box, is the width and height of the real frame, is the width and height of the predicted box, and IoU is the intersection-over-union ratio between the predicted box and the real box. The coordinate value of the prediction box in the t direction, is the coordinate value of the prediction box in the t direction, is the coordinate value of the real frame in the t direction, where the t direction is the x-axis or y-axis direction, and c tIt is the width of the minimum bounding rectangle of the predicted box and the real box in the x-axis direction or the height in the y-axis direction.

[0015] In a second aspect, an embodiment of the present application provides a lane segmentation device, comprising:

[0016] An acquisition module is used to acquire image information of a road, wherein the image information includes: lane lines;

[0017] A segmentation module is used to input the image information into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is the image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features with the frequency features to obtain initial fused features, and output the initial fused features to the neck network. The neck network is used to perform feature fusion on the initial fused features to obtain fused features, and output the fused features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fused features.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement at least one of the methods in the first aspect or the third aspect.

[0021] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0022] The lane line segmentation method provided in an embodiment of the present application obtains image information of a road, wherein the image information includes: lane lines; the image information is input into a pre-established lane line segmentation network to obtain the lane lines in the image information, the lane line segmentation network includes: a backbone network, a neck network and a detection head network, the input of the backbone network is the image information, the backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and the frequency features to obtain initial fusion features, and output the initial fusion features to the neck network, the neck network is used to perform feature fusion on the initial fusion features to obtain fusion features, and output the fusion features to the detection head network, the detection head network is used to output the lane lines in the image information based on the fusion features, which can improve the lane line segmentation accuracy.

[0023] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic flow chart of a lane line segmentation method provided by the present application;

[0026] Figure 2 A schematic diagram of the number of road lane line categories provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of a lane segmentation network provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a multi-scale entanglement learning module provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a global adaptive attention mechanism module provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a lane line segmentation device provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0034] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0035] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if it is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting" or "in response to detecting," depending on the context.

[0036] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0038] Before introducing the embodiments of the present application, a brief introduction to the related technologies and problems in the related technologies is given:

[0039] With the rapid development of intelligent driving and advanced driver assistance systems, lane segmentation, as a crucial component of road scene understanding, has garnered widespread attention. The accuracy and robustness of lane segmentation directly impact vehicle path planning and decision-making control. However, due to the complex shape characteristics of lanes, dynamic lighting changes, and the influence of scene interference (such as shadows, worn markings, and occlusions), traditional methods have limitations when handling diverse scenarios. In recent years, deep learning-based lane segmentation methods have made significant progress. By leveraging convolutional neural networks (CNNs), attention mechanisms, and task-specific optimization, these methods have achieved significant results in improving the accuracy and robustness of lane segmentation. However, these methods still face the following challenges: 1. Inadequate shape feature capture: Existing methods have limited ability to capture the complex shapes of lanes, particularly those with significant curvature or discontinuity. 2. Inadequate fusion of global and local information: Lane segmentation requires capturing both global road layout information and local details, placing higher demands on model design. 3. Loss function optimization problem: Traditional IoU (Intersection over Union) loss is insufficient in the segmentation of small objects or dense areas, resulting in inaccurate edge details in lane line prediction.

[0040] Based on the technical issues of related technologies, the embodiments of this application provide a lane segmentation method that can be applied to electronic devices. These electronic devices may include: mobile phones, tablet computers, in-vehicle devices, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of this application do not impose any restrictions on the specific type of electronic devices.

[0041] The present invention provides a lane segmentation method. Figure 1 The present application provides a flow chart of a lane line segmentation method, as shown in FIG. Figure 1 As shown, the method includes:

[0042] Step S101: Acquire image information of a road, wherein the image information includes lane lines.

[0043] In the embodiment of the present application, the image information of the road is the original road data obtained through the vehicle-mounted camera, lidar or high-precision map, including lane lines, curbs, vehicles and other elements.

[0044] In step S102, the image information is input into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and frequency features to obtain initial fusion features, and output the initial fusion features to the neck network. The neck network is used to fuse the initial fusion features to obtain fusion features, and output the fusion features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fusion features.

[0045] In this embodiment, the lane segmentation network is an end-to-end deep learning model that takes image data as input and outputs pixel-level lane masks (e.g., solid white lines, dashed yellow lines, etc.). The lane segmentation network can be represented as an AE-LaneNet network. By integrating multi-scale local features with frequency features through the backbone network, the segmentation of curved and discontinuous lanes can be resolved.

[0046] In embodiments of the present application, a lane segmentation network can be trained using a dataset. For example, traffic markings and road signs on urban roads can be collected and analyzed. All images in this dataset are derived from real-world vehicle-mounted scenes and contain nearly 20,000 high-resolution annotated images, each with a resolution of 8 megapixels. High-resolution images provide more detailed lane information, particularly capturing more detail in subtle lane markings, varying road conditions, and complex traffic environments. The dataset includes 28 different types of lane marking annotations. These categories cover common lane marking types such as solid white, solid yellow, and double solid white lines, as well as special marking types such as variable lane markings, curbs, and speed bumps. This diversity ensures that the lane segmentation network can accurately segment lanes across a wide range of road conditions, applicable not only to highways but also to urban streets and complex road sections. The dataset utilizes high-quality manual annotation methods to ensure that each image is accurately annotated and fully reflects the shape and variability of lane markings in the real world. This provides a solid foundation for subsequent lane segmentation networks. In order to clearly show the general characteristics and distribution of various lane lines, detailed data statistics were conducted for each category, and the results were presented in the form of charts. Figure 2 A schematic diagram of the number of road lane line categories provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the figure includes the quantity corresponding to each category.

[0047] After obtaining the dataset, the dataset can be cleaned and its annotated labels can be converted into the format required for model training. The dataset can be divided into training set and validation set in an 8:2 ratio. The lane segmentation network can be trained with the training set, and the lane segmentation network can be validated with the validation set to ensure that the lane segmentation network can also show good generalization ability on unseen data.

[0048] To enhance the robustness and adaptability of the lane segmentation network, the data can be augmented before being fed into the network. This includes, but is not limited to, random image resizing, adding random noise, and simulating different weather conditions to simulate various scenarios encountered in the real world. The augmented images are then fed into the network, enabling the lane segmentation network to learn stably under a wider range of conditions.

[0049] During the training process of the lane segmentation network, gradient descent is used to iteratively update model parameters, finding the optimal solution by minimizing the loss function. As training progresses, the model gradually learns how to make accurate predictions across various tasks. After training, the final model file is saved, containing information about the network structure and optimized weight parameters, ensuring that it can be directly loaded and used in future application deployments.

[0050] In an embodiment of the present application, by pre-establishing a lane line segmentation network, after obtaining image information, it can be input into the lane line segmentation network to achieve lane line segmentation.

[0051] The lane line segmentation method provided in an embodiment of the present application obtains image information of a road, wherein the image information includes: lane lines; the image information is input into a pre-established lane line segmentation network to obtain the lane lines in the image information, and the lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and frequency features to obtain initial fused features, and output the initial fused features to the neck network. The neck network is used to perform feature fusion on the initial fused features to obtain fused features, and output the fused features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fused features, which can improve the lane line segmentation accuracy.

[0052] In some embodiments, the backbone network includes: five convolution modules, a first cross-stage partial connection layer and three multi-scale entanglement learning modules (MSEL). The convolution module is used to extract basic features, the first cross-stage partial connection layer is used to enhance feature fusion, and the multi-scale entanglement learning module is used to extract multi-scale local features and frequency features, and fuse them to obtain initial fusion features. The five convolution modules include: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a fifth convolution module. The three multi-scale entangled learning modules include: a first multi-scale entangled learning module, a second multi-scale entangled learning module and a third multi-scale entangled learning module. The input of the first convolution module is image information, the input of the second convolution module is the output of the first convolution module, the input of the first cross-stage partial connection layer is the input of the second convolution module, the output of the first cross-stage partial connection layer is the input of the third convolution module, the input of the first multi-scale entangled learning module is the output of the third convolution module, the output of the first multi-scale entangled learning module is the input of the neck network and the fourth convolution module, the output of the fourth convolution module is the input of the second multi-scale entangled learning module, the output of the second multi-scale entangled learning module is the input of the neck network and the fifth convolution module, the output of the fifth convolution module is the input of the third multi-scale entangled learning module, and the output of the third multi-scale entangled learning module is the input of the neck network.

[0053] In the embodiment of this application, Figure 3 A schematic diagram of the structure of a lane segmentation network provided in an embodiment of the present application is shown as follows: Figure 3 As shown in the figure, the convolution module is represented by ConvModule, which is used for basic feature extraction and downsampling. The cross-stage partial connection layer is represented by CSPLayer. The first cross-stage partial connection layer can reduce computational redundancy and enhance gradient flow. The first cross-stage partial connection layer can split the features into two parts, with only half involved in the calculation. The multi-scale entangled learning module is represented by MSEL. The first convolution module, the second convolution module, the third convolution module, the fourth convolution module, and the fifth convolution module are respectively: the first ConvModule, the second ConvModule, the third ConvModule, the fourth ConvModule, and the fifth ConvModule. The first cross-stage partial connection layer is the first CSPLayer. The first multi-scale entangled learning module, the second multi-scale entangled learning module, and the third multi-scale entangled learning module are respectively: the first MSEL, the second MSEL, and the third MSEL.

[0054] In some embodiments, the multi-scale entanglement learning module includes: a multi-scale local feature extraction unit, a frequency operation unit and an output unit. The input of the multi-scale local feature extraction unit is feature information. The multi-scale local feature extraction unit is used to extract multi-scale features based on parallel convolution layers, and splice and fuse the multi-scale features, and input the spliced ​​multi-scale features into the frequency operation unit and the output unit. The frequency operation unit is used to perform Fourier transform on the spliced ​​multi-scale features to obtain frequency domain features, and perform point convolution processing on the frequency domain features and then perform inverse Fourier transform to obtain frequency domain enhanced features, and output the frequency domain enhanced features to the output unit. The output unit is used to perform residual connection fusion on the frequency domain enhanced features and the spliced ​​multi-scale features, and adjust the channel weights to generate initial fusion features, and output the initial fusion features to the neck network.

[0055] In the embodiment of this application, Figure 4 A schematic diagram of the structure of a multi-scale entanglement learning module provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the multi-scale local feature extraction unit consists of a step size of 2 Convolution and four parallel operations with a stride of 1 and a size of 、 、 In this embodiment, a 5×5 convolution is used to capture the local continuity of lane lines, a 7×7 convolution is used to perceive curvature changes in the medium range, and a 9×9 convolution is used to model the regularity of long-distance dashed line spacing.

[0056] In the embodiment of the present application, the frequency domain operation is performed in sequence with a step size of 2. Convolution, Fourier transform (FT), step size 1 It is connected to the inverse Fourier transform (IFT), and the output unit includes: residual, which is connected to the output through the residual.

[0057] In the embodiment of the present application, not only more complex features are captured by large convolution kernels and features under different receptive fields are integrated, but also the influence of noise is reduced, the feature extraction capability can be more effectively improved, and there are more advantages in lane line recognition.

[0058] Assume that the input features are , the output of the multi-scale local feature unit is , the output of the output unit is , then the MSEL module can be expressed as:

[0059] For the multi-scale local feature extraction unit, the multi-scale feature extraction is expressed as Splicing fusion means: The convolution results of different scales (kernel size is 5, 7, 9) are spliced ​​and fused to form multi-scale features. .

[0060] For the frequency operation unit, the frequency domain features obtained by Fourier transform can be expressed as: , which represents the Perform two-dimensional Fourier transform to obtain frequency domain features ; Then process it using the following formula: , which represents the linear transformation of the real part of the frequency domain feature (weight , bias ) and convolution operations (weights , bias ),get ,

[0061] The inverse Fourier transform can be expressed as: , which represents the Perform two-dimensional inverse Fourier transform to restore time domain features .

[0062] For the output unit, the residual connection fusion can be expressed as: After obtaining the fusion features of the residual connection, the formula Processing, this formula represents the convolution operation on the real part of the fusion result (weight , bias ), and get the output .

[0063] In the embodiment of the present application, convolution is good at extracting local features, but the perception of global features requires a deeper network or a larger receptive field. The Fourier transform can directly obtain the global frequency information of the image in the frequency domain. The MSEL module can help the model better understand the image content and accurately segment the target through its ability to fuse local and global features; the Fourier transform converts the convolution operation from the spatial domain to the frequency domain, turning the convolution operation into a point multiplication, which complements the smoothness of the convolution extracted features, makes the network more adaptable to interference, and greatly reduces the amount of calculation. In the case of segmentation of multiple targets and complex lane objects, the MSEL module can significantly improve the accuracy and robustness of segmentation.

[0064] In some embodiments, the neck network includes: the neck network includes: multiple upsampling modules, multiple connection modules, multiple cross-stage partial connection layers, multiple convolution modules and multiple global adaptive attention mechanism modules (GAAM, Global adaptive attention mechanism mechanism), the upsampling module is used to increase the size of the feature map, the connection module is used to perform a splicing operation and merge multiple feature maps along the channel dimension, the cross-stage partial connection layer is used to enhance feature fusion, the global adaptive attention mechanism module is used to fuse the global information of the channel and space, and output the fused features to the detection head network, multiple upsampling modules include: a first upsampling module and a second upsampling module, multiple connection modules include: a first connection module, a second connection module, a third connection module and a fourth connection module, multiple cross-stage partial connection layers include: a second cross-stage partial connection layer, a third cross-stage partial connection layer, a fourth cross-stage partial connection layer and a fifth cross-stage partial connection layer, multiple convolution modules include: a sixth convolution module and a seventh convolution module, multiple global adaptive attention mechanism modules include: a first global adaptive attention mechanism module, a second global adaptive attention mechanism module and a third global adaptive attention mechanism module, wherein the input of the first connection module is the output of the second multi-scale entangled learning module and the first upsampling module, the input of the first upsampling module is the output of the third multi-scale entangled learning module, the second cross-stage partial The input of the partial connection layer is the output of the first connection module, the output of the second cross-stage partial connection layer is the input of the second upsampling module and the fourth connection module, the input of the second connection module is the output of the second upsampling module and the first multi-scale entanglement learning module, the output of the second connection module is the input of the third cross-stage partial connection layer, the output of the third cross-stage partial connection layer is the input of the first global adaptive attention mechanism module and the seventh convolution module, the output of the first global adaptive attention mechanism module is the input of the detection head network, the output of the seventh convolution module is the input of the fourth connection module, the output of the fourth connection module is the input of the fourth cross-stage partial connection layer, the output of the fourth cross-stage partial connection layer is the input of the second global adaptive attention mechanism module and the sixth convolution module, the output of the second global adaptive attention mechanism module is the input of the detection head network, the output of the sixth convolution module is the input of the third connection module, the output of the third connection module is the input of the fifth cross-stage partial connection layer, the output of the fifth cross-stage partial connection layer is the input of the third global adaptive attention mechanism module, and the output of the third global adaptive attention mechanism module is the input of the detection head network.

[0065] Continue to see Figure 2The second cross-stage partially connected layer is represented as the second CSPLayer, the third cross-stage partially connected layer is represented as the third CSPLayer, the seventh convolution module is represented as the seventh ConvModule, and the sixth convolution module is represented as the sixth ConvModule. The fourth cross-stage partially connected layer is represented as the fourth CSPLaye, and the fifth cross-stage partially connected layer is represented as the fifth CSPLayer. The first global adaptive attention mechanism module, the second global adaptive attention mechanism module, and the third global adaptive attention mechanism module are represented as the first GAAM, the second GAAM, and the third GAAM, respectively. The connection module can be represented as a Concat module.

[0066] In an embodiment of the present application, the connection module is used for multi-level feature fusion: deep semantic features (from upsampling) are combined with shallow geometric features (from MSEL), and some connection layers across stages can replace standard convolutions with depthwise separable convolutions to reduce the amount of computation.

[0067] In some embodiments, the global adaptive attention mechanism module includes: a multi-scale channel attention mechanism unit and a multi-headed axial mixed self-attention mechanism unit. The multi-scale channel attention mechanism unit is used to input a feature map and output a channel-weighted feature map. The input of the multi-headed axial mixed self-attention mechanism unit is a channel-weighted feature map. The output of the multi-headed axial mixed self-attention mechanism unit is a fused feature after spatial attention weighting, and the fused feature is output to the detection head network.

[0068] In an embodiment of the present application, a multi-scale channel attention mechanism unit is used to extract features through multiple parallel convolutions, and to splice the extracted features to obtain spliced ​​feature information, and to perform global average pooling and maximum pooling on the spliced ​​feature information, and to generate channel weights through a multi-layer perceptron, and to perform point multiplication based on the channel weights and the feature information input into the multi-scale channel attention mechanism unit to obtain channel-weighted feature information, and to output the channel-weighted feature information to a multi-headed axial hybrid self-attention mechanism unit, and the multi-headed axial hybrid self-attention mechanism unit is used to perform axial feature decomposition on the channel-weighted feature information to obtain each axial feature, and to generate a query matrix, a key matrix and a value matrix based on each axial feature, and to calculate an attention score based on the query matrix and the key matrix, and to multiply the attention features of each axis based on the attention score and the value matrix, and to multiply the attention features of each axis, and to generate spatial weights based on the multiplied attention features, and to perform point multiplication on the spatial weights and the channel-weighted feature information to obtain a fused feature.

[0069] Figure 5 A schematic diagram of the structure of a global adaptive attention mechanism module provided in an embodiment of the present application is shown as follows: Figure 5As shown in the figure, the global adaptive attention mechanism module includes: Multi scale channel attention mechanism (MSCA) and Multi head axial self-attention mechanism (MASA). The MSCA module consists of multiple convolution kernels with a stride of 1 and a size of The convolution is performed by splicing, channel transformation, pooling and other operations to obtain features under different receptive fields, fuse the channel information of average pooling and maximum pooling, and finally generate through the MLP fully connected layer The channel weight of the size is multiplied by the input feature to generate a new feature map with channel weights; the multi-head axial hybrid self-attention mechanism includes average pooling in the X-axis and Y-axis directions, multiple convolution kernels with a step size of 1 and a The convolution kernel generates three learnable matrices: query, key, and value. The attention score is calculated by the query matrix and the key matrix to represent the correlation with other positions in the feature map. Then, the attention features of each axis are generated by weighted dot product of the value matrix. Finally, the attention features of the X-axis and Y-axis are multiplied and generated by softmax. The spatial attention weights of size are multiplied by the input to obtain the final output matrix.

[0070] Assume that the input features are , the MSCA output is , the MASA output is , after splicing the extracted features, the spliced ​​feature information can be expressed by the following formula: , extract multi-scale features through convolution operations at different levels, and splice the original features with the features of each scale to enhance feature diversity, represents the splicing features along the channel dimension, , , : Represents the convolution kernel weights at different levels, used to extract multi-scale features.

[0071] After obtaining the spliced ​​feature information, it is processed using the following formula: , this formula is used to perform convolution, batch normalization and nonlinear activation on the concatenated features to generate high-order fusion features ,in represents batch normalization, : represents the convolution kernel weight, which is used to fuse the concatenated features. is the activation function.

[0072] Global average pooling and maximum pooling can be expressed as: , in this formula is the global average pooling, It is a global maximum pooling that captures channel importance by combining the global information of average and maximum pooling.

[0073] The calculation of channel attention weight can be expressed as: ,in, , is the weight of the fully connected layer (used for dimensionality reduction and dimensionality increase); 、 is the bias term, The channel attention weight is generated through the fully connected layer and nonlinear transformation to represent the importance of each channel.

[0074] The dot product of the channel weight and the feature information input into the multi-scale channel attention mechanism unit can be expressed as: , Represents channel-by-channel dot product. This formula adjusts the input features using attention weights to enhance the response of important channels.

[0075] Height direction feature extraction:

[0076] ;

[0077] The query matrix is ​​expressed as: ;

[0078] The bond matrix is ​​represented as: ;

[0079] The value matrix is ​​represented as: ;

[0080] Generate a high degree of directional attention representation as:

[0081] ,This formula captures highly directional spatial dependencies through the self-attention mechanism and generates attention-weighted features.

[0082] Similarly, the width direction attention can be expressed as:

[0083] ;

[0084] Multiplying the attention features of each axis can be expressed as:

[0085] ;

[0086] The generated spatial weight can be expressed as follows:

[0087] is the total number of attention heads, Softmax: normalizes along the spatial dimension to generate a probability distribution, Global spatial attention weight,This formula generates global spatial weights by fusing multi-head attention and normalizing,to identify important areas in the image.

[0088] The dot product of the spatial weight and channel-weighted feature information can be expressed by the following formula: ,This formula uses global spatial weights to adjust features, enhance the response of key areas (such as lane lines), and suppress irrelevant background.

[0089] In the embodiment of the present application, the global adaptive attention mechanism module (GAAM) has significant advantages in the recognition of lane lines, and can efficiently fuse the global information of channels and spaces in the feature extraction process. The features of lane lines at different perspectives may have scale differences. In the MSCA module, the rich features under different receptive fields are effectively captured through multi-scale channel feature extraction operations, and the average pooling and maximum pooling strategies are combined to further enhance the expression ability of channel dimension information. In addition, lane lines are usually ductile in the image along the longitudinal and transverse directions. The MASA module uses a multi-head axial self-attention mechanism to model spatial dependencies along the X-axis and Y-axis respectively, to achieve a comprehensive perception of the position correlation in the feature map. Through the learning of query, key, value matrices and the calculation of attention scores, the global correlation between different positions of the feature map is accurately captured, which improves the model's learning of multiple feature representations in different subspaces. This weight mechanism is particularly suitable for processing complex road conditions with multiple lane lines or different types of lane lines, and enhances the generalization ability of the model. Finally, the channel and spatial weights are fused into a new feature representation to improve the model's representation ability and robustness for lane lines.

[0090] In some embodiments, the regression loss function of the lane segmentation network includes:

[0091]

[0092] in, , is a fixed constant, , , , is the square of the Euclidean distance between the center point of the real box and the predicted box, is the square of the diagonal length of the rectangle surrounding the real box and the predicted box, is the width and height of the real frame, is the width and height of the predicted box, and IoU is the intersection-over-union ratio between the predicted box and the real box. is the coordinate value of the prediction box in the t direction, is the coordinate value of the ground truth in the t direction, where the t direction is the x-axis or y-axis direction, and c t It is the width of the minimum bounding rectangle of the predicted box and the real box in the x-axis direction or the height in the y-axis direction.

[0093] In this embodiment of the present application, the detection head network includes a detection head and a segmentation head. The detection head is used for object detection and outputs bounding boxes and category labels for objects in the image. The segmentation head is used for semantic segmentation and outputs category labels for each pixel, generating pixel-level masks.

[0094] The method provided in the embodiments of this application designs the AE-LaneNet network by addressing the challenges of multi-scale feature fusion, detailed feature identification, and adjacent target differentiation in lane segmentation tasks. By introducing the Multi-Scale Entangled Learning Module (MSEL), the Global Adaptive Attention Mechanism Module (GAAM), and the Distance Penalized Intersection-over-Union (DPIoU) Loss, the lane segmentation task is comprehensively optimized from three aspects: feature extraction, feature fusion, and loss design. The Multi-Scale Entangled Learning Module leverages the correlation between features at different scales, significantly improving the network's ability to capture lane details in complex scenes. The Global Adaptive Attention Mechanism combines multi-scale channel attention with shared spatial self-attention, strengthening the network's global modeling capabilities for lane features. The Distance Penalized IoU Loss effectively resolves the confusion problem of adjacent lane annotations by introducing directional distance constraints, thereby improving positioning accuracy.

[0095] In this embodiment, the multi-scale entangled learning module employs a Fourier convolution hybrid technique to effectively identify and segment lanes with high curvature. It maintains high segmentation accuracy across various lane shapes (e.g., curves and intersections), effectively addressing the limitations of traditional methods in complex scenarios. The global adaptive attention mechanism module aims to achieve a deep fusion of global road layout and local detail features within the model. This module utilizes multi-scale feature extraction and a self-attention mechanism to capture local detail while preserving global context. This adaptive multi-level information fusion approach enables more accurate lane segmentation in complex scenarios, ensuring that the model understands both the overall road structure and the local lane features, enabling it to address challenging areas such as complex road conditions and intersections. The regression loss function introduces an additional distance penalty, which better accounts for lane detail and distinguishes adjacent lanes that are too close together, particularly in densely packed areas, thereby improving lane segmentation accuracy.

[0096] The lane segmentation network provided in the embodiment of the present application can not only efficiently adapt to complex road conditions, but also has stronger robustness and generalization capabilities, providing a more efficient and accurate technical solution for practical application scenarios such as intelligent driving.

[0097] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0098] According to the aforementioned embodiments, an embodiment of the present application provides a lane line segmentation device. The modules included in the device, and the units included in each module, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit. During implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0099] The embodiment of the present application provides a lane line segmentation device, Figure 6 A schematic diagram of the structure of a lane line segmentation device provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the lane line segmentation device 600 includes:

[0100] The acquisition module 601 is used to acquire image information of the road, wherein the image information includes: lane lines;

[0101] The segmentation module 602 is used to input the image information into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is the image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features with the frequency features to obtain initial fusion features, and output the initial fusion features to the neck network. The neck network is used to perform feature fusion on the initial fusion features to obtain fusion features, and output the fusion features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fusion features.

[0102] In some embodiments, the backbone network includes: five convolution modules, a first cross-stage partial connection layer and three multi-scale entanglement learning modules, the convolution module is used to extract basic features, the first cross-stage partial connection layer is used to enhance feature fusion, and the multi-scale entanglement learning module is used to extract multi-scale local features and frequency features, and fuse them to obtain initial fusion features, wherein the five convolution modules include: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a fifth convolution module, and the three multi-scale entanglement learning modules include: a first multi-scale entanglement learning module, a second multi-scale entanglement learning module and a third multi-scale entanglement learning module, the input of the first convolution module is image information, and the input of the second convolution module is The output of the first convolution module and the input of the first cross-stage partially connected layer are the input of the second convolution module, the output of the first cross-stage partially connected layer is the input of the third convolution module, the input of the first multi-scale entangled learning module is the output of the third convolution module, the output of the first multi-scale entangled learning module is the input of the neck network and the fourth convolution module, the output of the fourth convolution module is the input of the second multi-scale entangled learning module, the output of the second multi-scale entangled learning module is the input of the neck network and the fifth convolution module, the output of the fifth convolution module is the input of the third multi-scale entangled learning module, and the output of the third multi-scale entangled learning module is the input of the neck network.

[0103] In some embodiments, the multi-scale entanglement learning module includes: a multi-scale local feature extraction unit, a frequency operation unit and an output unit. The input of the multi-scale local feature extraction unit is feature information. The multi-scale local feature extraction unit is used to extract multi-scale features based on parallel convolution layers, and splice and fuse the multi-scale features, and input the spliced ​​multi-scale features into the frequency operation unit and the output unit. The frequency operation unit is used to perform Fourier transform on the spliced ​​multi-scale features to obtain frequency domain features, and perform point convolution processing on the frequency domain features and then perform inverse Fourier transform to obtain frequency domain enhanced features, and output the frequency domain enhanced features to the output unit. The output unit is used to perform residual connection fusion on the frequency domain enhanced features and the spliced ​​multi-scale features, and adjust the channel weights to generate initial fusion features, and output the initial fusion features to the neck network.

[0104] In some embodiments, the neck network includes: multiple upsampling modules, multiple connection modules, multiple cross-stage partial connection layers, multiple convolution modules and multiple global adaptive attention mechanism modules, the upsampling module is used to increase the size of the feature map, the connection module is used to perform a splicing operation and merge multiple feature maps along the channel dimension, the cross-stage partial connection layer is used to enhance feature fusion, the global adaptive attention mechanism module is used to fuse the global information of the channel and space, and output the fused features to the detection head network, the multiple upsampling modules include: a first upsampling module and a second upsampling module, the multiple connection modules include: a first connection module, a second connection module, a third connection module and the fourth connection module, multiple cross-stage partial connection layers include: the second cross-stage partial connection layer, the third cross-stage partial connection layer, the fourth cross-stage partial connection layer and the fifth cross-stage partial connection layer, multiple convolution modules include: the sixth convolution module and the seventh convolution module, multiple global adaptive attention mechanism modules include: the first global adaptive attention mechanism module, the second global adaptive attention mechanism module and the third global adaptive attention mechanism module, wherein the input of the first connection module is the output of the second multi-scale entangled learning module and the first upsampling module, the input of the first upsampling module is the output of the third multi-scale entangled learning module, The input of the second cross-stage partially connected layer is the output of the first connection module, the output of the second cross-stage partially connected layer is the input of the second upsampling module and the fourth connection module, the input of the second connection module is the output of the second upsampling module and the first multi-scale entangled learning module, the output of the second connection module is the input of the third cross-stage partially connected layer, the output of the third cross-stage partially connected layer is the input of the first global adaptive attention mechanism module and the seventh convolution module, the output of the first global adaptive attention mechanism module is the input of the detection head network, the output of the seventh convolution module is the input of the fourth connection module, the output of the fourth connection module is the input of the fourth cross-stage partially connected layer, the output of the fourth cross-stage partially connected layer is the input of the second global adaptive attention mechanism module and the sixth convolution module, the output of the second global adaptive attention mechanism module is the input of the detection head network, the output of the sixth convolution module is the input of the third connection module, the output of the third connection module is the input of the fifth cross-stage partially connected layer, the output of the fifth cross-stage partially connected layer is the input of the third global adaptive attention mechanism module, and the output of the third global adaptive attention mechanism module is the input of the detection head network.

[0105] In some embodiments, the global adaptive attention mechanism module includes: a multi-scale channel attention mechanism unit and a multi-headed axial mixed self-attention mechanism unit. The multi-scale channel attention mechanism unit is used to input a feature map and output a channel-weighted feature map. The input of the multi-headed axial mixed self-attention mechanism unit is a channel-weighted feature map. The output of the multi-headed axial mixed self-attention mechanism unit is a fused feature after spatial attention weighting, and the fused feature is output to the detection head network.

[0106] In some embodiments, the multi-scale channel attention mechanism unit is used to extract features through multiple parallel convolutions, and to splice the extracted features to obtain spliced ​​feature information, and to perform global average pooling and maximum pooling on the spliced ​​feature information, and to generate channel weights through a multi-layer perceptron, and to perform point multiplication based on the channel weights and the feature information input into the multi-scale channel attention mechanism unit to obtain channel-weighted feature information, and to output the channel-weighted feature information to the multi-headed axial hybrid self-attention mechanism unit, and the multi-headed axial hybrid self-attention mechanism unit is used to perform axial feature decomposition on the channel-weighted feature information to obtain each axial feature, generate a query matrix, a key matrix and a value matrix based on each axial feature, and calculate an attention score based on the query matrix and the key matrix, and multiply the attention features of each axis based on the attention score and the value matrix, and multiply the attention features of each axis, and generate spatial weights based on the multiplied attention features, and perform point multiplication of the spatial weights and the channel-weighted feature information to obtain the fused feature.

[0107] In some embodiments, the regression loss function of the lane segmentation network includes:

[0108]

[0109] in, , is a fixed constant, , , , is the square of the Euclidean distance between the center point of the real box and the predicted box, is the square of the diagonal length of the rectangle surrounding the real box and the predicted box, is the width and height of the real frame, is the width and height of the predicted box, and IoU is the intersection-over-union ratio between the predicted box and the real box. is the coordinate value of the prediction box in the t direction, is the coordinate value of the real frame in the t direction, where the t direction is the x-axis or y-axis direction, and c t It is the width of the minimum bounding rectangle of the predicted box and the real box in the x-axis direction or the height in the y-axis direction.

[0110] in addition, Figure 6 The lane line segmentation shown can be a software unit, a hardware unit, or a combination of software and hardware units built into existing electronic devices, or can be integrated into electronic devices as independent accessories, or can exist as an independent terminal device.

[0111] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0113] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 7 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Figure 7 Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps of any of the above-mentioned method embodiments are implemented, or when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0114] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more of which are stored in the memory 31 and executed by the processor 30 to implement the present application. The one or more modules / units may be a series of computer program 32 instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0115] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0116] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process steps in the above-mentioned method embodiments can be implemented by computer program 32 instructing the relevant hardware. Computer program 32 can be stored in a computer-readable storage medium. When executed by processor 30, computer program 32 can implement the steps of each of the above-mentioned method embodiments. Computer program 32 includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. Computer-readable media can include at least: any entity or device capable of carrying computer program code to a terminal, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0118] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0119] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0123] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0124] The personal information processed by the Applicant will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The Applicant will treat the user's personal information and its processing with a high degree of diligence.

[0125] The Applicant attaches great importance to the security of user personal information and has taken reasonable and feasible security measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

Claims

1. A lane line segmentation method, characterized in that: include: Acquiring image information of a road, wherein the image information includes: lane lines; The image information is input into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is the image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and the frequency features to obtain initial fusion features, and output the initial fusion features to the neck network. The neck network is used to perform feature fusion on the initial fusion features to obtain fusion features, and output the fusion features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fusion features; the backbone network includes: five convolution modules, a first cross-stage partial connection layer and three multi-scale entanglement learning modules. The convolution module is used to extract basic features. The first cross-stage partial connection layer is used to enhance feature fusion. The multi-scale entanglement learning module is used to extract multi-scale local features and frequency features, and fuse them to obtain initial fusion features. Among them, the five convolution modules include : The first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module, the three multi-scale entangled learning modules include: the first multi-scale entangled learning module, the second multi-scale entangled learning module and the third multi-scale entangled learning module, the input of the first convolution module is image information, the input of the second convolution module is the output of the first convolution module, the input of the first cross-stage partial connection layer is the input of the second convolution module, the output of the first cross-stage partial connection layer is the input of the third convolution module, the input of the first multi-scale entangled learning module is the output of the third convolution module, the output of the first multi-scale entangled learning module is the input of the neck network and the fourth convolution module, the output of the fourth convolution module is the input of the second multi-scale entangled learning module, the output of the second multi-scale entangled learning module is the input of the neck network and the fifth convolution module, the output of the fifth convolution module is the input of the third multi-scale entangled learning module, and the output of the third multi-scale entangled learning module is the input of the neck network.

2. The method according to claim 1, characterized in that The multi-scale entanglement learning module includes: a multi-scale local feature extraction unit, a frequency operation unit and an output unit. The input of the multi-scale local feature extraction unit is feature information. The multi-scale local feature extraction unit is used to extract multi-scale features based on parallel convolution layers, and splice and fuse the multi-scale features, and input the spliced ​​multi-scale features into the frequency operation unit and the output unit. The frequency operation unit is used to perform Fourier transform on the spliced ​​multi-scale features to obtain frequency domain features, and perform point convolution processing on the frequency domain features and then perform inverse Fourier transform to obtain frequency domain enhanced features, and output the frequency domain enhanced features to the output unit. The output unit is used to perform residual connection fusion on the frequency domain enhanced features and the spliced ​​multi-scale features, and adjust the channel weights to generate initial fusion features, and output the initial fusion features to the neck network.

3. The method according to claim 1, characterized in that The neck network includes: multiple upsampling modules, multiple connection modules, multiple cross-stage partial connection layers, multiple convolution modules and multiple global adaptive attention mechanism modules, the upsampling module is used to increase the size of the feature map, the connection module is used to perform a splicing operation and merge multiple feature maps along the channel dimension, the cross-stage partial connection layer is used to enhance feature fusion, the global adaptive attention mechanism module is used to fuse the global information of the channel and space, and output the fused features to the detection head network, the multiple upsampling modules include: a first upsampling module and a second upsampling module, the multiple connection modules include: a first connection module, a second connection module, a third connection module and a third connection module Four connection modules, multiple cross-stage partial connection layers include: a second cross-stage partial connection layer, a third cross-stage partial connection layer, a fourth cross-stage partial connection layer and a fifth cross-stage partial connection layer, multiple convolution modules include: a sixth convolution module and a seventh convolution module, multiple global adaptive attention mechanism modules include: a first global adaptive attention mechanism module, a second global adaptive attention mechanism module and a third global adaptive attention mechanism module, wherein the input of the first connection module is the output of the second multi-scale entangled learning module and the first upsampling module, the input of the first upsampling module is the output of the third multi-scale entangled learning module, the second cross The input of the stage partial connection layer is the output of the first connection module, the output of the second cross-stage partial connection layer is the input of the second upsampling module and the fourth connection module, the input of the second connection module is the output of the second upsampling module and the first multi-scale entangled learning module, the output of the second connection module is the input of the third cross-stage partial connection layer, the output of the third cross-stage partial connection layer is the input of the first global adaptive attention mechanism module and the seventh convolution module, the output of the first global adaptive attention mechanism module is the input of the detection head network, the output of the seventh convolution module is the input of the fourth connection module, the output of the fourth connection module is the input of the fourth cross-stage partial connection layer, the output of the fourth cross-stage partial connection layer is the input of the second global adaptive attention mechanism module and the sixth convolution module, the output of the second global adaptive attention mechanism module is the input of the detection head network, the output of the sixth convolution module is the input of the third connection module, the output of the third connection module is the input of the fifth cross-stage partial connection layer, the output of the fifth cross-stage partial connection layer is the input of the third global adaptive attention mechanism module, and the output of the third global adaptive attention mechanism module is the input of the detection head network.

4. The method according to claim 3, characterized in that The global adaptive attention mechanism module includes: a multi-scale channel attention mechanism unit and a multi-head axial hybrid self-attention mechanism unit. The multi-scale channel attention mechanism unit is used to input a feature map and output a channel-weighted feature map. The input of the multi-head axial hybrid self-attention mechanism unit is a channel-weighted feature map. The output of the multi-head axial hybrid self-attention mechanism unit is a fused feature weighted by spatial attention, and the fused feature is output to the detection head network.

5. The method according to claim 4, characterized in that The multi-scale channel attention mechanism unit is used to extract features through multiple parallel convolutions, and to splice the extracted features to obtain spliced ​​feature information, and to perform global average pooling and maximum pooling on the spliced ​​feature information and generate channel weights through a multi-layer perceptron, and to perform point multiplication based on the channel weights and the feature information input into the multi-scale channel attention mechanism unit to obtain channel-weighted feature information, and to output the channel-weighted feature information to the multi-headed axial hybrid self-attention mechanism unit, and the multi-headed axial hybrid self-attention mechanism unit is used to perform axial feature decomposition on the channel-weighted feature information to obtain each axial feature, generate a query matrix, a key matrix and a value matrix based on each axial feature, and calculate an attention score based on the query matrix and the key matrix, and multiply the attention features of each axis based on the attention score and the value matrix, and multiply the attention features of each axis, and generate spatial weights based on the multiplied attention features, and perform point multiplication of the spatial weights and the channel-weighted feature information to obtain the fused feature.

6. The method according to claim 1, characterized in that The regression loss function of the lane segmentation network includes: in, , is a fixed constant, , , , is the square of the Euclidean distance between the center point of the real box and the predicted box, is the square of the diagonal length of the rectangle surrounding the real box and the predicted box, is the width and height of the real frame, is the width and height of the predicted box, and IoU is the intersection-over-union ratio between the predicted box and the real box. is the coordinate value of the prediction box in the t direction, is the coordinate value of the real frame in the t direction, where the t direction is the x-axis or y-axis direction, and c t It is the width of the minimum bounding rectangle of the predicted box and the real box in the x-axis direction or the height in the y-axis direction.

7. A lane line segmentation device, characterized in that: include: An acquisition module is used to acquire image information of a road, wherein the image information includes: lane lines; A segmentation module is used to input the image information into a pre-established lane line segmentation network to obtain the lane lines in the image information. The lane line segmentation network includes: a backbone network, a neck network and a detection head network. The input of the backbone network is the image information. The backbone network is used to extract multi-scale local features and frequency features in the image information, and fuse the local features and the frequency features to obtain initial fusion features, and output the initial fusion features to the neck network. The neck network is used to perform feature fusion on the initial fusion features to obtain fusion features, and output the fusion features to the detection head network. The detection head network is used to output the lane lines in the image information based on the fusion features. The backbone network includes: five convolution modules, a first cross-stage partial connection layer and three multi-scale entanglement learning modules. The convolution module is used to extract basic features. The first cross-stage partial connection layer is used to enhance feature fusion. The multi-scale entanglement learning module is used to extract multi-scale local features and frequency features, and fuse them to obtain initial fusion features. Among them, the five convolutions The module includes: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a fifth convolution module. The three multi-scale entangled learning modules include: a first multi-scale entangled learning module, a second multi-scale entangled learning module and a third multi-scale entangled learning module. The input of the first convolution module is image information, the input of the second convolution module is the output of the first convolution module, the input of the first cross-stage partial connection layer is the input of the second convolution module, the output of the first cross-stage partial connection layer is the input of the third convolution module, the input of the first multi-scale entangled learning module is the output of the third convolution module, the output of the first multi-scale entangled learning module is the input of the neck network and the fourth convolution module, the output of the fourth convolution module is the input of the second multi-scale entangled learning module, the output of the second multi-scale entangled learning module is the input of the neck network and the fifth convolution module, the output of the fifth convolution module is the input of the third multi-scale entangled learning module, and the output of the third multi-scale entangled learning module is the input of the neck network.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images