Lane line detection method and device, electronic equipment and storage medium

Abstract

The application provides a lane line detection method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a semantic feature image and a detail feature image according to an obtained to-be-recognized lane line image; aggregating the semantic feature image and the detail feature image to obtain an aggregated feature image; performing multi-scale feature alignment fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determining a lane line in the to-be-recognized lane line image according to an affinity domain corresponding to the multi-scale spatial feature image. The method is used to solve the defect that the existing lane line detection method cannot accurately detect lane lines, and effectively improves the detection accuracy of lane lines in the to-be-recognized lane line image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a lane line detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In a complex and changeable natural scene, lane lines are inevitably affected by natural factors (such as sun exposure, rain, snow cover, etc.) and / or human factors (such as vehicle or pedestrian occlusion, wear, etc.), causing the boundary of the lane line to be blurred or even disappear.

[0003] In the existing lane line detection method, after an electronic device obtains a to-be-identified lane line image, the electronic device can identify each pixel in the to-be-identified lane line image based on a semantic segmentation lane line detection network to obtain a lane line. However, the lane line detection method has a large amount of calculation and is not easy to extract lane line pixels in the to-be-identified lane line image, which can easily lead to inaccurate lane lines obtained by the electronic device. SUMMARY

[0004] The present application provides a lane line detection method, device, electronic equipment and storage medium to solve the problem that the existing lane line detection method cannot accurately detect lane lines and to effectively improve the detection accuracy of lane lines in a to-be-identified lane line image.

[0005] The present application provides a lane line detection method, comprising:

[0006] According to the obtained to-be-identified lane line image, a semantic feature image and a detail feature image are determined;

[0007] The semantic feature image and the detail feature image are aggregated to obtain an aggregated feature image;

[0008] The aggregated feature image is subjected to multi-scale feature alignment fusion to obtain a multi-scale spatial feature image;

[0009] According to the affinity domain corresponding to the multi-scale spatial feature image, lane lines in the to-be-identified lane line image are determined.

[0010] According to the lane line detection method provided by the present application, according to the obtained to-be-identified lane line image, a semantic feature image and a detail feature image are determined, comprising: obtaining the to-be-identified lane line image; based on a semantic segmentation backbone network in a residual lane line detection model, performing semantic segmentation on the to-be-identified lane line image to obtain the semantic feature image; based on a lane line spatial convolution network in the residual lane line detection model, performing detail feature extraction on the to-be-identified lane line image to obtain the detail feature image.

[0011] According to a lane line detection method provided by the present invention, a multi-scale feature alignment and fusion is performed on the aggregated feature image to obtain a multi-scale spatial feature image, comprising: performing pooling and channel compression processing on the aggregated feature image based on the spatial position attention module in the residual lane line detection model to obtain a first spatial feature image; performing sampling processing on the first spatial feature image based on the feature adaptation module in the residual lane line detection model to obtain a second spatial feature image; and aligning the deep feature image corresponding to the second spatial feature image with the shallow feature image based on the spatial feature adaptive alignment module in the residual lane line detection model to obtain the multi-scale spatial feature image.

[0012] According to a lane line detection method provided by the present invention, a first spatial feature image is obtained by performing pooling and channel compression processing on the aggregated feature image, including: performing average pooling processing on the aggregated feature image to obtain a first pooling result; performing max pooling processing on the aggregated feature image to obtain a second pooling result; summarizing the first pooling result and the second pooling result in a cascaded manner to obtain a target pooling result; performing channel compression processing on the target pooling result to obtain a channel compression result; and obtaining the first spatial feature image based on the channel compression result.

[0013] According to a lane line detection method provided by the present invention, a first spatial feature image is sampled to obtain a second spatial feature image, comprising: sampling the first spatial feature image according to a preset offset vector guide table to obtain the second spatial feature image.

[0014] According to a lane line detection method provided by the present invention, the deep feature image corresponding to the second spatial feature image is aligned with the shallow feature image to obtain the multi-scale spatial feature image, including: aligning the deep feature image corresponding to the second spatial feature image with the shallow feature image, and obtaining pixel pairs at corresponding positions between the aligned deep feature image and the aligned shallow feature image; and fusing the pixel pairs to obtain the multi-scale spatial feature image.

[0015] According to a lane line detection method provided by the present invention, lane lines in a lane line image to be identified are determined based on the affinity domain corresponding to the multi-scale spatial feature image. The method includes: slicing the multi-scale spatial feature image to obtain slicing results; performing convolution processing on the slicing results to obtain convolution results; and determining lane lines in the lane line image to be identified based on a clustering method based on the affinity domain, according to the pixel association information and mask information corresponding to the convolution results.

[0016] The present invention also provides a lane line detection device, comprising:

[0017] The processing module is used to determine the semantic feature image and the detail feature image based on the lane line image to be identified acquired by the acquisition module; to aggregate the semantic feature image and the detail feature image to obtain an aggregated feature image; to perform multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and to determine the lane line in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lane line detection method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lane line detection method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the lane line detection method as described above.

[0021] The lane line detection method, apparatus, electronic device, and storage medium provided by this invention determine semantic feature images and detail feature images based on an acquired lane line image to be identified; aggregate the semantic feature images and detail feature images to obtain an aggregated feature image; perform multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determine the lane lines in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image. This method addresses the deficiency of existing lane line detection methods in accurately detecting lane lines, effectively improving the detection accuracy of lane lines in lane line images to be identified. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of the lane line detection method provided by the present invention;

[0024] Figure 2 This is a schematic diagram illustrating the determination of aggregated feature images from aggregated semantic feature images and detail feature images provided by the present invention;

[0025] Figure 3This is a schematic diagram of determining the first spatial feature image based on the Spatial Location Attention (PAM) module provided by the present invention;

[0026] Figure 4 This is a schematic diagram of the LFAM (Landing Alignment Module Based on Spatial Features) used in this invention to determine multi-scale spatial feature images.

[0027] Figure 5 This is a schematic diagram of the slicing results determined by the Lane Line Spatial Feature Extraction Neural Network (LSCNN) provided by the present invention;

[0028] Figure 6 This is a schematic diagram of a scenario for the lane line detection method provided by the present invention;

[0029] Figure 7 This is a schematic diagram of the lane line detection device provided by the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] It should be noted that the execution subject involved in the embodiments of the present invention can be a feed stability detection device or an electronic device. Optionally, the electronic device may include a computer, a mobile terminal, and a wearable device.

[0033] The embodiments of the present invention will be further described below using an electronic device as an example.

[0034] like Figure 1 The diagram shown is a flowchart of the lane line detection method provided by the present invention, which may include:

[0035] 101. Based on the acquired lane line image to be identified, determine the semantic feature image and the detail feature image.

[0036] The lane line image to be identified refers to an image that includes lane lines to be identified.

[0037] Semantic feature images refer to the contextual semantic information corresponding to lane lines;

[0038] Detail feature images refer to the spatial shape details corresponding to lane lines.

[0039] The electronic device first acquires an image of the lane line to be identified; then, based on the image of the lane line to be identified, the electronic device can determine the corresponding semantic feature image in the semantic branch and the corresponding detail feature image in the detail branch.

[0040] Optionally, the semantic feature image and the detail feature image can be referred to as the bilateral feature image.

[0041] It should be noted that the timing of the electronic device determining the semantic feature image and the electronic device determining the detail feature image is not limited.

[0042] In some embodiments, the electronic device determines a semantic feature image and a detail feature image based on the acquired lane line image to be identified. This may include: the electronic device acquiring the lane line image to be identified; the electronic device performing semantic segmentation on the lane line image to be identified based on the semantic segmentation backbone network in the residual lane line detection model to obtain a semantic feature image; and the electronic device extracting detail features from the lane line image to be identified based on the lane line spatial convolutional network in the residual lane line detection model to obtain a detail feature image.

[0043] Among them, the residual lane line detection model refers to the model that can accurately determine the lane lines in the lane line image after detecting the lane line image to be identified.

[0044] The semantic segmentation backbone network does not have fully connected layers, which can accurately perform semantic segmentation on the lane line image to be identified and obtain a relatively accurate semantic feature image.

[0045] The lane line spatial convolutional network has three stages. The first layer of each stage consists of a convolutional layer with a stride of s=2 and a kernel size of 3*3. The other layers in each stage have the same number of filters and the same output feature image size. Based on these three stages, the lane line spatial convolutional network has a high ability to extract lane line spatial shape features, and can effectively and accurately extract detailed features from the lane line image to be identified, obtaining a relatively accurate detailed feature image.

[0046] In other words, after acquiring the lane line image to be identified, the electronic device can input the lane line image to be identified into the residual lane line detection model; then, the electronic device obtains a high-quality semantic feature image based on the semantic segmentation backbone network in the residual lane line detection model, and obtains an effective detail feature image based on the lane line spatial convolutional network in the residual lane line detection model.

[0047] Optionally, the electronic device performs semantic segmentation on the lane line image to be identified based on the semantic segmentation backbone network in the residual lane line detection model to obtain a semantic feature image; the electronic device performs semantic segmentation on the lane line image to be identified based on the semantic segmentation backbone network in the residual lane line detection model to obtain a first feature image; the electronic device performs multiple deconvolution processes on the first feature image to obtain a semantic feature image.

[0048] Because the electronic device is based on a semantic segmentation backbone network, the resolution of the first feature image obtained is smaller than that of the detail feature image, resulting in a small size for the first feature image. This makes it difficult for the electronic device to subsequently aggregate the first feature image and the detail feature image. Therefore, after acquiring the first feature image, the electronic device can perform multiple deconvolution processes on it. While maintaining the quality of the first feature image, this effectively increases its size, resulting in a semantic feature image that is compatible with the detail feature image. In other words, the final size of the semantic feature image obtained by the electronic device is the same as the size of the detail feature image, facilitating the subsequent effective aggregation of the semantic feature image and the detail feature image.

[0049] Optionally, the electronic device determines the semantic feature image and the detail feature image based on the acquired lane line image to be identified. This may include: the electronic device acquiring the lane line image to be identified; the electronic device preprocessing the lane line image to be identified; and determining the semantic feature image and the detail feature image based on the preprocessed lane line image to be identified.

[0050] The pretreatment may include at least one of the following: noise reduction treatment, expansion treatment, or corrosion treatment.

[0051] Denoising refers to removing environmental noise from the image of lane lines to be identified;

[0052] Dilation refers to expanding the highlighted area in the lane line image to be identified, that is, the lane line image to be identified after dilation has a larger highlighted area than the lane line image to be identified before dilation.

[0053] Erosion refers to the process of eroding the bright areas in the lane line image to be identified, meaning that the lane line image after erosion has a smaller bright area than the lane line image before erosion.

[0054] After acquiring the lane line image to be identified, the electronic device can preprocess the lane line image to be identified because the image quality of the lane line image is poor, so as to obtain a preprocessed lane line image to be identified with better image quality. Then, based on the preprocessed lane line image to be identified, the electronic device can accurately determine the corresponding semantic feature image and detail feature image.

[0055] 102. Aggregate the semantic feature image and the detail feature image to obtain the aggregated feature image.

[0056] After acquiring semantic feature images and detail feature images, electronic devices can aggregate the semantic feature images and detail feature images to obtain an aggregated feature image corresponding to the lane line image to be identified.

[0057] Wherein, semantic feature image ∈ R H×W×C ; Detail feature image ∈ R H×W×C R represents a real number; H×W×C represents the three spatial dimensions of the real number R; H represents the number of pixels in a row of the image, H≥1; W represents the number of pixels in a column of the image, W≥1; C represents the number of pixel channels in the image, C≥1.

[0058] Optionally, the electronic device may aggregate the semantic feature image and the detail feature image to obtain an aggregated feature image, which may include: the electronic device obtaining the aggregated feature image based on the aggregation feature formula in the residual lane detection model.

[0059] The aggregation characteristic formula is o=f(H) d +H s );

[0060] o represents the aggregated feature image; H d =I c +sigmoid(S avg ), representing a detailed feature image; I c S represents the result of the first 1*1 convolution; avg H represents the result of the first average pooling process; s =I avg +sigmoid(S c ), representing a semantic feature image; I avg S represents the result of the second 1*1 convolution; c This represents the result of the second average pooling process; sigmoid(·) represents the nonlinear action function.

[0061] It should be noted that the result of the first average pooling is the average pixel value of all pixels in the detail feature image; the result of the second average pooling is the average pixel value of all pixels in the semantic feature image.

[0062] For example, such as Figure 2 The diagram shown is a schematic representation of the method for determining aggregated feature images from aggregated semantic feature images and detail feature images provided by the present invention. Figure 2 In this process, the electronic device inputs a semantic feature image into a 3*3 convolutional network layer (3*3Conv), which, through a batch normalization (BN) layer and a ReLU activation function layer, yields a first 1*1 convolutional result (1*1Conv) and a first average pooling result (3*3Avgpooling). The electronic device then inputs a detail feature image into the same 3*3 convolutional network layer (3*3Conv), which, through a BN layer and a ReLU layer, yields a second 1*1 convolutional result (1*1Conv) and a second average pooling result (3*3Avgpooling). Finally, the electronic device aggregates (Sum) the first 1*1Conv, the first 3*3Avgpooling, the second 1*1Conv, and the second 3*3Avgpooling, and inputs this aggregated feature image into a 3*3 convolutional network layer, which, through a BN layer, yields an aggregated feature image ∈ R. H×W×C .

[0063] Electronic devices based on the aggregation feature formula in the residual lane line detection model can effectively aggregate semantic feature images and detail feature images to obtain highly accurate aggregated feature images.

[0064] 103. Perform multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image.

[0065] After acquiring the aggregated feature image, the electronic device can perform multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image corresponding to the lane line image to be identified.

[0066] In some embodiments, the electronic device performs multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image. This may include: the electronic device performing pooling and channel compression processing on the aggregated feature image based on the Position Attention Model (PAM) module in the residual lane detection model to obtain a first spatial feature image; the electronic device performing sampling processing on the first spatial feature image based on the Self-adaptation Model (SAM) module in the residual lane detection model to obtain a second spatial feature image; and the electronic device aligning the deep feature image corresponding to the second spatial feature image with the shallow feature image based on the Lane Feature Self-adaptation Alignment Model (LFAM) module in the residual lane detection model to obtain a multi-scale spatial feature image.

[0067] Pooling refers to the process of processing all pixels in the aggregated feature image. Optionally, the pooling process may include average pooling and max pooling.

[0068] Channel compression processing refers to compressing the pixel channels corresponding to all pixels in the aggregated feature image to obtain a first spatial feature map. The number of pixel channels in the first spatial feature map is less than the number of pixel channels in the aggregated feature image, which can effectively reduce the information loss in the aggregated feature image.

[0069] Sampling processing refers to the process of processing all pixels in the first spatial feature image. Optionally, the sampling processing may include upsampling processing and downsampling processing.

[0070] Electronic devices can obtain multi-scale spatial feature images with better image quality by performing a series of processing steps on aggregated feature images based on PAM, SAM and LFAM in the residual lane detection model.

[0071] In some embodiments, the electronic device performs pooling and channel compression processing on the aggregated feature image to obtain a first spatial feature image, which may include: the electronic device performing average pooling processing on the aggregated feature image to obtain a first pooling result; the electronic device performing max pooling processing on the aggregated feature image to obtain a second pooling result; the electronic device using a concatenation method to summarize the first pooling result and the second pooling result to obtain a target pooling result; the electronic device performing channel compression processing on the target pooling result to obtain a channel compression result; and the electronic device obtaining the first spatial feature image based on the channel compression result.

[0072] In this context, average pooling refers to determining the average pixel value U1 of all pixels in the aggregated feature image; that is, the first pooling result is this average pixel value U1, where the average pixel value U1 ∈ R. H×W×C ;

[0073] Max pooling refers to determining the maximum pixel value U2 among all pixels in the aggregated feature image; that is, the second pooling result is this maximum pixel value U2, where the maximum pixel value U2 ∈ R. H×W×C .

[0074] Optionally, the maximum pixel value U2 and the average pixel value U1 can be the same or different; no specific limitation is made here.

[0075] After performing average pooling on the aggregated feature image to obtain a first pooling result, and then performing max pooling on the aggregated feature image to obtain a second pooling result, the electronic device can use a cascaded approach to summarize the first pooling result and the second pooling result to obtain a target pooling result U, i.e., U can include U1 and U2. Then, the electronic device performs channel compression on the target pooling result, aggregates feature maps on the spatial dimension (H×W) of the target pooling result to obtain a channel compression result, and then obtains a first spatial feature image. The channel compression result is a channel descriptor. The function of the channel descriptor is to generate a globally distributed embedding of the channel feature response corresponding to the target pooling result, allowing all layers of the network in the residual lane detection model to use information from the global receptive field.

[0076] It should be noted that the timing of the electronic device determining the first pooling result and the electronic device determining the second pooling result is not limited.

[0077] For example, such as Figure 3 The diagram shown is a schematic of the method for determining the first spatial feature image based on the Spatial Location Attention (PAM) module provided by the present invention. Figure 3 In this process, the electronic device obtains a first pooling result (MaxPool) and a second pooling result (AvgPool) based on the aggregated feature image; then, the electronic device concatenates the first pooling result and the second pooling result to obtain a target pooling result, which ∈ R. 1×1×C Next, the electronic device performs non-linear activation (sigmoid) processing on the target pooling result to obtain the first spatial feature image.

[0078] Optionally, the electronic device performs channel compression processing on the target pooling result to obtain the channel compression result, which may include: the electronic device obtaining the channel compression result based on the channel compression processing formula.

[0079] The channel compression formula is as follows:

[0080] u c (i, j) represents the pooling result in the i-th row and j-th column of the target pooling result U;

[0081] z c =F c (u c ) indicates the channel compression result.

[0082] Electronic devices can obtain accurate channel compression results based on the channel compression processing formula.

[0083] Optionally, the electronic device obtains the first spatial feature image based on the channel compression result, which may include: the electronic device obtaining the first spatial feature image based on an attention formula.

[0084] The attention formula is s = F a (z c ,W)=δ(W2(W1 z c ));

[0085] s represents the first spatial feature image; δ(·) is the sigmoid(·); C1≥1, r1 represents the first attention layer; C2≥1, r2 represents the number of the second attention layer.

[0086] Optionally, W1 and W2 can be the same or different; no specific restrictions are imposed here.

[0087] Electronic devices can obtain a relatively accurate first spatial feature image based on the attention formula.

[0088] In some embodiments, the electronic device performs sampling processing on the first spatial feature image to obtain a second spatial feature image, which may include: the electronic device performs sampling processing on the first spatial feature image according to a preset offset vector guide table to obtain a second spatial feature image.

[0089] During the sampling process of the first spatial feature image according to the preset offset vector guide table, the electronic device can make the edge pixels of the first spatial feature image more likely to be assigned to the correct category, so as to obtain a more accurate second spatial feature image.

[0090] Optionally, the electronic device may sample the first spatial feature image to obtain a second spatial feature image, which may include: the electronic device obtaining the second spatial feature image based on a pixel mapping formula.

[0091] Wherein, the pixel mapping formula is

[0092] Represents the pixel coordinates in the first spatial feature image; Represents the row pixel coordinates in the first spatial feature image; Represents the column pixel coordinates in the first spatial feature image; These are the pixel coordinates in the second spatial feature image; Represents the row pixel coordinates in the second spatial feature image; Represents the column pixel coordinates in the second spatial feature image; Represents the preset feature map transformation matrix; α ij This represents the preset feature map transformation parameters.

[0093] Electronic devices can accurately obtain the second spatial feature image based on the pixel mapping formula.

[0094] Optionally, the electronic device obtains the second spatial feature image based on the pixel mapping formula, which may include: the electronic device establishing a feature map index between the first spatial feature image and the second spatial feature image according to the pixel mapping formula; the electronic device obtaining the second spatial feature image according to the feature map index and the first formula or the second formula.

[0095] The first formula is:

[0096] The second formula is:

[0097] Out pix Represents the second spatial feature image; ε h ε represents the first sampling coordinate offset; w Indicates the offset of the second sampling coordinate; This represents the Kronecker delta function; max indicates taking the maximum value.

[0098] It should be noted that the output of both the first and second formulas uses distance as the weight and takes the distance in pixels. The weighted summation of the four most recent pixel values ​​can effectively alleviate the problem of misclassification of boundary pixels caused by sampling processing, thereby effectively improving the accuracy of image segmentation. In other words, the electronic device can obtain a more accurate second spatial feature image based on the feature map index and the first formula or the second formula.

[0099] Optionally, the first formula and the second formula can be collectively referred to as the adaptive difference formula, which can effectively fill pixels in the first spatial feature image.

[0100] In some embodiments, the electronic device aligns the deep feature image corresponding to the second spatial feature image with the shallow feature image to obtain a multi-scale spatial feature image. This may include: the electronic device aligning the deep feature image corresponding to the second spatial feature image with the shallow feature image, and obtaining pixel pairs at corresponding positions between the aligned deep feature image and the aligned shallow feature image; the electronic device then fuses the pixel pairs to obtain the multi-scale spatial feature image.

[0101] Among them, the multi-scale spatial feature image is a spatial feature pyramid with multiple layers of feature information.

[0102] Electronic devices can align the deep feature image corresponding to the second spatial feature image with the shallow feature image via LFAM, and then fuse the pixel pairs at corresponding positions of the two aligned images to obtain a multi-scale spatial feature image.

[0103] Optionally, the electronic device may fuse pixel pairs to obtain a multi-scale spatial feature image, which may include: after fusing the pixel pairs, the electronic device may fuse the fusion result with a new shallow feature image to obtain a multi-scale spatial feature image.

[0104] The fusion result is fused with the new shallow feature image at least once.

[0105] For example, such as Figure 4 The diagram shown is a schematic of how the LFAM (Landscape Alignment Module) based on spatial features determines multi-scale spatial feature images, as provided by this invention. Figure 4 In this process, the electronic device obtains a multi-scale spatial feature image by performing at least one pixel pair fusion based on the spatial feature adaptive alignment module LFAM and the second spatial feature image.

[0106] 104. Based on the affinity domain corresponding to the multi-scale spatial feature image, determine the lane lines in the lane line image to be identified.

[0107] In this context, the affinity region refers to the area in a multi-scale spatial feature image where pixels are relatively similar.

[0108] After acquiring a multi-scale spatial feature image, the electronic device can first determine the affinity region corresponding to the multi-scale feature image; then, the electronic device clusters the affinity region to obtain lane lines in the lane line image to be identified with higher accuracy.

[0109] In some embodiments, the electronic device determines the lane lines in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image. This may include: the electronic device slicing the multi-scale spatial feature image to obtain a slicing result; the electronic device performing convolution processing on the slicing result to obtain a convolution result; and the electronic device determining the lane lines in the lane line image to be identified based on the affinity domain clustering method, according to the pixel association information and mask information corresponding to the convolution result.

[0110] For example, such as Figure 5 The diagram shown is a schematic representation of the slicing result determined by the Lane Line Spatial Feature Extraction Neural Network (LSCNN) provided by this invention. Figure 5 In this process, after acquiring a multi-scale spatial feature image, the electronic device can use a Lane Spatial Convolutional Neural Network (LSCNN) to slice the multi-scale spatial feature image according to the row dimension H and column dimension W to obtain the sliced ​​result. Then, the electronic device sends the sliced ​​result to a convolutional layer with a kernel size of C*h for convolution processing to obtain the convolution result, where h represents the kernel width of the convolutional layer. Based on the pixel correlation information and mask information corresponding to the convolution result, the electronic device clusters the lane line pixels in the convolution result into horizontal and vertical directions to obtain the lane lines in the lane line image to be identified.

[0111] Among them, LSCNN is the first recurrent neural network.

[0112] For example, the electronic device slices the multi-scale spatial feature image according to the row dimension H to obtain H slice results.

[0113] Optionally, after the electronic device performs convolution processing on the slicing results to obtain the convolution result, the method may further include: the electronic device performs slicing processing on the convolution result again to obtain a new slicing result; the electronic device performs convolution processing on the new slicing result again to obtain a new convolution result, and repeats the steps of obtaining a new slicing result and obtaining a new convolution result until the last slicing result is obtained, and then stops repeating the operation.

[0114] Different convolutional kernels have different kernel weights, which are shared across all slice results.

[0115] Optionally, the electronic device uses a lane line spatial feature extraction neural network LSCNN to slice the multi-scale spatial feature image according to the row and column dimensions to obtain the slicing result. This can include the electronic device obtaining the slicing result based on an update formula.

[0116] The update formula is as follows:

[0117]

[0118] K m,p,n This represents the weight between pixels in the last slice channel m and pixels in the current slice channel p, with the offset between the two pixels being K columns. p represents the channel index in the multi-scale spatial feature image, q represents the row pixel index, and n represents the column pixel index. p,q,l M′ represents the three-dimensional tensor corresponding to a multi-scale spatial feature image; p,q,l This indicates the sliced ​​result.

[0119] Optionally, the electronic device can determine the lane lines in the lane line image to be identified based on the affinity domain clustering method, according to the pixel correlation information and mask information corresponding to the convolution result. This can include: the electronic device can determine the lane lines in the lane line image to be identified based on the affinity domain loss function.

[0120] Wherein, the affinity domain loss function is L total =L BCE +L loU +L AF ;

[0121]

[0122]

[0123]

[0124] L total L represents the total loss function; BCE L represents the cross-entropy loss function; loU L represents the joint loss function; AF Represents the regression loss function;

[0125] N represents the total number of pixels; y represents the penalty weight for increasing the foreground pixels; t i Represents the value of the i-th pixel; o i N represents the i-th aggregated feature image; fg This indicates the total number of foreground pixels; This represents the value of the i-th pixel in the horizontal affinity domain; This represents the i-th aggregated feature image of the horizontal affinity domain; This represents the value of the i-th pixel in the vertical affinity domain; This represents the i-th aggregated feature image of the vertical affinity domain.

[0126] Electronic devices utilize an affinity domain loss function to minimize the distance between pixel embeddings belonging to the same lane line and maximize the distance between pixel embeddings belonging to different lane lines. This causes pixel embeddings from the same lane to cluster together, forming a unique cluster for each lane, enabling affinity domain clustering to detect a variable number of lane lines. In other words, the lane lines obtained by this electronic device based on the affinity domain loss function are relatively accurate.

[0127] In this embodiment of the invention, semantic feature images and detail feature images are determined based on the acquired lane line image to be identified; the semantic feature images and detail feature images are aggregated to obtain an aggregated feature image; multi-scale feature alignment and fusion are performed on the aggregated feature image to obtain a multi-scale spatial feature image; and lane lines in the lane line image to be identified are determined based on the affinity domain corresponding to the multi-scale spatial feature image. This method addresses the deficiency of existing lane line detection methods in accurately detecting lane lines, thereby effectively improving the detection accuracy of lane lines in lane line images to be identified.

[0128] For example, such as Figure 6 The image shown is a schematic diagram of a scenario for the lane line detection method provided by this invention. Figure 6 In the diagram, C1, C2, and C3 represent detailed feature images; S1, S2, and S3 represent semantic feature images; G1, G2, and G3 represent aggregated feature images; and H1 and H2 represent multi-scale spatial feature images. The electronic device determines the lane lines in the lane line image to be identified based on the affinity domain corresponding to these multi-scale spatial feature images.

[0129] Specifically, the residual lane detection model first extracts features from the lane line image to be identified. The lane line spatial convolutional network is constructed by stacking simple convolution operations. The semantic segmentation backbone network uses a pre-trained residual network with pooling and fully connected layers removed to adapt to the lane line detection task. Simultaneously, a lane line feature extraction module is added to the semantic segmentation backbone network. Because the semantic segmentation backbone network performs faster downsampling compared to the lane line spatial convolutional network, a mismatch occurs between the resolution of the semantic feature image and the resolution of the detail feature image. Therefore, a deconvolution block is added to the semantic segmentation backbone network to match the resolution of the semantic feature image with that of the detail feature image.

[0130] Then, the semantic feature image and the detail feature image are aggregated to obtain the aggregated feature image. After passing through the spatial pyramid multi-scale fusion module, starting from the deepest feature image, it is aligned with the shallow feature image through the feature adaptive alignment fusion module. Then, the corresponding elements are added together to obtain the multi-scale spatial feature image. In this way, the new multi-scale spatial feature image is then passed through the feature adaptive alignment fusion module and fused with the shallower feature image.

[0131] Finally, an end-to-end lane detection network is formed by integrating the affinity domain loss function with a feedforward network. Lane line pixels are clustered into corresponding lane instances in both the horizontal and vertical directions using inter-pixel correlation information and binary masks. The affinity domain loss function minimizes the distance between pixel embeddings belonging to the same lane line and maximizes the distance between pixel embeddings belonging to different lane lines. This ensures that pixel embeddings of the same lane line cluster together, forming a unique cluster for each lane line, allowing affinity domain clustering to detect a variable number of lane lines.

[0132] The lane line detection device provided by the present invention is described below. The lane line detection device described below can be referred to in correspondence with the lane line detection method described above.

[0133] like Figure 7 The diagram shown is a structural schematic of the lane line detection device provided by the present invention, which may include:

[0134] The processing module 702 is used to determine a semantic feature image and a detail feature image based on the lane line image to be identified acquired by the acquisition module 701; aggregate the semantic feature image and the detail feature image to obtain an aggregated feature image; perform multi-scale feature alignment fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determine the lane line in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image.

[0135] Optionally, the acquisition module 701 is specifically used to acquire the image of the lane line to be identified;

[0136] The processing module 702 is specifically used to perform semantic segmentation on the lane line image to be identified based on the semantic segmentation backbone network in the residual lane line detection model to obtain the semantic feature image; and to extract detailed features from the lane line image to be identified based on the lane line spatial convolutional network in the residual lane line detection model to obtain the detailed feature image.

[0137] Optionally, the processing module 702 is specifically used to perform pooling and channel compression processing on the aggregated feature image based on the spatial position attention module in the residual lane detection model to obtain a first spatial feature image; to perform sampling processing on the first spatial feature image based on the feature adaptation module in the residual lane detection model to obtain a second spatial feature image; and to align the deep feature image corresponding to the second spatial feature image with the shallow feature image based on the spatial feature adaptive alignment module in the residual lane detection model to obtain the multi-scale spatial feature image.

[0138] Optionally, the processing module 702 is specifically used to perform average pooling on the aggregated feature image to obtain a first pooling result; perform max pooling on the aggregated feature image to obtain a second pooling result; summarize the first pooling result and the second pooling result in a cascaded manner to obtain a target pooling result; perform channel compression on the target pooling result to obtain a channel compression result; and obtain the first spatial feature image based on the channel compression result.

[0139] Optionally, the processing module 702 is specifically used to perform sampling processing on the first spatial feature image according to a preset offset vector guide table to obtain the second spatial feature image.

[0140] Optionally, the processing module 702 is specifically used to align the deep feature image and the shallow feature image corresponding to the second spatial feature image, and obtain the pixel pairs at corresponding positions between the aligned deep feature image and the aligned shallow feature image; and fuse the pixel pairs to obtain the multi-scale spatial feature image.

[0141] Optionally, the processing module 702 is specifically used to perform slicing processing on the multi-scale spatial feature image to obtain slicing results; to perform convolution processing on the slicing results to obtain convolution results; and to determine the lane lines in the lane line image to be identified based on the affinity domain-based clustering method, according to the pixel correlation information and mask information corresponding to the convolution results.

[0142] like Figure 8 The diagram shows the structure of an electronic device provided by the present invention. This electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a lane line detection method. This method includes: determining a semantic feature image and a detail feature image based on an acquired lane line image to be identified; aggregating the semantic feature image and the detail feature image to obtain an aggregated feature image; performing multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determining the lane line in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image.

[0143] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the lane line detection method provided by the above methods. The method includes: determining a semantic feature image and a detail feature image based on an acquired lane line image to be identified; aggregating the semantic feature image and the detail feature image to obtain an aggregated feature image; performing multi-scale feature alignment fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determining the lane line in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image.

[0145] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the lane line detection method provided by the above methods. The method includes: determining a semantic feature image and a detail feature image based on an acquired lane line image to be identified; aggregating the semantic feature image and the detail feature image to obtain an aggregated feature image; performing multi-scale feature alignment fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determining the lane line in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image.

[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lane line detection method, characterized in that, include: Based on the acquired lane line image to be identified, determine the semantic feature image and the detail feature image; The semantic feature image and the detail feature image are aggregated to obtain an aggregated feature image; The aggregated feature image is subjected to multi-scale feature alignment and fusion to obtain a multi-scale spatial feature image; Based on the affinity domain corresponding to the multi-scale spatial feature image, the lane lines in the lane line image to be identified are determined; The step of performing multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image includes: Based on the spatial position attention module in the residual lane detection model, the aggregated feature image is subjected to pooling and channel compression to obtain the first spatial feature image. Based on the feature adaptive module in the residual lane detection model, the first spatial feature image is sampled to obtain the second spatial feature image. Based on the spatial feature adaptive alignment module in the residual lane detection model, the deep feature image corresponding to the second spatial feature image is aligned with the shallow feature image to obtain the multi-scale spatial feature image. The step of determining the lane lines in the lane line image to be identified based on the affinity region corresponding to the multi-scale spatial feature image includes: The multi-scale spatial feature image is sliced ​​to obtain the slicing result; The sliced ​​results are subjected to convolution processing to obtain the convolution result; The affinity-based clustering method determines the lane lines in the lane line image to be identified based on the pixel association information and mask information corresponding to the convolution result.

2. The method according to claim 1, characterized in that, The step of determining the semantic feature image and detail feature image based on the acquired lane line image to be identified includes: Acquire the image of the lane line to be identified; Based on the semantic segmentation backbone network in the residual lane line detection model, the lane line image to be identified is semantically segmented to obtain the semantic feature image. Based on the lane line spatial convolutional network in the residual lane line detection model, detail features are extracted from the lane line image to be identified to obtain the detail feature image.

3. The method according to claim 1, characterized in that, The process of performing pooling and channel compression on the aggregated feature image to obtain a first spatial feature image includes: The aggregated feature image is subjected to average pooling to obtain the first pooling result; Max pooling is performed on the aggregated feature image to obtain the second pooling result; By using a cascading approach, the first pooling result and the second pooling result are summarized to obtain the target pooling result; The target pooling result is subjected to channel compression processing to obtain the channel compression result; The first spatial feature image is obtained based on the channel compression result.

4. The method according to claim 1, characterized in that, The step of sampling the first spatial feature image to obtain the second spatial feature image includes: According to the preset offset vector guide table, the first spatial feature image is sampled to obtain the second spatial feature image.

5. The method according to claim 1, characterized in that, The step of aligning the deep feature image corresponding to the second spatial feature image with the shallow feature image to obtain the multi-scale spatial feature image includes: Align the deep feature image corresponding to the second spatial feature image with the shallow feature image, and obtain the pixel pairs at corresponding positions between the aligned deep feature image and the aligned shallow feature image; The pixel pairs are fused to obtain the multi-scale spatial feature image.

6. A lane line detection device, characterized in that, include: The processing module is used to determine a semantic feature image and a detail feature image based on the lane line image to be identified acquired by the acquisition module; aggregate the semantic feature image and the detail feature image to obtain an aggregated feature image; perform multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image; and determine the lane lines in the lane line image to be identified based on the affinity domain corresponding to the multi-scale spatial feature image. The step of performing multi-scale feature alignment and fusion on the aggregated feature image to obtain a multi-scale spatial feature image includes: Based on the spatial position attention module in the residual lane detection model, the aggregated feature image is subjected to pooling and channel compression to obtain the first spatial feature image. Based on the feature adaptive module in the residual lane detection model, the first spatial feature image is sampled to obtain the second spatial feature image. Based on the spatial feature adaptive alignment module in the residual lane detection model, the deep feature image corresponding to the second spatial feature image is aligned with the shallow feature image to obtain the multi-scale spatial feature image. The step of determining the lane lines in the lane line image to be identified based on the affinity region corresponding to the multi-scale spatial feature image includes: The multi-scale spatial feature image is sliced ​​to obtain the slicing result; The sliced ​​results are subjected to convolution processing to obtain the convolution result; The affinity-based clustering method determines the lane lines in the lane line image to be identified based on the pixel association information and mask information corresponding to the convolution result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the lane line detection method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the lane detection method as described in any one of claims 1 to 5.

Images