A method and system for processing lane lines

Abstract

This invention relates to a method and system for processing lane lines. The method includes: receiving a first image; performing three-level feature extraction processing on the first image based on a feature extraction backbone network to obtain corresponding first-level and second-level feature images; performing lane line start point detection processing based on a key point detection network according to the first-level and second-level feature images to obtain a corresponding second image; performing lane line recognition processing on lane lines in the second image with each first start point as the starting position according to a lane line recognition network to obtain corresponding first lane line pixel sequences; and marking lane lines on the second image according to each first lane line pixel sequence. This invention reduces the computational load for lane line recognition, thereby improving recognition efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for processing lane lines. Background Technology

[0002] Autonomous vehicles need to identify the location information of themselves and surrounding obstacles during operation. This location information includes not only the map coordinates of the vehicle and surrounding obstacles, but also the lane information of their respective lanes. Determining lane information requires identifying the lane lines in the environment surrounding the vehicle and its surrounding obstacles. However, if the radar point cloud output by the vehicle's perception system is used as the identification object when identifying lane lines, the three-dimensional nature of the point cloud data leads to excessive computational load and low identification efficiency. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, system, electronic device, and computer-readable storage medium for lane line processing. This invention uses a two-dimensional environmental image (with a data structure complexity far less than point cloud data) as the overall recognition object to achieve lane line recognition. During processing, this invention performs feature extraction and key point (lane line starting point) identification based on the two-dimensional image, constructs ray proposals based on the key points, and performs ray-lane line confidence assessment and ray-lane line offset estimation on the ray proposals. Finally, based on the corresponding ray-lane line offset, it modulates the pixel coordinates of the ray with the highest confidence to obtain the final lane line pixel sequence. This invention significantly reduces the computational load for lane line recognition, thereby improving recognition efficiency.

[0004] To achieve the above objectives, a first aspect of the present invention provides a method for processing lane lines, the method comprising:

[0005] Receive the first image;

[0006] Based on the feature extraction backbone network, the first image is subjected to three-level feature extraction processing to obtain the corresponding first-level and second-level feature images;

[0007] Based on the key point detection network, lane line start point detection processing is performed on the first-level and second-level feature images to obtain the corresponding second image; the second image includes multiple first start points.

[0008] Based on the lane line recognition network, the lane lines in the second image are recognized and processed with each of the first starting points as the starting position to obtain the corresponding first lane line pixel sequence.

[0009] Lane lines are marked on the second image based on the sequence of each first lane line pixel.

[0010] Preferably, the feature extraction backbone network has a feature pyramid network structure, consisting of a bottom-up downsampling residual network side and a top-down upsampling feature extraction network side. The downsampling residual network side includes first, second, and third level sampling network layers from bottom to top. The upsampling feature extraction network side includes third, second, and first level feature extraction layers from top to bottom. The first, second, and third level sampling network layers are connected sequentially. The third, second, and first level feature extraction layers are connected sequentially. The first level sampling network layer is also connected to the first level feature extraction layer. The second level sampling network layer is also connected to the second level feature extraction layer. The third level sampling network layer is also connected to the third level feature extraction layer.

[0011] Preferably, the step of performing three-level feature extraction processing on the first image based on the feature extraction backbone network to obtain corresponding first-level and second-level feature images specifically includes:

[0012] The first image is input into the feature extraction backbone network;

[0013] The first-level sampling network layer of the feature extraction backbone network downsamples the first image according to the specified feature dimension to expand the feature dimension and generate the corresponding first-level sampling image; the second-level sampling network layer performs downsampling residual operation on the first-level sampling image to generate the corresponding second-level sampling image; and the third-level sampling network layer performs downsampling residual operation on the second-level sampling image to generate the corresponding third-level sampling image.

[0014] The third-level feature extraction layer performs feature extraction processing on the third-level sampled image to generate a corresponding third-level feature image; the second-level feature extraction layer upsamples the third-level feature image, fuses the upsampled image with the second-level sampled image, and performs feature extraction processing on the fused image to generate the corresponding second-level feature image; the first-level feature extraction layer upsamples the second-level feature image, fuses the upsampled image with the first-level sampled image, and performs feature extraction processing on the fused image to generate the corresponding first-level feature image; and the first-level and second-level feature images are output.

[0015] Preferably, the key point detection network consists of a first convolutional network module, a second convolutional network module, a key point recognition module, and an image fusion module connected in sequence; the first convolutional network module has a kernel size of 3, a stride of 1, and padding of 1; the second convolutional network module has a kernel size of 3, a stride of 1, and padding of 1; the output image of the second convolutional network module is a thermal image, and its output image feature dimension is 1.

[0016] Preferably, the key point detection network performs lane line start point detection processing based on the first-level and second-level feature images to obtain the corresponding second image, specifically including:

[0017] The primary and secondary feature images are input into the key point detection network;

[0018] The first convolutional network module of the key point detection network performs convolution operations on the secondary feature image to generate the corresponding first convolutional image.

[0019] The second convolutional network module then performs convolution operations on the first convolutional image to generate the corresponding first heatmap.

[0020] The key point identification module then performs key point identification on the first heat map to obtain multiple first key points.

[0021] The image fusion module then enlarges the first heatmap according to the size of the first-level feature image to generate a corresponding second heatmap. On the enlarged second heatmap, pixels matching the first key points before enlargement are designated as corresponding second key points. On the first-level feature image, pixels corresponding to each second key point are recorded as first starting points, and the pixel values ​​of each first starting point are set to preset starting point pixel values. The modified first-level feature image is then output as the corresponding second image.

[0022] Preferably, the lane line recognition network comprises a lane line proposal module, a first multi-layer perception network module, a second multi-layer perception network module, and a lane line output module; the lane line proposal module is connected to the first and second multi-layer perception network modules respectively; the first and second multi-layer perception network modules are connected to the lane line output module respectively.

[0023] The first multilayer perceptron module consists of a first input layer, a first fully connected layer, a second fully connected layer, and a first output layer; the first input layer includes h*w neurons; the second fully connected layer includes 2 neurons; the classification function of the first output layer is the softmax function; the first output layer outputs a pair of positive and negative class scores; the positive and negative class scores are mutually exclusive; h is the total number of vertical pixels in the second image, and w is the pixel feature dimension of the second image;

[0024] The second multilayer perceptual network module consists of a second input layer, a third fully connected layer, and a fourth fully connected layer; the second input layer includes h*w neurons; the second fully connected layer includes 1+h neurons.

[0025] Preferably, the process of identifying lane lines in the second image based on the lane line recognition network, with each of the first starting points as the starting position, to obtain the corresponding first lane line pixel sequence, specifically includes:

[0026] The second image is input into the lane line recognition network;

[0027] The lane line proposal module of the lane line recognition network draws a semicircle of 0-180° on the second image with each of the first starting points as the center and h as the radius, and records it as the corresponding first semicircle. According to the specified ray angle rules, rays are drawn from each of the first starting points to the arc of the corresponding first semicircle to obtain multiple first rays. The w-dimensional pixel features of each first ray are extracted to form the corresponding first ray vector. The first ray vector is formed by sequentially concatenating h first ray point vectors. The feature dimension of the first ray point vector is w. The first first ray point vector is composed of the w-dimensional pixel features of the corresponding first starting point. The first ray point vector whose coordinates exceed the boundary of the second image is a unified w-dimensional preset feature vector.

[0028] The first multilayer perception network module then performs lane line confidence scoring on all the first ray vectors corresponding to each first starting point, generating corresponding positive and negative class scores.

[0029] The second multilayer perception network module then estimates the effective lane length and pixel offset for all the first ray vectors corresponding to each first starting point, generating a corresponding first effective length L and a first offset sequence; wherein, the first offset sequence includes multiple first offset distances Δs. i , 1≤i≤h;

[0030] The lane line output module selects the first ray with the maximum positive class score (preset positive class score) from all the first rays corresponding to each first starting point as the first matching ray; and trims the ray length according to the first effective length L corresponding to the first matching ray to generate the corresponding second matching ray; and calculates the first offset distance Δs corresponding to each first matching ray. i For the corresponding pixel point p on the second matching ray j x-coordinate j＝i Adjustments are made to obtain the new pixel p' j ; and process all the pixels p' in order from nearest to farthest. j The pixels are sorted to form the corresponding first lane line pixel sequence; wherein, the second matching ray consists of L pixels p j Composition, the pixel point pj The pixel coordinates are (x j ,y j ), 1≤j≤L; the first lane line pixel sequence consists of L pixels p' j Composed of, the pixel p' j The pixel coordinates are (x' j＝i ,y' j ), x' j＝i =x j＝i +△s i y' j =y j .

[0031] A second aspect of the present invention provides a system for implementing the lane line processing method described in the first aspect above. The system includes: a data receiving module, a feature extraction backbone network processing module, a key point detection network processing module, a lane line recognition network processing module, and a lane line marking processing module.

[0032] The data receiving module is used to receive the first image;

[0033] The feature extraction backbone network processing module is used to perform three-level feature extraction processing on the first image based on the feature extraction backbone network to obtain the corresponding first-level and second-level feature images;

[0034] The key point detection network processing module is used to perform lane line start point detection processing based on the first-level and second-level feature images to obtain a corresponding second image; the second image includes multiple first start points.

[0035] The lane line recognition network processing module is used to recognize lane lines in the second image with each of the first starting points as the starting position based on the lane line recognition network to obtain the corresponding first lane line pixel sequence.

[0036] The lane marking processing module is used to mark lanes on the second image according to the sequence of each first lane pixel.

[0037] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0038] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0039] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0041] This invention provides a method, system, electronic device, and computer-readable storage medium for processing lane lines. Using an environmental image (a first image) output by a perception system as the recognition object, a feature extraction backbone network based on a feature pyramid network (FPN) of three-level residual networks (ResNets) is used to perform three-level feature extraction on the first image, outputting first-level and second-level feature maps. Two convolutional neural network (CNN) modules of a keypoint detection network are used to perform a one-dimensional heat map transformation on the second-level feature map, and the keypoints on the heat map are used as lane line starting points and fused into the first-level feature map to obtain an environmental feature map (a second image) with multiple lane starting points. A lane line proposal module of a lane line recognition network is used to construct corresponding ray proposals for each lane line starting point on the second image; these proposals consist of multiple first rays. The first multi-layer perception of the lane line recognition network is then used. The multi-layer perceptron (MLP) network module of the lane line recognition network scores the lane line confidence of multiple first rays corresponding to each lane line starting point, and selects the ray with the highest score as the first matching ray that best matches the real lane line at that starting point. The second multi-layer perceptron (MLP) network module of the lane line recognition network then estimates the pixel-level offset relationship between the multiple first rays corresponding to each lane line starting point and the real lane line at that starting point, obtaining the corresponding lane line length L and pixel-level offset sequence. The lane line output module of the lane line recognition network then performs lane line length clipping and lane line offset modulation on the first matching rays corresponding to each lane line starting point based on their corresponding lane line length L and pixel-level offset sequence, thereby obtaining the first lane line pixel sequence that best matches the real lane line at each starting point. Finally, lane lines are marked on a second image with multiple lane starting points based on these first lane line pixel sequences. This invention not only significantly reduces the computational load of lane line recognition but also further improves the efficiency and accuracy of lane line recognition. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a lane line processing method provided in Embodiment 1 of the present invention;

[0043] Figure 2a This is a module structure diagram of the feature extraction backbone network provided in Embodiment 1 of the present invention;

[0044] Figure 2b This is a module structure diagram of the key point detection network provided in Embodiment 1 of the present invention;

[0045] Figure 2c This is a module structure diagram of the lane line recognition network provided in Embodiment 1 of the present invention;

[0046] Figure 3 This is a modular structure diagram of a lane line processing system provided in Embodiment 2 of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0049] Embodiment 1 of the present invention provides a method for processing lane lines, such as... Figure 1 The schematic diagram shows a lane line processing method provided in Embodiment 1 of the present invention. The method mainly includes the following steps:

[0050] Step 1: Receive the first image.

[0051] Here, the first image is a two-dimensional image of the vehicle and the surrounding environment, output by the vehicle perception system. This image is obtained by the image sensor (such as a camera) of the vehicle perception system capturing the current environment.

[0052] Step 2: Perform three-level feature extraction processing on the first image based on the feature extraction backbone network to obtain the corresponding first-level and second-level feature images;

[0053] Here, in the first embodiment of the present invention, the first image is actually extracted by the feature extraction backbone network of the feature pyramid network, and the first and second level features are output as the corresponding first and second level feature images.

[0054] Specifically, this includes: Step 21, inputting the first image into the feature extraction backbone network;

[0055] The feature extraction backbone network has a feature pyramid structure, consisting of a bottom-up downsampling residual network and a top-down upsampling feature extraction network. The downsampling residual network includes first, second, and third level sampling network layers from bottom to top; the upsampling feature extraction network includes third, second, and first level feature extraction layers from top to bottom. The first, second, and third level sampling network layers are connected sequentially; the third, second, and first level feature extraction layers are connected sequentially; the first level sampling network layer is connected to the first level feature extraction layer; the second level sampling network layer is also connected to the second level feature extraction layer; and the third level residual network layer is also connected to the third level feature extraction layer.

[0056] Here, the feature extraction backbone network is as follows: Figure 2a The diagram shows the module structure of the feature extraction backbone network provided in Embodiment 1 of the present invention. The network structure of the feature extraction backbone network is actually the feature pyramid network structure. This structure is a well-known structure for feature extraction based on multi-size fusion features. This structure consists of a bottom-up downsampling residual network side and a top-down upsampling feature extraction network side. It is named the feature pyramid network because its shape resembles a pyramid. In conventional feature extraction methods, the repeated use of convolution may cause excessive loss of original image features. However, the use of the feature pyramid network can effectively alleviate this problem, allowing more original feature information to be retained in the output feature map.

[0057] It should be noted that in the implementation of the feature extraction backbone network, i.e., the feature pyramid network, in Embodiment 1 of the present invention, the first three levels of the ResNet101 network are used to implement the first, second, and third level sampling network layers on the downsampling residual network side. As we know from the known ResNet101 network structure, it is specifically composed of five levels: conv1, conv2_x, conv3_x, conv4_x, and conv5_x. In the specific implementation of Embodiment 1 of the present invention, the first three levels, namely conv1, conv2_x, and conv3_x, are used to construct the first, second, and third level sampling network layers on the downsampling residual network side. In this way, Embodiment 1 of the present invention can reduce the amount of feature loss through multi-size feature data and reduce the overall computational load by controlling the depth of the residual network.

[0058] Step 22: The first-level sampling network layer of the feature extraction backbone network downsamples the first image according to the specified feature dimension to expand the feature dimension and generate the corresponding first-level sampling image; the second-level sampling network layer performs downsampling residual operation on the first-level sampling image to generate the corresponding second-level sampling image; and the third-level sampling network layer performs downsampling residual operation on the second-level sampling image to generate the corresponding third-level sampling image.

[0059] Here, the first-level sampling network layer is actually the conv1 layer of the ResNet101 network. The first-level sampling network layer is actually based on a 7×7 matrix with a dimension of 64. It performs high-dimensional convolution on the input first image with a sliding stride of 2 to obtain a feature map with 64 feature dimensions, which is the first-level sampling image. The specified feature dimension here is 64 dimensions. Since the sliding stride is 2, the shape ratio between the first-level sampling image and the first image remains unchanged, but the size and area become 1 / 4 of the first image.

[0060] The second-level sampling network layer is actually the conv2_x layer of the ResNet101 network. The second-level sampling network layer first performs pooling operation on the input first-level sampled image based on a 3×3 matrix with a sliding stride of 2 to obtain a downsampled feature map with 64 feature dimensions. Here, because the sliding stride is 2, the shape ratio and feature dimensions of this downsampled feature map remain unchanged from the first-level sampled image, but the size and area become 1 / 4 of the first-level sampled image, which is 1 / 16 of the first image. After obtaining this downsampled feature map, the second-level sampling network layer performs residual operation on it based on 3 consecutive residual modules and expands its feature dimensions from 64 to 128, thus obtaining a second-level sampled image with a feature dimension of 128 and a size of 1 / 16 of the first image.

[0061] The third-level sampling network layer is actually the conv3_x layer of the ResNet101 network. The third-level sampling network layer first performs pooling operation on the input second-level sampled image based on a 3×3 matrix with a sliding stride of 2 to obtain a downsampled feature map with 128 feature dimensions. Here, because the sliding stride is 2, the shape ratio and feature dimensions of this downsampled feature map remain unchanged with the second-level sampled image, but the size and area become 1 / 4 of the second-level sampled image, which is 1 / 64 of the first image. After obtaining this downsampled feature map, the second-level sampling network layer performs residual operation on it based on 4 sets of consecutive residual modules and expands its feature dimensions from 128 to 256, thus obtaining a third-level sampled image with a feature dimension of 256 and a size of 1 / 64 of the first image.

[0062] Step 23: The third-level feature extraction layer performs feature extraction processing on the third-level downsampled image to generate the corresponding third-level feature image; the second-level feature extraction layer upsamples the third-level feature image, fuses the upsampled image with the second-level downsampled image, and performs feature extraction processing on the fused image to generate the corresponding second-level feature image; the first-level feature extraction layer upsamples the second-level feature image, fuses the upsampled image with the first-level downsampled image, and performs feature extraction processing on the fused image to generate the corresponding first-level feature image; and the first-level and second-level feature images are output.

[0063] Here, as can be seen from the known step-by-step feature extraction method of the feature pyramid network, when the third-level feature extraction layer performs feature extraction processing on the third-level downsampled image, it actually performs convolution operation on the third-level downsampled image based on a preset 3×3 feature matrix to obtain the corresponding third-level feature image.

[0064] When the second-level feature extraction layer upsamples the third-level feature image, it actually first performs a 2× upsampling on the third-level feature image and reduces its dimension from 256 to 128, resulting in an upsampled feature map with a size of 1 / 16 of the first image and a feature dimension of 128. This upsampled feature map has the same size and dimension as the second-level downsampled image. The second-level feature extraction layer then fuses the features of this upsampled feature map and the second-level downsampled image to obtain the corresponding fused image. Finally, the second-level feature extraction layer performs a convolution operation on the fused image based on a preset 3×3 feature matrix to obtain the corresponding second-level feature image.

[0065] When the first-level feature extraction layer upsamples the second-level feature image, it actually performs a 2× upsampling on the second-level feature image and reduces its dimension from 128 to 64, resulting in an upsampled feature map with a size of 1 / 4 of the first image and a feature dimension of 64. This upsampled feature map has the same size and dimension as the first-level downsampled image. The first-level feature extraction layer then fuses the features of this upsampled feature map and the first-level downsampled image to obtain the corresponding fused image. The first-level feature extraction layer then performs a convolution operation on the fused image based on a preset 3×3 feature matrix to obtain the corresponding first-level feature image.

[0066] It should be noted that, before performing three-level feature extraction processing on the first image based on the feature extraction backbone network in Embodiment 1 of the present invention, the feature extraction backbone network needs to be trained to set the corresponding parameters for the 7×7 matrix, 3×3 matrix, and residual module used in each sampling network layer in the above steps; and to set the corresponding parameters for the 3×3 feature matrix and the relevant parameters of the 2× upsampling operation used in each feature extraction layer in the above steps; the training method is similar to the conventional feature pyramid network method, and will not be described further here.

[0067] Step 3: Based on the key point detection network, lane start point detection is performed on the first-level and second-level feature images to obtain the corresponding second image;

[0068] The second image includes multiple first starting points;

[0069] Here, in the first embodiment of the present invention, the heatmap of the secondary feature image is actually converted by two layers of convolution operation, and the key points on the heatmap are used as lane starting points. Then, the heatmap and the primary feature image are fused to obtain an environmental feature map with lane starting point markings, which is the second image.

[0070] Specifically, this includes: Step 31, inputting the primary and secondary feature images into the keypoint detection network;

[0071] The keypoint detection network consists of a first convolutional network module, a second convolutional network module, a keypoint recognition module, and an image fusion module connected sequentially. The first convolutional network module has a kernel size of 3, a stride of 1, and padding of 1. The second convolutional network module also has a kernel size of 3, a stride of 1, and padding of 1. The input image feature dimension of the first convolutional network module is consistent with the feature dimension of the secondary feature image, and its output feature map feature dimension is reduced to half of the feature dimension of the secondary feature image. The output image feature dimension of the second convolutional network module is 1, and the output image of the second convolutional network module is a thermal image.

[0072] Here, the key point detection network is as follows: Figure 2b The block structure diagram of the key point detection network provided in Embodiment 1 of the present invention is shown below;

[0073] Step 32: The first convolutional network module of the key point detection network performs convolution operation on the secondary feature image to generate the corresponding first convolutional image;

[0074] Specifically, it includes: a first convolutional network module using a first convolutional filter composed of 64 convolutional kernels of size 3×3 to perform convolution operations on the secondary feature image with a stride of 1 and padding of 1, thereby obtaining 64 feature maps with a feature dimension of 1 and the same length and width as the secondary feature image; and then concatenating these 64 feature maps to obtain a first convolutional image with the same length and width as the secondary feature image but with a feature dimension of 64.

[0075] Here, padding with 1 actually means adding one pixel with a set value (default is 0) to each of the four sides of the secondary feature image in a symmetrical manner (top and bottom, left and right). This ensures that the feature map size obtained after convolution with a 3×3 kernel and a stride of 1 is consistent with the original image, i.e., the secondary feature image. The feature dimension of the input image of the first convolutional network module is the same as that of the secondary feature image, which is 128. To reduce the feature dimension of its output feature map to half of that of the secondary feature image, i.e., 64, a convolutional filter consisting of 64 kernels of size 3×3 is needed to convolve the secondary feature image.

[0076] Step 33: The second convolutional network module performs convolution operations on the first convolutional image to generate the corresponding first heatmap;

[0077] Specifically, it includes: a second convolutional network module, using a second convolutional filter composed of a 3×3 convolutional kernel, performing convolution operations on the first convolutional image with a stride of 1 and padding of 1, thereby obtaining a feature map with a feature dimension of 1 and the same length and width as the first convolutional image, and then using this feature map as the corresponding first heatmap;

[0078] Here, padding with 1 actually means adding one pixel with a set value (default is 0) to each of the four sides of the first convolutional image in a symmetrical manner (top and bottom, left and right). This ensures that the feature map size obtained after convolution with a 3×3 kernel and a stride of 1 is consistent with the original image, i.e., the first convolutional image. The feature dimension of the input image of the second convolutional network module is the same as that of the first convolutional image, which is 64. To reduce the feature dimension of its output feature map to 1, a convolutional filter consisting of a 3×3 kernel is needed to convolve the first convolutional image.

[0079] After continuous convolution in steps 31-32 above, the feature map finally output by the second convolutional network module fuses the 128-dimensional features of the secondary feature image into a single feature value. This processing method is similar to that of a heat map. Therefore, the feature map finally output by the second convolutional network module is denoted as the first heat map, and the size of the first heat map is the same as that of the secondary feature image.

[0080] Step 34: The key point recognition module performs key point recognition on the first heat map to obtain multiple first key points;

[0081] Specifically, this includes: Step 341, normalizing the pixel values ​​of each pixel in the first heatmap to obtain the corresponding normalized heatmap;

[0082] Here, the normalized heatmap has the same size as the first heatmap;

[0083] Step 342: Construct a 9×9 symmetric two-dimensional Gaussian distribution space with 5 pixels as the Gaussian kernel radius; and construct the corresponding Gaussian kernel matrix as the specified Gaussian kernel based on the weight distribution relationship of the two-dimensional Gaussian distribution space.

[0084] The two-dimensional Gaussian distribution space includes a center point; the four neighboring points of the center point are: left, top, right, and bottom neighbors; there are two straight lines passing through the center point, namely line 1 and line 2. Line 1 passes through the left neighbor point - center point - right neighbor point, and line 2 passes through the top neighbor point - center point - bottom neighbor point; in the two-dimensional Gaussian distribution space, the Gaussian weight of the center point is the maximum weight, which is set to 1 by default. The Gaussian weight distribution relationship between the other points on lines 1 and 2 and the center point satisfies the characteristics of a symmetric Gaussian distribution. The Gaussian weight of points not on lines 1 and 2 is set to 0.

[0085] Step 343: Perform convolution operation on the normalized heatmap using a specified Gaussian kernel with a stride of 1 and a padding of 4 to obtain a normalized feature map with the same size as the first heatmap.

[0086] Here, because lane lines are typically marked with bright, vivid colors, and higher brightness and more vivid colors result in larger pixel values, Embodiment 1 of this invention uses the aforementioned specified Gaussian kernel to convolve the normalized heatmap, which essentially weights the pixel values ​​of each pixel according to a Gaussian distribution. The purpose is to increase the pixel difference between pixels, thereby enhancing the features of lane line pixels, weakening the features of pixels surrounding the lane lines, and improving the recognizability of lane line pixels. The padding of 4 is actually applied to the normalized heatmap... Figure 4 Four pixels with a set value (default 0) are added symmetrically to each edge, both vertically and horizontally. This ensures that the size of the normalized feature map obtained after convolution with a specified Gaussian kernel of 9×9 and a stride of 1 is consistent with the original image, i.e., the normalized heatmap. This is because the size of the normalized heatmap is consistent with the size of the first heatmap. Figure 1 Therefore, the size of the normalized feature map is the same as the size of the first heatmap;

[0087] Step 344: Use a 5×5 sliding window to slide on the normalized heatmap with a sliding step size of 1. Each time it slides, calculate the sum of pixel values ​​of all pixels in the sliding window to obtain the corresponding first sliding window sum. If the first sliding window sum exceeds the preset specified sum threshold, then the pixel with the largest pixel value in the current sliding window is recorded as the first key point.

[0088] Here, in Embodiment 1 of the present invention, the non-maximum suppression (NMS) method is used to screen for maxima in the normalized heatmap, and the screened pixels are recorded as the first key points;

[0089] Step 35: The image fusion module enlarges the first heatmap according to the size of the first-level feature image to generate a corresponding second heatmap; on the enlarged second heatmap, the pixels that match each first key point before enlargement are taken as the corresponding second key points; on the first-level feature image, the pixels corresponding to each second key point are recorded as the first starting point, and the pixel values ​​of each first starting point are set to the preset starting point pixel values; and the modified first-level feature image is output as the corresponding second image.

[0090] Here, the size of the first heatmap is the same as that of the second-level feature image, and the size of the second-level feature image is 1 / 4 of that of the first-level feature image. In Embodiment 1 of the present invention, the first heatmap is enlarged based on the bilinear interpolation algorithm to make its size consistent with that of the first-level feature image. After the image is enlarged, the coordinates of the first key point are migrated according to the enlargement relationship to obtain the coordinates of the second key point. In order to improve the recognition of the starting point, Embodiment 1 of the present invention will also set the pixel value of each first starting point to a higher pixel value, namely the starting point pixel value.

[0091] Step 4: Based on the lane line recognition network, the lane lines in the second image with each first starting point as the starting position are recognized to obtain the corresponding first lane line pixel sequence.

[0092] Here, in Embodiment 1 of the present invention, rays are first drawn within the 0-180° region based on each first starting point to obtain corresponding ray proposals; then, two parallel fully connected networks are used to score the lane line confidence of each ray in the proposals and to estimate the lane line length and lane line offset of each ray; then, the ray with the highest score is subjected to lane line clipping and pixel modulation based on the corresponding lane line length and lane line offset to obtain the pixel sequence that best matches the real lane line, namely the first lane line pixel sequence;

[0093] Specifically, this includes: Step 41, inputting the second image into the lane line recognition network;

[0094] The lane line recognition network comprises a lane line proposal module, a first multi-layer perception network module, a second multi-layer perception network module, and a lane line output module; the lane line proposal module is connected to the first and second multi-layer perception network modules respectively; the first and second multi-layer perception network modules are connected to the lane line output module respectively.

[0095] The first multilayer perceptron module consists of a first input layer, a first fully connected layer, a second fully connected layer, and a first output layer. The first input layer includes h*w neurons; the second fully connected layer includes 2 neurons; the classification function of the first output layer is the softmax function; the first output layer outputs a pair of positive and negative class scores; the positive and negative class scores are mutually exclusive; h is the total number of vertical pixels in the second image, and w is the pixel feature dimension of the second image; here, h is also the height of the second image, and the pixel feature dimension w of the second image is consistent with the feature dimension of the first-level feature image, which is 64. The positive and negative class scores are both 0 or 1 by default, and the positive and negative class scores are mutually exclusive, that is, when the positive class score is 1, the negative class score is 0, and when the positive class score is 0, the negative class score is 1.

[0096] The second multilayer perceptron module consists of a second input layer, a third fully connected layer, and a fourth fully connected layer; the second input layer includes h*w neurons; the fourth fully connected layer includes 1+h neurons; here, because the input of the second multilayer perceptron module is the same as that of the first multilayer perceptron module, the number of neurons in the second input layer is the same as that in the first input layer.

[0097] Here, the lane line recognition network is like... Figure 2c The module structure diagram of the lane line recognition network provided in Embodiment 1 of the present invention is shown below;

[0098] Step 42: The lane line proposal module of the lane line recognition network draws a semicircle of 0-180° on the second image with each first starting point as the center and h as the radius, and marks it as the corresponding first semicircle; and draws rays from each first starting point to the arc of the corresponding first semicircle according to the specified ray angle rules to obtain multiple first rays; and extracts the w-dimensional pixel features of each first ray to form the corresponding first ray vector.

[0099] The first ray vector is formed by sequentially concatenating h first ray point vectors. The feature dimension of the first ray point vector is w. The first first ray point vector is composed of the w-dimensional pixel features of the corresponding first starting point. The first ray point vector whose ray point coordinates exceed the boundary of the second image is a unified w-dimensional preset feature vector.

[0100] Here, if a horizontal line parallel to the bottom edge of the second image is drawn through the first starting point, the area above the first starting point is the corresponding 0-180° region. Numerous rays can be obtained by drawing rays from each first starting point to the corresponding first semicircle. Therefore, Embodiment 1 of this invention specifically defines a ray angle rule to restrict ray extraction. This ray angle rule is as follows: within the range of 0-50° (excluding 50°), a ray is drawn every 10° starting from 0° to obtain 5 first rays; within the range of 50-130° (excluding 130°), a ray is drawn every 5° starting from 50° to obtain 16 first rays; within the range of 130-180° (including 180°), a ray is drawn every 10° starting from 130° to obtain 6 first rays. Finally, according to the specified... The ray angle rule is to draw rays from each first starting point to the arc of the corresponding first semicircle, resulting in 5 + 16 + 6 = 27 first rays; the length of each first ray is set to h, and the pixel feature dimension of the second image is known to be w = 64, so the first ray vector corresponding to each first ray is naturally composed of h pixel vectors with feature dimension w = 64, which are also the first ray vectors; in actual processing, some first rays may actually exceed the boundary of the second image. For the first ray vector corresponding to such first rays, Embodiment 1 of the present invention treats the ray points that exceed the boundary of the second image as empty points, and sets the first ray vector of each empty point as a unified w-dimensional preset feature vector. This unified w-dimensional preset feature vector is a zero vector with dimension w = 64 by default;

[0101] Step 43: The first multilayer perception network module performs lane line confidence scoring on all first ray vectors corresponding to each first starting point, and generates corresponding positive and negative class scores.

[0102] Here, the first multilayer perception network module in the lane line recognition network is actually a classification prediction processing module with two fully connected layers and a normalization classifier; the input of the first input layer corresponds to the representation vector of a first ray, i.e., the first ray vector; the first fully connected layer calculates the overall lane line features of the first ray vector to generate an intermediate vector; the second fully connected layer estimates the probability that the first ray is a positive sample (lane line) and a negative sample (non-lane line) based on the intermediate vector to obtain two probability scores; the first output layer calculates the normalization index of these two probability scores based on the softmax function and performs classification prediction based on the calculation result and a preset threshold to obtain a pair of positive and negative classifications. The positive and negative class scores are either positive score = 1 and negative score = 0, or positive score = 0 and negative score = 1. As can be seen from the aforementioned ray angle rule, each first starting point corresponds to 27 first rays. After the first multilayer perception network module performs lane line confidence scoring on the 27 first ray vectors corresponding to each first starting point, it will obtain 27 pairs of positive and negative class scores. It should be noted that among these 27 pairs of positive and negative class scores, at most one pair is positive score = 1 and negative score = 0. In Embodiment 1 of the present invention, before using the first multilayer perception network module, its ray scoring performance is trained based on the real lane line labels of the road image. During training, the classification prediction loss is calculated based on the focus loss function.

[0103] Step 44: The second multilayer perception network module estimates the effective lane line length and pixel offset for all first ray vectors corresponding to each first starting point, and generates the corresponding first effective length L and first offset sequence.

[0104] The first offset sequence includes multiple first offset distances Δs. i , 1≤i≤h;

[0105] Here, the second multilayer perception network module in the lane line recognition network is actually a lane line length and deviation prediction processing module with two fully connected layers. The input of the second multilayer perception network module is the same as that of the first multilayer perception network module, so the number of neurons in the second input layer is the same as that in the first input layer. The third fully connected layer calculates the features of each ray point on the first ray to generate a multi-point multi-dimensional feature vector with multi-point multi-dimensional features. The fourth fully connected layer estimates the number of real lane line points based on the multi-point multi-dimensional feature vector to obtain a first effective length L as the output, and estimates the lateral offset of each point from the real lane line based on the multi-point multi-dimensional feature vector to obtain h first offset distances Δs. i As output, these h first offset distances △s i If i > L, then the first offset distance △s iThe default value is 0; In Embodiment 1 of the present invention, before using the second multilayer perception network module, the lane length and lane offset prediction performance of the network are trained based on the real lane label of the road image, and the length and offset prediction loss are calculated based on the mean absolute error L1 loss function (L1 loss) during training.

[0106] Step 45: The lane line output module selects the first ray with the preset maximum positive class score from all first rays corresponding to each first starting point as the first matching ray; and trims the ray length according to the first effective length L corresponding to the first matching ray to generate the corresponding second matching ray; and calculates the first offset distance Δs corresponding to each first matching ray. i For the corresponding pixel p on the second matching ray j x-coordinate j＝i Adjustments are made to obtain the new pixel p' j And process all pixels p' in order from nearest to farthest. j The pixels are sorted to form the corresponding first lane line pixel sequence.

[0107] The default maximum value for the positive class score is 1; the second matching ray consists of L pixels p. j Composition, pixel p j The pixel coordinates are (x j ,y j ), 1≤j≤L; the first lane line pixel sequence consists of L pixels p' j Composition, pixel p' j The pixel coordinates are (x' j＝i ,y' j ), x' j＝i =x j＝i +△s i y' j =y j .

[0108] Here, the lane line output module obtains 27 sets of positive and negative class scores for each first starting point from the first multilayer perception network module. In this embodiment of the invention, the positive class score is used as the preset maximum value of the positive class score (default is 1) as the filtering condition. That is, the first ray corresponding to the set of 27 sets (positive class score = 1, negative class score = 0) is selected as the ray that best matches the real lane line, i.e., the first matching ray. In addition, the lane line output module also obtains 27 sets (first effective length L + h first offset distances Δs) for each first starting point from the second multilayer perception network module. iIn this embodiment of the invention, the group corresponding to the first matching ray is selected for lane line pixel modulation. During lane line pixel modulation, this embodiment first trims the first matching ray based on the first effective length L, reducing its length from the original length h to length L. Then, h first offset distances Δs are used to... i Choose the first offset distance △s where 1≤i≤L i The lateral coordinates of the corresponding pixels on the second matching ray are modulated to obtain the pixel sequence that best matches the real lane line, which is the first lane line pixel sequence.

[0109] Step 5: Mark lane lines on the second image according to the sequence of pixels of each first lane line.

[0110] Here, through the above steps 1-4, multiple first starting points and a sequence of first lane line pixels corresponding to each first starting point can be obtained on the second image; when marking lane lines, pixels are marked on the second image according to each sequence of first lane line pixels and the marked pixels are connected sequentially to obtain the corresponding marked lane lines; in order to effectively distinguish each lane line, different marking colors can be used for each marked lane line in batches.

[0111] Figure 3 This is a modular structure diagram of a lane line processing system provided in Embodiment 2 of the present invention. This system can be a system, terminal device, or server implementing the aforementioned method embodiment 1, or it can be an apparatus capable of enabling the aforementioned system, terminal device, or server to implement the aforementioned method embodiment 1. For example, the apparatus can be a device or chip system of the aforementioned terminal device or server. Figure 3 As shown, the system includes: a data receiving module 201, a feature extraction backbone network processing module 202, a key point detection network processing module 203, a lane line recognition network processing module 204, and a lane line marking processing module 205.

[0112] The data receiving module 201 is used to receive the first image.

[0113] The feature extraction backbone network processing module 202 is used to perform three-level feature extraction processing on the first image based on the feature extraction backbone network to obtain the corresponding first-level and second-level feature images.

[0114] The key point detection network processing module 203 is used to perform lane line start point detection processing based on the first-level and second-level feature images using the key point detection network to obtain the corresponding second image; the second image includes multiple first start points.

[0115] The lane line recognition network processing module 204 is used to recognize lane lines in the second image based on each first starting point to obtain the corresponding first lane line pixel sequence.

[0116] The lane marking processing module 205 is used to mark lanes on the second image according to the sequence of each first lane pixel.

[0117] The lane line processing system provided in Embodiment 2 of the present invention can execute the method steps in Embodiment 1 of the above method. Its implementation principle and technical effect are similar, and will not be described again here.

[0118] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the data receiving module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0119] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0120] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0121] Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of Embodiment 1 of the present invention. Figure 4 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in Embodiment 3 of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0122] exist Figure 4The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0123] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0124] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0125] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0126] This invention provides a method, system, electronic device, and computer-readable storage medium for processing lane lines. Using an environmental image (first image) output by a perception system as the recognition object, a feature extraction backbone network based on a three-level residual network is used to perform three-level feature extraction on the first image, outputting first-level and second-level feature maps. Two convolutional network modules of a keypoint detection network are used to perform a one-dimensional heatmap transformation on the second-level feature map, and the keypoints on the heatmap are used as lane line starting points and fused into the first-level feature map to obtain an environmental feature map (second image) with multiple lane starting points. A lane line proposal module of a lane line recognition network is used to construct corresponding ray proposals for each lane line starting point on the second image; these proposals consist of multiple first rays. Finally, a first multilayer perception network module of the lane line recognition network is used to process the corresponding lane line starting points. Multiple first rays are used to score lane line confidence, and the ray with the highest score is selected as the first matching ray that best matches the real lane line at the starting point. The second multilayer perceptron module of the lane line recognition network is then used to estimate the pixel-level offset relationship between the multiple first rays corresponding to each lane line starting point and the real lane line at that starting point, obtaining the corresponding lane line length L and pixel-level offset sequence. The lane line output module of the lane line recognition network is then used to perform lane line length clipping and lane line offset modulation on the first matching rays corresponding to each lane line starting point, based on their corresponding lane line length L and pixel-level offset sequence, thereby obtaining the first lane line pixel sequence that best matches the real lane line at each starting point. Lane lines are then marked on a second image with multiple lane starting points based on these first lane line pixel sequences. This invention not only significantly reduces the computational load for lane line recognition but also further improves the efficiency and accuracy of lane line recognition.

[0127] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0128] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of processing a lane line, characterized by, The method comprises: receiving a first image; performing three-level feature extraction processing on the first image based on a feature extraction backbone network to obtain corresponding first-level and second-level feature images; performing lane line starting point detection processing on the first-level and second-level feature images based on a key point detection network to obtain a corresponding second image; the second image comprises a plurality of first starting points; performing identification processing on lane lines with each first starting point as a starting position in the second image based on a lane line identification network to obtain a corresponding first lane line pixel point sequence; performing lane line marking on the second image according to each first lane line pixel point sequence; wherein the lane line identification network comprises a lane line proposal module, a first multi-layer perception network module, a second multi-layer perception network module, and a lane line output module; the lane line proposal module is connected with the first and second multi-layer perception network modules respectively; the first and second multi-layer perception network modules are connected with the lane line output module respectively; the first multi-layer perception network module is composed of a first input layer, a first full connection layer, a second full connection layer, and a first output layer; the first input layer comprises h*w neurons; the second full connection layer comprises 2 neurons; the classification function of the first output layer is a softmax function; the first output layer outputs a pair of positive and negative class scores; the values of the positive and negative class scores are mutually exclusive; h is the total number of vertical pixel points of the second image, and w is the pixel point feature dimension of the second image; the second multi-layer perception network module is composed of a second input layer, a third full connection layer, and a fourth full connection layer; the second input layer comprises h*w neurons; the second full connection layer comprises 1+h neurons; the lane line identification network performs identification processing on lane lines with each first starting point as a starting position in the second image to obtain a corresponding first lane line pixel point sequence, specifically comprising: inputting the second image into the lane line identification network; and by the lane line proposal module of the lane line identification network, taking each first starting point as the center and h as the radius to draw a semicircle of 0-180° on the second image as a corresponding first semicircle; and according to a specified ray angle rule, drawing a ray from each first starting point to the circular arc of the corresponding first semicircle to obtain a plurality of first rays; and extracting the w-dimensional pixel point features of the pixel points passed by each first ray to form a corresponding first ray vector; wherein the first ray vector is sequentially spliced by h first ray point vectors, the feature dimension of the first ray point vector is w, the first first ray point vector is composed of the w-dimensional pixel point features of the corresponding first starting point, and the first ray point vector with a ray point coordinate beyond the boundary of the second image is a uniform w-dimensional preset feature vector; and by the first multi-layer perception network module, performing lane line confidence score on all first ray vectors corresponding to each first starting point respectively to generate corresponding positive and negative class scores; And by the second multi-layer perception network module, all the first ray vectors corresponding to each of the first starting points are respectively estimated for effective lane line length and pixel point offset to generate corresponding first effective length L and first offset sequence; wherein the first offset sequence includes a plurality of first offset distances △s i , 1≤i≤h; And the lane line output module, from all the first rays corresponding to each of the first starting points, selects the first ray with the positive class score being the preset positive class score maximum value as a first matching ray; and according to the first effective length L corresponding to the first matching ray, performs ray length clipping to generate a corresponding second matching ray; and according to each of the first offset distances △s corresponding to the first matching ray i , adjusts the horizontal coordinate x j of the corresponding pixel point p j＝i on the second matching ray to obtain a new pixel point p j ’ ; and sorts all the pixel points p'j in order from near to far to form a corresponding first lane line pixel point sequence; wherein the second matching ray is composed of L pixel points p j , the pixel point coordinate of the pixel point p j is (x j , y j ), and 1≤j≤L; the first lane line pixel point sequence is composed of L pixel points p ’ j , the pixel point coordinate of the pixel point p ’ j is (x ’ j＝i , y j ’ ), x ’ j＝i =x j＝i +△s i , y ’ j =y j .

2. The lane line processing method according to claim 1, characterized in that, The network structure of the feature extraction backbone network is a feature pyramid network structure, which is composed of a bottom-up down-sampling residual network side and a top-down up-sampling feature extraction network side; the down-sampling residual network side comprises first, second and third level sampling network layers from bottom to top; the up-sampling feature extraction network side comprises third, second and first level feature extraction layers from top to bottom; the first, second and third level sampling network layers are sequentially connected; the third, second and first level feature extraction layers are sequentially connected; the first level sampling network layer is further connected with the first level feature extraction layer; the second level sampling network layer is further connected with the second level feature extraction layer; and the third level sampling network layer is further connected with the third level feature extraction layer.

3. The method of processing a lane line according to claim 2, wherein, The feature extraction backbone network is used for performing three-level feature extraction processing on the first image to obtain corresponding first, second and third level feature images, specifically including: inputting the first image into the feature extraction backbone network; and using the first level sampling network layer of the feature extraction backbone network to perform down-sampling feature dimension expansion on the first image according to a specified feature dimension to generate a corresponding first level sampling image; using the second level sampling network layer to perform down-sampling residual operation on the first level sampling image to generate a corresponding second level sampling image; and using the third level sampling network layer to perform down-sampling residual operation on the second level sampling image to generate a corresponding third level sampling image; and using the third level feature extraction layer to perform feature extraction processing on the third level sampling image to generate a corresponding third level feature image; using the second level feature extraction layer to perform up-sampling on the third level feature image, performing image fusion on the up-sampling image and the second level sampling image, and performing feature extraction processing on the fused image to generate a corresponding second level feature image; using the first level feature extraction layer to perform up-sampling on the second level feature image, performing image fusion on the up-sampling image and the first level sampling image, and performing feature extraction processing on the fused image to generate a corresponding first level feature image; and outputting the first and second level feature images.

4. The lane line processing method according to claim 1, characterized in that the key point detection network is composed of a first convolutional network module, a second convolutional network module, a key point identification module and an image fusion module which are sequentially connected; the convolution kernel size of the first convolutional network module is 3, the step is 1, and the padding is 1; the convolution kernel size of the second convolutional network module is 3, the step is 1, and the padding is 1; and the output image of the second convolutional network module is a heat map image, and the output image feature dimension thereof is 1.

5. The method of processing a lane line according to claim 4, wherein, The key point detection network is used for performing lane line starting point detection processing on the first and second level feature images to obtain a corresponding second image, specifically including: inputting the first and second level feature images into the key point detection network; and using the first convolutional network module of the key point detection network to perform convolution operation on the second level feature image to generate a corresponding first convolutional image; And by the second convolutional network module, the first convolutional image is convoluted to generate a corresponding first heat map; And by the key point identification module, key points are identified from the first heat map to obtain a plurality of first key points; And by the image fusion module, the first heat map is enlarged according to the size of the primary feature image to generate a corresponding second heat map; and on the enlarged second heat map, the pixel points matched with the first key points before enlargement are taken as corresponding second key points; and on the primary feature image, the pixel points corresponding to the second key points are taken as the first starting points, and the pixel values of the first starting points are set as preset starting point pixel values; and the modified primary feature image is taken as the corresponding second image.

6. A system for implementing the processing method of the lane line according to any one of claims 1-5, characterized in that, The system comprises a data receiving module, a feature extraction backbone network processing module, a key point detection network processing module, a lane line identification network processing module, and a lane line marking processing module. The data receiving module is configured to receive a first image. The feature extraction backbone network processing module is configured to perform three-level feature extraction processing on the first image based on a feature extraction backbone network to obtain corresponding primary and secondary feature images. The key point detection network processing module is configured to perform lane line starting point detection processing on the primary and secondary feature images based on a key point detection network to obtain a corresponding second image; the second image comprises a plurality of first starting points. The lane line identification network processing module is configured to identify lane lines starting from the first starting points in the second image based on a lane line identification network to obtain a corresponding first lane line pixel point sequence. The lane line marking processing module is configured to mark lane lines on the second image according to the first lane line pixel point sequence.

7. An electronic device, comprising: It comprises: a memory, a processor, and a transceiver; The processor is configured to couple with the memory, read and execute instructions in the memory to implement the method of any one of claims 1-5; The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transmission and reception.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer executes the method of any one of claims 1-5. The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer executes the method of any one of claims 1-5.

Images