Lane line recognition method, device, computer equipment and storage medium

Abstract

The present application relates to a lane line recognition method, device, computer equipment and storage medium. The method comprises: obtaining a road image to be recognized; performing feature extraction on the road image to be recognized to obtain low-level image features and high-level image features of the road image to be recognized; performing feature enhancement processing on the high-level image features to obtain enhanced high-level image features; fusing the enhanced high-level image features and the low-level image features to obtain fused image features; and obtaining a lane line recognition result according to the fused image features. The method can improve the accuracy of lane line recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a lane line recognition method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology, artificial intelligence has developed rapidly in all walks of life, including machine vision, fingerprint recognition, face recognition, expert systems, intelligent control, robotics, language and image understanding, etc. One of the important specific applications of intelligent control is vehicle intelligent driving. Intelligent driving refers to the technology of machines helping people to drive, and completely replacing human driving in special circumstances. The basis of intelligent driving technology includes identifying lane lines in the road around the vehicle to achieve intelligent control of the vehicle on the road.

[0003] At present, the commonly used lane line recognition is based on visual methods, that is, image recognition processing is performed based on road images to identify lane lines. However, when recognizing lane lines, visual methods are easily affected by multiple factors such as occlusion, blur and weather, resulting in limited accuracy of lane line recognition results. Summary of the invention

[0004] Based on this, it is necessary to provide a lane line recognition method, device, computer equipment and storage medium that can improve the accuracy of lane line recognition in response to the above technical problems.

[0005] A lane line recognition method, the method comprising:

[0006] Acquire a road image to be identified;

[0007] Extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0008] Performing feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0009] Fusion enhances high-level image features and low-level image features to obtain fused image features;

[0010] According to the fused image features, the lane line recognition result is obtained.

[0011] In one embodiment, the high-level image features are enhanced to obtain enhanced high-level image features, including:

[0012] Divide the high-level image features to obtain the sub-feature graph;

[0013] Split the feature extraction factors corresponding to the high-level image features to obtain the sub-slice factors corresponding to the sub-slice feature graph;

[0014] Perform feature processing on the sub-feature map of the slice through the slice factor to obtain the image sub-feature;

[0015] The sub-features of each image are fused to obtain enhanced high-level image features.

[0016] In one embodiment, the sub-features of the image are obtained by performing feature processing on the sub-feature map of the slice through the slice factor, and the sub-features of the image include:

[0017] Obtaining image sub-features corresponding to a previous layer of slice sub-feature map, where the previous layer of slice sub-feature map is a slice sub-feature map that is adjacent to the slice sub-feature map and is one layer above the slice sub-feature map;

[0018] Combine the image sub-features corresponding to the previous layer of the slice sub-feature map with the slice sub-feature map to obtain an updated slice sub-feature map;

[0019] The updated slice sub-feature map is processed by the slice sub-factor to obtain the image sub-feature.

[0020] In one embodiment, after fusing and enhancing the high-level image features and the low-level image features to obtain the fused image features, the method further includes:

[0021] The fused image features are enhanced and the fused image features after the enhanced processing are used as the fused image features.

[0022] In one embodiment, obtaining a lane line recognition result based on the fused image features includes:

[0023] The fused image features are enlarged and restored to obtain a semantic segmentation image. The lane line recognition result includes the semantic segmentation image.

[0024] In one embodiment, the lane line recognition method is implemented based on a pre-trained generator network, and the generator network is obtained by the following processing:

[0025] The real image sample data is processed by the generator to obtain generated image data, and the generated image data is combined with the real image sample data to obtain fake sample data;

[0026] Based on the generative adversarial model, the discriminator is trained with fake sample data and real image sample data, and the generator is updated according to the output of the discriminator, and the real image sample data is processed by the generator to obtain the steps of generating image data;

[0027] When the generator and the discriminator meet the preset equilibrium conditions, the training ends and the generator is used as the generator network.

[0028] In one embodiment, terminating the training when the generator and the discriminator meet a preset equilibrium condition includes:

[0029] According to the training output of the discriminator, the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability is determined;

[0030] Determine the cost function of the generator and the cost function of the discriminator according to the distribution probability distance;

[0031] When the cost function of the generator and the cost function of the discriminator meet the preset equilibrium condition, the training ends.

[0032] A lane line recognition device, the device comprising:

[0033] The module for acquiring the image to be identified is used to acquire the road image to be identified;

[0034] A feature extraction module is used to extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0035] A feature enhancement module is used to perform feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0036] A feature fusion module is used to fuse and enhance high-level image features and low-level image features to obtain fused image features;

[0037] The recognition result acquisition module is used to obtain the lane line recognition result based on the fused image features.

[0038] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Acquire a road image to be identified;

[0040] Extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0041] Performing feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0042] Fusion enhances high-level image features and low-level image features to obtain fused image features;

[0043] According to the fused image features, the lane line recognition result is obtained.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0045] Acquire a road image to be identified;

[0046] Extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0047] Performing feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0048] Fusion enhances high-level image features and low-level image features to obtain fused image features;

[0049] According to the fused image features, the lane line recognition result is obtained.

[0050] The lane line recognition method, device, computer equipment and storage medium perform feature enhancement processing on the high-level image features obtained by feature extraction in the road image to be identified, fuse the enhanced high-level image features obtained after the feature enhancement processing and the low-level image features, and obtain the lane line recognition result based on the fused image features obtained by the fusion. By fusing the enhanced high-level image features obtained after the feature enhancement processing with the low-level image features, the lane line features in the image can be effectively extracted, and the correlation of the image segmentation pixels can be strengthened, so that the image segmentation is more continuous, and the occluded and blurred parts in the image can be effectively predicted, thereby improving the accuracy of lane line recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A diagram showing an application environment of a lane line recognition method in an embodiment;

[0052] Figure 2 A schematic diagram of a process flow of a lane line recognition method in one embodiment;

[0053] Figure 3 It is a schematic diagram of a process of performing feature enhancement processing on high-level image features in one embodiment;

[0054] Figure 4 Schematic diagram of a flow chart of a generator network training process in one embodiment;

[0055] Figure 5 is a schematic diagram of the structure of a discriminator in one embodiment;

[0056] Figure 6 is a structural block diagram of a lane line recognition device in an embodiment;

[0057] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The lane line recognition method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the vehicle is traveling on the road, and the vehicle shoots the road through the vehicle-mounted camera (not shown) to obtain the road image to be identified, and sends the road image to be identified to the server 104 set on the roadside or cloud through the vehicle-mounted terminal (not shown), such as a vehicle-mounted unit, such as a roadside unit. The server 104 performs feature enhancement processing on the high-level image features obtained by feature extraction in the received road image to be identified, and fuses the enhanced high-level image features and low-level image features obtained after the feature enhancement processing, and obtains the lane line recognition result according to the fused image features obtained by fusion. In addition, the lane line recognition process can also be performed separately by the vehicle-mounted terminal. Among them, the vehicle-mounted terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers, vehicle-mounted units and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a lane line recognition method is provided, and the method is applied to Figure 1 The server or vehicle terminal in the example is used to illustrate, including the following steps:

[0061] Step S201: Acquire a road image to be identified.

[0062] The road image to be identified is an image that needs to be detected and identified for lane lines, which can be captured by a vehicle-mounted camera or a camera installed on the roadside.

[0063] Step S203: extracting features of the road image to be identified, and obtaining low-level image features and high-level image features of the road image to be identified.

[0064] After obtaining the road image to be identified, feature extraction is performed on the image to obtain low-level image features and high-level image features of the road image to be identified. In specific implementation, feature extraction can be performed on the road image to be identified through artificial neural networks, such as VGG (Visual Geometry Group Network), ResNet101, Xception, etc., and low-level image features and high-level image features can be extracted from the hidden layer of the network. Generally, high-level image features can effectively reflect the strong semantic information of the image, while low-level image features can effectively reflect the fine-grained features of the image. For example, when extracting features of the road image to be identified through the residual network ResNet50, the low-level fine-grained features output by the conv3 layer can be used as low-level image features, and the high-level semantic features output by the conv5 layer can be used as high-level image features.

[0065] Step S205: performing feature enhancement processing on the high-level image features to obtain enhanced high-level image features.

[0066] After extracting the high-level image features, feature enhancement processing is performed. For example, the high-level image features can be spatially structured through a preset spatial convolution structure to obtain spatial combination information. The spatial convolution structure can enhance the correlation of lane segmentation pixels, making the lane segmentation more continuous, thereby better predicting the lane line blocked by obstacles and blurred parts such as dotted lines. Through feature enhancement processing, such as introducing a spatial convolution structure to build a network, transfer spatial information, and learn spatial relationships, strong prior lane line structure information can be obtained, thereby enhancing the adaptability to lane lines. Specifically, the high-level image features can be transposed and segmented, and then the corresponding segmentation sub-factors can be used to perform feature processing on each segmented sub-feature map to obtain enhanced high-level image features.

[0067] Step S207: Fusing and enhancing high-level image features and low-level image features to obtain fused image features.

[0068] After obtaining the enhanced high-level image features, they are fused with the low-level image features to obtain fused image features. The fused image features reflect the image features of the lane lines in the road image to be identified, and the lane lines can be identified from the road image to be identified by fusing the image features. In the specific implementation, considering that the sizes of the enhanced high-level image features and the low-level image features may be different, the enhanced high-level image features can be processed first, such as using multi-level upsampling and bilinear interpolation to make their size the same as that of the low-level image features, and then fused to obtain the fused image features. By fusing multi-level spatial features, the obtained fused image features combine the strong semantic information of the high-level and the fine-grained features of the low-level, thereby ensuring the accuracy of lane line recognition.

[0069] Step S209: Obtain lane line recognition results based on fused image features.

[0070] After the fused image features are obtained, the lane line recognition result is obtained according to the fused image features. Specifically, the obtained fused image features can be magnified and restored through multiple upsampling layers to obtain the lane line recognition result.

[0071] In the lane line recognition method, the high-level image features obtained by feature extraction in the road image to be recognized are subjected to feature enhancement processing, the enhanced high-level image features obtained after the feature enhancement processing are fused with the low-level image features, and the lane line recognition result is obtained based on the fused image features obtained by fusion. By fusing the enhanced high-level image features obtained after the feature enhancement processing with the low-level image features, the lane line features in the image can be effectively extracted, and at the same time, the correlation of the image segmentation pixels is strengthened, so that the image segmentation is more continuous, and the occluded and blurred parts in the image are effectively predicted, thereby improving the accuracy of lane line recognition.

[0072] In one of the embodiments, feature enhancement processing is performed on high-level image features to obtain enhanced high-level image features, including: segmenting the high-level image features to obtain segmentation sub-feature maps; splitting feature extraction factors corresponding to the high-level image features to obtain segmentation sub-factors corresponding to the segmentation sub-feature maps; performing feature processing on the segmentation sub-feature maps through the segmentation sub-factors to obtain image sub-features; and fusing the image sub-features to obtain enhanced high-level image features.

[0073] In this embodiment, feature enhancement processing is performed on high-level image features through a spatial convolution structure with strong structural information. Specifically, when performing feature enhancement processing, the high-level image features are segmented to obtain segmentation sub-feature maps. In specific implementation, the high-level image features can be feature matrices, then the feature matrix can be transposed first, and then the transposed feature matrix can be segmented to obtain segmentation sub-feature maps, and at this time the segmentation sub-feature maps are feature sub-matrices. Then, the feature extraction factors corresponding to the high-level image features are queried, and the feature extraction factors are split to obtain segmentation sub-factors corresponding to the segmentation sub-feature maps. Among them, the feature extraction factor is the feature extraction factor corresponding to the high-level image features extracted when the feature extraction is performed on the road image to be identified. In a specific implementation, if the feature extraction is performed on the road image to be identified through a residual network, such as ResNet50, the feature extraction factor can be the convolution kernel of the corresponding convolution layer, and the convolution sub-kernel, i.e., the segmentation sub-factor, is obtained by splitting the convolution kernel. The slicing sub-factor corresponds to the slicing sub-feature map, and specifically may correspond to the position of the slicing sub-feature map in the high-level image features, that is, the position of the slicing sub-feature map in the high-level image features is the same as the position of the slicing sub-factor in the feature extraction factor.

[0074] After splitting the high-level image features and feature extraction factors respectively to obtain the sliced ​​sub-feature map and the sliced ​​sub-factor, the sliced ​​sub-feature map is feature processed by the sliced ​​sub-factor to obtain the image sub-feature. Specifically, when the sliced ​​sub-factor is a convolution sub-kernel, the sliced ​​sub-feature map is convolved by the sliced ​​sub-factor to obtain the image sub-feature. After obtaining the image sub-features corresponding to each sliced ​​sub-feature map, each image sub-feature is fused to obtain the enhanced high-level image feature. Specifically, each image sub-feature can be subjected to the inverse processing of sequential segmentation and transposition, that is, each image sub-feature is combined and subjected to the inverse transposition processing, so as to achieve the feature enhancement processing of the high-level image feature and obtain the enhanced high-level image feature.

[0075] In one of the embodiments, feature processing is performed on a slice sub-feature map through a slice sub-factor to obtain image sub-features, including: obtaining image sub-features corresponding to a previous slice sub-feature map, the previous slice sub-feature map being a slice sub-feature map adjacent to and on the previous layer of the slice sub-feature map; combining the image sub-features corresponding to the previous slice sub-feature map and the slice sub-feature map to obtain an updated slice sub-feature map; and feature processing is performed on the updated slice sub-feature map through a slice sub-factor to obtain image sub-features.

[0076] In this embodiment, feature processing is performed on each sliced ​​sub-feature map and the sliced ​​sub-factor of each slice, and after feature processing, the sub-feature map is summed with the next layer of sliced ​​sub-feature maps as the sliced ​​sub-feature map for feature processing of the next layer. Specifically, when feature processing is performed on the sliced ​​sub-feature map by the sliced ​​sub-factor, the image sub-features corresponding to the previous layer of sliced ​​sub-feature map are obtained, and the previous layer of sliced ​​sub-feature map is a sliced ​​sub-feature map adjacent to the sliced ​​sub-feature map and on the previous layer of the sliced ​​sub-feature map. Then, the image sub-features corresponding to the previous layer of sliced ​​sub-feature map and the sliced ​​sub-feature map are combined to obtain an updated sliced ​​sub-feature map. Specifically, the image sub-features corresponding to the previous layer of sliced ​​sub-feature map and the sliced ​​sub-feature map can be summed, and the sum can be used as the updated sliced ​​sub-feature map. The updated sliced ​​sub-feature map is then feature processed by the sliced ​​sub-factor to obtain image sub-features. For example, the updated sliced ​​sub-feature map is convolved by the sliced ​​sub-factor to obtain image sub-features. In specific implementation, for the sliced ​​sub-feature map of the first layer, the image sub-features corresponding to the previous layer of sliced ​​sub-feature map can be set to an empty matrix.

[0077] In one embodiment, Figure 3 As shown in the figure, it is a schematic diagram of feature enhancement processing for high-level image features. Generally, in convolutional neural networks, since lane lines are relatively long, ordinary convolution can only sense local parts of the image. In order to enhance the adaptability of the convolution structure to lane lines, a spatial convolution structure with strong structural information is introduced. Figure 3As shown in the figure, the high-level image features are the hidden layer outputs in the convolutional neural network, that is, the feature matrix. The high-level image features are transposed and segmented, and then the segmented sub-feature maps and segmented sub-factors of each segment are convolved. After the convolution, they are summed with the next layer of segmented sub-feature maps as the next layer of convolutional sub-feature maps. Finally, all the output features of the spatial convolution, that is, the image sub-features, are combined to form an overall enhanced feature matrix to obtain enhanced high-level image features. Among them, the circle plus sign represents convolution addition, C represents the number of channels of the feature, H represents the height of the feature matrix, and W represents the width of the feature matrix.

[0078] In one of the embodiments, after fusing and enhancing the high-level image features and the low-level image features to obtain the fused image features, the method further includes: performing feature enhancement processing on the fused image features, and using the fused image features after the feature enhancement processing as the fused image features.

[0079] In this embodiment, after fusing and enhancing the high-level image features and the low-level image features, feature enhancement processing is further performed on the obtained fused image features to further improve the feature extraction effect. Specifically, after obtaining the fused image features, feature enhancement processing is performed on the fused image features, and the fused image features after feature enhancement processing are used as the fused image features. In specific implementation, feature enhancement processing can also be implemented through a spatial convolution structure.

[0080] In one of the embodiments, obtaining a lane line recognition result based on the fused image features includes: enlarging and restoring the fused image features to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image.

[0081] In this embodiment, considering the size difference between the fused image features and the road image to be identified, the fused image features are enlarged and restored to obtain a lane line recognition result of the same size as the road image to be identified. Specifically, the fused image features are enlarged and restored, and multi-level upsampling can be used to enlarge and restore the features in a bilinear interpolation manner to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image. In addition, the fused image features can also be enlarged and restored by combining the deconvolution layer information to obtain a semantic segmentation image.

[0082] In one of the embodiments, the lane line recognition method is implemented based on a pre-trained generator network. The generator network can be the generator part in a generative adversarial network. Generative adversarial networks, referred to as GAN (Generative Adversarial Networks), are composed of two networks, a generator network and a discriminator network, and these two networks can be neural networks (from convolutional neural networks, recurrent neural networks to autoencoders). The generator network converts a random noise vector from a latent space (not all GAN ​​samples come from a latent space) to a sample of a real data set. Training a GAN is a very intuitive process. By training these two networks at the same time, they will both get better over time.

[0083] In this embodiment, a semantic segmentation image is generated by using a generator network of a generative adversarial network. The semantic segmentation image segments lane lines and non-lane lines. For example, the pixels of lane lines can be set to A, and the pixels of other non-lane lines can be set to B, so as to identify lane lines. During training, the generator and the discriminator are trained using a generative adversarial model, and the real original road image is input into the generator, and the generator outputs a semantic segmentation image. The discriminator is then used to distinguish the authenticity of the classified generated image and the original label. After the training is completed, the road image to be identified is input into the generator network, and the corresponding semantic segmentation image can be output by the generator network. The semantic segmentation image has the same size as the road image to be identified, and the lane line pixels and non-lane line pixels in the road image to be identified are segmented, thereby achieving accurate recognition of the lane lines of the road image to be identified.

[0084] Specifically, Figure 4 As shown in the figure, the generator network training process, the generator network is obtained through the following processing:

[0085] Step S401: The real image sample data is processed by a generator to obtain generated image data, and the generated image data is combined with the real image sample data to obtain fake sample data.

[0086] In this embodiment, the generator and discriminator as components of the generative adversarial network are first constructed. In a specific implementation, the basic network of the generator can be ResNet50, and the discriminator can be constructed through multi-level convolutional layers. Figure 5 As shown in Figure 1, it is a schematic diagram of the structure of the discriminator in a specific application. Figure 5As shown in the figure, the input of the discriminator is processed by 5 convolutional layers, and finally the true or false judgment result is output, and the corresponding cost function Loss is obtained. The discriminator has fewer network layers, and training the discriminator multiple times is conducive to fast network fitting. When training a generative adversarial network, an overlapping training method is adopted, that is, the generator is trained once and the discriminator is trained multiple times. Specifically, the parameters of the discriminator are fixed first, the generator is trained, and the gradient is updated. Then the generator parameters are fixed, the discriminator is trained, and the parameters of the discriminator are updated. The discriminator is trained n times and the generator is trained once. The discriminator model is small, which is conducive to the rapid convergence of the overall network.

[0087] Specifically, the real image sample data is processed by the generator to obtain the generated image data, and the generated image data is combined with the real image sample data to obtain the fake sample data. The generated image data is a semantic segmentation image. By combining the semantic segmentation image output by the generator with the real image sample data, the fake sample data as the discriminator network training set can be obtained, and the discriminator network training set also includes the real image sample data.

[0088] Step S403: Based on the generative adversarial model, the discriminator is trained by using fake sample data and real image sample data, and the generator is updated according to the output of the discriminator, and the process returns to the step of processing the real image sample data through the generator to obtain generated image data.

[0089] When training the generator and discriminator, based on the generative adversarial model, the discriminator is trained with fake sample data and real image sample data, and the generator is updated according to the output of the discriminator, and the real image sample data is processed by the generator to obtain the steps of generating image data. In this way, overlapping training of the generator and the discriminator is achieved until the training end condition is met, that is, the training ends when both the generator and the discriminator are good enough.

[0090] In specific implementation, when training the discriminator, the input label of the fake sample data is 0, that is, fake, and the input label of the real image sample data is 1, that is, true. The sum of the cost functions corresponding to the output of the fake sample data and the real image sample data is used as the cost function of the discriminator. The sum of the cost function corresponding to the discriminator when the fake sample data is input and the output is shown as true, and the sum of the cost function corresponding to the output of the real image sample data are used as the cost function of the generator.

[0091] Step S405: When the generator and the discriminator meet the preset equilibrium condition, the training is terminated and the generator is used as the generator network.

[0092] When determining whether the training is terminated, it is determined whether the generator and the discriminator meet the preset equilibrium condition, that is, when the output result of the discriminator is close to 0.5, the training is terminated and the generator is used as the generator network.

[0093] In a specific application, the generator is improved based on ResNet50. Specifically, the last few layers of ResNet50 use spatial convolution, and restore to the original image size through multiple upsampling layers, and then output. Specifically, the features of conv3 and conv5 in ResNet50 are fused, and more fine-grained features can be obtained through multi-level spatial feature fusion.

[0094] In one of the embodiments, terminating training when the generator and the discriminator satisfy a preset equilibrium condition includes: determining the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability based on the training output of the discriminator; determining the cost function of the generator and the cost function of the discriminator based on the distribution probability distance; terminating the training when the cost function of the generator and the cost function of the discriminator satisfy the preset equilibrium condition.

[0095] In this embodiment, the generator cost function and the discriminator cost function are used to determine whether the preset equilibrium condition is met, that is, whether the training end condition is met. Specifically, according to the training output of the discriminator, the distribution probability distance between the real data distribution probability input by the discriminator and the generated data distribution probability is determined, and the distribution probability distance can be the Wasssertein distance. It can be understood that if the real data distribution probability input by the discriminator and the marginal distribution of the generated data distribution probability do not overlap, that is, the KL divergence or JS divergence is large, the original generative adversarial network will cause the generator cost function to face the gradient vanishing problem under the (approximate) optimal discriminator, or cause optimization confusion, gradient instability, and imbalance in the punishment of diversity and accuracy, resulting in model collapse. In order to generate stable training of adversarial networks, Wasssertein distance can be applied, and Wasssertein distance can reflect the distance or overlap ratio of two distributions. If the distribution does not overlap, the distribution of the two data is brought closer by Wasssertein. By introducing the Wasssertein distance, the original data information is overlapped in space, and the difficulty of generating adversarial networks to converge can be solved at the same time. The Wasssertein distance can be obtained by the following formula (1):

[0096]

[0097] Among them, inf means taking the lower bound of the expectation; P r ,P g Respectively represent the real data distribution probability and generated data distribution probability of the discriminator input; x~∏(P r ,P g ) means x belongs to P r ,P gJoint probability distribution; ||xy|| represents the distance between the real sample and the generated sample.

[0098] After obtaining the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability, the cost function of the generator and the cost function of the discriminator are determined according to the distribution probability distance. Generally, when the model satisfies the Laplace distribution || f w (x1)]-f w (x2)||≤C(w)g||x1-x2||, the model is insensitive to input perturbations. Cannot be solved directly, definition: Where sup means taking an upper bound, |f|L means satisfying a specific Laplace constraint, and N means a constant. Right now satisfies Lipschitz continuity. Therefore, definition Among them, N is a constant, so L can be used as the cost function of the discriminator. For the generator, The formula has nothing to do with the generator, so the generation cost function can be defined as When designing the cost function of the discriminator, the pixel blocks of the local receptive field are used to determine the authenticity of real images and images generated by the generative adversarial network, which can ensure high segmentation accuracy.

[0099] In a specific application, the cost function G_loss of the generator is as follows (2):

[0100]

[0101] Among them, f w (x) represents the output of the generator; x represents the real RGB image as the network input; Indicates the expected value of the input x belonging to the generated probability distribution; It represents the expected value of the probability that the input x belongs to the real data distribution; min{} means minimizing the distribution.

[0102] The cost function D_loss of the discriminator can be taken as formula (3),

[0103]

[0104] Among them, f w (x) represents the network output of the discriminator, and x represents the RGB image combined with the output of the generator and the RGB image combined with the label as the network input.

[0105] It should be understood that although Figure 2 , 4The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0106] In one embodiment, Figure 6 As shown, a lane line recognition device is provided, including: an image acquisition module 601 to be recognized, a feature extraction module 603, a feature enhancement module 605, a feature fusion module 607 and a recognition result acquisition module 609, wherein:

[0107] The to-be-recognized image acquisition module 601 is used to acquire the to-be-recognized road image;

[0108] A feature extraction module 603 is used to extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0109] A feature enhancement module 605 is used to perform feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0110] A feature fusion module 607 is used to fuse and enhance high-level image features and low-level image features to obtain fused image features;

[0111] The recognition result acquisition module 609 is used to obtain the lane line recognition result according to the fused image features.

[0112] In one embodiment, the feature enhancement module 605 includes a feature segmentation unit, a factor splitting unit, a sub-feature acquisition unit and a sub-feature fusion unit; wherein: the feature segmentation unit is used to segment the high-level image features to obtain a segmentation sub-feature map; the factor splitting unit is used to split the feature extraction factors corresponding to the high-level image features to obtain segmentation sub-factors corresponding to the segmentation sub-feature map; the sub-feature acquisition unit is used to perform feature processing on the segmentation sub-feature map through the segmentation sub-factors to obtain image sub-features; the sub-feature fusion unit is used to fuse each image sub-feature to obtain an enhanced high-level image feature.

[0113] In one embodiment, the sub-feature acquisition unit includes an upper-layer sub-feature unit, a sub-feature updating unit and a sub-feature processing unit; wherein: the upper-layer sub-feature unit is used to obtain the image sub-features corresponding to the upper-layer slice sub-feature map, and the upper-layer slice sub-feature map is a slice sub-feature map adjacent to the slice sub-feature map and one layer above the slice sub-feature map; the sub-feature updating unit is used to combine the image sub-features corresponding to the upper-layer slice sub-feature map and the slice sub-feature map to obtain an updated slice sub-feature map; the sub-feature processing unit is used to perform feature processing on the updated slice sub-feature map through the slice sub-factor to obtain the image sub-features.

[0114] In one of the embodiments, a fusion enhancement module is further included, which is used to perform feature enhancement processing on the fused image features, and use the fused image features after the feature enhancement processing as the fused image features.

[0115] In one embodiment, the recognition result acquisition module 609 includes an enlargement and restoration module for enlarging and restoring the fused image features to obtain a semantic segmentation image, and the lane line recognition result includes a semantic segmentation image.

[0116] In one of the embodiments, it also includes a fake sample module, an adversarial training module and a generator determination module; wherein: the fake sample module is used to process real image sample data through a generator to obtain generated image data, and combine the generated image data with real image sample data to obtain fake sample data; the adversarial training module is used to train the discriminator through fake sample data and real image sample data based on a generative adversarial model, and update the generator according to the output of the discriminator, and return to the step of processing the real image sample data through the generator to obtain generated image data; the generator determination module is used to end the training when the generator and the discriminator meet a preset equilibrium condition, and use the generator as a generator network.

[0117] In one embodiment, the generator determination module includes a probability distance determination unit, a cost function determination unit and an equilibrium condition unit; wherein: the probability distance determination unit is used to determine the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability according to the training output of the discriminator; the cost function determination unit is used to determine the cost function of the generator and the cost function of the discriminator according to the distribution probability distance; the equilibrium condition unit is used to end the training when the cost function of the generator and the cost function of the discriminator meet the preset equilibrium condition.

[0118] For the specific definition of the lane line recognition device, please refer to the definition of the lane line recognition method above, which will not be repeated here. Each module in the above lane line recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0119] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a lane line recognition method is implemented.

[0120] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0122] Acquire a road image to be identified;

[0123] Extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0124] Performing feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0125] Fusion enhances high-level image features and low-level image features to obtain fused image features;

[0126] According to the fused image features, the lane line recognition result is obtained.

[0127] In one embodiment, when the processor executes the computer program, the following steps are also implemented: segmenting high-level image features to obtain segmentation sub-feature maps; splitting feature extraction factors corresponding to the high-level image features to obtain segmentation sub-factors corresponding to the segmentation sub-feature maps; performing feature processing on the segmentation sub-feature maps through the segmentation sub-factors to obtain image sub-features; and fusing the image sub-features to obtain enhanced high-level image features.

[0128] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining image sub-features corresponding to the previous layer of slice sub-feature map, the previous layer of slice sub-feature map is a slice sub-feature map adjacent to the slice sub-feature map and on the previous layer of the slice sub-feature map; combining the image sub-features corresponding to the previous layer of slice sub-feature map and the slice sub-feature map to obtain an updated slice sub-feature map; performing feature processing on the updated slice sub-feature map through the slice sub-factor to obtain image sub-features.

[0129] In one embodiment, when the processor executes the computer program, the following steps are further implemented: performing feature enhancement processing on the fused image features, and using the fused image features after the feature enhancement processing as the fused image features.

[0130] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the fused image features are enlarged and restored to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image.

[0131] In one embodiment, the lane line recognition method is implemented based on a pre-trained generator network, and the processor also implements the following steps when executing the computer program: processing real image sample data through the generator to obtain generated image data, and combining the generated image data with the real image sample data to obtain false sample data; based on the generative adversarial model, training the discriminator through false sample data and real image sample data, and updating the generator according to the output of the discriminator, returning to the step of processing the real image sample data through the generator to obtain generated image data; ending the training when the generator and the discriminator meet the preset equilibrium condition, and using the generator as the generator network.

[0132] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the training output of the discriminator, the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability is determined; based on the distribution probability distance, the cost function of the generator and the cost function of the discriminator are determined; when the cost function of the generator and the cost function of the discriminator meet the preset equilibrium condition, the training is terminated.

[0133] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0134] Acquire a road image to be identified;

[0135] Extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified;

[0136] Performing feature enhancement processing on high-level image features to obtain enhanced high-level image features;

[0137] Fusion enhances high-level image features and low-level image features to obtain fused image features;

[0138] According to the fused image features, the lane line recognition result is obtained.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: segmenting high-level image features to obtain segmentation sub-feature maps; splitting feature extraction factors corresponding to the high-level image features to obtain segmentation sub-factors corresponding to the segmentation sub-feature maps; performing feature processing on the segmentation sub-feature maps through the segmentation sub-factors to obtain image sub-features; and fusing the image sub-features to obtain enhanced high-level image features.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining image sub-features corresponding to the previous layer of slice sub-feature map, the previous layer of slice sub-feature map is a slice sub-feature map adjacent to the slice sub-feature map and on the previous layer of the slice sub-feature map; combining the image sub-features corresponding to the previous layer of slice sub-feature map and the slice sub-feature map to obtain an updated slice sub-feature map; performing feature processing on the updated slice sub-feature map through the slice sub-factor to obtain image sub-features.

[0141] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: performing feature enhancement processing on the fused image features, and using the fused image features after the feature enhancement processing as the fused image features.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the fused image features are enlarged and restored to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image.

[0143] In one embodiment, the lane line recognition method is implemented based on a pre-trained generator network, and the computer program also implements the following steps when executed by the processor: processing real image sample data through the generator to obtain generated image data, and combining the generated image data with the real image sample data to obtain false sample data; based on the generative adversarial model, training the discriminator through false sample data and real image sample data, and updating the generator according to the output of the discriminator, returning to the step of processing the real image sample data through the generator to obtain generated image data; ending the training when the generator and the discriminator meet the preset equilibrium condition, and using the generator as the generator network.

[0144] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the training output of the discriminator, the distribution probability distance between the real data distribution probability of the discriminator input and the generated data distribution probability is determined; based on the distribution probability distance, the cost function of the generator and the cost function of the discriminator are determined; when the cost function of the generator and the cost function of the discriminator meet the preset equilibrium condition, the training is terminated.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A lane line recognition method, the method comprising: Acquire a road image to be identified; Extracting features of the road image to be identified to obtain low-level image features and high-level image features of the road image to be identified; Segmenting the high-level image features to obtain segmented sub-feature graphs; Splitting the feature extraction factors corresponding to the high-level image features to obtain the slice sub-factors corresponding to the slice sub-feature graph; Obtaining image sub-features corresponding to a previous layer of slice sub-feature map, wherein the previous layer of slice sub-feature map is a slice sub-feature map adjacent to the slice sub-feature map and a layer above the slice sub-feature map; Combining the image sub-features corresponding to the previous-layer slice sub-feature map with the slice sub-feature map to obtain an updated slice sub-feature map; Performing feature processing on the updated slice sub-feature map by using the slice sub-factor to obtain image sub-features; Fusing the image sub-features to obtain enhanced high-level image features; Fusing the enhanced high-level image features and the low-level image features to obtain fused image features; A lane line recognition result is obtained according to the fused image features.

2. The method according to claim 1, characterized in that After fusing the enhanced high-level image features and the low-level image features to obtain fused image features, the method further includes: The fused image features are subjected to feature enhancement processing, and the fused image features after the feature enhancement processing are used as the fused image features.

3. The method according to claim 1, characterized in that The obtaining of a lane line recognition result according to the fused image features includes: The fused image features are enlarged and restored to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image.

4. The method according to any one of claims 1 to 3, characterized in that: The method is implemented based on a pre-trained generator network, which is obtained by the following processing: Processing the real image sample data by a generator to obtain generated image data, and combining the generated image data with the real image sample data to obtain fake sample data; Based on the generative adversarial model, a discriminator is trained by the fake sample data and the real image sample data, and the generator is updated according to the output of the discriminator, and the real image sample data is processed by the generator to obtain the step of generating image data; The training is terminated when the generator and the discriminator meet a preset equilibrium condition, and the generator is used as the generator network.

5. The method according to claim 4, characterized in that The step of ending the training when the generator and the discriminator meet a preset equilibrium condition includes: Determining, based on the training output of the discriminator, a distribution probability distance between a distribution probability of real data input to the discriminator and a distribution probability of generated data; Determine a cost function of the generator and a cost function of the discriminator according to the distribution probability distance; When the cost function of the generator and the cost function of the discriminator meet a preset equilibrium condition, the training is terminated.

6. A lane line recognition device, characterized in that: The device comprises: The module for acquiring the image to be identified is used to acquire the road image to be identified; A feature extraction module, used to extract features of the road image to be identified, and obtain low-level image features and high-level image features of the road image to be identified; The feature space structuring module is used to segment the high-level image features to obtain segmentation sub-feature maps; split the feature extraction factors corresponding to the high-level image features to obtain segmentation sub-factors corresponding to the segmentation sub-feature maps; obtain image sub-features corresponding to the previous-level segmentation sub-feature maps, wherein the previous-level segmentation sub-feature maps are segmentation sub-feature maps adjacent to the segmentation sub-feature maps and one layer above the segmentation sub-feature maps; combine the image sub-features corresponding to the previous-level segmentation sub-feature maps with the segmentation sub-feature maps to obtain an updated segmentation sub-feature map; perform feature processing on the updated segmentation sub-feature map through the segmentation sub-factors to obtain image sub-features; and fuse the image sub-features to obtain enhanced high-level image features; A feature fusion module, used for fusing the enhanced high-level image features and the low-level image features to obtain fused image features; The recognition result acquisition module is used to obtain the lane line recognition result according to the fused image features.

7. The device according to claim 6, characterized in that The device also includes: The fusion enhancement module is used to perform feature enhancement processing on the fused image features, and use the fused image features after the feature enhancement processing as the fused image features.

8. The device according to claim 6, characterized in that The recognition result acquisition module is also used to enlarge and restore the fused image features to obtain a semantic segmentation image, and the lane line recognition result includes the semantic segmentation image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images