Lane line semantic segmentation method, device and model based on OCRNet and storage medium

Abstract

The invention relates to an OCRNet-based lane line semantic segmentation method, device and model and a storage medium. The method comprises the following steps: acquiring different features output by different hierarchical networks of an OCRNet-based lane line semantic segmentation model; according to different features, obtaining a lane line first semantic segmentation result output by a lane line semantic segmentation model; according to the first semantic segmentation result of the lane line and different features, respectively calculating a loss function value corresponding to each hierarchical network of different hierarchical networks; calculating a loss function value of a lane line semantic segmentation model according to the loss function value corresponding to each hierarchical network; and if the loss function value of the lane line semantic segmentation model is smaller than a set loss function threshold value, according to the lane line first semantic segmentation result, obtaining a lane line semantic segmentation result which is output by the lane line semantic segmentation model and contains the lane line image. According to the scheme provided by the invention, image features containing more detail information can be obtained, and an accurate lane line semantic segmentation result is obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lane line semantic segmentation method, device, model and storage medium based on OCRNet. Background Art

[0002] When an autonomous vehicle is driving or making a high-precision map, it needs to know the exact location of the lane lines in order to determine the safe driving area or make some decisions based on the lane lines.

[0003] In the related technology, lane line semantic segmentation based on deep neural network can be used to extract image features using the powerful feature extraction capability of deep neural network, and then the lane line and background are distinguished through pixel-level classification to obtain the lane line semantic segmentation result, and the lane line is identified based on the lane line semantic segmentation result. The related technology is based on the lane line semantic segmentation of deep neural network. The deep neural network has poor training effect and too large receptive field, which causes the loss of detailed information of image features and leads to inaccurate lane line semantic segmentation results. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related technology, the present application provides a lane line semantic segmentation method, device, model and storage medium based on OCRNet, which can obtain image features containing more detailed information and obtain accurate lane line semantic segmentation results.

[0005] In a first aspect, the present application provides a lane line semantic segmentation method based on OCRNet, the method comprising: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet to obtain different features of the lane line semantic segmentation model output by different layers of the network; Obtaining, according to the different features, a first semantic segmentation result of the lane line including the lane line image output by the lane line semantic segmentation model; According to the first semantic segmentation result of the lane line and the different features, respectively calculate the loss function value corresponding to each layer network of the different layers of the lane line semantic segmentation model; Calculating the loss function value of the lane semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks; If the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, the lane line semantic segmentation result of the lane line image output by the lane line semantic segmentation model is obtained according to the first semantic segmentation result of the lane line including the lane line image.

[0006] Preferably, the method further comprises: If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, adjust the network parameters corresponding to each level network of different levels of the lane line semantic segmentation model.

[0007] Preferably, the step of inputting the lane line image into the lane line semantic segmentation model based on OCRNet and obtaining different features output by networks at different levels of the lane line semantic segmentation model includes: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet, and obtaining pixel semantic information and pixel features of the lane line image output by the lane line semantic segmentation model; Obtaining the category region features output by the first-level network of the lane line semantic segmentation model according to the pixel semantic information and pixel features of the lane line image; Obtaining the similarity between pixels and regions output by the second-level network of the lane semantic segmentation model according to the category region features; Obtaining context representation features output by a third-level network of the lane semantic segmentation model according to the similarity between the pixel and the region; Obtain enhanced pixel features of the lane line image output by a fourth-level network of the lane line semantic segmentation model based on the context representation features.

[0008] Preferably, the step of inputting the image containing lane lines into a lane line semantic segmentation model based on OCRNet to obtain pixel features of the image containing lane lines output by the lane line semantic segmentation model comprises: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet, preprocessing the lane line image to obtain a feature map of a set dimension; Extracting the first layer of the feature of the first-level network of the lane semantic segmentation model, and obtaining a 4-fold downsampled first pixel feature map and an 8-fold downsampled first pixel feature map according to the set dimension feature map; Through the feature extraction second layer of the second layer network of the lane semantic segmentation model, according to the 4-fold downsampled first pixel feature map and the 8-fold downsampled first pixel feature map, a 4-fold downsampled second pixel feature map, an 8-fold downsampled second pixel feature map, and a 16-fold downsampled second pixel feature map are obtained respectively; Through the feature extraction third layer of the third-level network of the lane semantic segmentation model, according to the 4-fold downsampled second pixel feature map, the 8-fold downsampled second pixel feature map, and the 16-fold downsampled second pixel feature map, a 4-fold downsampled third pixel feature map, an 8-fold downsampled third pixel feature map, a 16-fold downsampled third pixel feature map, and a 32-fold downsampled third pixel feature map are obtained respectively; Through the fourth layer of feature extraction of the fourth-level network of the lane line semantic segmentation model, according to the 4-fold down-sampled third pixel feature map, 8-fold down-sampled third pixel feature map, 16-fold down-sampled third pixel feature map, and 32-fold down-sampled third pixel feature map, a 4-fold down-sampled pixel feature map, an 8-fold down-sampled pixel feature map, a 16-fold down-sampled pixel feature map, and a 32-fold down-sampled pixel feature map of the lane line image are obtained respectively; The lane line semantic segmentation model is used to fuse the 4-fold downsampled pixel feature map, the 8-fold downsampled pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled pixel feature map to obtain the pixel features of the lane line image output by the lane line semantic segmentation model.

[0009] Preferably, the step of calculating the loss function value corresponding to each level network of different levels of the lane line semantic segmentation model according to the first lane line semantic segmentation result and the different features includes: Calculating a first loss function value of the first hierarchical network using a cross entropy loss function according to the pixel semantic information and pixel features of the lane line image and the category region features; Calculating a second loss function value of the second-level network using the cross entropy loss function according to the category region feature and the similarity between the pixel and the region; Calculating a third loss function value of the third-level network using the cross entropy loss function according to the similarity between the pixel and the region and the context representation feature; According to the context representation features and the enhanced pixel features of the lane line image, the cross entropy loss function is used to calculate the fourth loss function value of the fourth-level network.

[0010] Preferably, the loss function value of the lane semantic segmentation model is calculated according to the loss function value corresponding to each level network of the different levels of networks, including: According to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, a weighted sum algorithm is used to calculate the loss function value of the lane line semantic segmentation model.

[0011] Preferably, if the loss function value of the lane line semantic segmentation model is greater than or equal to a set loss function threshold, adjusting the network parameters corresponding to each level network of different levels of the lane line semantic segmentation model includes: If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the first level network and the first layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the second layer network and the second layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the third level network and the third layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the fourth-level network and the fourth-layer feature extraction of the lane line semantic segmentation model.

[0012] A second aspect of the present application provides a lane semantic segmentation device based on OCRNet, the device comprising: A first acquisition module is used to input the lane line image into the lane line semantic segmentation model based on OCRNet, and obtain different features of the lane line semantic segmentation model output by different layers of the network; A second acquisition module, configured to obtain a first lane line semantic segmentation result of the lane line image output by the lane line semantic segmentation model according to different features acquired by the first acquisition module; A first calculation module, used to calculate the loss function value corresponding to each layer network of the lane line semantic segmentation model according to the first semantic segmentation result of the lane line obtained by the second acquisition module and the different features obtained by the first acquisition module; A second calculation module, used to calculate the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level network of different levels of networks calculated by the first calculation module; A segmentation result acquisition module is used to obtain the lane line semantic segmentation result including the lane line image output by the lane line semantic segmentation model based on the first semantic segmentation result of the lane line including the lane line image obtained by the second acquisition module if the loss function value of the lane line semantic segmentation model calculated by the second calculation module is less than a set loss function threshold.

[0013] The third aspect of the present application provides a lane line semantic segmentation model based on OCRNet, including: Processor; and The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of a lane semantic segmentation model based on OCRNet, the processor executes the method described above.

[0015] The technical solution provided by this application may have the following beneficial effects: The technical solution of the present application inputs an image containing lane lines into a lane line semantic segmentation model based on OCRNet, and obtains different features output by different levels of networks of the lane line semantic segmentation model; obtains the first lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the different features; calculates the loss function value corresponding to each level of the network of different levels of the lane line semantic segmentation model according to the first lane line semantic segmentation result and different features; calculates the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level of the network of different levels; if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, obtains the lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the first lane line semantic segmentation result of the lane line containing the lane line image. Results: Different features of the lane line image are extracted through different levels of networks based on OCRNet, and the loss function value corresponding to each level of the lane line semantic segmentation model is calculated according to the input and output of the different levels of networks. The loss function value of the lane line semantic segmentation model is calculated according to the loss function value corresponding to each level of the different levels of networks. Each level of the lane line semantic segmentation model is supervised to improve the accuracy of the output of each level of the lane line semantic segmentation model, so that each level of the lane line semantic segmentation model can output more accurate features, obtain features containing more detailed information, and enable the lane line semantic segmentation model to output more accurate lane line semantic segmentation results, and obtain accurate lane line semantic segmentation results.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0018] Figure 1 It is a flowchart of a lane semantic segmentation method based on OCRNet shown in an embodiment of the present application; Figure 2 is another flowchart of the lane semantic segmentation method based on OCRNet shown in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of different feature extraction layers corresponding to different levels of networks of the lane semantic segmentation model based on OCRNet shown in an embodiment of the present application; Figure 4 is a structural schematic diagram of a lane semantic segmentation device based on OCRNet shown in an embodiment of the present application; Figure 5 is another structural schematic diagram of a lane semantic segmentation device based on OCRNet shown in an embodiment of the present application; Figure 6 Schematic diagram of the structure of the lane semantic segmentation model based on OCRNet shown in the embodiment of the present application; Figure 7 It is a schematic diagram comparing the IOU of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application and the IOU of the lane line semantic segmentation model of the related technology. DETAILED DESCRIPTION

[0019] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0020] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0022] When an autonomous vehicle is driving or making a high-precision map, it needs to know the exact location of the lane lines in order to determine the safe driving area or make some decisions based on the lane lines.

[0023] In the related art, the lane line semantic segmentation based on the deep neural network can be performed by using the powerful feature extraction ability of the deep neural network to extract image features, and then the lane line and the background are distinguished through pixel-level classification to obtain the lane line semantic segmentation result, and the lane line is identified according to the lane line semantic segmentation result. In the related art, the lane line semantic segmentation based on the deep neural network (for example, OCRNet) is performed. Although the image features from low resolution to high resolution are obtained through multiple parallel branches, the receptive field of the multiple parallel branches that obtain the image features from low resolution to high resolution gradually increases, and the deep neural network is trained by calculating a loss function value through the final output of the deep neural network. The deep neural network may still have poor training effect; the poor training effect of the deep neural network and the excessive receptive field of multiple parallel branches cause the loss of detailed information of the image features, resulting in inaccurate lane line semantic segmentation results.

[0024] In response to the above problems, an embodiment of the present application provides a lane line semantic segmentation method based on OCRNet, which can obtain image features containing more detailed information and obtain accurate lane line semantic segmentation results.

[0025] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0026] Figure 1 It is a flowchart of the lane line semantic segmentation method based on OCRNet shown in an embodiment of the present application.

[0027] See also Figure 1 , a lane line semantic segmentation method based on OCRNet, including: Step 101, inputting the lane line image into the lane line semantic segmentation model based on OCRNet to obtain different features of the lane line semantic segmentation model output by different layers of the network.

[0028] In one embodiment, an image containing lane lines is input into a lane line semantic segmentation model based on OCRNet, and different levels of networks of the lane line semantic segmentation model based on OCRNet sequentially output pixel semantic information (Soft Object Regions) and pixel features (Pixel Representations) of the lane line image according to the lane line image, output category region features (Object Region Representations) according to the pixel semantic information and pixel features of the lane line image, output similarity between pixels and regions (Pixel-Region Relation) according to the category region features, output contextual representation features (Object-Contextual Representations) according to the similarity between pixels and regions, and output enhanced pixel features (Augmented Representations) of the lane line image according to the contextual representation features.

[0029] In one embodiment, pixel features of different resolutions of a lane line image can be obtained by different feature extraction layers corresponding to different levels of networks of a lane line semantic segmentation model based on OCRNet, and pixel features of different resolutions can be fused to obtain pixel features of a lane line image.

[0030] Step 102: Obtain a first lane line semantic segmentation result including a lane line image output by a lane line semantic segmentation model according to different features.

[0031] In one embodiment, the lane line semantic segmentation model based on OCRNet can output a first lane line semantic segmentation result of the lane line image containing the lane line image according to the pixel semantic information and pixel features, category region features, similarity between pixels and regions, context representation features, and enhanced pixel features of the lane line image; and obtain the first lane line semantic segmentation result of the lane line image output by the lane line semantic segmentation model based on OCRNet.

[0032] Step 103, according to the first semantic segmentation result of the lane line and different features, respectively calculate the loss function value corresponding to each level network of the different levels of the lane line semantic segmentation model based on OCRNet.

[0033] In one embodiment, the loss function value corresponding to each level network of the lane line semantic segmentation model based on OCRNet can be calculated according to the annotated image (Ground Truth) containing the lane line image, the pixel semantic information and pixel features of the lane line image output containing the lane line image, the category region features, the similarity between pixels and regions, the context representation features, the enhanced pixel features, the first semantic segmentation result of the lane line, and the loss function of the different levels network of the lane line semantic segmentation model based on OCRNet.

[0034] Step 104, calculating the loss function value of the lane semantic segmentation model according to the loss function value corresponding to each level network of different levels of networks.

[0035] In one embodiment, the loss function value of the lane line semantic segmentation model based on OCRNet can be calculated according to the loss function value corresponding to each level network of different levels of the lane line semantic segmentation model based on OCRNet.

[0036] Step 105: If the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, a lane line semantic segmentation result including a lane line image output by the lane line semantic segmentation model is obtained based on the first lane line semantic segmentation result including the lane line image.

[0037] In one embodiment, if the loss function value of the lane line semantic segmentation model based on OCRNet is less than the set loss function threshold, the first semantic segmentation result of the lane line outputted by the lane line semantic segmentation model based on OCRNet can be obtained as the lane line semantic segmentation result including the lane line image.

[0038] The lane line semantic segmentation method based on OCRNet in the embodiment of the present application inputs the lane line image into the lane line semantic segmentation model based on OCRNet, and obtains different features output by different levels of networks of the lane line semantic segmentation model; obtains the first lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the different features; calculates the loss function value corresponding to each level network of the different levels of networks of the lane line semantic segmentation model according to the first lane line semantic segmentation result and different features; calculates the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks; if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, obtains the lane line image output by the lane line semantic segmentation model according to the first lane line semantic segmentation result containing the lane line image. Lane line semantic segmentation results; extract different features of the lane line image through different levels of networks of the lane line semantic segmentation model based on OCRNet, calculate the loss function value corresponding to each level of the lane line semantic segmentation model according to the input and output of the different levels of networks, calculate the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level of the different levels of networks, supervise each level of the lane line semantic segmentation model, improve the accuracy of the output of each level of the lane line semantic segmentation model, so that each level of the lane line semantic segmentation model can output more accurate features, obtain features containing more detailed information, so that the lane line semantic segmentation model outputs more accurate lane line semantic segmentation results, and obtains accurate lane line semantic segmentation results.

[0039] Figure 2 is another flowchart of the lane semantic segmentation method based on OCRNet shown in an embodiment of the present application; Figure 3 It is a structural schematic diagram of different feature extraction layers corresponding to different levels of networks of the lane line semantic segmentation model based on OCRNet shown in an embodiment of the present application.

[0040] See also Figure 2 , a lane line semantic segmentation method based on OCRNet, including: Step 201: Input an image containing lane lines into a lane line semantic segmentation model based on OCRNet to obtain pixel features of the image containing lane lines output by the lane line semantic segmentation model.

[0041] In one embodiment, a lane line semantic segmentation model based on OCRNet including a preprocessing layer and networks of different levels may be constructed based on OCRNet.

[0042] In one embodiment, the image containing lane lines can be preprocessed by the preprocessing layer, the image containing lane lines can be upsampled and / or downsampled, and the resolution, size, and dimension of the image containing lane lines can be adjusted to obtain a feature map of a set dimension. The preprocessing layer can include two convolution blocks, and the convolution block includes a convolution layer, a batch normalization layer, and a convolution layer. Normalization , BN) module, Relu function, the convolution layer can be a convolution layer with kernel (kernel) of 3, stride (step length) of 2, and padding (padding) of 1.

[0043] In one embodiment, the different-level networks of the lane line semantic segmentation model based on OCRNet include a first-level network of the lane line semantic segmentation model corresponding to the first-level network of OCRNet, a second-level network of the lane line semantic segmentation model corresponding to the second-level network of OCRNet, a third-level network of the lane line semantic segmentation model corresponding to the third-level network of OCRNet, and a fourth-level network of the lane line semantic segmentation model corresponding to the fourth-level network of OCRNet.

[0044] See also Figure 3 The different-level networks of the lane line semantic segmentation model include different feature extraction layers corresponding to the different-level networks, and the different feature extraction layers include the first feature extraction layer 301 of the first-level network of the lane line semantic segmentation model, the second feature extraction layer 302 of the second-level network, the third feature extraction layer 303 of the third-level network, and the fourth feature extraction layer 304 of the fourth-level network; the first feature extraction layer 301 includes the 11th module, the 11th branch, and the 12th branch; the second feature extraction layer 302 includes the 21st module, the 22nd module, the 23rd module, the 21st branch, the 22nd branch, and the 23rd branch; the third feature extraction layer 303 includes the 31st module, the 32nd module, the 33rd module, the 34th module, the 31st branch, the 32nd branch, the 33rd branch, and the 34th branch; the fourth feature extraction layer 304 includes the 41st module, the 42nd module, the 43rd module, the 44th module, and the 45th module.

[0045] In one embodiment, the first feature extraction layer 301, the second feature extraction layer 302, the third feature extraction layer 303 and the fourth feature extraction layer 304 may be neural networks having functions of upsampling, downsampling, convolution, normalization, fusion and the like of images.

[0046] In one embodiment, the image containing the lane lines can be preprocessed through a preprocessing layer to obtain a feature map of a set dimension; through the first layer of feature extraction of the first-level network of the lane line semantic segmentation model, a 4-fold downsampled first pixel feature map and an 8-fold downsampled first pixel feature map are obtained according to the feature map of the set dimension; through the second layer of feature extraction of the second-level network of the lane line semantic segmentation model, a 4-fold downsampled second pixel feature map, an 8-fold downsampled second pixel feature map, and a 16-fold downsampled second pixel feature map are obtained according to the 4-fold downsampled first pixel feature map and the 8-fold downsampled first pixel feature map; through the third layer of feature extraction of the third-level network of the lane line semantic segmentation model, a 4-fold downsampled second pixel feature map, an 8-fold downsampled second pixel feature map, and a 16-fold downsampled second pixel feature map are obtained according to the 4-fold downsampled second pixel feature map, the 8-fold downsampled second pixel feature map, and the 16-fold downsampled second pixel feature map. The fourth layer of feature extraction of the fourth-level network of the lane line semantic segmentation model is used to obtain a 4-fold downsampled pixel feature map, an 8-fold downsampled pixel feature map, an 16-fold downsampled pixel feature map, and a 32-fold downsampled pixel feature map containing the lane line image according to the 4-fold downsampled third pixel feature map, the 8-fold downsampled third pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled third pixel feature map; the 4-fold downsampled pixel feature map, the 8-fold downsampled pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled pixel feature map are fused through the lane line semantic segmentation model to obtain the pixel features of the lane line image output by the lane line semantic segmentation model.

[0047] In one embodiment, the 11th module of the first layer of feature extraction can be used to perform multi-scale group convolution on the set dimension feature map to obtain a 4-fold downsampled 11th intermediate pixel feature map with 4N channels, where N is the number of channels of the set dimension feature map; the 11th branch of the first layer of feature extraction can be used to obtain a 4-fold downsampled first pixel feature map based on the 4-fold downsampled 11th intermediate pixel feature map; the 12th branch of the first layer of feature extraction can be used to downsample the 4-fold downsampled 11th intermediate pixel feature map by 2 times to obtain an 8-fold downsampled first pixel feature map.

[0048] In one embodiment, a 21st module of the second feature extraction layer may be used to perform multi-scale group convolution on the 4-fold downsampled first pixel feature map to obtain a 21st intermediate pixel feature map of the 4-fold downsampled image; a 22nd module of the second feature extraction layer may be used to perform multi-scale group convolution on the 8-fold downsampled first pixel feature map to obtain a 21st intermediate pixel feature map of the 8-fold downsampled image; a 23rd module of the second feature extraction layer may be used to perform multi-scale convolution on the 21st intermediate pixel feature map of the 4-fold downsampled image and the 21st intermediate pixel feature map of the 8-fold downsampled image, respectively, to obtain a 4 The 22nd intermediate pixel feature map is downsampled by times and the 22nd intermediate pixel feature map is downsampled by 8 times; through the 21st branch of the second layer of feature extraction, a 4-fold downsampled second pixel feature map is obtained according to the 4-fold downsampled 22nd intermediate pixel feature map; through the 22nd branch of the second layer of feature extraction, an 8-fold downsampled second pixel feature map is obtained according to the 8-fold downsampled 22nd intermediate pixel feature map; through the 23rd branch of the second layer of feature extraction, the 8-fold downsampled 22nd intermediate pixel feature map is downsampled to obtain a 16-fold downsampled second pixel feature map.

[0049] In one embodiment, the 23rd module of the second layer of feature extraction can perform 2 times upsampling on the 21st intermediate pixel feature map of 8 times downsampling to obtain a 4 times intermediate pixel feature map with the same scale as the 21st intermediate pixel feature map of 4 times downsampling; the 4 times intermediate pixel feature map is fused with the 21st intermediate pixel feature map of 4 times downsampling by addition, and the 22nd intermediate pixel feature map of 4 times downsampling is obtained through the Relu function. The 23rd module of the second layer of feature extraction can perform 2 times downsampling on the 21st intermediate pixel feature map of 4 times downsampling to obtain an 8 times intermediate pixel feature map with the same scale as the 21st intermediate pixel feature map of 8 times downsampling; the 8 times intermediate pixel feature map is fused with the 21st intermediate pixel feature map of 8 times downsampling by addition, and the 22nd intermediate pixel feature map of 8 times downsampling is obtained through the Relu function.

[0050] In one embodiment, a 4-fold downsampled second pixel feature map may be subjected to a multi-scale group convolution through a 31st module of the third layer of feature extraction to obtain a 4-fold downsampled 31st intermediate pixel feature map; a 8-fold downsampled second pixel feature map may be subjected to a multi-scale group convolution through a 32nd module of the third layer of feature extraction to obtain an 8-fold downsampled 31st intermediate pixel feature map; a 16-fold downsampled second pixel feature map may be subjected to a multi-scale group convolution through a 33rd module of the third layer of feature extraction to obtain a 16-fold downsampled 31st intermediate pixel feature map; a 4-fold downsampled 31st intermediate pixel feature map, an 8-fold downsampled 31st intermediate pixel feature map, and a 16-fold downsampled 31st intermediate pixel feature map may be subjected to a multi-scale convolution through a 34th module of the third layer of feature extraction, respectively. A 4-fold downsampled 32nd intermediate pixel feature map, an 8-fold downsampled 32nd intermediate pixel feature map and a 16-fold downsampled 32nd intermediate pixel feature map are obtained; through the 31st branch of the third layer of feature extraction, a 4-fold downsampled third pixel feature map is obtained according to the 4-fold downsampled 32nd intermediate pixel feature map; through the 32nd branch of the third layer of feature extraction, an 8-fold downsampled third pixel feature map is obtained according to the 8-fold downsampled 32nd intermediate pixel feature map; through the 33rd branch of the third layer of feature extraction, a 16-fold downsampled third pixel feature map is obtained according to the 16-fold downsampled 32nd intermediate pixel feature map; through the 34th branch of the third layer of feature extraction, the 16-fold downsampled 32nd intermediate pixel feature map is downsampled to obtain a 32-fold downsampled third pixel feature map.

[0051] In one embodiment, the 34th module of the third feature extraction layer can be used to upsample the 8-fold downsampled 31st intermediate pixel feature map by 2 times to obtain a 4-fold first intermediate pixel feature map with the same scale as the 4-fold downsampled 31st intermediate pixel feature map; the 16-fold downsampled 31st intermediate pixel feature map is upsampled by 4 times to obtain a 4-fold second intermediate pixel feature map with the same scale as the 4-fold downsampled 31st intermediate pixel feature map; the 4-fold first intermediate pixel feature map, the 4-fold second intermediate pixel feature map and the 4-fold downsampled 31st intermediate pixel feature map are fused by addition, and the 4-fold downsampled 32nd intermediate pixel feature map is obtained through the Relu function. The 34th module of the third layer of feature extraction can be used to downsample the 4-fold downsampled 31st intermediate pixel feature map by 2 times to obtain an 8-fold first intermediate pixel feature map with the same scale as the 8-fold downsampled 31st intermediate pixel feature map; the 16-fold downsampled 31st intermediate pixel feature map is upsampled by 2 times to obtain an 8-fold second intermediate pixel feature map with the same scale as the 8-fold downsampled 31st intermediate pixel feature map; the 8-fold first intermediate pixel feature map, the 8-fold second intermediate pixel feature map and the 8-fold downsampled 31st intermediate pixel feature map are fused in an additive manner, and the 8-fold downsampled 32nd intermediate pixel feature map is obtained through a Relu function. The 34th module of the third layer of feature extraction can be used to downsample the 4-fold downsampled 31st intermediate pixel feature map by 4 times to obtain a 16-fold first intermediate pixel feature map with the same scale as the 16-fold downsampled 31st intermediate pixel feature map; the 8-fold downsampled 31st intermediate pixel feature map is downsampled by 2 times to obtain a 16-fold second intermediate pixel feature map with the same scale as the 16-fold downsampled 31st intermediate pixel feature map; the 16-fold first intermediate pixel feature map, the 16-fold second intermediate pixel feature map and the 16-fold downsampled 31st intermediate pixel feature map are fused in an additive manner, and the 16-fold downsampled 32nd intermediate pixel feature map is obtained through a Relu function.

[0052] In one embodiment, a 41st module of the fourth layer of feature extraction may be used to perform multi-scale group convolution on the 4-fold downsampled third pixel feature map to obtain a 41st intermediate pixel feature map of the 4-fold downsampled; a 42nd module of the fourth layer of feature extraction may be used to perform multi-scale group convolution on the 8-fold downsampled third pixel feature map to obtain a 41st intermediate pixel feature map of the 8-fold downsampled; a 43rd module of the fourth layer of feature extraction may be used to perform multi-scale group convolution on the 16-fold downsampled third pixel feature map to obtain a 16-fold downsampled 41st intermediate pixel feature map; a 44th module of the fourth layer of feature extraction may be used to perform multi-scale group convolution on the 16-fold downsampled third pixel feature map to obtain a 44th intermediate pixel feature map of the 16-fold downsampled; Block, multi-scale group convolution is performed on the 32-fold downsampled third pixel feature map to obtain a 32-fold downsampled 41st intermediate pixel feature map; through the 45th module of the fourth layer of feature extraction, multi-scale convolution is performed on the 41st intermediate pixel feature map of the 4-fold downsampled, the 41st intermediate pixel feature map of the 8-fold downsampled, the 41st intermediate pixel feature map of the 16-fold downsampled, and the 41st intermediate pixel feature map of the 32-fold downsampled, respectively, to obtain a 4-fold downsampled pixel feature map, an 8-fold downsampled pixel feature map, a 16-fold downsampled pixel feature map, and a 32-fold downsampled pixel feature map containing the lane line image, respectively.

[0053] In one embodiment, the 45th module of the fourth layer of feature extraction can be used to upsample the 41st intermediate pixel feature map of the 8-fold downsampled image by 2 times to obtain a 4-fold downsampled first intermediate pixel feature map of the same scale as the 41st intermediate pixel feature map of the 4-fold downsampled image; the 41st intermediate pixel feature map of the 16-fold downsampled image is upsampled by 4 times to obtain a 4-fold downsampled second intermediate pixel feature map of the same scale as the 41st intermediate pixel feature map of the 4-fold downsampled image; the 41st intermediate pixel feature map of the 32-fold downsampled image is upsampled by 8 times to obtain a 4-fold downsampled third intermediate pixel feature map of the same scale as the 41st intermediate pixel feature map of the 4-fold downsampled image; the 4-fold downsampled first intermediate pixel feature map, the 4-fold downsampled second intermediate pixel feature map, the 4-fold downsampled third intermediate pixel feature map and the 4-fold downsampled 41st intermediate pixel feature map are fused by addition, and a 4-fold downsampled pixel feature map containing a lane line image is obtained through a Relu function.

[0054] In one embodiment, the 45th module of the fourth layer of feature extraction is used to downsample the 41st intermediate pixel feature map of the 4th downsampled layer by 2 times to obtain an 8x downsampled first intermediate pixel feature map with the same scale as the 8x downsampled 41st intermediate pixel feature map; the 41st intermediate pixel feature map of the 16th downsampled layer is upsampled by 2 times to obtain an 8x downsampled second intermediate pixel feature map with the same scale as the 8x downsampled 41st intermediate pixel feature map; the 41st intermediate pixel feature map of the 32th downsampled layer is upsampled by 4 times to obtain an 8x downsampled third intermediate pixel feature map with the same scale as the 8x downsampled 41st intermediate pixel feature map; the 8x downsampled first intermediate pixel feature map, the 8x downsampled second intermediate pixel feature map, the 8x downsampled third intermediate pixel feature map and the 8x downsampled 41st intermediate pixel feature map are fused by addition, and the 8x downsampled pixel feature map containing the lane line image is obtained through a Relu function.

[0055] In one embodiment, the 45th module of the fourth layer of feature extraction is used to downsample the 41st intermediate pixel feature map of the 4th downsampled layer by 4 times to obtain a 16-fold downsampled first intermediate pixel feature map with the same scale as the 16-fold downsampled 41st intermediate pixel feature map; the 41st intermediate pixel feature map of the 8th downsampled layer is downsampled by 2 times to obtain a 16-fold downsampled second intermediate pixel feature map with the same scale as the 16-fold downsampled 41st intermediate pixel feature map; the 41st intermediate pixel feature map of the 32th downsampled layer is upsampled by 2 times to obtain a 16-fold downsampled third intermediate pixel feature map with the same scale as the 16-fold downsampled 41st intermediate pixel feature map; the 16-fold downsampled first intermediate pixel feature map, the 16-fold downsampled second intermediate pixel feature map, the 16-fold downsampled third intermediate pixel feature map and the 16-fold downsampled 41st intermediate pixel feature map are fused in an additive manner, and a 16-fold downsampled pixel feature map containing a lane line image is obtained through a Relu function.

[0056] In one embodiment, the 45th module of the fourth layer of feature extraction is used to downsample the 41st intermediate pixel feature map of the 4th downsampled pixel by 8 times to obtain a 32-fold downsampled first intermediate pixel feature map with the same scale as the 41st intermediate pixel feature map of the 32-fold downsampled pixel; the 41st intermediate pixel feature map of the 8th downsampled pixel is downsampled by 4 times to obtain a 32-fold downsampled second intermediate pixel feature map with the same scale as the 41st intermediate pixel feature map of the 32-fold downsampled pixel; the 41st intermediate pixel feature map of the 16th downsampled pixel is downsampled by 2 times to obtain a 32-fold downsampled third intermediate pixel feature map with the same scale as the 41st intermediate pixel feature map of the 32-fold downsampled pixel; the 32-fold downsampled first intermediate pixel feature map, the 32-fold downsampled second intermediate pixel feature map, the 32-fold downsampled third intermediate pixel feature map and the 32-fold downsampled 41st intermediate pixel feature map are fused by addition, and a 32-fold downsampled pixel feature map containing a lane line image is obtained through a Relu function.

[0057] In one embodiment, the lane line semantic segmentation model based on OCRNet can be used to fuse a 4x downsampled pixel feature map, an 8x downsampled pixel feature map, a 16x downsampled pixel feature map, and a 32x downsampled pixel feature map to obtain pixel features of a lane line image output by the lane line semantic segmentation model.

[0058] Step 202, obtaining different features of the network outputs at different levels of the lane semantic segmentation model.

[0059] In one embodiment, an image containing lane lines can be input into a lane line semantic segmentation model based on OCRNet to obtain pixel semantic information of the image containing lane lines output by the lane line semantic segmentation model; based on the pixel semantic information and pixel features of the image containing lane lines, the category region features output by the first-level network of the lane line semantic segmentation model based on the pixel semantic information and pixel features of the image containing lane lines are obtained; the similarity between pixels and regions output by the second-level network of the lane line semantic segmentation model based on the category region features is obtained; the context representation features output by the third-level network of the lane line semantic segmentation model based on the similarity between pixels and regions are obtained; and the enhanced pixel features of the image containing lane lines output by the fourth-level network of the lane line semantic segmentation model based on the context representation features are obtained.

[0060] In one embodiment, an image containing lane lines can be input into a lane line semantic segmentation model based on OCRNet to obtain pixel semantic information of the lane line image output by the lane line semantic segmentation model; the first-level network of the lane line semantic segmentation model takes pixel semantic information and pixel features of the lane line image as input, and outputs category region features of the lane line image; the second-level network of the lane line semantic segmentation model takes category region features of the lane line image as input, and outputs the similarity between pixels and regions of the lane line image; the third-level network of the lane line semantic segmentation model takes the similarity between pixels and regions of the lane line image as input, and outputs context representation features of the lane line image; the fourth-level network of the lane line semantic segmentation model takes context representation features of the lane line image as input, and outputs enhanced pixel features of the lane line image.

[0061] Step 203: Obtain a first lane line semantic segmentation result including a lane line image output by the lane line semantic segmentation model according to different features.

[0062] In one embodiment, the lane line semantic segmentation model based on OCRNet can output a first semantic segmentation result of the lane line containing the lane line image based on the pixel semantic information and pixel features of the lane line image, category region features, similarity between pixels and regions, context representation features, and enhanced pixel features.

[0063] Step 204, based on the first semantic segmentation result of the lane line and different features, respectively calculate the loss function value corresponding to each level network of the different levels of the lane line semantic segmentation model.

[0064] In one embodiment, the loss function value corresponding to each level network of the lane line semantic segmentation model can be calculated based on the annotated image (Ground Truth) containing the lane line image, pixel semantic information and pixel features containing the lane line image, category region features, similarity between pixels and regions, context representation features, enhanced pixel features, the first semantic segmentation result of the lane line, and the loss function corresponding to each level network of the different level networks.

[0065] In one embodiment, a first loss function value of a first-level network can be calculated using a cross-entropy loss function based on pixel semantic information and pixel features of a lane line image and category region features; a second loss function value of a second-level network can be calculated using a cross-entropy loss function based on category region features and the similarity between pixels and regions; a third loss function value of a third-level network can be calculated using a cross-entropy loss function based on the similarity between pixels and regions and context representation features; and a fourth loss function value of a fourth-level network can be calculated using a cross-entropy loss function based on context representation features and enhanced pixel features of a lane line image.

[0066] In one embodiment, the image containing the lane lines can be One-Hot encoded (one-bit effective encoding) to obtain a labeled image (Ground Truth) containing the lane line image; based on the labeled image (Ground Truth) containing the lane line image, the input and output of the first-layer network, the second-layer network, the third-layer network and the fourth-layer network, the cross entropy loss function is used to calculate the loss function values ​​of the first-layer network, the second-layer network, the third-layer network and the fourth-layer network, respectively.

[0067] In one embodiment, a first loss function value of a first-level network can be calculated using a cross-entropy loss function based on pixel semantic information and pixel features of an image containing lane lines, as well as category region features; a second loss function value of a second-level network can be calculated using a cross-entropy loss function based on category region features and the similarity between pixels and regions; a third loss function value of a third-level network can be calculated using a cross-entropy loss function based on the similarity between pixels and regions and context representation features; and a fourth loss function value of a fourth-level network can be calculated using a cross-entropy loss function based on context representation features, enhanced pixel features of an image containing lane lines, and a first semantic segmentation result of lane lines.

[0068] Step 205 , calculate the loss function value of the lane semantic segmentation model using a weighted sum algorithm based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0069] In one embodiment, a weighted summation algorithm may be used to perform weighted summation on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value to obtain the loss function value of the lane line semantic segmentation model.

[0070] Step 206, determine whether the loss function value of the lane line semantic segmentation model is less than the set loss function threshold; if yes, execute step 207; if not, execute step 208.

[0071] In one embodiment, it can be determined whether the loss function value of the lane line semantic segmentation model is less than the set loss function threshold; if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, execute step 207; if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, execute step 208.

[0072] Step 207, obtaining a lane line semantic segmentation result including a lane line image output by a lane line semantic segmentation model according to the first lane line semantic segmentation result including the lane line image.

[0073] In one embodiment, if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, the first lane line semantic segmentation result including the lane line image can be used as the lane line semantic segmentation result including the lane line image output by the lane line semantic segmentation model.

[0074] Step 208 , adjusting the network parameters corresponding to each level network of different levels of the lane line semantic segmentation model; executing step 201 .

[0075] In one embodiment, if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model can be back-propagated to adjust the network parameters corresponding to each level network of the lane line semantic segmentation model; execute step 201 until the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, and obtain the lane line semantic segmentation result containing the lane line image output by the lane line semantic segmentation model based on OCRNet.

[0076] In one embodiment, if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the first-level network and the first-level feature extraction of the lane line semantic segmentation model; and / or, if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the second-level network and the second-level feature extraction of the lane line semantic segmentation model; and / or, if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the third-level network and the third-level feature extraction of the lane line semantic segmentation model; and / or, if the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the fourth-level network and the fourth-level feature extraction of the lane line semantic segmentation model.

[0077] The lane line semantic segmentation method based on OCRNet in the embodiment of the present application inputs the lane line image into the lane line semantic segmentation model based on OCRNet, and obtains different features output by different levels of networks of the lane line semantic segmentation model; obtains the first lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the different features; calculates the loss function value corresponding to each level network of the different levels of networks of the lane line semantic segmentation model according to the first lane line semantic segmentation result and different features; calculates the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks; if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, obtains the lane line image output by the lane line semantic segmentation model according to the first lane line semantic segmentation result containing the lane line image. Lane line semantic segmentation results; extract different features of the lane line image through different levels of networks of the lane line semantic segmentation model based on OCRNet, calculate the loss function value corresponding to each level of the lane line semantic segmentation model according to the input and output of the different levels of networks, calculate the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level of the different levels of networks, supervise each level of the lane line semantic segmentation model, improve the accuracy of the output of each level of the lane line semantic segmentation model, so that each level of the lane line semantic segmentation model can output more accurate features, obtain features containing more detailed information, so that the lane line semantic segmentation model outputs more accurate lane line semantic segmentation results, and obtains accurate lane line semantic segmentation results.

[0078] Furthermore, the lane line semantic segmentation method based on OCRNet in the embodiment of the present application inputs the image containing the lane line into the lane line semantic segmentation model based on OCRNet, pre-processes the image containing the lane line to obtain a feature map of a set dimension; extracts the first layer of the feature of the first-level network of the lane line semantic segmentation model, and obtains a 4-fold downsampled first pixel feature map and an 8-fold downsampled first pixel feature map according to the feature map of the set dimension; extracts the second layer of the feature of the second-level network of the lane line semantic segmentation model, and obtains the 4-fold downsampled first pixel feature map and the 8-fold downsampled first pixel feature map according to the feature map of the set dimension. The 4-fold downsampled second pixel feature map, 8-fold downsampled second pixel feature map, and 16-fold downsampled second pixel feature map are obtained respectively; through the third layer of feature extraction of the third layer network of the lane line semantic segmentation model, according to the 4-fold downsampled second pixel feature map, 8-fold downsampled second pixel feature map, and 16-fold downsampled second pixel feature map, the 4-fold downsampled third pixel feature map, 8-fold downsampled third pixel feature map, 16-fold downsampled third pixel feature map, and 32-fold downsampled third pixel feature map are obtained respectively; through the fourth layer of feature extraction of the fourth layer network of the lane line semantic segmentation model, according to the 4-fold downsampled second pixel feature map, The three-pixel feature map, the 8-fold downsampled third pixel feature map, the 16-fold downsampled third pixel feature map, and the 32-fold downsampled third pixel feature map are used to obtain the 4-fold downsampled pixel feature map, 8-fold downsampled pixel feature map, 16-fold downsampled pixel feature map, and 32-fold downsampled pixel feature map containing the lane line image respectively; the lane line semantic segmentation model is used to fuse the 4-fold downsampled pixel feature map, the 8-fold downsampled pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled pixel feature map to obtain the pixel features of the lane line image output by the lane line semantic segmentation model; the first-level network based on OCRNet The first layer network, the second layer network, the third layer network and the fourth layer network of OCRNet are used to add the corresponding first feature extraction layer to the first layer network of OCRNet, the corresponding second feature extraction layer to the second layer network, the corresponding third feature extraction layer to the third layer network and the corresponding fourth feature extraction layer to the fourth layer network. When performing lane line semantic segmentation based on the lane line semantic segmentation model of OCRNet, multi-scale group convolution and / or multi-scale convolution are performed on the pixel feature map containing the lane line image through the newly added different auxiliary feature extraction layers corresponding to the different layers of the network, which can reduce the loss of scale features;The pixel feature maps are fused through different feature extraction layers corresponding to different layers of the network. When fusing pixel feature maps of multiple scales, the feature information of the original scale is maintained, and the feature information of different scales is fused, thereby enhancing the feature representation capability at that scale, so that the fused image feature map has a stronger feature representation, and the pixel features containing the lane line image contain as much information as possible required to obtain the lane line semantic segmentation result, so that image features containing more detailed information can be obtained, and accurate lane line semantic segmentation results can be obtained. ;

[0079] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a lane line semantic segmentation device, model and corresponding embodiments based on OCRNet.

[0080] Figure 4 It is a structural diagram of a lane semantic segmentation device based on OCRNet shown in an embodiment of the present application.

[0081] See also Figure 4 A lane semantic segmentation device based on OCRNet includes a first acquisition module 401, a second acquisition module 402, a first calculation module 403, a second calculation module 404, and a segmentation result acquisition module 405.

[0082] The first acquisition module 401 is used to input the lane line image into the lane line semantic segmentation model based on OCRNet, and obtain different features of the lane line semantic segmentation model output by different levels of the network.

[0083] In one embodiment, the first acquisition module 401 inputs the lane line image into the lane line semantic segmentation model based on OCRNet; the different levels of the lane line semantic segmentation model based on the lane line image sequentially output pixel semantic information (Soft Object Regions) and pixel features (PixelRepresentations) of the lane line image according to the lane line image, output category region features (Object Region Representations) according to the pixel semantic information and pixel features of the lane line image, output the similarity between pixels and regions (Pixel-Region Relation) according to the category region features, output contextual representation features (Object-ContextualRepresentations) according to the similarity between pixels and regions, and output enhanced pixel features (Augmented Representations) of the lane line image according to the contextual representation features.

[0084] In one embodiment, the first acquisition module 401 can obtain pixel features of different resolutions of the lane line image through different feature extraction layers corresponding to different levels of networks of the lane line semantic segmentation model based on OCRNet, and fuse the pixel features of different resolutions to obtain pixel features of the lane line image.

[0085] The second acquisition module 402 is used to obtain a first lane line semantic segmentation result including a lane line image output by the lane line semantic segmentation model according to the different features acquired by the first acquisition module 401.

[0086] In one embodiment, the lane line semantic segmentation model based on OCRNet can output a first lane line semantic segmentation result containing a lane line image based on pixel semantic information and pixel features, category region features, similarity between pixels and regions, context representation features, and enhanced pixel features. The second acquisition module 402 can obtain the first lane line semantic segmentation result containing a lane line image output by the lane line semantic segmentation model based on OCRNet.

[0087] The first calculation module 403 is used to calculate the loss function value corresponding to each level network of the lane line semantic segmentation model according to the first semantic segmentation result of the lane line obtained by the second acquisition module 402 and the different features obtained by the first acquisition module 401.

[0088] In one embodiment, the first calculation module 403 can calculate the loss function value corresponding to each level network of the lane line semantic segmentation model based on OCRNet according to the annotated image (GroundTruth) containing the lane line image, the pixel semantic information and pixel features of the lane line image output containing the lane line image, the category region features, the similarity between pixels and regions, the context representation features, the enhanced pixel features, the first semantic segmentation result of the lane line, and the loss function of the different levels of the lane line semantic segmentation model based on OCRNet.

[0089] The second calculation module 404 is used to calculate the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks calculated by the first calculation module 403.

[0090] In one embodiment, the second calculation module 404 may calculate the loss function value of the lane line semantic segmentation model based on OCRNet according to the loss function value corresponding to each level network of different levels of the lane line semantic segmentation model based on OCRNet.

[0091] The segmentation result acquisition module 405 is used to obtain the lane line semantic segmentation result including the lane line image output by the lane line semantic segmentation model according to the first lane line semantic segmentation result including the lane line image obtained by the second acquisition module 402 if the loss function value of the lane line semantic segmentation model calculated by the second calculation module 404 is less than the set loss function threshold.

[0092] In one embodiment, if the loss function value of the lane line semantic segmentation model based on OCRNet is less than the set loss function threshold, the segmentation result acquisition module 405 can use the lane line semantic segmentation model based on OCRNet to output the first lane line semantic segmentation result containing the lane line image as the lane line semantic segmentation result containing the lane line image.

[0093] The technical solution of the embodiment of the present application inputs the lane line image into the lane line semantic segmentation model based on OCRNet, and obtains different features output by different levels of networks of the lane line semantic segmentation model; obtains the first lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the different features; calculates the loss function value corresponding to each level of the network of different levels of the lane line semantic segmentation model according to the first lane line semantic segmentation result and different features; calculates the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level of the network of different levels; if the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, obtains the lane line semantic segmentation result output by the lane line semantic segmentation model containing the lane line image according to the first lane line semantic segmentation result. Results: Different features of the lane line image are extracted through different levels of networks based on OCRNet. According to the input and output of different levels of networks, the loss function value corresponding to each level of the lane line semantic segmentation model is calculated. According to the loss function value corresponding to each level of the different levels of networks, the loss function value of the lane line semantic segmentation model is calculated, and each level of the lane line semantic segmentation model is supervised to improve the accuracy of the output of each level of the lane line semantic segmentation model, so that each level of the lane line semantic segmentation model can output more accurate features, obtain features containing more detailed information, so that the lane line semantic segmentation model outputs more accurate lane line semantic segmentation results, and obtains accurate lane line semantic segmentation results.

[0094] Figure 5 It is another structural schematic diagram of the lane semantic segmentation device based on OCRNet shown in an embodiment of the present application.

[0095] See also Figure 5A lane line semantic segmentation device based on OCRNet includes a first acquisition module 401, a second acquisition module 402, a first calculation module 403, a second calculation module 404, a segmentation result acquisition module 405, and an adjustment module 501.

[0096] The first acquisition module 401 is used to input the lane line image into the lane line semantic segmentation model based on OCRNet, and obtain the pixel semantic information and pixel features of the lane line image output by the lane line semantic segmentation model; obtain the category region features output by the first-level network of the lane line semantic segmentation model based on the pixel semantic information and pixel features of the lane line image; obtain the similarity between pixels and regions output by the second-level network of the lane line semantic segmentation model based on the category region features; obtain the context representation features output by the third-level network of the lane line semantic segmentation model based on the similarity between pixels and regions; obtain the enhanced pixel features of the lane line image output by the fourth-level network of the lane line semantic segmentation model based on the context representation features.

[0097] In one embodiment, the first acquisition module 401 is also used to input the image containing lane lines into the lane line semantic segmentation model based on OCRNet, pre-process the image containing lane lines through the lane line semantic segmentation model, and obtain a feature map of a set dimension; through the first layer of feature extraction of the first layer network of the lane line semantic segmentation model, according to the feature map of the set dimension, obtain a 4-fold down-sampled first pixel feature map and an 8-fold down-sampled first pixel feature map respectively; through the second layer of feature extraction of the second layer network of the lane line semantic segmentation model, according to the 4-fold down-sampled first pixel feature map and the 8-fold down-sampled first pixel feature map, obtain a 4-fold down-sampled second pixel feature map, an 8-fold down-sampled second pixel feature map, and a 16-fold down-sampled second pixel feature map respectively; through the third layer of feature extraction of the third layer network of the lane line semantic segmentation model, according to the 4-fold down-sampled second pixel feature map and the 8-fold down-sampled second pixel feature map The lane line semantic segmentation model is used to extract the lane line image by the fourth layer of the fourth-level network, and the 4-fold downsampled third pixel feature map, the 8-fold downsampled third pixel feature map, the 16-fold downsampled third pixel feature map, and the 32-fold downsampled third pixel feature map are obtained respectively; the lane line semantic segmentation model is used to extract the fourth layer of the fourth-level network, and the 4-fold downsampled third pixel feature map, the 8-fold downsampled third pixel feature map, the 16-fold downsampled third pixel feature map, and the 32-fold downsampled third pixel feature map are obtained respectively; the lane line semantic segmentation model is used to fuse the 4-fold downsampled pixel feature map, the 8-fold downsampled pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled pixel feature map to obtain the pixel features of the lane line image output by the lane line semantic segmentation model.

[0098] The second acquisition module 402 is used to obtain a first semantic segmentation result of the lane line containing the lane line image output by the lane line semantic segmentation model based on OCRNet according to the pixel semantic information and pixel features of the lane line image, category region features, similarity between pixels and regions, context representation features, and enhanced pixel features.

[0099] The first calculation module 403 is used to calculate the first loss function value of the first layer network by using the cross entropy loss function according to the pixel semantic information and pixel features of the lane line image and the category region features; calculate the second loss function value of the second layer network by using the cross entropy loss function according to the category region features and the similarity between the pixels and the regions; calculate the third loss function value of the third layer network by using the cross entropy loss function according to the similarity between the pixels and the regions and the context representation features; calculate the fourth loss function value of the fourth layer network by using the cross entropy loss function according to the context representation features and the enhanced pixel features of the lane line image.

[0100] The second calculation module 404 is used to calculate the loss function value of the lane line semantic segmentation model using a weighted sum algorithm based on the first loss function value, the second loss function value, the third loss function value and the fourth loss function value.

[0101] The segmentation result acquisition module 405 is used to obtain the lane line semantic segmentation result including the lane line image output by the lane line semantic segmentation model according to the first lane line semantic segmentation result including the lane line image obtained by the second acquisition module 402 if the loss function value of the lane line semantic segmentation model calculated by the second calculation module 404 is less than the set loss function threshold.

[0102] The adjustment module 406 is used to adjust the network parameters corresponding to each level network of the lane line semantic segmentation model if the loss function value of the lane line semantic segmentation model calculated by the second calculation module 404 is greater than or equal to the set loss function threshold.

[0103] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0104] Figure 6 It is a structural diagram of the lane line semantic segmentation model based on OCRNet shown in an embodiment of the present application. Figure 7 It is a schematic diagram comparing the IOU of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application and the IOU of the lane line semantic segmentation model of the related technology.

[0105] See also Figure 7 According to the IOU (Intersection over Union) curve 701 of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application and the IOU curve 702 of the lane line semantic segmentation model of the related technology, the IOU of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application is larger than the IOU of the lane line semantic segmentation model of the related technology, the accuracy of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application is better than the accuracy of the lane line semantic segmentation model of the related technology, and the lane line semantic segmentation effect of the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application is better than the lane line semantic segmentation effect of the lane line semantic segmentation model of the related technology. Therefore, the lane line semantic segmentation model based on OCRNet shown in the embodiment of the present application extracts different features of the lane line image through different levels of networks based on OCRNet, and calculates the loss function value corresponding to each level of the lane line semantic segmentation model according to the input and output of the different levels of networks. According to the loss function value corresponding to each level of the different levels of networks, the loss function value of the lane line semantic segmentation model is calculated, and each level of the lane line semantic segmentation model is supervised, thereby improving the accuracy of the output of each level of the lane line semantic segmentation model, so that each level of the lane line semantic segmentation model can output more accurate features, and obtain features containing more detailed information, so that the lane line semantic segmentation model can output more accurate lane line semantic segmentation results, and obtain accurate lane line semantic segmentation results.

[0106] See also Figure 6 , the lane line semantic segmentation model 1000 based on OCRNet includes a memory 1010 and a processor 1020.

[0107] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0108] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0109] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0110] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0111] Alternatively, the present application can also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) on which an executable code (or a computer program or a computer instruction code) is stored. When the executable code (or a computer program or a computer instruction code) is executed by a processor of a lane line semantic segmentation model based on OCRNet (or a server, or an electronic device, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0112] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A lane semantic segmentation method based on OCRNet, characterized in that: include: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet to obtain different features of the lane line semantic segmentation model output by different layers of the network; Obtaining, according to the different features, a first semantic segmentation result of the lane line including the lane line image output by the lane line semantic segmentation model; According to the first semantic segmentation result of the lane line and the different features, respectively calculate the loss function value corresponding to each layer network of the different layers of the lane line semantic segmentation model; Calculating the loss function value of the lane semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks; If the loss function value of the lane line semantic segmentation model is less than the set loss function threshold, the lane line semantic segmentation result of the lane line image output by the lane line semantic segmentation model is obtained according to the first semantic segmentation result of the lane line including the lane line image.

2. The method according to claim 1, characterized in that The method further comprises: If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, adjust the network parameters corresponding to each level network of different levels of the lane line semantic segmentation model.

3. The method according to claim 2, characterized in that The lane line image is input into the lane line semantic segmentation model based on OCRNet, and different features of the lane line semantic segmentation model output by different layers of the network are obtained, including: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet, and obtaining pixel semantic information and pixel features of the lane line image output by the lane line semantic segmentation model; Obtaining the category region features output by the first-level network of the lane line semantic segmentation model according to the pixel semantic information and pixel features of the lane line image; Obtaining the similarity between pixels and regions output by the second-level network of the lane semantic segmentation model according to the category region features; Obtaining context representation features output by a third-level network of the lane semantic segmentation model according to the similarity between the pixel and the region; Obtain enhanced pixel features of the image containing lane lines output by a fourth-level network of the lane line semantic segmentation model based on the context representation features.

4. The method according to claim 3, characterized in that The step of inputting the lane line image into the lane line semantic segmentation model based on OCRNet, and obtaining the pixel features of the lane line image output by the lane line semantic segmentation model, comprises: Inputting the lane line image into the lane line semantic segmentation model based on OCRNet, preprocessing the lane line image to obtain a feature map of a set dimension; Extracting the first layer of the feature of the first-level network of the lane semantic segmentation model, and obtaining a 4-fold downsampled first pixel feature map and an 8-fold downsampled first pixel feature map according to the set dimension feature map; Through the feature extraction second layer of the second layer network of the lane semantic segmentation model, according to the 4-fold downsampled first pixel feature map and the 8-fold downsampled first pixel feature map, a 4-fold downsampled second pixel feature map, an 8-fold downsampled second pixel feature map, and a 16-fold downsampled second pixel feature map are obtained respectively; Through the feature extraction third layer of the third-level network of the lane semantic segmentation model, according to the 4-fold downsampled second pixel feature map, the 8-fold downsampled second pixel feature map, and the 16-fold downsampled second pixel feature map, a 4-fold downsampled third pixel feature map, an 8-fold downsampled third pixel feature map, a 16-fold downsampled third pixel feature map, and a 32-fold downsampled third pixel feature map are obtained respectively; Through the fourth layer of feature extraction of the fourth-level network of the lane line semantic segmentation model, according to the 4-fold down-sampled third pixel feature map, 8-fold down-sampled third pixel feature map, 16-fold down-sampled third pixel feature map, and 32-fold down-sampled third pixel feature map, a 4-fold down-sampled pixel feature map, an 8-fold down-sampled pixel feature map, a 16-fold down-sampled pixel feature map, and a 32-fold down-sampled pixel feature map of the lane line image are obtained respectively; The lane line semantic segmentation model is used to fuse the 4-fold downsampled pixel feature map, the 8-fold downsampled pixel feature map, the 16-fold downsampled pixel feature map, and the 32-fold downsampled pixel feature map to obtain the pixel features of the lane line image output by the lane line semantic segmentation model.

5. The method according to claim 4, characterized in that The step of calculating the loss function value corresponding to each layer network of the different layers of the lane line semantic segmentation model according to the first lane line semantic segmentation result and the different features includes: Calculating a first loss function value of the first hierarchical network using a cross entropy loss function according to the pixel semantic information and pixel features of the lane line image and the category region features; Calculating a second loss function value of the second-level network using the cross entropy loss function according to the category region feature and the similarity between the pixel and the region; Calculating a third loss function value of the third-level network using the cross entropy loss function according to the similarity between the pixel and the region and the context representation feature; According to the context representation features and the enhanced pixel features of the lane line image, the cross entropy loss function is used to calculate the fourth loss function value of the fourth-level network.

6. The method according to claim 5, characterized in that The calculating the loss function value of the lane semantic segmentation model according to the loss function value corresponding to each level network of the different levels of networks includes: According to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, a weighted sum algorithm is used to calculate the loss function value of the lane line semantic segmentation model.

7. The method according to claim 6, characterized in that If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, adjusting the network parameters corresponding to each level network of the different levels of the lane line semantic segmentation model includes: If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the first level network and the first layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the second layer network and the second layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, back-propagate the loss function value of the lane line semantic segmentation model, and adjust the network parameters of the third level network and the third layer of feature extraction of the lane line semantic segmentation model; and / or, If the loss function value of the lane line semantic segmentation model is greater than or equal to the set loss function threshold, the loss function value of the lane line semantic segmentation model is back-propagated to adjust the network parameters of the fourth-level network and the fourth-layer feature extraction of the lane line semantic segmentation model.

8. A lane line semantic segmentation device based on OCRNet, characterized in that: include: A first acquisition module is used to input the lane line image into the lane line semantic segmentation model based on OCRNet, and obtain different features of the lane line semantic segmentation model output by different layers of the network; A second acquisition module, configured to obtain a first lane line semantic segmentation result of the lane line image output by the lane line semantic segmentation model according to different features acquired by the first acquisition module; A first calculation module, used to calculate the loss function value corresponding to each layer network of the lane line semantic segmentation model according to the first semantic segmentation result of the lane line obtained by the second acquisition module and the different features obtained by the first acquisition module; A second calculation module, used to calculate the loss function value of the lane line semantic segmentation model according to the loss function value corresponding to each level network of different levels of networks calculated by the first calculation module; A segmentation result acquisition module is used to obtain the lane line semantic segmentation result including the lane line image output by the lane line semantic segmentation model based on the first semantic segmentation result of the lane line including the lane line image obtained by the second acquisition module if the loss function value of the lane line semantic segmentation model calculated by the second calculation module is less than a set loss function threshold.

9. A lane semantic segmentation model based on OCRNet, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of a lane line semantic segmentation model based on OCRNet, the processor is caused to execute a method as described in any one of claims 1 to 7.

Images