Lane line segmentation method and device based on image labeling model

Abstract

The application relates to a lane line segmentation method and device based on an image labeling model. The method comprises the following steps: acquiring a sample image, wherein the sample image comprises a road image with lane line labeling; inputting the sample image into a conditional random field algorithm to obtain a first result; inputting the sample image into a preset neural network to obtain a sample output result; inputting the sample output result into the conditional random field algorithm to obtain a second result; calculating a loss function of the preset neural network according to the first result and the second result; iteratively optimizing the parameters of the preset neural network based on the loss function to obtain a target image labeling model; and determining the lane line of a to-be-detected road image according to the to-be-detected road image and the target image labeling model. The scheme provided in the application can improve the lane line segmentation efficiency of the road image.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and in particular to a lane line segmentation method and apparatus based on image annotation models. Background Technology

[0002] Autonomous vehicles need to know the precise location of lane lines when driving or creating high-precision maps to determine safe driving zones or make decisions based on lane lines. Currently, various image segmentation models have emerged for segmenting various types of images. One important image segmentation model is the semantic image segmentation model. Semantic image segmentation models typically classify images at the pixel level, allowing for fine-grained segmentation. Before semantically segmenting lane lines, each pixel in the lane line portion of the image needs to be manually labeled. Labeling is difficult to achieve pixel-level precision, making it hard to guarantee that every pixel is accurately labeled. Therefore, manually labeled data often contains some incorrectly labeled pixels. To achieve pixel-level precision, on the one hand, labelers need to spend more time, increasing labeling costs; on the other hand, even with significant time investment, it's still difficult to guarantee that every pixel is correctly labeled. Summary of the Invention

[0003] To address or partially address the problems existing in related technologies, this application provides a lane line segmentation method and apparatus based on an image annotation model, thereby improving the efficiency of segmenting lane lines from road images.

[0004] The first aspect of this application discloses a lane line segmentation method based on an image annotation model, comprising: Acquire sample images, including road images with lane line markings; The sample image is input into the Conditional Random Field algorithm to obtain the first result; The sample image is input into a preset neural network to obtain the sample output result. The sample output result is then input into a conditional random field algorithm to obtain a second result. Based on the first and second results, the loss function of the preset neural network is calculated, and the parameters of the preset neural network are iteratively optimized based on the loss function to obtain the target image annotation model. Based on the annotation model of the road image to be detected and the target image, the lane lines of the road image to be detected are determined.

[0005] Optionally, acquiring sample images includes: Acquire road images and perform grayscale processing on the road images; The lane lines in the grayscale processed road image are identified and labeled to obtain a labeled image.

[0006] Optionally, identifying lane lines in the grayscale processed road image and labeling the lane lines includes: Edge detection is performed on the grayscale processed road image to identify pixels with lane line features; The pixels representing lane line features are labeled with a first identifier, and the remaining pixels in the road image are labeled with a second identifier.

[0007] Optionally, the sample image is input into a conditional random field algorithm to obtain a first result, including: Set the feature function of the conditional random field, and calculate the feature score of lane lines in the sample image based on the feature function; The feature scores of each road image in the sample images are weighted and summed to determine the feature function set in the sample images; Based on the feature function set, the probability that a pixel in the sample image is a lane line pixel is obtained to get the first result.

[0008] Optionally, the preset neural network is a semantic segmentation model, and the method further includes constructing the preset neural network, which includes: The system is configured with a feature extraction module and a category output module. The feature extraction module is used to extract the depth features of the sample image, and the category output module is used to output the probability of the sample image belonging to each category based on the depth features. Configure the upsampling function and number of upsampling layers for the feature extraction module, and configure the depthwise convolution function for the category output module.

[0009] Optionally, the method also includes constructing a loss function, which includes: Based on the first result, obtain the first probability that a pixel is predicted to be a lane line; Based on the second result, obtain the second probability that a pixel is predicted to be a lane line; Set category labels for lane line pixels and non-lane line pixels; Based on the first probability, the second probability, and the class label, construct the loss function.

[0010] Optionally, the preset neural network is iteratively optimized multiple times based on the loss function to obtain the target image annotation model, including: Based on the loss function results, the preset neural network is trained iteratively multiple times until the preset neural network converges, thus obtaining the target image annotation model. In this process, each iteration of training the pre-defined neural network includes: Based on the preliminary results of the loss function calculation and the sample annotations in the sample images, the error of the preset neural network annotations is calculated, and the model parameters of the preset neural network are corrected and optimized using the stochastic gradient descent method.

[0011] A second aspect of this application provides a lane segmentation device based on an image annotation model, comprising: The acquisition module is used to acquire sample images, including road images with lane line markings; The first acquisition module is used to input the sample image into the conditional random field algorithm to obtain the first result; The second acquisition module is used to input the sample image into a preset neural network to obtain the sample output result, and input the sample output result into a conditional random field algorithm to obtain the second result; The training module is used to calculate the loss function of the preset neural network based on the first result and the second result, and to iteratively optimize the parameters of the preset neural network based on the loss function to obtain the target image annotation model. The segmentation module is used to determine the lane lines of the road image to be detected based on the annotation model of the road image to be detected and the target image.

[0012] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0013] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0014] The technical solution provided in this application can include the following beneficial effects: On the one hand, by predicting sample images based on the Conditional Random Field algorithm, fewer labeled images can be used to obtain the data training set of the pre-defined neural network, thereby improving the training efficiency of the pre-defined neural network; on the other hand, since the image results obtained by the Conditional Random Field algorithm may contain more category information, it is equivalent to selecting more valuable pixels in the sample images for labeling, thus reducing the impact on the training of the image semantic segmentation model. Furthermore, since only a portion of pixels are selected from the sample images, compared to labeling all pixels in the sample images, the labeling method in this application embodiment relatively increases the uncertainty of the labeled objects, thereby increasing the uncertainty in the model training process, which is beneficial to improving the generalization ability of the image semantic segmentation model.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0017] Figure 1 This is a schematic flowchart illustrating the lane line segmentation method based on an image annotation model as shown in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the lane line segmentation device based on the image annotation model shown in the embodiments of this application; Figure 3 This is a schematic diagram of the vehicle structure shown in the embodiments of this application. Detailed Implementation

[0018] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, the terms involved in this application are introduced below.

[0022] 1. Artificial Intelligence (AI): This refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0023] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0024] 2. Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0025] 3. Convolutional Neural Network (CNN): A type of feedforward neural network whose artificial neurons can respond to a portion of the surrounding units within their coverage area, exhibiting excellent performance in large-scale image processing. A CNN consists of one or more convolutional layers and a fully connected layer at the top (corresponding to a classic neural network), and also includes associated weights and pooling layers.

[0026] 4. Deep features: Image features extracted through deep networks contain abstract information about the image.

[0027] 5. Semantic segmentation: Assign a corresponding category label to each pixel in the image based on the object of interest to which it belongs.

[0028] 6. Semantic Image: The result obtained by assigning a category tag to each pixel in the image.

[0029] 7. Mask image: In this embodiment of the application, the mask image is used to represent the road image. For example, the mask image can be a binary image. The binary image includes a first type of pixel with a first value and a second type of pixel with a second value. For example, if the value of a pixel in the binary image is 0, it means that the pixel is not selected. If the value of a pixel in the binary image is 1, it means that the pixel is selected.

[0030] 8. Conditional Generative Adversarial Nets (CGAN): An improvement on GAN, CGAN adds additional conditional information to the generator and discriminator of the original GAN ​​to achieve a conditional generative model. This additional conditional information can be class labels or other auxiliary information.

[0031] 9. ImageNet database: A large-scale database containing 1,000 categories.

[0032] 10. MobileNetV2: A commonly used lightweight network model architecture that is trained on the ImageNet database and can be used to extract image features.

[0033] 11. Image Classification and Categories: Image classification refers to an image processing method that distinguishes different categories of targets based on their different features reflected in image information. It utilizes computers to quantitatively analyze images, classifying each pixel or region within an image into one of several categories to replace human visual interpretation. Categories are also called classifications. In the embodiments of this application, there can be two or more categories, such as vehicles, roads, etc. When the image semantic segmentation model is applicable to different scenarios, the corresponding categories to be labeled can be different. Each target in an image is actually composed of pixels; the category of a pixel corresponds to the category of that target.

[0034] 12. Sample image and target image: Both are images. In this embodiment, the image used to train the model is called the sample image, and the image used for subsequent processing by the model is called the target image.

[0035] 13. Edge Information and Edge Pixels: Edge information describes pixels in an image whose grayscale values ​​do not change continuously within their neighborhood. These pixels are called edge pixels. Edge information can specifically include the grayscale value of each edge pixel and the shape formed by these edge pixels. Edges are widely present between objects and the background, and between objects themselves. Edge information in an image can be obtained through image edge detection.

[0036] 14. Conditional random fields (CRFs) are discriminative probabilistic models, a type of random field, commonly used for labeling or analyzing sequence data, such as natural language text or biological sequences. A CRF is a conditional probability distribution model P(Y|X), representing a Markov random field where, given a set of input random variables X, there is another set of output random variables Y. In other words, a characteristic of CRFs is the assumption that the output random variables constitute a Markov random field. CRFs can be seen as a generalization of the maximum entropy Markov model to the labeling problem. Like Markov random fields, CRFs are undirected graphical models where vertices represent random variables, and the lines connecting vertices represent dependencies between random variables. In a CRF, the distribution of random variable Y is a conditional probability, and the given observation is the random variable X. In principle, the graphical model layout of a Conditional Random Field (CRF) can be arbitrarily given. A commonly used layout is a chained architecture, which offers efficient algorithms for training, inference, and decoding. A CRF is a typical discriminative model, and its joint probability can be written as a product of several potential functions, with the linear-chain CRF being the most common.

[0037] Autonomous vehicles need to know the precise location of lane lines when driving or creating high-precision maps to determine safe driving zones or make decisions based on lane lines. Currently, various image segmentation models have emerged for segmenting various types of images. One important image segmentation model is the semantic image segmentation model. Semantic image segmentation models typically classify images at the pixel level, allowing for fine-grained segmentation. Before semantically segmenting lane lines, each pixel in the lane line portion of the image needs to be manually labeled. Labeling is difficult to achieve pixel-level precision, making it hard to guarantee that every pixel is accurately labeled. Therefore, manually labeled data often contains some incorrectly labeled pixels. To achieve pixel-level precision, on the one hand, labelers need to spend more time, increasing labeling costs; on the other hand, even with significant time investment, it's still difficult to guarantee that every pixel is correctly labeled.

[0038] To address the aforementioned issues, this application provides a lane line segmentation method based on an image annotation model, which improves the efficiency of segmenting lane lines from road images.

[0039] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0040] See Figure 1 This is a flowchart illustrating the lane line segmentation method based on an image annotation model according to an embodiment of this application, mainly including steps S101 to S104, as described below: Step S101: Obtain sample images, which include road images with lane line markings.

[0041] In this embodiment, the road image is captured by an onboard camera and includes surrounding lane lines; the road image in the sample image carries lane line annotations. Acquiring the sample image includes: acquiring the road image; performing grayscale processing on the road image; identifying lane lines in the road image; and annotating the lane lines to obtain an annotated image.

[0042] In one embodiment, identifying lane lines in a road image and annotating the lane line image includes: performing edge detection on the grayscale processed road image to determine pixels with lane line features; marking the pixels with lane line features with a first identifier and marking the remaining pixels in the road image with a second identifier.

[0043] In this embodiment, edge detection includes: detecting edge information of the sample image; and based on the edge information, determining edge pixels in the sample image that have lane line edge features, which correspondingly determines the lane line pixels included in the image. Edge pixels can be understood as pixels whose pixel values ​​change significantly based on lane line features. Specifically, the sample image is converted into a grayscale image, and then the edge information of the image is extracted based on a preset edge detection algorithm. The preset edge detection algorithm is, for example, the Canny edge detection algorithm. The edge detection process is illustrated below using the Canny edge detection algorithm as an example. S1.2 performs Gaussian filtering on the grayscale-processed sample image.

[0044] The main purpose of Gaussian filtering is to reduce noise in grayscale processed image samples. Applying Gaussian filtering to a grayscale processed image can be understood as performing a weighted average; that is, the grayscale value of each pixel in the grayscale processed image is obtained by weighted averaging the grayscale values ​​of that pixel and its neighboring pixels. Because Gaussian filtering performs a weighted average of the grayscale values ​​of each pixel in the grayscale processed image, it filters out some noise, making the overall contours of the grayscale processed image relatively blurred, resulting in a smoother overall image and thus relatively increasing the width of the contours. The gradient value and gradient direction in the Gaussian-filtered image are then calculated. An edge can be understood as a set of pixels with significant grayscale value changes. For example, if one edge is black and the other is white, the area between the black and white edges is generally considered an edge. In practice, grayscale value changes can be detected to find edges in the image. The gradient value represents the degree of grayscale value change, and the gradient direction represents the direction of grayscale value change.

[0045] S1.3: Filter out non-maximum values.

[0046] Since the contour widths in the image are actually magnified, this may affect the accuracy of edge detection. Therefore, S1.3 is mainly used to filter out pixels that do not belong to the edges. Specifically, if the first server determines that the gradient value of a pixel in the gradient direction is the largest, then the pixel is determined to be a suspected edge pixel; if it determines that the gradient value of a pixel in the gradient direction is not the largest, then the pixel is determined not to be an edge pixel, and so on, thereby eliminating some pixels that do not belong to the edges. Suspected edge pixels can be understood as initially identified as edge pixels, but can be further confirmed. Optionally, suspected edge pixels can be directly used as edge pixels to obtain the edge pixels in the grayscale sample image.

[0047] S1.4: Edge Determination Using Upper Thresholds. To determine more accurate edge pixels, in this embodiment, suspected edge pixels in S1.3 can be further filtered using high and low thresholds. The high threshold is greater than the low threshold. Specifically, if the first server determines that the gradient value of a suspected edge pixel is greater than the high threshold, then the suspected edge pixel is determined to be an edge pixel; if the gradient value of a suspected edge pixel is less than the high threshold but greater than the low threshold, then the suspected edge pixel is determined to be an edge pixel; if the gradient value of a suspected edge pixel is less than or equal to the low threshold, then the suspected edge pixel is determined not to be an edge pixel. In this way, all edge pixels in the grayscale-processed sample image can be determined, thereby obtaining an edge image. An edge image can be understood as an image that identifies edge pixels and non-edge pixels. Non-edge pixels can be understood as pixels in the image that do not belong to the edge pixels.

[0048] After obtaining the edge image, the edge pixels are then identified based on their grayscale values. For example, pixels with a preset grayscale value are considered edge pixels. After obtaining the edge image, the number of edge pixels in each sample image can naturally be determined. It should be noted that the number of edge pixels in each sample image can be 0, 1, or multiple.

[0049] After determining the edge image, lane line pixels are labeled with a first identifier, and pixels in multiple sample images other than lane lines are labeled with a second identifier, thus obtaining a mask image. The first and second identifiers are different identifiers. They can be of the same type but different identifiers within the same type. For example, both the first and second identifiers can be represented by color; specifically, the first identifier could be white (represented by "1"), and the second identifier could be black (represented by "0"). Having the first and second identifiers of the same type facilitates subsequent analysis of the mask image by the device. However, the first and second identifiers can also belong to different types; this application does not impose this limitation.

[0050] Step S102: Input the sample image into the Conditional Random Field algorithm to obtain the first result.

[0051] In one embodiment, obtaining a first result based on a conditional random field algorithm and the sample image includes: setting a feature function of the conditional random field; calculating feature scores of road images in the sample image according to the feature function; weighted summing of the feature scores of each road image in the sample image to determine a feature function set in the sample image; and obtaining the probability that a pixel in the sample image is a lane line pixel according to the feature function set to obtain the first result.

[0052] Specifically, each element in the sample image is compared with the label assigned in the labeled image. A feature function (which can be a probability matrix) is introduced using a Conditional Random Field (CRF) to calculate the transition probability of each pixel becoming a lane line pixel, thus obtaining the label and probability score corresponding to each pixel in the sample image. CRFs are discriminative probabilistic models, a type of random field, commonly used for labeling or analyzing sequence data, such as natural language text or biological sequences. A CRF is a conditional probability distribution model P(Y|X), representing a Markov random field where, given a set of input random variables X, there is another set of output random variables Y. In other words, a characteristic of CRFs is the assumption that the output random variables constitute a Markov random field.

[0053] In this embodiment, a feature function is introduced through a conditional random field, and the initial score is input into the feature function to calculate the probability that each pixel in the sample image becomes a lane line pixel, thereby obtaining a probability score to obtain the label probability distribution of each pixel, that is, to obtain the similarity between each element and the label.

[0054] S2.1: Based on the probability score and label probability score of each pixel, obtain the total score for each pixel.

[0055] S2.2: By progressively accumulating the total score of each pixel using the Viterbi algorithm, the optimal solution path for each direction is obtained.

[0056] S2.3: Combine the optimal solution paths of each pixel to obtain the optimal solution path of the first result, and generate a feature function set based on the optimal solution path.

[0057] Specifically, the Viterbi algorithm predicts the optimal label sequence by progressively accumulating the total score of each element. When updating the label, since the label is fixed, the optimal distribution of each element is predicted in reverse by using the total score of the label.

[0058] Step S103: Input the sample image into a preset neural network to obtain the sample output result, and input the sample output result into a conditional random field algorithm to obtain the second result.

[0059] In one embodiment, the preset neural network is a semantic segmentation model. Obtaining the preset neural network includes: setting a feature extraction module and a category output module. The feature extraction module is used to extract the depth features of the sample image, and the category output module is used to output the probability that the sample image belongs to each category based on the depth features; setting the upsampling function and the number of upsampling layers of the feature extraction module, and setting the depth convolution function of the category output module.

[0060] Specifically, the feature extraction module can be implemented using a MobileNet network, such as MobileNetV2. MobileNetV2 can be pre-trained using images from the ImageNet database. The category output module can be implemented using convolutional layers, activation layers, and upsampling layers. Optionally, the category output module includes a first activation layer, a first convolutional layer, a second activation layer, a second convolutional layer, a third activation layer, a first upsampling layer, a fourth activation layer, a second upsampling layer, a fifth activation layer, a third upsampling layer, a sixth activation layer, a fourth upsampling layer, a seventh activation layer, and a fifth upsampling layer, connected in sequence.

[0061] Specifically, the MobileNetV2 network, acting as a feature extraction module, extracts convolutional features from sample images, outputs corresponding feature maps, and then performs convolutional processing on these feature maps to output new feature maps. Five upsampling layers enlarge the scale of these feature maps. Each upsampling layer inserts zeros into the middle of its input feature map to enlarge the image, then performs a convolution operation on the enlarged image to output an enlarged feature map. For example, the width and height of the output feature map can be doubled compared to the input feature map. The last sampling layer has C output channels, each corresponding to the probability of each pixel belonging to each category, thus obtaining the probability of each pixel belonging to different categories.

[0062] For example, the sample image, annotation results, and mask image can be processed to a preset size, such as 640×360×3, where 3 represents the number of channels. The processed sample image is input into the image semantic segmentation model in this embodiment. After processing by the MobileNetV2 network, a first feature map is obtained. This first feature map is then passed sequentially through a first activation layer and a first convolutional layer. The first convolutional layer has a kernel size of 4, a stride of 2, and outputs a second feature map with 512 channels.

[0063] Similarly, the second feature map is then passed through the second activation layer and the second convolutional layer to obtain a third feature map with 512 channels. The third feature map is then input into the third activation layer and the first upsampling layer to obtain a fourth feature map with 512 channels. The fourth feature map is then input into the fourth activation layer and the second upsampling layer to obtain a fifth feature map with 256 channels. The fifth feature map is then input into the fifth activation layer and the third upsampling layer to obtain a sixth feature map with 128 channels. The seventh feature map is then input into the sixth activation layer and the fourth upsampling layer to obtain an eighth feature map with 64 channels. The eighth feature map is then input into the sixth activation layer and the fifth upsampling layer to obtain a semantic segmentation image with 7 channels, which has a size of 640×360×7.

[0064] In this embodiment, for the sample output of the preset neural network, the Conditional Random Field (CRF) algorithm is applied to obtain the probability value of each pixel in the output second result as a lane line. Introducing the CRF algorithm converts the semantic segmentation result into probability value labels, thus transforming the comparison between the first and second results into a comparison of the probability values ​​of the first and second results, thereby making the loss function of the preset neural network more accurate.

[0065] Step S104: Based on the first result and the second result, calculate the loss function of the preset neural network, and iteratively optimize the parameters of the preset neural network based on the loss function to obtain the target image annotation model.

[0066] In one embodiment, the method further includes constructing a loss function, which includes: obtaining a first probability value that a pixel is predicted as a lane line based on a first result; obtaining a second probability value that a pixel is predicted as a lane line based on a second result; setting category labels for lane line pixels and non-lane line pixels; and constructing a loss function based on the first probability value, the second probability value, and the category labels.

[0067] In one embodiment, a target image annotation model is obtained by iteratively optimizing a preset neural network based on a loss function. This includes: training the preset neural network iteratively multiple times based on the loss function results until the preset neural network converges, thereby obtaining the target image annotation model. Each iteration of the training process includes: calculating the error of the preset neural network annotation based on the preliminary result of the loss function calculation and the sample annotations in the sample image, and adjusting the model parameters of the preset neural network using a stochastic gradient descent method.

[0068] Specifically, based on the pixel probability values ​​of the first result and the pixel probability values ​​of the second result (semantic segmentation result), the model parameters of the image semantic segmentation model are adjusted. The following example illustrates how to adjust the model parameters of the image semantic segmentation model: The loss function can be determined based on the annotation results of each pixel in the first result of the sample image (the probability value of the pixel being a lane line) and the semantic segmentation results of the semantic segmentation results corresponding to at least one pixel in the first data. Then, based on the value of the loss function, the model parameters of the image semantic segmentation model are adjusted to reduce the difference between the annotation results and the semantic segmentation results of the image semantic segmentation model. This process continues until the image semantic segmentation model converges, resulting in a trained image semantic segmentation model.

[0069] Specifically, when training the image semantic model iteratively multiple times, if the image semantic segmentation model converges, the training is considered complete, and the trained image semantic segmentation model is obtained. Convergence of the image semantic segmentation model can be achieved by the loss function value being less than a loss threshold, or by the number of iterations reaching a preset number, etc. This application does not impose specific restrictions on the convergence conditions.

[0070] In this embodiment, the image semantic segmentation model is trained using the first result. Since the first result is generated from a random training field, a large number of data samples can be generated from a small number of labeled images. Furthermore, because the first result does not annotate pixels in the sample images, the possibility of overfitting in the image semantic segmentation model is reduced, resulting in better generalization performance of the trained semantic segmentation model. Since the sample images are not fully annotated, the uncertainty of the annotation results is increased, thereby improving the processing power of the image semantic segmentation model and ultimately enhancing the accuracy of the segmentation results obtained by the image semantic segmentation model. Step S105: Determine the lane lines of the road image to be detected based on the annotation model of the road image to be detected and the target image.

[0071] In this embodiment, the road image to be detected is input into the target image annotation model to generate the probability that each pixel in the road image to be detected is a lane line pixel. According to a preset threshold, the pixels that are lane line pixels are determined and extracted to obtain the target lane line.

[0072] On the one hand, predicting sample images using the Conditional Random Field (CRF) algorithm allows for the acquisition of a training data set for the pre-defined neural network with fewer labeled images, thereby improving the training efficiency of the pre-defined neural network. On the other hand, since the image results obtained by the CRF algorithm may contain more category information, it is equivalent to selecting more valuable pixels from the sample images for labeling, thus reducing the impact on the training of the image semantic segmentation model. Furthermore, because only a portion of pixels are selected from the sample images, compared to labeling all pixels in the sample images, the labeling method in this embodiment relatively increases the uncertainty of the labeled objects, thereby increasing the uncertainty during model training and ultimately improving the generalization ability of the image semantic segmentation model.

[0073] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a lane line segmentation device, vehicle, and corresponding embodiments based on an image annotation model.

[0074] Figure 2 This is a schematic diagram illustrating the structure of a lane segmentation device based on an image annotation model, as shown in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiments of this application are shown. Figure 2The example device mainly includes a data acquisition module 201, a first acquisition module 202, a second acquisition module 203, a training module 204, and a segmentation module 205, wherein: Acquisition module 201 is used to acquire sample images, including road images with lane line markings; The first acquisition module 202 inputs the sample image into the conditional random field algorithm to obtain the first result; The second acquisition module 203 is used to input the sample image into a preset neural network to obtain the sample output result, and input the sample output result into a conditional random field algorithm to obtain the second result; The training module 204 is used to calculate the loss function of the preset neural network based on the first result and the second result, and to iteratively optimize the parameters of the preset neural network based on the loss function to obtain the target image annotation model. The segmentation module 205 is used to determine the lane lines of the road image to be detected based on the annotation model of the road image to be detected and the target image.

[0075] See Figure 3 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. The electronic device 300 includes a memory 310 and a processor 320.

[0076] The processor 320 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0077] Memory 310 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 320 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 310 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 310 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0078] The memory 310 stores executable code, which, when processed by the processor 320, can cause the processor 320 to execute part or all of the methods described above.

[0079] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0080] Alternatively, this application may also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of a vehicle (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0081] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A lane line segmentation method based on an image annotation model, characterized in that, The method includes: Acquire sample images, including road images with lane line markings; The sample image is input into a Conditional Random Field (CRF) algorithm to obtain a first result. This includes: setting a feature function for the CRF; calculating the feature score of lane lines in the sample image based on the feature function; weighted summing of the feature scores for each road image in the sample image to determine a feature function set in the sample image; and obtaining the probability that a pixel in the sample image is a lane line pixel based on the feature function set to obtain the first result. The feature function is used to calculate the probability that each pixel in the sample image is a lane line pixel. The sample image is input into a preset neural network to obtain the sample output result, and the sample output result is input into the conditional random field algorithm to obtain the second result; Based on the first result and the second result, the loss function of the preset neural network is calculated, and the parameters of the preset neural network are iteratively optimized based on the loss function to obtain the target image annotation model; Based on the road image to be detected and the target image annotation model, the lane lines of the road image to be detected are determined.

2. The method according to claim 1, characterized in that, The acquisition of sample images includes: Acquire road images and perform grayscale processing on the road images; Lane lines in a grayscale processed road image are identified and annotated to obtain an annotated image.

3. The method according to claim 2, characterized in that, The process of identifying lane lines in a grayscale-processed road image and annotating the lane lines includes: Edge detection is performed on the grayscale processed road image to identify pixels with lane line features; The pixels of the lane line feature are marked with a first identifier, and the remaining pixels in the road image are marked with a second identifier.

4. The method according to claim 1, characterized in that, The preset neural network is a semantic segmentation model, and the method further includes constructing the preset neural network, which includes: The system includes a feature extraction module and a category output module. The feature extraction module is used to extract the depth features of the sample image, and the category output module is used to output the probability that the sample image belongs to each category based on the depth features. Configure the upsampling function and number of upsampling layers for the feature extraction module, and configure the depthwise convolution function for the category output module.

5. The method according to claim 1, characterized in that, The method further includes constructing a loss function, which includes: Based on the first result, obtain the first probability that a pixel is predicted to be a lane line; Based on the second result, obtain the second probability that a pixel is predicted to be a lane line; Set category labels for lane line pixels and non-lane line pixels; The loss function is constructed based on the first probability, the second probability, and the category label.

6. The method according to claim 1, characterized in that, The step of iteratively optimizing the parameters of the preset neural network based on the loss function to obtain the target image annotation model includes: Based on the loss function results, the preset neural network is trained iteratively multiple times until the preset neural network converges, thereby obtaining the target image annotation model. In this process, each iteration of training the pre-defined neural network includes: Based on the preliminary results of the loss function calculation and the error of the preset neural network annotation calculated from the sample annotations in the sample images, the model parameters of the preset neural network are corrected and optimized using the stochastic gradient descent method.

7. A lane segmentation device based on an image annotation model, characterized in that, The device includes: The acquisition module is used to acquire sample images, including road images with lane line markings; The first acquisition module is used to input the sample image into a conditional random field algorithm to obtain a first result; including: setting a feature function of the conditional random field; calculating the feature score of lane lines in the sample image according to the feature function; weighted summing of the feature scores of each road image in the sample image to determine the feature function set in the sample image; obtaining the probability that a pixel in the sample image is a lane line pixel according to the feature function set to obtain the first result; the feature function is used to calculate the probability that each pixel in the sample image is a lane line pixel; The second acquisition module is used to input the sample image into a preset neural network to obtain a sample output result, and input the sample output result into the conditional random field algorithm to obtain a second result; The training module is used to calculate the loss function of the preset neural network based on the first result and the second result, and to iteratively optimize the parameters of the preset neural network based on the loss function to obtain the target image annotation model; The segmentation module is used to determine the lane lines of the road image to be detected based on the road image to be detected and the target image annotation model.

8. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 6.

Images