Road marking detection method, device, vehicle and storage medium

Abstract

This invention discloses a method, device, vehicle, and storage medium for detecting road markings. The method includes: acquiring an image captured by a vehicle-mounted camera; converting the image to obtain a road surface top view; inputting the road surface top view into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view; performing attribute recognition on the road surface top view based on the trained neural network model to obtain attribute recognition results; performing frame length and width regression detection on the road surface top view based on the trained neural network model to obtain frame length and width regression detection results; and determining the road marking detection results in the road surface top view based on the number and location of key points, the attribute recognition results, and the frame length and width regression detection results. This invention detects road markings in the road surface top view based on a neural network model, thereby effectively covering various lane lines and road markings and providing more accurate detection results.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more specifically, to a method, apparatus, vehicle, and storage medium for detecting road markings. Background Technology

[0002] With rapid economic development and rising living standards, my country's car ownership has increased rapidly, leading to increasingly prominent traffic safety issues. Vehicle safety driver assistance systems have become one of the hottest research topics in the global traffic safety field. Over the past few decades, experts and scholars both domestically and internationally have made significant contributions to the research of intelligent driving assistance systems. Currently, relatively mature advanced driver assistance systems have emerged abroad and have been commercialized. Compared to foreign technologies, domestic research started relatively late. Lane recognition is a crucial component of intelligent driving and autonomous driving systems. Current lane line and other road marking detection technologies are based on forward and side camera images, and can only detect the rough position and attributes of lane lines in the images. Summary of the Invention

[0003] In view of the above problems, the present invention proposes a road marking detection method, device, vehicle and storage medium to overcome or at least partially solve the problems of the prior art.

[0004] In a first aspect, embodiments of the present invention provide a road marking detection method, the method comprising: acquiring an image captured by a vehicle-mounted camera; converting the image to obtain a road surface top view; inputting the road surface top view into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view; performing attribute recognition on the road surface top view according to the trained neural network model to obtain attribute recognition results of the road surface top view; performing frame length and width regression detection on the road surface top view according to the trained neural network model to obtain frame length and width regression detection results of the road surface top view; and determining the road marking detection result in the road surface top view based on the number and location of key points in the road surface top view, the attribute recognition results, and the frame length and width regression detection results.

[0005] Secondly, embodiments of the present invention provide a road marking detection device, which includes: an image acquisition module for acquiring images captured by a vehicle-mounted camera; an image conversion module for converting the images to obtain a road surface top view; a key point detection module for inputting the road surface top view into a trained neural network model for key point detection, and determining the number and location of key points in the road surface top view; an attribute recognition module for performing attribute recognition on the road surface top view according to the trained neural network model, and obtaining the attribute recognition result of the road surface top view; a regression detection module for performing frame length and width regression detection on the road surface top view according to the trained neural network model, and obtaining the frame length and width regression detection result of the road surface top view; and a result output module for determining the road marking detection result in the road surface top view based on the number and location of key points in the road surface top view, the attribute recognition result, and the frame length and width regression detection result.

[0006] Thirdly, embodiments of the present invention provide a vehicle including a camera disposed on the exterior of the vehicle body. The vehicle further includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the above-described method.

[0007] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.

[0008] This invention provides a road marking detection method, device, vehicle, and storage medium. The method includes: acquiring an image captured by a vehicle-mounted camera; converting the image to obtain a road surface top view; inputting the road surface top view into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view; performing attribute recognition on the road surface top view based on the trained neural network model to obtain attribute recognition results; performing frame length and width regression detection on the road surface top view based on the trained neural network model to obtain frame length and width regression detection results; and determining the road marking detection results in the road surface top view based on the number and location of key points, attribute recognition results, and frame length and width regression detection results. Thus, by acquiring a road surface top view from an image captured by a vehicle-mounted camera and performing key point detection on the road surface top view based on a neural network model to determine the road marking detection results, this method effectively covers various lane lines and road markings, including various types such as points, lines, and frames, and provides more accurate detection results. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic flowchart of a road marking detection method provided in an embodiment of the present invention is shown;

[0011] Figure 2 A flowchart illustrating another road marking detection method provided by an embodiment of the present invention is shown;

[0012] Figure 3 A flowchart of the model inference process provided in an embodiment of the present invention is shown;

[0013] Figure 4 A schematic flowchart of a road marking detection method provided in an embodiment of the present invention is shown;

[0014] Figure 5 A flowchart of the model training process provided in an embodiment of the present invention is shown;

[0015] Figure 6 A flowchart of a detection method provided by an embodiment of the present invention is shown;

[0016] Figure 7 This diagram shows a block diagram of a road marking detection device provided in an embodiment of the present invention.

[0017] Figure 8 A block diagram of a vehicle used to perform a road marking detection method according to an embodiment of the present invention is shown.

[0018] Figure 9 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0020] With rapid economic development and the continuous improvement of people's living standards, my country's car ownership has increased rapidly, and traffic safety issues have become increasingly prominent. Vehicle safety driving assistance systems have become one of the hot research topics in the global traffic safety field. Over the past few decades, experts and scholars both domestically and internationally have made significant contributions to the research of intelligent driving assistance systems. Currently, relatively mature advanced driver assistance systems have emerged abroad and have been put into commercial application. Compared to foreign technologies, domestic research started relatively late. Lane line recognition is an important component of intelligent driving and autonomous driving systems. Current lane line and other road marking detection technologies are based on forward and side camera images. For lane line detection, only the rough position information and attributes of the lane lines in the image can be detected. For other road markings, only forward camera images are used for detection, resulting in significant limitations. Furthermore, existing technologies can only cover lane lines and driving markings within a small area near the vehicle; the results for distant or side recognition are poor or impossible. Simultaneously, only the lateral information of the lane lines is detected, and the longitudinal start and end points of the lane lines cannot be detected.

[0021] To address the aforementioned problems, the inventors have proposed a road marking detection method, device, vehicle, and storage medium as provided in this invention. By acquiring a top-down view of the road surface from images captured by an onboard camera, and performing key point detection on the road surface top-down view based on a neural network model, the detection results of road markings are determined. This effectively covers various lane lines and road markings, including various types such as points, lines, and frames, and provides more accurate detection results. The specific road marking detection method will be described in detail in subsequent embodiments.

[0022] Please see Figure 1 , Figure 1 A schematic flowchart of a road marking detection method provided by an embodiment of the present invention is shown. The following will focus on... Figure 1 The process shown is described in detail. The road marking detection method may specifically include the following steps S110 to S160.

[0023] Step S110: Acquire images captured by the vehicle-mounted camera.

[0024] Existing technologies typically rely solely on images captured by a forward-facing camera for lane line detection, resulting in only a rough approximation of lane lines and detection of lane lines within a small area near the vehicle. Therefore, this embodiment employs images captured by a front-facing main camera and three cameras on the left and right sides. This means that when the vehicle is photographed, the solution can acquire images from at least the front, left, and right sides of the vehicle. Compared to existing technologies, this allows for the detection of road markings from the side or at a distance. Furthermore, due to the increased identifiable range, this solution can detect not only the lateral information of lane lines but also their longitudinal start and end points, leading to more accurate lane line detection results.

[0025] In some implementations, the vehicle-mounted camera captures images of the road surface either in real-time and transmits them to the vehicle, or at regular intervals. For example, the camera might capture an image every five minutes, or take multiple consecutive images every five minutes to ensure image integrity. Furthermore, the camera may also begin capturing images of the road surface when preset conditions are met. For instance, it might start capturing images when the vehicle starts, or it might start capturing images when the vehicle detects that it is traveling on the road. These methods are merely examples and are not intended to limit the scope of the implementation.

[0026] In some implementations, when acquiring images from the vehicle-mounted camera, the vehicle can acquire the images in real time and process them accordingly; alternatively, it can acquire images at regular intervals, such as every minute. Furthermore, the vehicle can acquire images when preset conditions are met, such as while the vehicle is in motion, or when it receives an instruction to acquire images from the vehicle-mounted camera. These methods are merely examples and are not intended to be limiting. The number of images acquired from the vehicle-mounted camera can be multiple.

[0027] Step S120: Convert the image to obtain a top view of the road surface.

[0028] When a vehicle acquires images from its onboard camera, it can process the images to better detect lane lines and other road markings, providing more accurate detection results to the user. In this embodiment, the images captured by the onboard camera can be transformed to obtain a top-down view of the road surface. The images captured by the onboard camera can include at least images from the front of the vehicle, the front left side of the vehicle, and the front right side of the vehicle. However, detecting road markings based on images from each direction may result in inaccuracies due to image distortion in different directions. Therefore, by transforming and stitching the images from the three directions, a top-down view of the road surface ahead can be obtained. Key point detection using this top-down view yields more accurate road marking detection results. Road markings can include lane lines, stop lines, zebra crossings, road arrows, road text, speed bumps, etc.

[0029] In some implementations, multiple images captured by the vehicle-mounted camera can be directly stitched together to obtain a top-down view of the road surface.

[0030] Furthermore, to eliminate distortions caused by images captured from different angles, the images captured by the vehicle-mounted camera can be processed first. Specifically, camera calibration can be performed on the vehicle-mounted camera. For example, each camera can be used to capture images of a black-and-white grid from multiple angles to obtain images of the camera at different angles. This allows the acquisition of the camera's intrinsic parameter matrix and distortion parameters. Based on the camera's intrinsic parameter matrix and distortion parameters, distortion correction can be performed on the acquired images. Then, perspective transformation can be performed on the multiple images captured by the vehicle-mounted camera to convert each image into a top view. The top views of the various images can be stitched together to obtain a top view of the road surface.

[0031] In some implementations, the vehicle can be equipped with an onboard 360-degree surround view camera, which can directly generate a top-down view of the road surface, thus eliminating the need for top-down view conversion and stitching, saving onboard computing resources. At the same time, the onboard 360-degree camera can acquire images from all directions, resulting in a more accurate top-down view that is less affected by the smoothness of the road surface or slopes.

[0032] Step S130: Input the road surface top view into the trained neural network model to detect key points and determine the number and location of key points in the road surface top view.

[0033] In this embodiment, after obtaining the road surface top view, key point detection can be performed on the road surface top view to determine the number and location of key points in the road surface top view. Specifically, the road surface top view can be input into a trained neural network model for key point detection, obtaining the number and location of key points in the road surface top view output by the trained neural network model. The trained neural network model can be obtained by training a neural network based on a large number of sample road surface top views and corresponding road markings. The key points of the road markings can be points uniformly sampled on linear objects, endpoints of road arrows, or center points of road text, etc.

[0034] Step S140: Perform attribute recognition on the road surface top view based on the trained neural network model to obtain the attribute recognition results of the road surface top view.

[0035] In this embodiment, simply detecting the key points in the road surface top view and obtaining the number and location of these key points is insufficient to determine the road markings within the top view. Therefore, after obtaining the number and location of the key points, further attribute recognition can be performed on the road surface top view using a trained neural network model to obtain the attribute recognition results. Attribute recognition can involve identifying the attributes of the road markings in the top view. Attributes may include whether they are boundary lane lines or deceleration lane lines. Based on the number and location of the key points, the approximate category of the road markings can be determined, and then the specific road markings can be determined based on the attribute recognition results.

[0036] Step S150: Perform frame length and width regression detection on the road surface top view based on the trained neural network model to obtain the frame length and width regression detection results of the road surface top view.

[0037] In this embodiment, since road markings include not only simple markings such as lane lines, but also complex markings such as road arrows and road text, complex markings in the road top view can be identified by performing frame length and width regression detection on the road top view. Specifically, frame length and width regression detection can be performed on the road top view using a trained neural network model to obtain the frame length and width regression detection results. The frame length and width regression detection results can include the length, width, and position of the framed objects in the road top view, to locate the size and orientation of the framed objects.

[0038] Step S160: Determine the road marking detection results in the road top view based on the number and location of key points in the road top view, the attribute recognition results, and the frame length and width regression detection results.

[0039] In this embodiment, a trained neural network model is used to detect road surface top views to obtain road marking detection results. Specifically, the trained neural network model includes networks for key point detection, attribute recognition, and frame length and width regression. Each network is used to detect the road surface top view, obtaining the number and location of key points, attribute recognition results, and frame length and width regression detection results, thus achieving more accurate road marking detection results in the road surface top view. For example, when only one key point is detected in the road surface top view, the road marking in the top view can be determined as a point-type object. Combined with the attribute recognition results of the top view, the specific road marking can then be identified.

[0040] The road marking detection method provided in the above embodiments acquires images captured by a vehicle-mounted camera; transforms the images to obtain a road surface top view; inputs the road surface top view into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view; performs attribute recognition on the road surface top view based on the trained neural network model to obtain attribute recognition results; performs frame length and width regression detection on the road surface top view based on the trained neural network model to obtain frame length and width regression detection results; and determines the road marking detection results in the road surface top view based on the number and location of key points, attribute recognition results, and frame length and width regression detection results. Thus, by acquiring a road surface top view from images captured by a vehicle-mounted camera and performing key point detection on the road surface top view based on a neural network model to determine the detection results of road markings, this method effectively covers various lane lines and road markings, including various types of points, lines, and frames, and provides more accurate detection results.

[0041] Please see Figure 2 , Figure 2 This diagram illustrates a further road marking detection method provided by an embodiment of the present invention. The following will focus on... Figure 2 The process shown is described in detail. The method may specifically include the following steps S210 to S280.

[0042] Step S210: Acquire images captured by the vehicle-mounted camera.

[0043] Step S220: Convert the image to obtain a top view of the road surface.

[0044] Step S230: Input the road surface top view into the trained neural network model to detect key points and determine the number and location of key points in the road surface top view.

[0045] Step S240: Perform attribute recognition on the road surface top view based on the trained neural network model to obtain the attribute recognition results of the road surface top view.

[0046] Step S250: Perform frame length and width regression detection on the road surface top view based on the trained neural network model to obtain the frame length and width regression detection results of the road surface top view.

[0047] For a detailed description of steps S210 to S250, please refer to steps S110 to S150, which will not be repeated here.

[0048] Step S260: If it is determined that the road surface top view includes linear objects based on the number and location of key points in the road surface top view, the attribute recognition results, and the frame length and width regression detection results, then a two-way vector field estimation is performed on the road surface top view according to the trained neural network model to obtain the two-way vector field estimation result of the linear objects. The two-way vector field estimation is obtained by estimating the direction of the next line segment for each pixel in the image to obtain a vector, and the vector is used to represent the connection relationship from the pixel to the next connection point.

[0049] In this embodiment, when two or more key points are detected in the road surface top view, clustering is performed on the key points based on their locations and the attribute recognition results of the road surface top view to obtain the line segment detection results of the road surface top view. When the clustering results indicate that the key points form a line segment, it can be determined that the road surface top view includes a linear object. Lane lines can include solid lane lines and dashed lane lines; solid lane lines and dashed lane lines represent different meanings. Therefore, as... Figure 3 As shown, the bidirectional vector field estimation of the road surface top view can be further performed based on the trained neural network model to obtain the bidirectional vector field estimation result of the linear object. The bidirectional vector field estimation is obtained by estimating the direction of the next line segment for each pixel in the image. The vector is used to represent the connection relationship from the pixel to the next connection point.

[0050] Step S270: If the bidirectional vector field estimation result indicates that the connection relationship of the linear object satisfies the dashed line connection relationship, then the linear object is determined to be a dashed line.

[0051] Step S280: If the bidirectional vector field estimation result indicates that the connection relationship of the linear object does not satisfy the dashed line connection relationship, then the linear object is determined to be a solid line.

[0052] In this embodiment, due to the different connection relationships between dashed and solid lines, the linear object in the road surface top view can be determined as a solid or dashed line based on the bidirectional vector field estimation result. The bidirectional vector length estimation obtains a vector by estimating the direction of the next line segment for each pixel in the image. This vector represents the connection relationship from the pixel to the next connection point. The combined effect of multiple pixels forms a vector field. Since solid lines are continuous, the bidirectional vector field estimation result can yield a single vector field. Dashed lines, however, are discontinuous, resulting in multiple vector fields. Therefore, if the bidirectional vector field estimation result yields multiple vector fields, it indicates that the connection relationship of the linear object satisfies the dashed line connection relationship, and the linear object can be determined to be a dashed line. Conversely, if the bidirectional vector field estimation result yields a single vector field, it indicates that the connection relationship of the linear object does not satisfy the dashed line connection relationship, and the linear object can be determined to be a solid line.

[0053] The road marking detection method provided in the above embodiments is compared with Figure 1 The road marking detection method shown in this embodiment further obtains a two-way vector field estimation of the road surface top view based on a trained neural network model, resulting in a two-way vector field estimation result for linear objects, thereby obtaining a more accurate road marking detection result. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 A schematic flowchart of a road marking detection method provided by an embodiment of the present invention is shown. The following will focus on... Figure 4 The process shown is described in detail. The method may specifically include the following steps S310 to S380.

[0054] Step S310: Acquire images captured by the vehicle-mounted camera.

[0055] Step S320: Convert the image to obtain a top view of the road surface.

[0056] For a detailed description of steps S310 to S320, please refer to steps S110 to S120, which will not be repeated here.

[0057] Step S330: Obtain a training sample set, which includes multiple sample road surface top views and road marking samples corresponding to the multiple sample road surface top views.

[0058] In this embodiment, when training the neural network model, a training sample set can be obtained first. The training sample set may include multiple sample road surface top views and road marking samples corresponding to the multiple sample road surface top views.

[0059] In some implementations, extensive data collection can be performed before acquiring the training sample set to obtain sufficient data for model training. This can involve collecting large amounts of data from databases. For example, in this scheme, which detects road markings in a top-down view of a road surface, a large number of images containing road surfaces can be found in the image database corresponding to intelligent vehicles as training data. Furthermore, if the dataset is not large enough, even more images can be collected, or multiple copies of the same image can be created to generate multiple variants, thereby obtaining a large number of additional images. To ensure the data quality in the dataset, the dataset can also be inspected to remove images of poor quality, thereby optimizing the model results.

[0060] After acquiring the dataset—that is, a large number of images containing road surfaces—the dataset can be labeled. Specifically, road markings in the images can be labeled, and the labels can be matched with the images. The dataset is then divided proportionally to obtain a training sample set and a test set for model training. The training sample set can include multiple sample road surface top views and the corresponding labeled road surface markings for each sample road surface top view.

[0061] In some implementations, step S330 may also include steps S331 to S333.

[0062] Step S331: Collect multiple sample image data.

[0063] In this embodiment, as Figure 5 The flowchart of the detection method shown illustrates that, when acquiring the training sample set, multiple sample image data can be collected first. As one implementation, multiple road surface images can be captured by the front-facing main camera and the left and right side triple-lens cameras of the test vehicle, and these captured road surface images can be used as multiple sample image data. As another implementation, the image database corresponding to the intelligent vehicle can include multiple road surface images captured by the front-facing main camera and the left and right side triple-lens cameras of different vehicles. A large number of road surface images captured by the front-facing main camera and the left and right side triple-lens cameras of different vehicles can be obtained from the image database, thereby enriching the dataset for model training.

[0064] Furthermore, road surface image data captured by the vehicle's front main camera and left and right side front triple cameras can be collected in the high-precision map coverage area, thereby ensuring image quality and optimizing model results.

[0065] Step S332: Convert the sample image data to obtain a top view of the sample road surface corresponding to multiple sample image data.

[0066] In this embodiment, the sample image data can be further processed to obtain a top view of the sample road surface corresponding to multiple sample image data.

[0067] In some implementations, when multiple road surface images are captured by the front main camera and the left and right front triple cameras of the test vehicle, the image attributes can be acquired simultaneously during image acquisition. The images are then transformed based on these attributes to obtain multiple top-down views of the road surface. Specifically, road surface images captured simultaneously by the front main camera, the left front camera, and the right front camera at the same time can be transformed to obtain a top-down view of the road surface at that moment. The image attributes can include the time and the camera that captured the image. For example, multiple road surface images are acquired by the front main camera and the left and right front triple cameras of the test vehicle. The attributes of the multiple road surface images are: Image 1 "17:00, front main camera", Image 2 "17:00, left front camera", Image 3 "17:00, right front camera", Image 4 "17:01, front main camera", Image 5 "17:01, left front camera", and Image 6 "17:01, right front camera". Images 1, 2, and 3 were captured at the same time, as were images 4, 5, and 6. Therefore, images 1, 2, and 3 can be converted and stitched together to obtain a top-down view of the road surface. Figure 1 Images 4, 5, and 6 are converted and stitched together to obtain a top-down view of the road surface. Figure 2 The method for converting images captured by the front-facing main camera and the left and right side front cameras into a top-down view of the road surface can be found in the above embodiments and will not be repeated here.

[0068] In other implementations, for road surface images captured by a large number of different vehicles' front main camera and left and right front triple cameras collected through the image database of intelligent vehicles, the image attributes can be acquired simultaneously when acquiring the images. The images can be transformed according to the image attributes to obtain multiple road surface top views. Specifically, road surface images captured simultaneously by the front main camera, left front, and right front cameras of the same vehicle at the same time can be transformed to obtain the road surface top view at that time. The image attributes may include vehicle model, time, and the camera that captured the image. For example, multiple road surface images are collected from an image database. The attributes of these images are as follows: Image 1 "Vehicle 1, 17:00, Front-facing main camera", Image 2 "Vehicle 1, 17:00, Left-facing front camera", Image 3 "Vehicle 1, 17:00, Right-facing front camera", Image 4 "Vehicle 2, 17:00, Front-facing main camera", Image 5 "Vehicle 2, 17:00, Left-facing front camera", Image 6 "Vehicle 2, 17:00, Right-facing front camera", Image 7 "Vehicle 1, 17:01, Front-facing main camera", Image 8 "Vehicle 1, 17:01, Left-facing front camera", and Image 9 "Vehicle 1, 17:01, Right-facing front camera". Images 1, 2, and 3 contain the same vehicle model and were captured at the same time; images 4, 5, and 6 contain the same vehicle model and were captured at the same time; and images 7, 8, and 9 contain the same vehicle model and were captured at the same time. Therefore, images 1, 2, and 3 can be converted and stitched together to obtain a top-down view of the road surface. Figure 1 Images 4, 5, and 6 are converted and stitched together to obtain a top-down view of the road surface. Figure 2 Images 7, 8, and 9 are converted and stitched together to obtain a top-down view of the road surface. Figure 3 .

[0069] Furthermore, for road surface image data captured by the vehicle's front main camera and left and right front triple cameras in the high-precision map coverage area, the road surface images captured simultaneously by the front main camera, left front, and right front cameras at the same time can be converted to obtain a top view of the road surface at that time.

[0070] Step S333: Draw points, lines, and boxes for the objects that need to be labeled in the top view of the sample road surface, and add attribute labels to obtain the road surface marking samples corresponding to the top view of the sample road surface.

[0071] In this embodiment, after obtaining multiple sample road surface top views, points, lines, and boxes can be drawn on the objects that need to be labeled to perform attribute annotation, thereby obtaining road marking samples corresponding to the multiple sample road surface top views.

[0072] Specifically, the objects to be labeled in the top view of the sample road surface can be road markings, which can be marked with points, lines, or frames depending on the type of road marking. For example, for road arrows, points can be drawn at the endpoints of the arrows for labeling, along with the arrow's attributes; for lane lines, lines can be drawn for lane lines, along with their attributes; for road text, frames can be drawn around the text, along with its attributes, etc., without further limitation. Specific labeling methods can correspond to different regulations. Furthermore, when labeling objects in the top view of the road surface, labeling can be done manually or by machine. Manual labeling improves accuracy, while machine labeling improves efficiency. Figure 5 As shown, after the annotation is completed, quality inspectors can also check the annotated content to further ensure the accuracy of the annotation and thus optimize the trained model.

[0073] Step S340: Input the training sample set into the backbone neural network and the multi-channel detection network for training to obtain a trained neural network model. The multi-channel detection network includes at least two detection models from keypoint detection, attribute recognition, frame length and width regression detection, and bidirectional vector field estimation. In this embodiment, as... Figure 6As shown, after obtaining the training sample set, it can be input into the backbone neural network and multi-channel detection network for training to obtain a trained neural model. During the model training phase, after obtaining the dataset, images and annotation information can be matched, and all labeled images can be divided into a training sample set and a test set according to a certain proportion. For example, 80% of the dataset can be used as the training sample set, and the remaining 20% ​​as the test set.

[0074] In some implementations, when the training sample set is obtained, the training data can be preprocessed. This preprocessing may include image scaling and label parsing. Specifically, the scaling ratio can be obtained first, and the images in the training data can be scaled proportionally. Then, the required pixel length for padding is calculated and the scaled image is padded to obtain the final image. Label parsing is then performed on the final image to obtain the preprocessed training data and labels. Simultaneously, data augmentation can be performed on the training data during training to increase the model's robustness and generalizability. For example, data augmentation can be based on grayscale, rotation, translation, noise, size, horizontal / vertical flipping, scaling, cropping, shearing, contrast, color jitter, etc.

[0075] Furthermore, preprocessed training set image data can be used to train deep neural networks on a cloud workstation cluster using GPUs. Specifically, a backbone network based on a deep residual network and a recognition network based on a multi-layer convolutional neural network can be employed. During the training of the neural network model, training is an iterative process. Each iteration involves a forward propagation of the network, obtaining the network output and loss function value. Then, stochastic gradient descent can be used to optimize based on the loss function, thereby obtaining the corresponding network parameter values ​​and ultimately the trained neural network model. The multi-channel detection network can include at least two detection networks from keypoint detection, attribute recognition, frame width and height regression detection, and bidirectional vector field estimation. By training each detection network, the loss function corresponding to each detection network can be obtained. The loss functions of each detection network are integrated to obtain the overall loss function, which is used to optimize the neural network model, resulting in the trained neural network model.

[0076] Furthermore, in some embodiments, step S340 may include steps S341 to S343.

[0077] Step S341: Establish a Gaussian kernel based on the center points of points, lines, and frames in the road marking sample.

[0078] In this embodiment, during the preprocessing stage of model training, a Gaussian kernel can be established based on the center of the point, line, or box object. Specifically, a second-order Gaussian kernel can be drawn in the image coordinate system with the key points of the point, line, or box as the center, so as to set the weight of each point when calculating the loss function, thereby balancing the ratio of positive samples to negative samples.

[0079] Step S342: Detect key points of the object based on a multi-channel detection network to obtain the key point detection results of the object.

[0080] In this embodiment, key points of an object can be detected based on a multi-channel detection network to obtain the key point detection results. The key points of an object can be the position of a point-type object, the center point of a frame-type object, or points uniformly sampled within a linear object.

[0081] Specifically, when detecting key points of an object, different detection results can be obtained depending on the object type. When the object is a point object, the key point of the point object is the point object itself, so the point object can be directly output as the detection result. When the object is a bounding box object, the key point of the bounding box object can be the center point of the bounding box object. When the object is a line object, the key points of the line object can be points uniformly sampled on the line object, that is, the line object can include multiple key points. By performing clustering based on Euclidean distance, the line segment detection result is obtained and output as the line segment detection result of the line object.

[0082] Step S343: Perform attribute recognition on the object based on the multi-channel detection network to obtain the attribute recognition result of the object.

[0083] In this embodiment, object attribute recognition can be performed based on a multi-channel detection network to obtain the object attribute recognition result. Attribute recognition can include recognizing the attributes of the object itself or recognizing the attributes of its surroundings. Recognizing the object's own attributes determines its approximate attributes, while recognizing the attributes of its surroundings allows for more precise identification of the object's attributes among objects with similar attributes, by considering the surrounding environment. For example, if the approximate attribute of an object is lane markings, recognizing the attributes of its surroundings determines whether the surrounding area is a road edge or the middle of the road, further identifying whether the lane markings are boundary lane markings or other types of lane markings.

[0084] Step S344: Perform frame length and width regression detection on the object based on the multi-channel detection network to obtain the frame length and width regression detection results of the object.

[0085] In this embodiment, furthermore, a multi-channel detection network can be used to perform frame length and width regression detection on the object to obtain the frame length and width regression detection of the object. Specifically, the frame length and width can be regressed for the frame-shaped object using key points as anchor points to achieve frame detection, and output the key points and dimensions of the frame-shaped object. Furthermore, the cross-entropy loss function can be used to classify the frame attributes to identify the frame-shaped object.

[0086] In some implementations, linear objects also include dashed linear objects, such as dashed lane lines, solid-dashed lines, and dashed-solid lines. Based on a bidirectional keypoint vector field, a deep neural network can learn the vector field information to estimate the connection relationships of corresponding detection points, and determine whether the linear object is a dashed line based on the connection relationships of the detection points. Specifically, the bidirectional keypoint vector field is obtained by estimating the direction of the next line segment for each pixel, which is represented as a vector field in the image. This vector can represent the connection relationship from the current point to the next connection point, thus obtaining a connection relationship diagram of the dashed lane line segments. Specifically, since solid lines are continuous, the connection relationship diagram can yield a single vector field, while dashed lines are discontinuous, therefore, the connection relationship diagram can yield multiple vector fields. Therefore, if the two-way vector field estimation result yields multiple vector fields, it indicates that the connection relationship of the linear object represented by the two-way vector field estimation result satisfies the dashed line connection relationship, and the linear object can be determined to be a dashed line; if the two-way vector field estimation result yields only one vector field, it indicates that the connection relationship of the linear object represented by the two-way vector field estimation result does not satisfy the dashed line connection relationship, and the linear object can be determined to be a solid line.

[0087] Step S345: Calculate the loss function based on the object detection results, attribute recognition results, frame length and width regression detection results, and Gaussian kernel.

[0088] In this embodiment, different networks generate certain loss functions during training. Therefore, the loss functions of multiple networks can be weighted and averaged, and the neural network can be optimized based on the integrated loss function to obtain a trained neural network model. Specifically, the loss function can be calculated based on the object detection results, attribute recognition results, frame length and width regression detection results, and Gaussian kernels.

[0089] In some implementations, the difference between the detected keypoint locations and their actual positions can be obtained from the keypoint detection results. This difference can be used to adjust the loss function, and then a deep neural network model can be trained based on the adjusted loss function to increase the model's robustness and accuracy. Furthermore, the loss function can be adjusted for point-type objects, bounding box-type objects, and line-type objects. Finally, the loss functions are integrated using a weighted average, and a deep neural network model is trained based on the integrated loss function.

[0090] Step S346: Optimize the multi-channel detection network using the loss function.

[0091] By weighted averaging and integrating all the aforementioned loss functions, a neural network is trained based on the integrated loss function to obtain a trained neural network model. In some implementations, after model training, the model can be deployed and inference can proceed. Specifically, the trained neural network model can be converted from OnNX to TensorRT format, and the format-converted neural network model can be deployed to the vehicle module, thereby serving as a functional module on the vehicle side and outputting the corresponding perception results.

[0092] During the inference phase of the trained neural network model, the vehicle-mounted camera can collect multi-camera data at a fixed frequency as the vehicle moves, namely the front main camera and the left and right front three-cameras. The top-down view is then converted at a preset frequency to obtain a road surface top-down view. For example, the vehicle-mounted camera can collect road surface images captured by the front main camera and the left and right front three-cameras at 20Hz, and convert the road surface images into a road surface top-down view at a frequency of 1Hz. The frequency can be set by the engineer or by the system; this is not limited here. Further, based on the deployed neural network model, a forward network propagation is performed on the current road surface top-down view, outputting structured detection results to the downstream module, which can output detection results for point objects, line objects, or box objects. Step S350: The road surface top-down view is input into the trained neural network model for key point detection to determine the number and location of key points in the road surface top-down view.

[0093] Step S360: Perform attribute recognition on the road surface top view based on the trained neural network model to obtain the attribute recognition results of the road surface top view.

[0094] Step S370: Perform frame length and width regression detection on the road surface top view based on the trained neural network model to obtain the frame length and width regression detection results of the road surface top view.

[0095] Step S380: Determine the road marking detection results in the road top view based on the number and location of key points in the road top view, the attribute recognition results, and the frame length and width regression detection results.

[0096] For a detailed description of steps S350-S380, please refer to steps S130-S160, which will not be repeated here.

[0097] The road marking detection method provided in the above embodiments is compared with Figure 1 The road marking detection method shown in this embodiment also obtains a training sample set, which includes multiple sample road surface top views and road marking samples corresponding to the multiple sample road surface top views respectively; the training sample set is input into a deep neural network model for training to obtain a trained neural network model, thereby obtaining a more accurate road marking detection result by training the deep neural network model and using the trained neural network model to perform key point detection on the road surface top view.

[0098] Please see Figure 7 , Figure 7 A block diagram of a road marking detection device 100 provided in an embodiment of the present invention is shown. The following will focus on... Figure 7 The block diagram illustrates that the road marking detection device 100 includes: an image acquisition module 110, an image conversion module 120, a key point detection module 130, an attribute recognition module 140, a regression detection module 150, and a result output module 160, wherein:

[0099] Image acquisition module 110 is used to acquire images captured by the vehicle-mounted camera.

[0100] Image conversion module 120 is used to convert images to obtain a top view of the road surface.

[0101] The key point detection module 130 is used to input the road surface top view into the trained neural network model to detect key points and determine the number and location of key points in the road surface top view.

[0102] The attribute recognition module 140 is used to perform attribute recognition on the road surface top view based on the trained neural network model, and obtain the attribute recognition result of the road surface top view.

[0103] The regression detection module 150 is used to perform frame length and width regression detection on the road surface top view based on the trained neural network model, and obtain the frame length and width regression detection results of the road surface top view.

[0104] The result output module 160 is used to determine the road marking detection results in the road top view based on the number and location of key points in the road top view, attribute recognition results, and frame length and width regression detection results.

[0105] Furthermore, the result output module 160 may include a vector field estimation submodule, a dashed line determination submodule, and a solid line determination submodule, wherein:

[0106] The vector field estimation submodule is used to perform bidirectional vector field estimation on the road surface top view based on the number and location of key points in the road surface top view, attribute recognition results, and the regression detection results of the frame length and width, if it is determined that the road surface top view includes linear objects. The result is obtained by estimating the direction of the next line segment for each pixel in the image. The vector is used to represent the connection relationship from the pixel to the next connection point.

[0107] The dashed line determination submodule is used to determine that a linear object is a dashed line if the connection relationship of the linear object represented by the bidirectional vector field estimation results satisfies the dashed line connection relationship.

[0108] Implement a determination submodule, which determines that if the connection relationship of the linear object represented by the bidirectional vector field estimation result does not satisfy the dashed line connection relationship, then the linear object is determined to be a solid line.

[0109] Furthermore, the road marking detection device 100 may also include a sample acquisition module and a model training module, wherein:

[0110] The sample acquisition module is used to acquire a training sample set, which includes multiple sample road surface top views and road marking samples corresponding to the multiple sample road surface top views.

[0111] Furthermore, the sample acquisition module includes a data acquisition submodule, a data conversion submodule, and an image annotation submodule, wherein:

[0112] The data acquisition submodule is used to acquire multiple sample image data.

[0113] The data conversion submodule is used to convert sample image data to obtain sample road surface top views corresponding to multiple sample image data.

[0114] The image annotation submodule is used to draw points, lines, and boxes for the objects that need to be annotated in the top view of the sample road surface, and to perform attribute annotations to obtain the road surface identification samples corresponding to the top view of the sample road surface.

[0115] The model training module is used to input the training sample set into the backbone neural network and the multi-channel detection network for training to obtain a trained neural network model. The multi-channel detection network includes at least two detection models among key point detection, attribute recognition, frame length and width regression detection, and bidirectional vector field estimation.

[0116] Furthermore, the model training module includes: a Gaussian kernel construction submodule, a keypoint detection submodule, an attribute recognition submodule, a regression detection submodule, a function computation submodule, and a network optimization submodule, wherein:

[0117] The Gaussian kernel establishment submodule is used to detect Gaussian kernels based on the center points of points, lines, and frames in road marking samples.

[0118] The key point detection submodule is used to detect key points of an object based on a multi-channel detection network and obtain the key point detection results of the object.

[0119] Furthermore, the keypoint detection submodule includes a point object detection unit, a box object detection unit, and a line object detection unit, wherein:

[0120] The point object detection unit is used to output the detection results of point objects when the object is a point object.

[0121] The box-shaped object detection unit is used to output the key points and dimensions of the box-shaped object when the object is a box-shaped object.

[0122] The linear object detection unit is used to output the line segment detection results of linear objects when the object is a linear object.

[0123] Furthermore, the key point detection submodule may also include: a connection relationship detection unit, a dashed line object detection unit, and a solid line object detection unit, wherein:

[0124] The connection relationship detection unit is used to estimate the direction of the next line segment for each pixel of the linear object to obtain a vector, and the vector is used to represent the connection relationship from the pixel to the next connection point.

[0125] The dashed line object detection unit is used to determine that a linear object is a dashed line when the connection relationship satisfies the dashed line segment connection relationship.

[0126] The solid line object detection unit is used to determine that a linear object is a solid line when the connection relationship does not satisfy the connection relationship of dashed line segments.

[0127] The attribute recognition submodule is used to perform attribute recognition on objects based on a multi-channel detection network, and obtain the attribute recognition results of the objects.

[0128] The regression detection submodule is used to perform frame length and width regression detection on objects based on a multi-channel detection network, and obtain the frame length and width regression detection results of the objects.

[0129] The function computation submodule is used to calculate the loss function based on the object detection results, attribute recognition results, frame length and width regression detection results, and Gaussian kernel.

[0130] The network optimization submodule is used to optimize multi-channel detection networks using a loss function.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0132] In several embodiments provided by the present invention, the coupling between modules can be electrical, mechanical or other forms of coupling.

[0133] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0134] Please see Figure 8 The diagram illustrates a structural block diagram of a vehicle 200 provided in an embodiment of the present invention. The vehicle 200 of the present invention may include one or more components such as a processor 210, a memory 230, and one or more application programs, wherein the one or more application programs may be stored in the memory 230 and configured to be executed by the one or more processors 210, and the one or more programs are configured to perform the methods described in the foregoing method embodiments.

[0135] The processor 210 may include one or more processing cores. The processor 210 connects to various parts of the vehicle 200 via various interfaces and lines, and performs various functions and processes data of the vehicle 200 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 230, and by calling data stored in the memory 230. Optionally, the processor 210 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 210 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 210 and may be implemented separately using a communication chip.

[0136] The memory 230 may include random access memory (RAM) or read-only memory (ROM). The memory 230 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 230 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the vehicle 200 (such as phonebook data, audio and video data, chat log data, etc.).

[0137] Please see Figure 9 This diagram illustrates a structural block diagram of a computer-readable storage medium according to an embodiment of the present invention. The computer-readable storage medium 300 stores program code, which can be invoked by a processor to execute the methods described in the above method embodiments.

[0138] The computer-readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.

[0139] In summary, the road marking detection method, device, vehicle, and storage medium provided in this embodiment of the invention acquire images captured by a vehicle-mounted camera; convert the images to obtain a road surface top view; input the road surface top view into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view; perform attribute recognition on the road surface top view based on the trained neural network model to obtain attribute recognition results; perform frame length and width regression detection on the road surface top view based on the trained neural network model to obtain frame length and width regression detection results; and determine the road marking detection results in the road surface top view based on the number and location of key points, attribute recognition results, and frame length and width regression detection results. Therefore, by acquiring a road surface top view from images captured by a vehicle-mounted camera and performing key point detection on the road surface top view based on a neural network model to determine the road marking detection results, this method can effectively cover various lane lines and road markings, including various types of points, lines, and frames, and provides more accurate detection results.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting road markings, characterized in that, The method includes: Acquire images captured by the vehicle's onboard camera; The image is converted to obtain a top view of the road surface; The road surface top view is input into a trained neural network model for key point detection to determine the number and location of key points in the road surface top view. The key points include at least one of the following: points uniformly sampled on linear objects, endpoints of road surface arrows, and center points of road surface text. The attributes of the road surface top view are identified based on the trained neural network model to obtain the attribute identification results of the road surface top view; The length and width of the frame of the road surface top view are regressed and detected based on the trained neural network model to obtain the length and width regression detection results of the road surface top view. Based on the number and location of key points in the road surface top view, the attribute recognition results and the frame length and width regression detection results determine the road marking detection results in the road surface top view.

2. The method according to claim 1, characterized in that, The determination of road marking detection results in the road surface top view based on the number and location of key points in the road surface top view, the attribute recognition results, and the frame length and width regression detection results includes: If, based on the number and location of key points in the road surface top view, the attribute recognition result, and the frame length and width regression detection result, it is determined that the road surface top view includes a linear object, then a two-way vector field estimation is performed on the road surface top view according to the trained neural network model to obtain the two-way vector field estimation result of the linear object. The two-way vector field estimation is obtained by estimating the direction of the next line segment for each pixel in the image to obtain a vector, and the vector is used to represent the connection relationship from the pixel to the next connection point. If the bidirectional vector field estimation result indicates that the connection relationship of the linear object satisfies the dashed line connection relationship, then the linear object is determined to be a dashed line; If the bidirectional vector field estimation result indicates that the connection relationship of the linear object does not satisfy the dashed line connection relationship, then the linear object is determined to be a solid line.

3. The method according to claim 1, characterized in that, Before inputting the road surface top view into the trained neural network model for key point detection and obtaining the road surface marking detection result output by the trained neural network model, the method further includes: Obtain a training sample set, which includes multiple sample road surface top views and road marking samples corresponding to the multiple sample road surface top views; The training sample set is input into the backbone neural network and the multi-channel detection network for training to obtain a trained neural network model. The multi-channel detection network includes at least two detection models among key point detection, attribute recognition, frame length and width regression detection, and bidirectional vector field estimation.

4. The method according to claim 3, characterized in that, The acquisition of the training sample set includes: Collect image data from multiple samples; The multiple sample image data are converted to obtain a sample road surface top view corresponding to the multiple sample image data; Points, lines, and boxes are drawn for the objects that need to be labeled in the top view of the sample road surface, and attribute labels are added to obtain the road surface identification samples corresponding to the top view of the sample road surface.

5. The method according to claim 4, characterized in that, The step of inputting the training sample set into the backbone neural network and the multi-channel detection network for training includes: A Gaussian kernel is established based on the center points of the points, lines, and frames in the road marking sample; The key points of the object are detected based on the multi-channel detection network to obtain the key point detection results of the object; The object's attributes are identified based on the multi-channel detection network to obtain the object's attribute identification results; The object's frame length and width regression detection is performed based on the multi-channel detection network to obtain the object's frame length and width regression detection results; The loss function is calculated based on the object detection results, the attribute recognition results, the frame length and width regression detection results, and the Gaussian kernel. The multi-channel detection network is optimized using the loss function.

6. The method according to claim 5, characterized in that, The key points of the object are detected based on the multi-channel detection network to obtain the key point detection results of the object, including: When the object is a point object, output the detection result of the point object; When the object is a frame-shaped object, output the key points of the frame-shaped object and the dimensions of the frame-shaped object; When the object is a linear object, output the line segment detection result of the linear object.

7. The method according to claim 6, characterized in that, When the object is a linear object, after outputting the line segment detection result of the linear object, the method further includes: For each pixel of the linear object, the direction of the next line segment is estimated to obtain a vector, which is used to represent the connection relationship from the pixel to the next connection point; When the connection relationship satisfies the connection relationship of dashed line segments, the linear object is determined to be a dashed line; When the connection relationship does not satisfy the connection relationship of dashed line segments, the linear object is determined to be a solid line.

8. A road marking detection device, characterized in that, The device includes: The image acquisition module is used to acquire images captured by the vehicle-mounted camera. An image conversion module is used to convert the image to obtain a top view of the road surface; The key point detection module is used to input the road surface top view into a trained neural network model to detect key points and determine the number and location of key points in the road surface top view. The key points include at least one of the following: points uniformly sampled on linear objects, endpoints of road surface arrows, and center points of road surface text. The attribute recognition module is used to perform attribute recognition on the road surface top view based on the trained neural network model, and obtain the attribute recognition result of the road surface top view; The regression detection module is used to perform frame length and width regression detection on the road surface top view based on the trained neural network model, and obtain the frame length and width regression detection results of the road surface top view; The result output module is used to determine the road marking detection result in the road surface top view based on the number and location of key points in the road surface top view, the attribute recognition result, and the frame length and width regression detection result.

9. A vehicle, characterized in that, The vehicle includes a camera, which is mounted on the exterior of the vehicle body. The vehicle also includes: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.

Images