Road marking detection methods and devices, and methods for training marking recognition networks.

Abstract

This disclosure provides a method and apparatus for detecting road markings, a method and apparatus for training a marking recognition network, a storage medium, and an electronic device. The method includes: acquiring a road image and a top view of the road image; using a marking recognition network to detect road markings on the road image and the top view, obtaining a first detection result and a second detection result; and determining the detection result of the road markings based on the first and second detection results. Therefore, by detecting road markings on the road image and its top view, and by cross-checking and supplementing the first detection result of the road marking in the road image and the second detection result in the top view, the detection result of the road markings is determined. This solves the problems of missed or false detections of road markings caused by deformation or occlusion of road markings in the road image or top view, thus improving the accuracy of the road marking detection results.

Description

Technical Field

[0001] This disclosure relates to autonomous driving technology, and more particularly to a road marking detection method and apparatus, a marking recognition network training method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] Road markings on the road are designed to assist drivers, such as speed limit signs and straight-ahead signs. Accurate identification of these road markings is crucial for safe driving and route planning in assisted or autonomous driving systems.

[0003] During driving, road markings are typically identified using images captured by the vehicle's own camera. However, due to variations in shooting angle and distance, road markings in the camera-captured images are subject to significant distortion. Furthermore, road markings are often obscured by other objects, resulting in poor accuracy when using camera-captured images for road marking detection. Summary of the Invention

[0004] In existing road marking detection, the accuracy of road marking detection results is poor due to the deformation of road markings in road images and the obstruction of road markings by other objects.

[0005] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a road marking detection method and apparatus, a marking recognition network training method and apparatus, a storage medium, and an electronic device.

[0006] According to one aspect of the present disclosure, a road marking detection method is provided, comprising: acquiring a road image and a top view of the road image; performing road marking detection on the road image using a marking recognition network to obtain a first detection result; wherein the first detection result includes a first detection box and a first classification confidence level of the road marking; performing road marking detection on the top view using the marking recognition network to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence level of the road marking; and determining a detection result of the road marking based on the first detection result and the second detection result, wherein the detection result of the road marking includes a third detection box and a third classification confidence level of the road marking.

[0007] According to another aspect of the present disclosure, a method for training a sign recognition network is provided, comprising: based on a neural network to be trained, using each sample road surface image in a sample road surface image set, obtaining a first predicted detection box and a first classification confidence prediction value for road signs in each sample road surface image; wherein, the sample road surface image includes road signs, and the sample road surface image is labeled with the first detection box and the first classification confidence ground truth value of the road signs; based on the neural network to be trained, using each sample top view in a sample top view image set, obtaining a second predicted detection box and a second classification confidence prediction value for road signs in each sample top view; wherein, the sample top view includes road signs, and the sample top view is labeled with the second detection box and the second classification confidence ground truth value of the road signs; training the neural network to be trained based on a first loss between the first classification confidence prediction value and the first classification confidence ground truth value of the road signs in each sample road surface image, and a second loss between the second classification confidence prediction value and the second classification confidence ground truth value of the road signs in each sample top view, until a preset training condition is met, thereby obtaining a sign recognition network.

[0008] According to another aspect of the present disclosure, a road marking detection device is provided, comprising: an acquisition module for acquiring a road image and a top view of the road image; a first detection module for performing road marking detection on the road image using a marking recognition network to obtain a first detection result; wherein the first detection result includes a first detection box and a first classification confidence level for the road marking; a second detection module for performing road marking detection on the top view using the marking recognition network to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence level for the road marking; and a determination module for determining a detection result for the road marking based on the first detection result and the second detection result, wherein the detection result for the road marking includes a third detection box and a third classification confidence level for the road marking.

[0009] According to another aspect of the present disclosure, a training apparatus for an identification recognition network is provided, comprising: a first acquisition module, configured to obtain, based on a neural network to be trained, a first predicted detection box and a first classification confidence prediction value for a road sign in each of the sample road surface images in a sample road surface image set; wherein, the sample road surface image includes a road sign, and the sample road surface image is labeled with the first detection box and the first classification confidence ground truth value of the road sign; a second acquisition module, configured to obtain, based on the neural network to be trained, a second predicted detection box and a second classification confidence prediction value for a road sign in each of the sample top-view images in a sample top-view image set; wherein, the sample top-view includes a road sign, and the sample top-view is labeled with the second detection box and the second classification confidence ground truth value of the road sign; and a training module, configured to train the neural network to be trained based on a first loss between the first classification confidence prediction value and the first classification confidence ground truth value of the road sign in each of the sample road surface images, and a second loss between the second classification confidence prediction value and the second classification confidence ground truth value of the road sign in each of the sample top-view images, until a preset training condition is met, thereby obtaining an identification recognition network.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described above.

[0011] According to one aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described above.

[0012] Based on the road marking detection method and apparatus, marking recognition network training method and apparatus, storage medium, and electronic device provided in the above embodiments of this disclosure, a road image and a top view of the road image are first acquired; then, the marking recognition network is used to detect road markings on the road image and the top view, obtaining a first detection result and a second detection result; subsequently, based on the first and second detection results, the detection result of the road marking is determined. Therefore, in the embodiments of this disclosure, by performing road marking detection on the road image and the top view of the road image, and by mutually verifying and supplementing the first detection result of the road marking in the road image and the second detection result in the top view, the detection result of the road marking is determined, improving the accuracy of the road marking detection result. This solves the problems in the prior art where road markings are missed or falsely detected due to deformation of the road markings in the road image or top view, or when the road markings are obscured by other objects. Furthermore, because the embodiments of this disclosure effectively improve the accuracy of the road marking detection result, reliable reference data is provided for subsequent route planning based on road markings, assisting drivers in judging the driving direction of their own vehicle or other vehicles, and establishing high-precision maps.

[0013] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0014] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 This is a scenario diagram to which the embodiments of this disclosure apply;

[0016] Figure 2 This is a schematic flowchart of a road marking detection method provided in an exemplary embodiment of this disclosure;

[0017] Figure 3 This is a flowchart illustrating step S204 provided in an exemplary embodiment of this disclosure;

[0018] Figure 4 This is a flowchart illustrating step S2042 provided in an exemplary embodiment of this disclosure;

[0019] Figure 5 This is a flowchart illustrating step S2042 provided in another exemplary embodiment of this disclosure;

[0020] Figure 6This is a flowchart illustrating step S2042 provided in yet another exemplary embodiment of this disclosure;

[0021] Figure 7 This is a schematic flowchart of a road marking detection method provided in another exemplary embodiment of this disclosure;

[0022] Figure 8 This is a schematic flowchart of a road marking detection method provided in yet another exemplary embodiment of this disclosure;

[0023] Figure 9 This is a flowchart illustrating step S202 provided in an exemplary embodiment of this disclosure;

[0024] Figure 10 This is a flowchart illustrating step S203 provided in an exemplary embodiment of this disclosure;

[0025] Figure 11 This is a schematic diagram of the structure of an identifier recognition network provided in an exemplary embodiment of this disclosure;

[0026] Figure 12 This is an overall flowchart of a road marking detection method provided in an exemplary embodiment of this disclosure;

[0027] Figure 13 This is a flowchart illustrating a training method for an identifier recognition network provided in an exemplary embodiment of this disclosure;

[0028] Figure 14 This is a flowchart illustrating a training method for an identifier recognition network provided in another exemplary embodiment of this disclosure;

[0029] Figure 15 This is a schematic diagram of the structure of a road marking detection device provided in an exemplary embodiment of this disclosure;

[0030] Figure 16 This is a schematic diagram of the structure of a road marking detection device provided in another exemplary embodiment of this disclosure;

[0031] Figure 17 This is a schematic diagram of the structure of a training apparatus for an identifier recognition network provided in an exemplary embodiment of this disclosure;

[0032] Figure 18 This is a schematic diagram of the structure of a training apparatus for an identifier recognition network provided in another exemplary embodiment of this disclosure;

[0033] Figure 19 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0034] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0035] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0036] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0037] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0038] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0039] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0040] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0041] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0042] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0043] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0044] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0045] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0046] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, standard programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0047] Application Overview

[0048] In the process of realizing this disclosure, the inventors discovered that during driving, the road markings in the road images captured by the vehicle's own camera are subject to significant deformation due to changes in the shooting angle and shooting distance. In addition, since the road markings on the road surface are often obscured by other objects, the accuracy of the road marking detection results when using road images captured by the camera is poor.

[0049] Exemplary System

[0050] The technical solution disclosed herein can be applied to vehicles with assisted driving or autonomous driving functions. It can assist in planning driving routes, determining the driving direction of the vehicle or other vehicles, and creating high-precision maps. For example, by determining road markings, the vehicle can plan its route based on these markings, thereby achieving assisted or automatic control of vehicle driving.

[0051] Figure 1 This is an exemplary application scenario of the road marking detection method provided in this disclosure. In this disclosure, as... Figure 1As shown, the vehicle is equipped with an image acquisition device. This device is communicatively connected to a computing platform. The image acquisition device captures road surface images and transmits them to the computing platform. The computing platform performs image conversion on the road surface images to obtain a top-down view. Then, it performs road marking detection on both the road surface images and the top-down view. For example, it identifies road markings such as straight arrows, U-turn arrows, and stop signs in both the road surface images and the top-down view, obtaining a first detection result and a second detection result. Based on the first and second detection results, it determines the detection result of the road marking and transmits this result to the vehicle's control platform. The image acquisition device can be a monocular camera, a binocular camera, etc.

[0052] In this embodiment, road marking detection is performed on a road surface image and a top view of the road surface image. The detection result of the road marking is determined by cross-checking and supplementing the first detection result of the road marking in the road surface image and the second detection result in the top view. This solves the problems of missed or false detection of road markings caused by deformation of road markings in the road surface image or top view, or by road markings being obscured by other objects, thus improving the accuracy of road marking detection results.

[0053] Exemplary methods

[0054] Figure 2 This is a schematic flowchart of a road marking detection method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, vehicles, etc., such as... Figure 2 As shown, it includes the following steps:

[0055] Step S201: Obtain a road surface image and a top view of the road surface image.

[0056] The road surface image can be acquired using an image acquisition device installed on the vehicle. The acquired road surface image is then transformed to obtain a top view of the road surface. For example, inverse perspective transformation or similar methods can be used to transform the road surface image to obtain the top view. Alternatively, a pre-defined correspondence between range distance values ​​and pixel sizes can be established. The range distance value represents the distance between the road markings and the image acquisition device in the real-world environment. Based on the required range distance value and the correspondence between range distance value and pixel size, the pixel size corresponding to the required range distance value is determined. The top view is then scaled so that the pixel size of the top view matches the pixel size corresponding to the required range distance value, thus obtaining a top view of the required range distance value. This top view of the required range distance value can be used as the top view corresponding to the road surface image.

[0057] Step S202: Use the sign recognition network to detect road signs in the road image and obtain the first detection result.

[0058] The first detection result includes the first detection frame and the first classification confidence level of the road markings.

[0059] The classification confidence score of road markings is used to characterize the probability that a road marking belongs to that category. The detection result obtained by passing the road image through the marking recognition network is called the first detection result. The detection box of the road marking in the road image obtained by passing the road image through the marking recognition network is called the first detection box of the road marking, and the classification confidence score of the road marking in the road image obtained by passing the road image through the marking recognition network is called the first classification confidence score of the road marking.

[0060] Road markings refer to signs on the road surface used to guide, restrict, or warn vehicles. Examples include left-turn arrows, right-turn arrows, straight-ahead arrows, speed reduction signs, lane dividers, stop lines, speed limit signs, bus lane signs, and no-stopping signs. The sign recognition network can be a pre-trained neural network for detecting road markings. This network can be a single neural network or a composite neural network consisting of multiple neural networks. For example, sign recognition networks can be CNN (Convolutional Neural Networks), SCNN (Spatial Convolutional Neural Networks), RNN (Recurrent Neural Network), or LSTM (Long Short-Term Memory).

[0061] It should be noted that a road surface image may include one or more road surface markings. When a road surface image includes multiple road surface markings, a marking recognition network is used to determine the first detection result for each marking. For example, when a road surface image includes multiple road surface markings, such as directional markings (left turn arrows, right turn arrows, or straight arrows), speed reduction markings, lane divider markings, stop line markings, speed limit markings, bus lane markings, no-stopping markings, etc., the markings in the road surface image are identified separately, and the first detection result for each marking is determined.

[0062] Step S203: Using a sign recognition network, road markings are detected on the top view to obtain a second detection result.

[0063] The second detection result includes the second detection bounding box and the second classification confidence score of the road markings. The detection result of the road markings in the top view obtained through the sign recognition network is determined as the second detection result. The detection bounding box of the road markings in the top view obtained through the sign recognition network is determined as the second detection bounding box of the road marking, and the classification confidence score of the road markings in the top view obtained through the sign recognition network is determined as the second classification confidence score of the road marking.

[0064] It should be noted that the top view may include one road marking or multiple road markings. When the top view includes multiple road markings, a second detection result for each road marking is determined by a marking recognition network.

[0065] Step S204: Based on the first and second detection results, determine the detection results of the road markings.

[0066] The detection results of the road marking include the third detection frame and the third classification confidence level of the road marking. The detection frame of the road marking obtained based on the first and second detection results is called the third detection frame of the road marking, and the classification confidence level of the road marking obtained based on the first and second detection results is called the third classification confidence level of the road marking.

[0067] The detection result of the road marking can be determined based on the first and second category confidence levels, as well as the first and second detection frames.

[0068] For example, the first and second classification confidence levels of the road marking can be compared, and the classification confidence level with the highest level can be determined as the third classification confidence level of the road marking. At the same time, the detection box in the detection result to which the classification confidence level with the highest level belongs can be determined as the third detection box of the road marking.

[0069] Alternatively, thresholds can be preset, and the first and second category confidence scores of the road markings can be compared with the preset thresholds respectively. The category confidence scores that are greater than or equal to the preset thresholds are determined as the third category confidence scores of the road markings, and the detection boxes in the detection results to which the category confidence scores belong are determined as the detection boxes of the road markings. When both the first and second category confidence scores are greater than or equal to the preset thresholds, the maximum of the first and second category confidence scores of the road markings is determined as the third category confidence scores of the road markings, and the detection boxes in the detection results to which the maximum category confidence scores belong are determined as the third detection boxes of the road markings. When both the first and second category confidence scores of the road markings are less than the preset thresholds, it is determined that the road markings have not been detected.

[0070] Alternatively, the average of the first and second category confidence scores of the road markings can be used as the third category confidence score of the road markings, and the detection box in the detection result belonging to the highest category confidence score between the first and second category confidence scores can be determined as the third detection box.

[0071] It should be noted that when a road surface image or top view includes multiple road surface markings, the detection result of each road surface marking needs to be determined based on the first and second detection results of each marking.

[0072] In this embodiment, road marking detection is performed on a road surface image and its top-view. The detection results are then cross-checked and supplemented by the first detection result of the road marking in the road surface image and the second detection result in the top-view, thus improving the accuracy of the road marking detection. This solves the problems of missed or false detections of road markings caused by deformation or occlusion of road markings in the road surface image or top-view in the prior art. Furthermore, because this embodiment effectively improves the accuracy of road marking detection results, it provides reliable reference data for subsequent route planning based on road markings, assisting drivers in determining the direction of their own vehicle or other vehicles, and building high-precision maps.

[0073] In one alternative embodiment, such as Figure 3 As shown, step S204 may further include:

[0074] Step S2401: Map the second detection box onto the road surface image to obtain the first mapping box.

[0075] The mapping box that maps the second detection box of the road marking onto the road image is called the first mapping box of the road marking.

[0076] For example, the coordinate values ​​of the four vertices of the second detection box of the road surface image can be obtained. Based on the coordinate values ​​of the four vertices of the second detection box, the four vertices of the second detection box can be mapped to the road surface image through inverse perspective transformation or other methods. Based on the four mapped vertices on the road surface image, the first mapping box of the road surface image can be determined.

[0077] Alternatively, the coordinates of the center point and two vertices of the second detection box of the road marking can be obtained, where the two vertices are located on the same side or on the same diagonal. Based on the coordinates of the two vertices and the center point of the second detection box, the two vertices and the center point of the second detection box are mapped to the road image using methods such as inverse perspective transformation. Based on the two mapped vertices and the center point on the road image, the first mapping box of the road image is determined.

[0078] Step S2402: Match the first mapping box with the first detection box, and determine the detection result of the road markings based on the matching result.

[0079] Specifically, the matching result of the first detection frame and the first mapping frame of the road marking can be determined based on the area intersection-union ratio or the perimeter intersection-union ratio. The detection result of the road marking is then determined based on the matching result of the first detection frame and the first mapping frame.

[0080] In this embodiment of the disclosure, by mapping the second detection box of the road marking to the road image, a first mapping box of the road marking is formed on the road image. This effectively combines the first and second detection results of the road marking in the road image and the top view, thereby making the detection result of the road marking determined based on the matching result of the first detection box and the first mapping box of the road marking more accurate.

[0081] In one alternative embodiment, such as Figure 4 As shown, step S2402 may include:

[0082] Step S301: Obtain the intersection-union ratio between the first mapping box and the first detection box.

[0083] Intersection-over-Union (IoU) is a concept used in object detection, representing the overlap rate between the generated candidate bounding boxes and the original labeled bounding boxes. In this embodiment, the IoU between the first mapped bounding box and the first detected bounding box is the overlap rate between the first mapped bounding box and the first detected bounding box.

[0084] The intersection-union ratio (IUGR) between the first mapped frame and the first detection frame can be either the area IUGR or the perimeter IUGR. This IUGR can be determined based on the coordinates of the four vertices of the first mapped frame and the four vertices of the first detection frame of the road marking. Alternatively, it can be determined based on the coordinates of two vertices and the center point of the first detection frame of the road marking, as well as the coordinates of the two vertices and the center point of the first mapped frame.

[0085] Step S302: In response to the crossover ratio being greater than or equal to the first threshold, the first detection result or the second detection result is determined as the detection result of the road marking.

[0086] A first threshold can be preset. The intersection-union ratio (IUGR) between the first mapped frame and the first detection frame of the road marking is compared with the first threshold. When the IUGR between the first mapped frame and the first detection frame of the road marking is greater than or equal to the first threshold, the first detection result or the second detection result of the road marking is determined as the result of the road marking. Alternatively, the first classification confidence score and the second classification confidence score of the road marking are compared, and the detection result with the highest classification confidence score is determined as the inspection result of the road marking.

[0087] In this embodiment of the disclosure, when the cross-union ratio between the first detection box and the first mapping box of the road marking is greater than or equal to the first threshold, the first detection result or the second detection result of the road marking is determined as the detection result of the road marking. This realizes the correction and supplementation of the detection results of the road marking in the top view and the road image by the cross-union ratio between the first detection box and the first mapping box, thereby improving the accuracy of the detection results of the road marking.

[0088] In one alternative embodiment, such as Figure 4 As shown, step S2402 may include:

[0089] Step S303: In response to the cross-union ratio being less than a first threshold and greater than a second threshold, the detection result of the road marking is determined based on the first classification confidence and the second classification confidence.

[0090] A second threshold can be preset, and the first threshold is greater than the second threshold. When the intersection-union ratio (IUR) between the first mapped frame and the first detection frame of the road marking is less than the first threshold, the IUR is compared with the second threshold. When the IUR is less than the first threshold but greater than the second threshold, the first classification confidence score and the second classification confidence score of the road marking are compared, and the detection result of the road marking is determined based on the comparison result.

[0091] For example, the detection result corresponding to the largest value between the first and second category confidence scores of the road marking can be determined as the detection result of the road marking. For instance, when the intersection-union ratio between the first bounding box and the first detection box of road marking A is less than a first threshold and greater than a second threshold, the first category confidence score of road marking A is 85% and the second category confidence score is 55%, then the first detection result of road marking A is determined as the detection result of road marking A.

[0092] In this embodiment of the disclosure, when the intersection-union ratio between the first detection box and the first mapping box of the road marking is less than a first threshold and greater than a second threshold, the detection result of the road marking is determined by the first classification confidence and the second classification confidence of the road marking, which improves the accuracy of the detection result of the road marking and effectively reduces the situation of missed detection and false detection of road marking caused by detecting road markings only by a single top view or road image.

[0093] In one alternative embodiment, such as Figure 5 As shown, step S2042 may further include:

[0094] Step S304: In response to the intersection-union ratio being less than or equal to the second threshold, determine whether road markings are detected in the road surface image and top view.

[0095] Specifically, when the intersection-union ratio between the first detection box and the first mapping box of the road marking is less than or equal to the second threshold, it is determined whether the road marking is detected in the road image and the top view, respectively.

[0096] For example, the detection status of a road sign in the top view and the road image can be determined by the first and second classification confidence levels of the road sign. For instance, if the road sign has a first classification confidence level but no second classification confidence level, it is determined that the road sign is not detected in the top view; if the road sign has a second classification confidence level but no first classification confidence level, it is determined that the road sign is not detected in the road image. Alternatively, a classification confidence threshold can be preset, and the first and second classification confidence levels of the road sign can be compared with the preset threshold. If the first classification confidence level of the road sign is less than or equal to the preset threshold, it is determined that the road sign is not detected in the road image; if the second classification confidence level of the road sign is less than or equal to the preset threshold, it is determined that the road image is not detected in the top view.

[0097] In step S305, in response to the detection of road markings in the road surface image and the absence of road markings in the top view, the first detection box is mapped to the top view to obtain the second mapping box.

[0098] The first detection box of the road marking is mapped onto the top view and called the second mapping box of the road marking.

[0099] The system can obtain the coordinates of the four vertices of the first detection box of a road marking. Based on these coordinates, the system uses inverse perspective transformation and other methods to map the four vertices of the first detection box onto a top view. The second mapping box of the road marking is then determined based on the mapped vertices in the top view. Alternatively, the system can obtain the coordinates of the center point and two vertices located on the same side or diagonally of the first detection box of the road image. Based on these coordinates, the system uses inverse perspective transformation and other methods to map the two vertices and the center point of the first detection box onto a top view. The second mapping box of the road image is then determined based on the mapped vertices and the center point in the top view.

[0100] Step S306: Match the first image feature corresponding to the second mapping box with the image feature corresponding to at least one road surface mark in the top view.

[0101] In this context, the image features of the region corresponding to the second bounding box are referred to as the first image features corresponding to the second bounding box. The region corresponding to the second bounding box can be cropped from the top-down view. The image features of the cropped region corresponding to the second bounding box can be extracted using a pre-trained neural network for image feature extraction, yielding the first image features corresponding to the second bounding box. Then, the image features of all road markings included in the top-down view can be extracted using the same pre-trained neural network, yielding the image features of each road marking included in the top-down view. Alternatively, a marking recognition network can be used to determine the first image features corresponding to the second bounding box, as well as the image features of all road markings in the top-down view. Finally, the first image features corresponding to the second bounding box are matched with the image features of each road marking in the top-down view.

[0102] Step S307: In response to the presence of image features in the top view that match the first image features of the road markings, the first detection result is determined as the detection result of the road markings.

[0103] Specifically, when there is an image feature of a road surface sign in the top view that matches the first image feature corresponding to the second mapping box, the first detection result of the road surface sign is determined as the detection result of the road surface sign.

[0104] Step S308: In response to the absence of image features in the top view that match the first image features of the road marking, it is determined that the road marking has not been detected.

[0105] Specifically, if there is no image feature in the top view that matches the first image feature corresponding to the second mapping box, it is determined that the road marking has not been detected in either the road image or the top view.

[0106] In this embodiment of the disclosure, when a road marking is detected in the road surface image but not in the top view, a second mapping box is obtained by mapping the first detection box of the road marking to the top view. The image features corresponding to the second mapping box are matched with the image features of at least one road marking in the top view. The detection result of the road marking is determined based on the matching result. This realizes the mutual verification of the detection result of the road marking in the road surface image by the detection result of the road marking in the top view, effectively avoiding the situation of missed detection and false detection of road markings when only the road surface image is used to detect road markings.

[0107] In one alternative embodiment, such as Figure 6 As shown, step S2042 may further include:

[0108] Step S309: In response to the detection of road markings in the top view and the absence of road markings in the road image, the second image feature corresponding to the first mapping box is matched with the image features of at least one road marking in the road image.

[0109] In this context, the image features of the region corresponding to the first mapping box are referred to as the second image features corresponding to the first mapping box. When the intersection-union ratio (IU) between the first detection box and the first mapping box of the road sign is less than or equal to a second threshold, it is determined whether the road sign is detected in both the road image and the top view. When the road sign is detected in the top view but not in the road image, the region corresponding to the first mapping box of the road sign is cropped from the road image. The image features of the cropped region corresponding to the first mapping box can be extracted using a pre-trained neural network for image feature extraction to obtain the second image features corresponding to the first mapping box; the image features of all road signs in the road image can be extracted using a pre-trained neural network for image feature extraction. Alternatively, a sign recognition network can be used to determine the second image features corresponding to the first mapping box and the image features of all road signs in the road image. The second image features corresponding to the first mapping box are then matched with the image features of all road signs in the road image.

[0110] Step S310: In response to the presence of image features in the road surface image that match the second image features of the road surface marker, the second detection result is determined as the detection result of the road surface marker.

[0111] Specifically, when there is an image feature of a road surface sign in the road surface image that matches the second image feature corresponding to the first mapping box, the second detection result of the road surface sign is determined as the detection result of the road surface sign.

[0112] Step S311: In response to the absence of image features in the road surface image that match the second image features of the road surface sign, it is determined that the road surface sign has not been detected.

[0113] Specifically, if there is no image feature in the road surface image that matches the second image feature corresponding to the first mapping box, it is determined that the road surface marker is not detected in either the road surface image or the top view.

[0114] In this embodiment of the disclosure, when a road marking is detected in the top view but not in the road image, the second image feature corresponding to the first mapping box is determined, and the second image feature is matched with the image features of all road markings in the road image. The road marking detection result is determined based on the matching result. This realizes mutual verification between the detection results of the road markings in the road image and the detection results of the road markings in the top view, effectively avoiding the situation of missed detection and false detection of road markings when only the top view is used to detect road markings.

[0115] In one alternative embodiment, such as Figure 7 As shown, the road marking detection method also includes:

[0116] Step S401: In response to the intersection-union ratio being less than a first threshold, obtain the position information of lane lines and road signs in the road surface image.

[0117] Specifically, when the intersection-over-union (IoU) ratio between the first mapping box and the first detection box of the road marking is less than a first threshold, the location information of the road marking and the location information of the lane lines in the road image are obtained. The location information of the road marking may include the coordinates of the four vertices and / or the coordinates of the center point of the first detection box of the road marking. The road image can be identified by a pre-trained neural network for lane line recognition to determine the detection boxes of the lane lines; the location information of the lane lines in the road image may include the coordinates of the four vertices and / or the coordinates of the center point of the detection box of the lane lines.

[0118] Step S402: In response to determining that the lane line and the road sign overlap in the road image based on the position information of the lane line and the position information of the road sign in the road image, the road sign is filtered out.

[0119] Specifically, in the road surface image, based on the location information of the road surface marker and the lane line, it is determined whether the road surface marker and the lane line overlap in the road surface image. If the road surface marker and the lane line overlap in the road surface image, the road surface marker is filtered out.

[0120] For example, a third threshold can be preset. The area intersection-union ratio (IUU) or perimeter intersection-union ratio (PCU) between the four vertices of the lane line detection box and the first detection box of the road sign in the road image can be determined using the coordinates of the four vertices of the lane line detection box and the first detection box of the road sign. When the IUU or PCU between the lane line detection box and the first detection box of the road sign is greater than or equal to the preset third threshold, it is determined that the road sign in the road image coincides with the lane line. In this case, the road sign in the road image is filtered out. When the road sign coincides with the lane line, it indicates that the lane line in the road image has been mistakenly detected as a road sign. Therefore, when the road sign in the road image coincides with the lane line, the road sign is filtered out, and subsequent road sign detection steps are not performed on it. When the IUU or PCU between the lane line detection box and the first detection box of the road sign is less than the preset third threshold, it is determined that the road sign in the road image does not coincide with the lane line. In this case, subsequent road sign detection steps are performed on the road sign.

[0121] A fourth threshold can also be preset. By using the coordinates of the center point of the lane line detection box in the road image and the coordinates of the center point of the first detection box of the road sign, the distance between the center points of the lane line detection box and the first detection box of the road sign can be determined. This distance can be Euclidean distance, etc. When the distance between the center points of the lane line detection box and the first detection box of the road sign is less than or equal to the preset fourth threshold, it is determined that the road sign in the road image coincides with the lane line, and the road sign in the road image is filtered out. When the distance between the center points of the lane line detection box and the first detection box of the road sign is greater than the preset fourth threshold, it is determined that the road sign in the road image does not coincide with the lane line, and the subsequent detection steps for the road sign continue.

[0122] In this embodiment of the disclosure, by using the position information of lane lines and road markings in the road image, road markings that overlap with lane lines in the road image are filtered out, thus solving the problem of misdetecting lane lines as road markings and effectively improving the accuracy of road marking detection results.

[0123] In one alternative embodiment, such as Figure 8 As shown, the road marking detection method also includes:

[0124] Step S403: In response to the intersection-to-merge ratio being less than the first threshold, obtain the position information of the lane lines and the position information of the road signs in the top view.

[0125] Specifically, when the intersection-over-union ratio (IoU) between the first mapping box and the first detection box of the road marking is less than a first threshold, the position information of the road marking and the position information of the lane lines in the top view are obtained. The position information of the road marking includes the coordinate values ​​of the four vertices and / or the coordinate value of the center point of the second detection box of the road marking; the top view can be identified by a pre-trained neural network for lane line recognition to determine the detection boxes of the lane lines in the top view; the position information of the lane lines in the top view may include the coordinate values ​​of the four vertices and / or the coordinate values ​​of the center point of the detection box of the lane lines.

[0126] Step S404: In response to determining that the lane line and the road sign overlap in the top view based on the position information of the lane line and the position information of the road sign in the top view, the road sign is filtered out.

[0127] In the top view, based on the location information of the road marking and the lane lines, it is determined whether the road marking and lane lines overlap in the top view. If the road marking and lane lines overlap in the top view, the road marking is filtered out.

[0128] For example, by using the coordinate values ​​of the four vertices of the lane line detection box in the top view and the coordinate values ​​of the four vertices of the second detection box of the road marking, the area intersection ratio or perimeter intersection ratio between the lane line detection box and the second detection box of the road marking in the top view is determined. When the area intersection ratio or perimeter intersection ratio between the lane line detection box and the second detection box of the road marking in the top view is greater than or equal to a preset third threshold, it is determined that the road marking in the top view coincides with the lane line, and the road marking in the top view is filtered out. When the road marking in the top view coincides with the lane line, the lane line in the top view is mistakenly detected as a road marking, so when the road marking in the top view coincides with the lane line, the road marking is filtered out and no further road marking detection steps are performed on the road marking. When the area intersection ratio or perimeter intersection ratio between the lane line detection box and the second detection box of the road marking in the top view is less than the preset third threshold, it is determined that the road marking in the top view does not coincide with the lane line, and the subsequent road marking detection steps are performed on the road marking.

[0129] The distance between the center point of the lane line detection box and the center point of the second detection box of the road marking can be determined by using the coordinates of the center point of the lane line detection box in the top view and the coordinates of the center point of the second detection box of the road marking. This distance can be Euclidean distance, etc. When the distance between the center point of the lane line detection box and the center point of the second detection box of the road marking is less than or equal to a preset fourth threshold, it is determined that the road marking coincides with the lane line in the top view, and the road marking is filtered out in the top view. When the distance between the center point of the lane line detection box and the center point of the second detection box of the road marking is greater than the preset fourth threshold, it is determined that the road marking does not coincide with the lane line in the top view, and the subsequent detection steps for the road marking continue.

[0130] In this embodiment of the disclosure, by using the position information of lane lines and road markings in the top view, road markings that overlap with lane lines in the top view are filtered out, thus solving the problem of misdetecting lane lines as road markings and effectively improving the accuracy of road marking detection results.

[0131] In one alternative embodiment, such as Figure 9 As shown, step S202 may include:

[0132] Step S2021: Use the shared sub-network in the identifier recognition network to extract features from the road surface image to obtain the image features of the road surface image.

[0133] The sign recognition network includes a shared sub-network and a first road surface sign recognition sub-network. The shared sub-network is used to extract image features from the road surface image and the top-down view, determining the image features of the road surface image and the top-down view. The shared sub-network can be CNN, SCNN, RNN, LSTM, etc. The road surface image is input into the shared sub-network, and the shared sub-network outputs the image features of the road surface image.

[0134] Step S2022: Using the first road marking recognition sub-network in the marking recognition network, the image features of the road image are recognized to obtain the first detection result.

[0135] The first road marking recognition subnetwork is used to detect image features of the road surface image. This subnetwork can be a CNN, SCNN, RNN, LSTM, etc. The image features of the road surface image are input into the first road marking recognition subnetwork, which then outputs the first detection result of the road markings.

[0136] In this embodiment of the disclosure, based on the powerful learning capabilities of the shared subnetwork and the first road marking recognition subnetwork, the shared subnetwork can extract image features from different road images, and the first road marking recognition subnetwork can detect image features from different road images, thereby obtaining accurate detection results of road markings in the road images.

[0137] In one alternative embodiment, such as Figure 10 As shown, step S203 may include:

[0138] Step S2031: Using the shared sub-network in the identifier recognition network, feature extraction is performed on the top view to obtain the image features of the top view.

[0139] Step S2032: Using the second road marking recognition subnetwork in the marking recognition network, the image features of the top view are recognized to obtain the second detection result.

[0140] The sign recognition network further includes a second road sign recognition subnetwork. A top-down view is input into a shared subnetwork, which outputs the image features of the top-down view. The second road sign recognition subnetwork is used to detect these image features. This second road sign recognition subnetwork can be a CNN, SCNN, RNN, LSTM, etc. The image features of the top-down view are input into the second road sign recognition subnetwork, which outputs a second detection result for the road signs.

[0141] For example, Figure 11 The structure of the identifier recognition network is shown. For example... Figure 11 As shown, the sign recognition network includes a shared subnetwork, a first road sign recognition subnetwork, and a second road sign recognition subnetwork. The road image and the top view are input into the shared subnetwork, and the image features of the road image and the top view are obtained through the shared subnetwork. The image features of the road image are input into the first road sign recognition subnetwork, and a first detection result is obtained through the first road sign recognition subnetwork. The image features of the top view are input into the second road sign recognition subnetwork, and a second detection result is obtained through the second road sign recognition subnetwork.

[0142] In this embodiment of the disclosure, based on the powerful learning capabilities of the shared subnetwork and the second road marking recognition subnetwork, the shared subnetwork can extract image features from different top views, and the second road marking recognition subnetwork can detect image features from different top views, thus obtaining accurate detection results of road markings in the top view.

[0143] In one optional example, Figure 12A flowchart of a road marking detection method provided by an exemplary embodiment of this disclosure is disclosed. The specific operations of each step have been described in detail above and will not be repeated here.

[0144] 1. Obtain the road surface image and its top view.

[0145] 2. Use a sign recognition network to detect road signs in the road image to obtain a first detection result, wherein the first detection result includes a first detection box and a first classification confidence of the road sign; use a sign recognition network to detect road signs in the top view to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence of the road sign.

[0146] 3. Map the second detection box of the road marking onto the road image to obtain the first mapping box of the road marking.

[0147] 4. Determine the intersection-union ratio between the first detection frame and the first mapping frame of the road marking.

[0148] 5. Compare the cross-union ratio between the first detection box and the first mapping box of the road marking with the first threshold. If the cross-union ratio between the first detection box and the first mapping box of the road marking is greater than or equal to the first threshold, proceed to step 6. If the cross-union ratio between the first detection box and the first mapping box of the road marking is less than the first threshold, proceed to step 7.

[0149] 6. Determine the first or second detection result of the road marking as the detection result of the road marking, wherein the detection result of the road marking includes the third detection frame and the third classification confidence level of the road marking, and then do not execute the subsequent process of this embodiment.

[0150] 7. Obtain the location information of the road markings and lane lines in the road image.

[0151] 8. Based on the location information of the road marking and the lane line in the road image, determine whether the road marking and the lane line overlap in the road image. If the road marking and the lane line overlap in the road image, proceed to step 9. If the road marking and the lane line do not overlap in the road image, proceed to step 10.

[0152] 9. Delete the road marking, and then do not execute the subsequent processes of this embodiment.

[0153] 10. Obtain the location information of the road markings and lane lines in the top view.

[0154] 11. Based on the location information of the road marking and the lane line in the top view, determine whether the road marking and the lane line overlap in the top view. If the road marking and the lane line overlap in the top view, proceed to step 12. If the road marking and the lane line do not overlap in the top view, proceed to step 13.

[0155] 12. Delete the road marking, and then do not execute the subsequent processes of this embodiment.

[0156] 13. Compare the cross-union ratio between the first detection box and the first mapping box of the road marking with the second threshold. If the cross-union ratio between the first detection box and the first mapping box of the road marking is greater than the second threshold, proceed to step 14. If the cross-union ratio between the first detection box and the first mapping box of the road marking is less than or equal to the second threshold, proceed to step 15.

[0157] 14. Determine the detection result of the road marking based on the first category confidence level and the second category confidence level, and then do not execute the subsequent process of this embodiment.

[0158] 15. Determine whether the road marking is detected in the road image and the top view. If the road marking is detected in the road image but not in the top view, proceed to step 16. If the road marking is not detected in the road image but is detected in the top view, proceed to step 20.

[0159] 16. Map the first detection frame of the road marking to the top view to obtain the second mapping frame of the road marking.

[0160] 17. Match the first image feature corresponding to the second mapping box with the image features of all road markings in the top view. If there is an image feature of a road marking in the top view that matches the first image feature corresponding to the second mapping box, proceed to step 18. If there is no image feature of a road marking in the top view that matches the first image feature corresponding to the second mapping box, proceed to step 19.

[0161] 18. The first detection result is determined as the detection result of the road marking, and the subsequent processes of this embodiment are not executed.

[0162] 19. If the road marking is not detected, the subsequent processes of this embodiment will not be executed.

[0163] 20. Match the second image feature corresponding to the first mapping box with the image features of all road markings in the road image. If there is an image feature of a road marking in the road image that matches the second image feature corresponding to the first mapping box, proceed to step 21. If there is no image feature of a road marking in the road image that matches the second image feature corresponding to the first mapping box, proceed to step 22.

[0164] 21. The second detection result is determined as the detection result of the road marking, and then the subsequent process of this embodiment is not executed.

[0165] 22. It was determined that the road marking was not detected.

[0166] Any of the road marking detection methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the road marking detection methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the road marking detection methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0167] Figure 13 This is a schematic flowchart of a training method for an identifier recognition network provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, etc. Figure 13 As shown, it includes the following steps:

[0168] Step S501: Based on the neural network to be trained, the first predicted detection box and the first classification confidence prediction value of the road surface markings in each sample road surface image are obtained using the sample road surface image set.

[0169] The sample road surface image set includes multiple sample road surface images. Each sample road surface image includes a road surface marker, and each sample road surface image is labeled with a first detection box and a first classification confidence ground truth value for the road surface marker. The detection box and the classification confidence ground truth value labeled with the road surface marker in the road surface image are respectively referred to as the first detection box and the first classification confidence ground truth value of the road surface marker in that road surface image.

[0170] Multiple raw road surface images with road markings can be acquired using an image acquisition device. These raw images may contain depth information. The road markings in each raw image can be labeled manually or using annotation software, such as Label Img. The labeled raw images are then used as sample road surface images. Alternatively, multiple sample road surface images can be obtained from an existing image library, forming a sample road surface image set. The neural network to be trained can be a single neural network or a composite network composed of multiple neural networks. For example, the neural network to be trained can be CNN, SCNN, RNN, or LSTM, etc.

[0171] Each sample road surface image in the sample road surface image set is input into the neural network to be trained. The neural network to be trained outputs the predicted detection box and classification confidence prediction value of the road surface markings in each sample road surface image. The predicted detection box and classification confidence prediction value of the road surface markings in each sample road surface image are respectively called the first predicted detection box and the first classification confidence prediction value of the road surface markings in each sample road surface image.

[0172] Step S502: Based on the neural network to be trained, the second predicted detection box and the second classification confidence prediction value of the road surface markings in each sample top view image set are obtained using the sample top view images.

[0173] Each sample top view includes road markings, and each sample top view is labeled with a second detection box for the road markings and the true value of the second classification confidence level.

[0174] The sample top-view image set includes multiple sample top-views. The detection boxes and ground truth classification confidence values ​​of the road markings in each sample top-view are referred to as the second detection boxes and second ground truth classification confidence values ​​of the road markings in that sample top-view, respectively. Multiple original top-views with road markings can be acquired using an image acquisition device. These original top-views may contain depth information. The road markings in each original top-view can be labeled manually or using labeling software, such as Label Img. The labeled original top-views are then used as sample top-views. Alternatively, image transformation methods such as inverse projection transformation can be used to transform each sample road image to obtain the sample top-view set. Each sample top-view in the sample top-view set is input into a neural network to be trained. The neural network outputs the predicted detection boxes and predicted classification confidence values ​​of the road markings in each sample top-view. These predicted detection boxes and predicted classification confidence values ​​of the road markings in each sample top-view are referred to as the second predicted detection boxes and second ground truth classification confidence values ​​of the road markings in each sample top-view, respectively.

[0175] Step S503: Based on the first loss between the first classification confidence prediction value and the first classification confidence ground truth value of the road signs in each sample road surface image, and the second loss between the second classification confidence prediction value and the second classification confidence ground truth value of the road signs in each sample top view, the neural network to be trained is trained until the preset training conditions are met, thus obtaining the sign recognition network. This sign recognition network is applied to claims 1 to 7.

[0176] Specifically, based on the true and predicted first-class confidence values ​​of road markings in each sample road surface image, the loss between the predicted and true first-class confidence values ​​of road markings in each sample road surface image is determined using loss functions such as mean squared error and cross-entropy error. This loss between the predicted and true first-class confidence values ​​of road markings in the sample road surface image is defined as the first loss between the predicted and true first-class confidence values ​​of road markings in the sample road surface image.

[0177] Based on the true and predicted confidence values ​​of the second category of road markings in the top view of each sample, the loss between the predicted and true confidence values ​​of the second category of road markings in the top view of each sample is determined using loss functions such as mean squared error and cross-entropy error. This loss between the predicted and true confidence values ​​of the second category of road markings in the top view of the sample is defined as the second loss between the predicted and true confidence values ​​of the second category of road markings in the top view of the sample.

[0178] The joint loss can be obtained by summing and averaging the individual first and second losses, or by summing the individual first losses to obtain a total first loss, and summing the individual second losses to obtain a total second loss. Weights corresponding to the sums of the first and second losses are pre-set. The sums of the first and second losses are then multiplied by their respective weights, summed, and averaged to obtain the joint loss. The neural network to be trained can be trained based on algorithms such as backpropagation and stochastic gradient descent, using the joint loss to obtain the identifier recognition network. For example, the gradient of each weight parameter of the neural network to be trained can be calculated. The gradient represents the direction in which the joint loss decreases the most. The weight parameters are adjusted along the gradient direction, and the above steps are repeated until the joint loss no longer decreases, confirming that the preset training conditions are met, thus obtaining the identifier recognition network.

[0179] For example, the neural network to be trained may include a shared sub-network to be trained, a first road surface marker recognition sub-network to be trained, and a second road surface marker recognition sub-network to be trained. The shared sub-network to be trained, the first road surface marker recognition sub-network to be trained, and the second road surface marker recognition sub-network to be trained may be CNN, SCNN, RNN, or LSTM, etc.

[0180] Each sample road surface image from the sample road surface image set and each sample top-view image from the sample top-view image set are input into the neural network to be trained. A shared sub-network outputs the predicted image features of each sample road surface image and each sample top-view image. The first road sign recognition sub-network to be trained identifies the predicted image features of each sample road surface image, obtaining the first predicted detection box and the first classification confidence prediction value for the road signs in each sample road surface image. The second road sign recognition sub-network to be trained identifies the predicted image features of each sample top-view image, obtaining the second predicted detection box and the second classification confidence prediction value for the road signs in each sample top-view image. Based on the first loss between the first classification confidence prediction value and the first classification confidence ground truth value of the road signs in each sample road surface image, and the second loss between the second classification confidence prediction value and the second classification confidence ground truth value of the road signs in each sample top-view image, the neural network to be trained is trained until the preset training conditions are met, resulting in the sign recognition network.

[0181] In this embodiment of the disclosure, the neural network to be trained is trained by using a sample road surface image set and a sample top view set, thereby obtaining a sign recognition network. This improves the generalization ability of the sign recognition network and thus improves the accuracy of the sign recognition network in detecting road signs.

[0182] In one alternative embodiment, such as Figure 14 As shown, the training methods for the identifier recognition network also include:

[0183] Step S601: Based on each sample road surface image in the sample road surface image set, determine the first sample top view corresponding to each sample road surface image.

[0184] In this process, the top view obtained by transforming the sample road surface image is determined as the first sample top view of that sample road surface image. Each sample road surface image can be transformed using image transformation methods such as inverse projection transformation to obtain the first sample top view of each sample road surface image.

[0185] Step S602: Scaling the first sample top view corresponding to each sample road surface image to obtain at least one second sample top view corresponding to each sample road surface image.

[0186] Each second sample top view includes road markings. The range distance values ​​of the first sample top view and the second sample top view corresponding to the same sample road image are different. The range distance value represents the distance between the road markings and the image acquisition device in the real environment.

[0187] The first sample top view corresponding to the scaled sample road surface image is determined as the second sample top view of the same sample road surface image. A pre-defined correspondence between range distance values ​​and pixel sizes can be established. For each first sample top view of a sample road surface image, the required range distance value is determined. Based on the correspondence between range distance values ​​and pixel sizes and the required range distance value, the pixel size corresponding to the required range distance value is determined. The first sample top view is then scaled so that its pixel size matches the pixel size corresponding to the required range distance value, thus obtaining the second sample top view corresponding to the required range distance value for the sample road surface image. The size of the second sample top view image can be cropped to meet the image size requirements of the neural network to be trained. For example, the center position of the road surface marker in the first sample top view can be determined based on the contour data of the road surface marker or the coordinates of the vertices of the detection box of the road surface marker. The first sample top view is then scaled around the center position of the road surface marker to obtain the second sample top view corresponding to the sample road surface image. It should be noted that when the first sample top view includes multiple road markings, the center position of all road markings can be determined based on the center position of the multiple road markings, and the first sample top view can be scaled with that center position as the center.

[0188] Step S603: Obtain the sample top view image set from the top views of each first sample and / or each second sample.

[0189] Specifically, the first and second sample top views corresponding to each sample road surface image can be respectively determined as the sample top views in the sample top view image set. Alternatively, the first and second sample top views corresponding to each sample road surface image can be filtered according to a preset ratio, and the filtered first and second sample top views corresponding to each sample road surface image can be respectively determined as the sample top views in the sample top view image set. Alternatively, either the first or second sample top view corresponding to each sample road surface image can be respectively determined as the sample top views in the sample top view image set.

[0190] In this embodiment of the disclosure, by acquiring the first sample top view and the second sample top view corresponding to each sample road surface image, and forming a sample top view image set by multiple first sample top views and multiple second sample top views, since the sample top view set contains sample top views with different range distance values, and the sample top views in the sample top view set correspond to the road surface top view, the generalization ability of the identification recognition network obtained by training the neural network to be trained through this sample top view set and this sample road surface image set is improved, and the accuracy of the identification recognition network in detecting road surface markings is further improved.

[0191] The training method for any of the identifier recognition networks provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the training method for any of the identifier recognition networks provided in this disclosure can be executed by a processor, such as by a processor executing the training method for any of the identifier recognition networks mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0192] Exemplary device

[0193] Figure 15 This is a structural block diagram of a road marking detection device according to one embodiment of this disclosure. Figure 15 As shown, the road marking detection device includes: an acquisition module 700, a first detection module 701, a second detection module 702, and a determination module 703.

[0194] The acquisition module 700 is used to acquire a road surface image and a top view of the road surface image;

[0195] The first detection module 701 is used to perform road marking detection on the road image using a marking recognition network to obtain a first detection result; wherein, the first detection result includes a first detection box and a first classification confidence score of the road marking;

[0196] The second detection module 702 is used to perform road marking detection on the top view using the marking recognition network to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence level of the road marking;

[0197] The determination module 703 is used to determine the detection result of the road marking based on the first detection result and the second detection result, wherein the detection result of the road marking includes a third detection frame and a third classification confidence level of the road marking.

[0198] In one embodiment of this disclosure, Figure 16 This is a structural block diagram of a road marking detection device in another embodiment of this disclosure, as shown below. Figure 16As shown, the determining module 703 includes:

[0199] The mapping submodule 7031 is used to map the second detection box to the road surface image to obtain a first mapping box;

[0200] The matching submodule 7032 is used to match the first mapping box with the first detection box and determine the detection result of the road marking based on the matching result.

[0201] In one embodiment of this disclosure, the matching submodule 7032 includes:

[0202] The acquisition unit 70321 (not shown in the figure) acquires the intersection-over-union ratio between the first mapping box and the first detection box;

[0203] Response unit 70322 (not shown in the figure), in response to the cross-union ratio being greater than or equal to a first threshold, determines the first detection result or the second detection result as the detection result of the road marking; or,

[0204] The response subunit 70322 is further configured to, in response to the cross-union ratio being less than the first threshold and greater than the second threshold, determine the detection result of the road marking based on the first classification confidence level and the second classification confidence level, wherein the first threshold is greater than the second threshold; or,

[0205] The response subunit 70322 is further configured to, in response to the intersection-union ratio being less than or equal to the second threshold, determine whether the road surface image and the top view detect the road surface marker; in response to the road surface image detecting the road surface marker and the top view not detecting the road surface marker, map the first detection box to the top view to obtain a second mapping box; match the first image feature corresponding to the second mapping box with the image feature corresponding to at least one road surface marker in the top view respectively; in response to the existence of an image feature of a road surface marker in the top view that matches the first image feature, determine the first detection result as the detection result of the road surface marker; in response to the absence of an image feature of a road surface marker in the top view that matches the first image feature, determine that the road surface marker has not been detected.

[0206] In one embodiment of this disclosure, the response subunit 70322 is further configured to, in response to the detection of the road marking in the top view and the absence of the road marking in the road image, match the second image feature corresponding to the first mapping frame with the image features of at least one road marking in the road image; in response to the presence of an image feature of a road marking in the road image that matches the second image feature, determine the second detection result as the detection result of the road marking; and in response to the absence of an image feature of a road marking in the road image that matches the second image feature, determine that the road marking has not been detected.

[0207] In one embodiment of this disclosure, the response subunit 70322 is further configured to, in response to the intersection-over-union ratio being less than the first threshold, acquire the position information of the lane lines and the position information of the road sign in the road image; in response to determining, based on the position information of the lane lines and the position information of the road sign in the road image, that the lane lines and the road sign in the road image overlap, and filter out the road sign; and / or, in response to the intersection-over-union ratio being less than the first threshold, acquire the position information of the lane lines and the position information of the road sign in the top view; in response to determining, based on the position information of the lane lines and the position information of the road sign in the top view, that the lane lines and the road sign in the top view overlap, and filter out the road sign.

[0208] In one embodiment of this disclosure, the first detection module 701 includes:

[0209] The first identification submodule 7011 is used to extract features from the road surface image using the shared subnetwork in the identification network to obtain the image features of the road surface image;

[0210] The second recognition submodule 7012 is used to recognize the image features of the road surface image using the first road surface recognition subnetwork in the recognition network to obtain the first detection result.

[0211] In one embodiment of this disclosure, the second detection module 702 includes:

[0212] The third identification submodule 7021 is used to extract features from the top view using the shared subnetwork in the identification network to obtain the image features of the top view;

[0213] The fourth recognition submodule 7022 is used to recognize the image features of the top view using the second road surface recognition subnetwork in the recognition network to obtain the second detection result.

[0214] Figure 17This is a structural block diagram of a training device for an identifier recognition network in one embodiment of this disclosure. Figure 17 As shown, the training device for the identifier recognition network includes: a first acquisition module 800, a second acquisition module 801, and a training module 803.

[0215] The first acquisition module 800 is used to obtain, based on the neural network to be trained, a first predicted detection box and a first classification confidence prediction value of the road surface markings in each of the sample road surface images in the sample road surface image set; wherein, the sample road surface image includes road surface markings, and the sample road surface image is labeled with the first detection box and the first classification confidence ground truth value of the road surface markings.

[0216] The second acquisition module 801, based on the neural network to be trained, uses the top views of each sample in the sample top view image set to obtain the second predicted detection box and the second classification confidence prediction value of the road surface markings in each sample top view; wherein, the sample top view includes the road surface markings, and the sample top view is labeled with the second detection box and the second classification confidence ground truth value of the road surface markings.

[0217] The training module 803 is used to train the neural network to be trained based on the first loss between the first classification confidence prediction value and the first classification confidence true value of the road sign in each of the sample road surface images, and the second loss between the second classification confidence prediction value and the second classification confidence true value of the road sign in each of the sample top view images, until the preset training conditions are met, and thus obtain the sign recognition network.

[0218] In one embodiment of this disclosure, such as Figure 18 The training device for the identifier recognition network shown also includes:

[0219] The first image conversion module 804 is used to determine the first sample top view corresponding to each sample road surface image based on each sample road surface image in the sample road surface image set.

[0220] The second image conversion module 805 is used to scale the first sample top view corresponding to each sample road surface image to obtain at least one second sample top view corresponding to each sample road surface image. The second sample top view includes the road surface markings. The range distance values ​​of the first sample top view and the second sample top view corresponding to the same sample road surface image are different. The range distance value represents the distance between the road surface markings and the image acquisition device in the real environment.

[0221] The set module 806 is used to obtain the sample top view image set from each first sample top view and / or each second sample top view.

[0222] Exemplary electronic devices

[0223] under, Figure 19 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0224] like Figure 19 As shown, the electronic device includes one or more processors 900 and a memory 901.

[0225] The processor 900 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0226] The memory 901 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 900 may execute the program instructions to implement the road marking detection method and marking recognition network training method of the various embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0227] In one example, the electronic device may also include an input device 902 and an output device 903, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0228] For example, input device 902 may be the microphone or microphone array described above, used to capture input signals from a sound source. Input device 902 may also include, for example, a keyboard, mouse, etc.

[0229] The output device 903 can output various information to the outside, including determined distance information, direction information, etc. The output device 903 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0230] Of course, for the sake of simplicity, Figure 19 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0231] Exemplary computer program products and computer-readable storage media

[0232] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the road marking detection method and marking recognition network training method according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0233] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0234] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the road marking detection method and marking recognition network training method according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0235] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0236] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0237] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0238] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0239] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0240] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0241] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0242] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for detecting road markings, comprising: Obtain a road surface image and a top view of the road surface image; Using a sign recognition network, road sign detection is performed on the road image to obtain a first detection result; wherein, the first detection result includes a first detection box and a first classification confidence score for the road sign; Using the aforementioned sign recognition network, road surface signs are detected in the top view to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence level for the road surface signs; Based on the first detection result and the second detection result, the detection result of the road marking is determined, wherein the detection result of the road marking includes a third detection box and a third classification confidence score of the road marking, including: in response to the intersection-union ratio (IU) between the first mapping box and the first detection box being less than a first threshold, obtaining the position information of the lane lines and the position information of the road marking in the road image; in response to determining that the lane lines and the road marking in the road image overlap based on the position information of the lane lines and the position information of the road marking in the road image, filtering out the road marking; and / or, in response to the intersection-union ratio being less than the first threshold, obtaining the position information of the lane lines and the position information of the road marking in the top view; in response to determining that the lane lines and the road marking in the top view overlap based on the position information of the lane lines and the position information of the road marking in the top view, filtering out the road marking, wherein the first mapping box is obtained by mapping the second detection box to the road image.

2. The method according to claim 1, wherein, The step of determining the detection result of the road marking based on the first detection result and the second detection result includes: The second detection box is mapped onto the road surface image to obtain the first mapping box; The first mapping box is matched with the first detection box, and the detection result of the road marking is determined based on the matching result.

3. The method according to claim 2, wherein, Matching the first mapping box with the first detection box, and determining the detection result of the road marking based on the matching result, includes: Obtain the intersection-union ratio (IU / U) between the first mapped frame and the first detected frame; In response to the cross-union ratio being greater than or equal to a first threshold, the first detection result or the second detection result is determined as the detection result of the road marking; or, In response to the intersection-union ratio being less than the first threshold and greater than the second threshold, the detection result of the road marking is determined based on the first classification confidence score and the second classification confidence score, wherein the first threshold is greater than the second threshold; or, In response to the intersection-union ratio being less than or equal to the second threshold, it is determined whether the road surface image and the top view detect the road surface markings; In response to the detection of the road marking in the road image and the absence of the road marking in the top view, the first detection box is mapped to the top view to obtain a second mapping box; Match the first image feature corresponding to the second mapping box with the image feature corresponding to at least one road surface mark in the top view respectively; In response to the presence of image features of a road surface marker in the top view that match the first image features, the first detection result is determined as the detection result of the road surface marker; In response to the absence of an image feature in the top view that matches the first image feature of the road marking, it is determined that the road marking has not been detected.

4. The method of claim 3, after determining whether the road surface image and the top view detect the road surface marking in response to the intersection-union ratio being less than or equal to the second threshold, the method further includes: In response to the detection of the road marking in the top view and the absence of the road marking in the road image, the second image feature corresponding to the first mapping box is matched with the image feature of at least one road marking in the road image; In response to the presence of a road surface feature in the road surface image that matches the second image feature, the second detection result is determined as the detection result of the road surface feature; In response to the absence of an image feature in the road surface image that matches the second image feature, it is determined that the road surface marker has not been detected.

5. The method according to any one of claims 1-4, wherein, The step of using a sign recognition network to detect road signs in the road image and obtaining a first detection result includes: The shared subnetwork in the identification network is used to extract features from the road surface image to obtain the image features of the road surface image; The first road marking recognition subnetwork in the marking recognition network is used to identify the image features of the road image to obtain the first detection result.

6. The method according to claim 1, wherein, The step of using the sign recognition network to detect road markings in the top view and obtaining a second detection result includes: By utilizing the shared subnetwork in the identifier recognition network, feature extraction is performed on the top view to obtain the image features of the top view; The second road marking recognition subnetwork in the marking recognition network is used to identify the image features of the top view to obtain the second detection result.

7. A method for training an identifier recognition network, comprising: Based on the neural network to be trained, the first predicted detection box and the first classification confidence prediction value of the road surface markings in each sample road surface image are obtained using the sample road surface image set; wherein, the sample road surface image includes road surface markings, and the sample road surface image is labeled with the first detection box and the first classification confidence ground truth value of the road surface markings; Based on the neural network to be trained, the second predicted detection box and the second classification confidence prediction value of the road surface markings in each sample top view image set are obtained using the sample top view images; wherein, the sample top view includes the road surface markings, and the sample top view is labeled with the second detection box and the second classification confidence ground truth value of the road surface markings. Based on the first loss between the first classification confidence prediction value and the first classification confidence true value of the road markings in each of the sample road surface images, and the second loss between the second classification confidence prediction value and the second classification confidence true value of the road markings in each of the sample top views, the neural network to be trained is trained until the preset training conditions are met to obtain the marking recognition network, which is used to perform the road marking detection method according to any one of claims 1-6.

8. The method according to claim 7, wherein, The method further includes: Based on each sample road surface image in the sample road surface image set, determine the first sample top view corresponding to each sample road surface image; The first sample top view corresponding to each sample road surface image is scaled to obtain at least one second sample top view corresponding to each sample road surface image. The second sample top view includes the road surface markings. The range distance values ​​of the first sample top view and the second sample top view corresponding to the same sample road surface image are different. The range distance value represents the distance between the road surface markings and the image acquisition device in the real environment. The sample top view image set is obtained from the top view of each first sample and / or the top view of each second sample.

9. A road marking detection device, comprising: The acquisition module is used to acquire a road surface image and a top view of the road surface image; The first detection module is used to perform road marking detection on the road image using a marking recognition network to obtain a first detection result; wherein, the first detection result includes a first detection box and a first classification confidence score of the road marking; The second detection module is used to perform road marking detection on the top view using the marking recognition network to obtain a second detection result, wherein the second detection result includes a second detection box and a second classification confidence level of the road marking; The determination module is used to determine the detection result of the road marking based on the first detection result and the second detection result, wherein the detection result of the road marking includes a third detection box and a third classification confidence level of the road marking; The determining module includes a matching submodule, and the matching submodule includes a response unit; A response unit is configured to: acquire the position information of lane lines and the position information of road markings in the road image in response to the intersection-union ratio (IU) between the first mapping box and the first detection box being less than a first threshold; determine that the lane lines and the road markings in the road image overlap based on the position information of the lane lines and the position information of the road markings in the road image, and filter out the road markings; and / or, acquire the position information of lane lines and the position information of road markings in the top view in response to the intersection-union ratio being less than the first threshold; determine that the lane lines and the road markings in the top view overlap based on the position information of the lane lines and the position information of the road markings in the top view, and filter out the road markings, wherein the first mapping box is obtained by mapping the second detection box onto the road image.

10. A training device for an identifier recognition network, comprising: The first acquisition module is used to obtain, based on the neural network to be trained, a first predicted detection box and a first classification confidence prediction value of the road surface markings in each of the sample road surface images in the sample road surface image set; wherein, the sample road surface image includes road surface markings, and the sample road surface image is labeled with the first detection box and the first classification confidence ground truth value of the road surface markings; The second acquisition module, based on the neural network to be trained, uses the top views of each sample in the sample top view image set to obtain the second predicted detection box and the second classification confidence prediction value of the road surface markings in each sample top view; wherein, the sample top view includes the road surface markings, and the sample top view is labeled with the second detection box and the second classification confidence ground truth value of the road surface markings. The training module is used to train the neural network to be trained based on a first loss between the first classification confidence prediction value and the first classification confidence true value of the road markings in each of the sample road surface images, and a second loss between the second classification confidence prediction value and the second classification confidence true value of the road markings in each of the sample top views, until a preset training condition is met, thereby obtaining a marking recognition network. The marking recognition network is used to perform the road marking detection method according to any one of claims 1-6.

11. A computer-readable storage medium storing a computer program for performing the method according to any one of claims 1-8.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

Images