Traffic marking recognition method, device, computer equipment and storage medium

Abstract

The present disclosure provides a method, device, computer equipment and storage medium for identifying traffic markings, wherein the method comprises: acquiring road point cloud data; rasterizing the road point cloud data to obtain local point cloud data contained in at least one target grid; determining the probability that the target grid belongs to a traffic marking category based on the local point cloud data contained in each target grid; and determining the marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each target grid belongs.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, computer equipment and storage medium for identifying traffic markings. Background Art

[0002] With the rapid development of computer vision technology, computer vision technology has been applied in more and more fields. In the field of autonomous driving technology, a camera device installed on a vehicle usually captures road images, and then determines the distance between the lane lines, road boundary lines and other traffic markings on the road and the vehicle based on the road images.

[0003] However, the lane lines and road boundary lines determined based on road images usually have problems of discontinuous and inaccurate line segments. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, device, computer equipment and storage medium for identifying traffic markings.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for identifying a traffic marking, comprising:

[0006] Obtain road point cloud data;

[0007] Rasterizing the road point cloud data to obtain local point cloud data contained in at least one target grid;

[0008] Based on the local point cloud data contained in each target grid, determining the probability that the target grid belongs to the traffic marking category;

[0009] Based on the probability of the traffic line category to which each of the target grids belongs, a line marking recognition result of the road point cloud data is determined.

[0010] In this implementation, since the road point cloud data photographed by the laser radar has morphological stability, the road point cloud data obtained is used to identify the traffic marking category, which can improve the continuity of the identified traffic markings. By rasterizing the road point cloud data, and then processing the local point cloud data corresponding to each target grid obtained after the rasterization (i.e., the local point cloud data contained in each target grid), the probability that each target grid belongs to the traffic marking category can be determined. Based on the probability of the traffic marking category to which each target grid belongs, each target grid belonging to the traffic marking category can be accurately determined from multiple target grids. Then, using each target grid belonging to the traffic marking category, each point cloud point in the road point cloud data belonging to the traffic marking category can be accurately determined, and then each category of traffic markings, i.e., the marking recognition result, can be accurately obtained.

[0011] In a possible implementation, the traffic marking category includes a lane line category and a road boundary line category; and determining, based on the local point cloud data contained in each target grid, the probability that the target grid belongs to the traffic marking category includes:

[0012] For the local point cloud data contained in each of the target grids, feature extraction is performed on the point cloud information of each point cloud point in the local point cloud data to generate road feature information of the target grid;

[0013] Based on the road feature information, a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category are determined.

[0014] In this implementation, by extracting features from the point cloud information of each point cloud point in the local point cloud data, the feature information related to the traffic marking category in each point cloud point can be fully extracted, and then the accurate road feature information of the target grid can be obtained. By using the road feature information, the first probability that the target grid belongs to the lane line category and the second probability that it belongs to the road boundary line category can be determined, and the target grid can be accurately divided into the corresponding traffic marking category, and then the accurate traffic marking category of the target grid can be obtained.

[0015] In a possible implementation, determining the road marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each of the target grids belongs includes:

[0016] For each of the target grids, determining whether there is a target probability greater than a preset threshold value in the first probability and the second probability of the target grid;

[0017] In the case where the target probability exists, determining the recognition result of the target grid according to the traffic line category associated with the target probability;

[0018] Based on the recognition results of each of the target grids, a line marking recognition result of the road point cloud data is determined.

[0019] In this implementation, a target probability with a larger probability value can be screened out by using a preset threshold value, and the traffic marking category associated with the target probability is more likely to be the traffic marking category to which the target grid corresponds, so that the recognition result determined based on the traffic marking category associated with the target probability is more accurate. In this way, the recognition result of each target grid is determined by using each target probability, and then the marking recognition result is determined based on each recognition result, which can improve the accuracy of the determined marking recognition result.

[0020] In a possible implementation manner, determining the recognition result of the target grid according to the traffic marking category associated with the target probability includes:

[0021] In a case where the target probability includes the first probability and the second probability, determining a maximum probability of the first probability and the second probability;

[0022] The traffic marking category associated with the maximum probability is used as the recognition result of the target grid.

[0023] In this implementation, when the target probability includes the first probability and the second probability, the traffic marking category associated with the maximum probability is used as the recognition result of the target grid, which can ensure the uniqueness of the obtained target probability and improve the rationality of the obtained recognition result.

[0024] In a possible implementation manner, the road point cloud data is collected by a driving device, and after determining the line marking recognition result of the road point cloud data, the method further includes:

[0025] The driving device is controlled to travel based on the line marking information of at least one traffic line marking indicated by the line marking recognition result, wherein the line marking information includes a line marking position and / or a line marking category.

[0026] In this implementation, the driving of the traveling device is controlled by determining the position and / or type of the road marking, so as to ensure that the traveling device drives in a reasonable area and improve the driving safety of the traveling device.

[0027] In a possible implementation, determining the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid includes:

[0028] The trained target neural network is used to determine the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid.

[0029] In this implementation manner, the trained target neural network has reliable prediction accuracy. By using the trained target neural network, the probability that the target grid belongs to each category of traffic markings can be accurately determined.

[0030] In a possible implementation, the target neural network is trained according to the following steps:

[0031] Get sample point cloud data;

[0032] Rasterizing the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determining the annotation label information of each sample grid;

[0033] Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, and generating a predicted probability that the sample grid belongs to the traffic marking category;

[0034] Based on the predicted probability of the traffic line category to which each of the sample grids belongs and the annotated label information of each of the sample grids, the neural network to be trained is iteratively trained until a training cutoff condition is met to obtain the target neural network.

[0035] In this implementation, the neural network to be trained is iteratively trained through the predicted probability that the sample grid belongs to the traffic marking category and the annotated label information of each sample grid, which can improve the consistency between the output predicted probability and the annotated label information, thereby obtaining a target neural network with reliable prediction accuracy.

[0036] In a possible implementation, the prediction probability includes a first prediction probability that the sample grid belongs to a lane line category, and a second prediction probability that the sample grid belongs to a road boundary line category;

[0037] The step of iteratively training the neural network to be trained based on the predicted probability of the traffic line category to which each of the sample grids belongs and the label information of each of the sample grids until a training cutoff condition is met to obtain the target neural network includes:

[0038] When the annotated label information of the sample grid indicates that the sample grid does not belong to the background, determining a first loss based on the first predicted probability of the sample grid, the second predicted probability, the annotated label information of the sample grid, and the annotated label information corresponding to the background label;

[0039] When the annotated label information of the sample grid indicates that the sample grid belongs to the background, determining a second loss based on the first predicted probability, the second predicted probability and the annotated label information corresponding to the background label;

[0040] Based on at least one of the first loss and the second loss, the neural network to be trained is iteratively trained until a training cutoff condition is met, so as to obtain the target neural network.

[0041] This implementation method, by determining each loss and then using the determined losses to iteratively train the neural network to be trained, can improve the consistency between the prediction probability output by the network and the annotated label information, thereby obtaining a target neural network with reliable prediction accuracy.

[0042] In a possible implementation manner, the first loss includes a first sub-loss and a second sub-loss;

[0043] The determining a first loss based on the first prediction probability of the sample grid, the second prediction probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label includes:

[0044] Determining, from the first predicted probability and the second predicted probability of the sample grid, a target predicted probability that matches the traffic line category indicated by the annotated label information of the sample grid;

[0045] Determining a first sub-loss based on the target prediction probability and the labeled label information of the sample grid;

[0046] A second sub-loss is determined based on other prediction probabilities except the target prediction probability in the first prediction probability and the second prediction probability, and the annotated label information corresponding to the background label.

[0047] In this implementation, the first sub-loss can characterize the difference between the predicted probability output by the network and the annotated label information, and the second sub-loss can characterize the difference between the predicted probability output by the network and the annotated label information corresponding to the background label. Using the first sub-loss and the second sub-loss to train the network can improve the consistency between the predicted probability output by the network and the annotated label information.

[0048] In a possible implementation, determining the annotation label information of each sample grid includes:

[0049] Generate a sample top view based on the local sample point cloud data respectively contained in each of the sample grids; wherein each of the sample grids corresponds to a pixel point in the sample top view;

[0050] Based on the pixel information of each pixel point in the sample top view, the annotation label information of the sample grid matching each pixel point is determined.

[0051] This implementation generates a sample overhead view based on the local sample point cloud data respectively contained in each sample grid, and can use the pixel information of the pixel points in the sample overhead view to characterize the local sample point cloud data corresponding to the sample grid; then use the pixel information of the pixel points to determine the annotation label information of the sample grid matching the pixel points, which can reduce the annotation difficulty and improve the annotation speed.

[0052] In a possible implementation, determining the annotation label information of the sample grid matching each pixel point based on the pixel information of each pixel point in the sample top view includes:

[0053] Determine, based on pixel information of each pixel point in the sample top view, annotation label information corresponding to each pixel point in the sample top view;

[0054] For a target pixel point indicated by the annotated label information as a traffic line category, based on a preset expansion width, the annotated label information of the neighboring pixel points of the target pixel point is adjusted to the annotated label information of the target pixel point;

[0055] Based on the adjusted label information of the adjacent pixels in the sample top view and the unadjusted label information of other pixels except the adjacent pixels, the label information of the sample grid matching each pixel is determined.

[0056] In this implementation, since each type of traffic marking has a certain width, by expanding the width, the label information of the adjacent pixels of the target pixel is adjusted, which can improve the accuracy and rationality of the label information of the adjacent pixels.

[0057] In a second aspect, the present disclosure also provides a traffic marking recognition device, including:

[0058] Acquisition module, used to acquire road point cloud data;

[0059] A division module, used for performing raster division on the road point cloud data to obtain local point cloud data contained in at least one target grid;

[0060] A first determination module, configured to determine, based on the local point cloud data contained in each target grid, the probability that the target grid belongs to a traffic marking category;

[0061] The second determination module is used to determine the traffic marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each of the target grids belongs.

[0062] In a possible implementation, the traffic marking line category includes a lane line category and a road boundary line category; the first determination module, when determining the probability that the target grid belongs to the traffic marking line category based on the local point cloud data contained in each target grid, is used to extract features from the point cloud information of each point cloud point in the local point cloud data contained in each target grid, and generate road feature information of the target grid;

[0063] Based on the road feature information, a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category are determined.

[0064] In a possible implementation manner, the second determination module, when determining the line marking recognition result of the road point cloud data based on the probability of the traffic line marking category to which each of the target grids belongs, is used to determine, for each of the target grids, whether there is a target probability greater than a preset threshold in the first probability and the second probability of the target grid;

[0065] In the case where the target probability exists, determining the recognition result of the target grid according to the traffic line category associated with the target probability;

[0066] Based on the recognition results of each of the target grids, a line marking recognition result of the road point cloud data is determined.

[0067] In a possible implementation manner, the second determination module, when determining the recognition result of the target grid according to the traffic marking category associated with the target probability, is used to determine the maximum probability of the first probability and the second probability when the target probability includes the first probability and the second probability;

[0068] The traffic marking category associated with the maximum probability is used as the recognition result of the target grid.

[0069] In a possible implementation manner, the road point cloud data is collected by a driving device, and the device further includes:

[0070] A control module is used to control the driving device to drive based on the marking information of at least one traffic marking indicated by the marking recognition result after determining the marking recognition result of the road point cloud data, wherein the marking information includes the marking position and / or the marking category.

[0071] In a possible implementation, the first determination module, when determining the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid, is used to use a trained target neural network to determine the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid.

[0072] In a possible implementation, the device further includes:

[0073] The training module is used to train the target neural network according to the following steps:

[0074] Get sample point cloud data;

[0075] Rasterizing the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determining the annotation label information of each sample grid;

[0076] Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, and generating a predicted probability that the sample grid belongs to the traffic marking category;

[0077] Based on the predicted probability of the traffic line category to which each of the sample grids belongs and the annotated label information of each of the sample grids, the neural network to be trained is iteratively trained until a training cutoff condition is met to obtain the target neural network.

[0078] In a possible implementation, the prediction probability includes a first prediction probability that the sample grid belongs to a lane line category, and a second prediction probability that the sample grid belongs to a road boundary line category;

[0079] The training module, when iteratively training the neural network to be trained based on the predicted probability of the traffic marking category to which each of the sample grids belongs and the labeled label information of each of the sample grids until a training cutoff condition is satisfied and the target neural network is obtained, is used to determine a first loss based on the first predicted probability of the sample grid, the second predicted probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label when the labeled label information of the sample grid indicates that the sample grid does not belong to the background;

[0080] When the annotated label information of the sample grid indicates that the sample grid belongs to the background, determining a second loss based on the first predicted probability, the second predicted probability and the annotated label information corresponding to the background label;

[0081] Based on at least one of the first loss and the second loss, the neural network to be trained is iteratively trained until a training cutoff condition is met, so as to obtain the target neural network.

[0082] In a possible implementation manner, the first loss includes a first sub-loss and a second sub-loss;

[0083] The training module is used to determine, when determining the first loss based on the first prediction probability of the sample grid, the second prediction probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label, a target prediction probability that matches the traffic line category indicated by the labeled label information of the sample grid from the first prediction probability and the second prediction probability of the sample grid;

[0084] Determining a first sub-loss based on the target prediction probability and the labeled label information of the sample grid;

[0085] A second sub-loss is determined based on other prediction probabilities except the target prediction probability in the first prediction probability and the second prediction probability, and the annotated label information corresponding to the background label.

[0086] In a possible implementation, the training module, when determining the annotation label information of each sample grid, is used to generate a sample top view based on the local sample point cloud data respectively contained in each sample grid; wherein each sample grid corresponds to a pixel point in the sample top view;

[0087] Based on the pixel information of each pixel point in the sample top view, the annotation label information of the sample grid matching each pixel point is determined.

[0088] In a possible implementation, the training module, when determining the annotation label information of the sample grid matching each pixel point based on the pixel information of each pixel point in the sample overhead view, is used to determine the annotation label information corresponding to each pixel point in the sample overhead view based on the pixel information of each pixel point in the sample overhead view;

[0089] For a target pixel point indicated by the annotated label information as a traffic line category, based on a preset expansion width, the annotated label information of the neighboring pixel points of the target pixel point is adjusted to the annotated label information of the target pixel point;

[0090] Based on the adjusted label information of the adjacent pixels in the sample top view and the unadjusted label information of other pixels except the adjacent pixels, the label information of the sample grid matching each pixel is determined.

[0091] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0092] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0093] For a description of the effects of the above-mentioned traffic marking recognition device, computer equipment, and computer-readable storage medium, please refer to the description of the above-mentioned traffic marking recognition method, which will not be repeated here.

[0094] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0096] Figure 1 A flow chart showing a method for identifying traffic markings provided by an embodiment of the present disclosure is shown;

[0097] Figure 2 A flow chart of a method for training a neural network to be trained provided by an embodiment of the present disclosure is shown;

[0098] Figure 3 A schematic diagram showing a traffic marking recognition device provided by an embodiment of the present disclosure is shown;

[0099] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0100] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0101] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0102] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0103] According to research, when using road images taken by cameras to identify traffic markings, it is necessary to convert the coordinates of the pixels belonging to traffic markings in the road images in the image coordinate system into coordinates in the world coordinate system, and then determine the traffic markings based on the coordinates of the pixels in the world coordinate system. This method not only increases the complexity of traffic marking determination, but also introduces errors in coordinate conversion, resulting in inaccurate and discontinuous traffic markings.

[0104] Based on the above research, the present disclosure provides a solution for identifying traffic markings. Since the road point cloud data photographed by laser radar has morphological stability, the road point cloud data obtained is used to identify the category of traffic markings, which can improve the continuity of the identified traffic markings. By rasterizing the road point cloud data, and then processing the local point cloud data corresponding to each target grid obtained after the rasterization, the probability that each target grid belongs to the category of traffic markings can be determined. Based on the probability corresponding to each target grid, each target grid belonging to the category of traffic markings can be accurately determined from multiple target grids. Then, using each target grid belonging to the category of traffic markings, each point cloud point in the road point cloud data that belongs to the category of traffic markings can be accurately determined, and then each category of traffic markings, that is, the marking recognition result, can be accurately obtained.

[0105] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0106] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0107] To facilitate understanding of this embodiment, a method for identifying traffic markings disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the method for identifying traffic markings provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, etc.; in some possible implementations, the method for identifying traffic markings may be implemented by a processor calling computer-readable instructions stored in a memory.

[0108] The following describes the traffic marking recognition method provided by the embodiment of the present disclosure by taking a computer device as an example of an execution subject.

[0109] like Figure 1 As shown, it is a flow chart of a method for identifying traffic markings provided by an embodiment of the present disclosure, which may include the following steps:

[0110] S101: Acquire road point cloud data.

[0111] Here, the road point cloud data can be obtained by using a laser radar installed on a driving device, and the road point cloud data can be a point cloud vector set, and the point cloud vector set includes point cloud information of multiple point cloud points. The point cloud information may include the coordinates of the point cloud points in a three-dimensional world coordinate system, the color information of the point cloud points, the reflection intensity information, the distance information, etc.

[0112] For example, during the driving process of the driving device, a laser radar installed on the driving device can be used to collect road point cloud data of the driving road.

[0113] S102: Rasterizing the road point cloud data to obtain local point cloud data contained in at least one target grid.

[0114] Here, the road point cloud data is rasterized and divided to obtain a plurality of grids, wherein the plurality of grids obtained may include empty grids and non-empty target grids. Each target grid may include local point cloud data.

[0115] The local point cloud data is part of the point cloud data in the road point cloud data, including at least one point cloud point in the road point cloud data and point cloud information of each point cloud point in the at least one point cloud point. The local point cloud data respectively contained in each target grid can constitute the above road point cloud data.

[0116] For example, the horizontal axis direction (i.e., the x-axis direction) in the world coordinate system can be the direction of the road, the vertical axis direction (i.e., the y-axis direction) in the world coordinate system can be perpendicular to the direction of the road, and the vertical axis direction (i.e., the z-axis direction) in the world coordinate system can be perpendicular to the direction of the road and pointing to the sky (or the ground).

[0117] After acquiring the road point cloud data, the road point cloud data may be rasterized according to a preset grid size. For example, the preset grid size may be L meters*M meters*N meters, where L is the length in the x-axis direction, M is the length in the y-axis direction, and N is the length in the z-axis direction. For example, the preset grid size may be 0.16 meters*0.16 meters*15 meters.

[0118] In specific applications, the values ​​corresponding to L, M, and N can be determined according to the parameters of the laser radar actually used, and the embodiments of the present disclosure do not specifically limit them. For example, they can be determined according to the maximum x value, maximum y value, and maximum z value in the coordinates of the point cloud points in the road point cloud data acquired by the laser radar.

[0119] In specific implementation, the number of divided target grids and the target grid where each point cloud point is located can be determined according to the preset grid size and the coordinates of each point cloud point in the road point cloud data in the world coordinate system, and each point cloud point located in the same target grid is regarded as the local point cloud data contained in the target grid.

[0120] Based on rasterizing the road point cloud data, at least one target grid and local point cloud data contained in each target grid in the at least one target grid can be obtained.

[0121] S103: Based on the local point cloud data contained in each target grid, determine the probability that the target grid belongs to the traffic marking category.

[0122] Here, the traffic markings may specifically be any markings on the road, for example, the traffic markings may be lane markings, road boundary lines, zebra crossings, turn indicator lines, and the like.

[0123] In specific implementation, for each target grid obtained after division, the probability that the target grid belongs to the traffic marking can be determined based on the point cloud information of each point cloud point in the local point cloud data contained in the target grid. Exemplarily, based on the point cloud information of each point cloud point of the target grid, it can be determined whether each point cloud point belongs to the traffic marking, and based on the ratio of the number of point cloud points belonging to the traffic marking in the target grid to the total number of point cloud points in the target grid, the probability that the target grid belongs to the traffic marking category is determined.

[0124] Alternatively, a first number of point cloud points belonging to traffic markings in the target grid and a second number of point cloud points belonging to the background in the target grid can be determined based on the point cloud information of each point cloud point of the target grid, and the probability that the target grid belongs to the traffic marking category is determined based on the ratio of the first number to the second number.

[0125] In one embodiment, the traffic marking category may include a lane line category and a road boundary line category, that is, the traffic marking may include traffic markings of the lane line category and traffic markings of the road boundary line category. Among them, the traffic markings of the road boundary line category include road boundary lines located at the edges of both sides of the road; the traffic markings of the lane line category are various traffic markings on the road other than traffic markings of the non-road boundary line category.

[0126] For the above S103, it can be implemented according to the following steps:

[0127] S103 - 1 : For the local point cloud data contained in each target grid, feature extraction is performed on the point cloud information of each point cloud point in the local point cloud data to generate road feature information of the target grid.

[0128] Here, the road feature information is high-dimensional feature information, for example, 64 dimensions, 128 dimensions, etc. The road feature information is feature information that can characterize whether the target grid is related to the traffic marking category on the road.

[0129] In specific implementation, for each target grid, feature extraction can be performed on the point cloud information of each point cloud point in the local point cloud data contained in the target grid, and target feature information related to the traffic marking line category in the point cloud information of each point cloud point can be extracted. For example, for each point cloud point, first feature information related to the lane line category can be extracted from the point cloud information of the point cloud point, and second feature information related to the road boundary line category can be extracted from the point cloud information of the point cloud point. Then, the first feature information and the second feature information can be used as the target feature information of the point cloud point.

[0130] Afterwards, the target feature information corresponding to each point cloud point can be fused to obtain the road feature information corresponding to the target grid.

[0131] S103 - 2 : Based on the road feature information, determine a first probability that the target grid belongs to a lane line category and a second probability that the target grid belongs to a road boundary line category.

[0132] Here, in the case where the traffic marking line category includes a lane line category and a road boundary line category, the probability that the target grid belongs to the traffic marking line category may include a first probability and a second probability. The first probability is used to characterize the probability that the point cloud points in the target grid belong to the lane line category; the second probability is used to characterize the probability that the point cloud points in the target grid belong to the road boundary line category.

[0133] In a specific implementation, the road feature information corresponding to the target grid may be subjected to convolution processing, and based on the result of the convolution processing, a first probability that the target grid belongs to a lane line category and a second probability that the target grid belongs to a road boundary line category may be determined.

[0134] In one embodiment, the above S103 can be executed using a trained target neural network. In specific implementation, after obtaining the local point cloud data contained in each target grid, the local point cloud data contained in each target grid can be input into the trained target neural network in a serial manner, and the local point cloud data contained in each target grid can be processed by the target neural network to output a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category. In this way, by inputting the data into the trained target neural network in a serial manner for processing, the processing pressure of the target neural network can be reduced, making the target neural network lighter.

[0135] Exemplarily, for each target grid, after the local point cloud data contained in the target grid is input into the trained target neural network, the feature extractor in the target neural network can be used to first extract the feature of the point cloud information of each point cloud point in the local point cloud data to obtain the high-dimensional road feature information of the target grid, and then feature process the road feature information to output the first probability that the target grid belongs to the lane line category and the second probability that it belongs to the road boundary line category. Specifically, the feature extractor can include two parts, one part is used to extract the road feature information, and the other part is used to feature process the road feature information and output the first probability and the second probability.

[0136] Among them, the part used to extract road feature information may include a fully connected layer, a batch normalization layer, a linear rectification function (ReLU) function layer and a maximum pooling layer. Specifically, the local point cloud data contained in the target grid can be input into the fully connected layer, and the point cloud information of each point cloud point in the local point cloud data can be fully connected to obtain the first intermediate feature information of each point cloud point; then the first intermediate feature information of each point cloud point is input into the batch normalization layer, and the range of the eigenvalue corresponding to each first intermediate feature information is converted to obtain the second intermediate feature information of each point cloud point; then the second intermediate feature information of each point cloud point is input into the ReLU function layer, and the eigenvalue corresponding to each second intermediate feature information is numerically converted using the ReLU function, and the eigenvalues ​​less than 0 are set to 0, and the eigenvalues ​​greater than 0 are retained, thereby obtaining the third intermediate feature information of each point cloud point; finally, the third intermediate feature information of each point cloud point can be input into the maximum pooling layer, and the third intermediate feature information of each point cloud point is feature fused using the maximum pooling layer to obtain the high-dimensional road feature information of the target grid.

[0137] The part of the feature extractor used to output the first probability and the second probability may also include multiple network layers, wherein the multiple network layers may be a 2D convolution layer, a batch normalization layer, a ReLU function layer, an upsampling layer, and a sigmoid activation function layer, and the multiple network layers constitute a full convolution network. The sigmoid function is used for the output of hidden layer neurons, and the value range is (0, 1). It can map a value to the interval of (0, 1) and can be used for binary classification.

[0138] After obtaining the road feature information of the target grid, the road feature information can be input into the 2D convolution layer, and the road feature information can be 2D convolved to obtain the fourth intermediate feature information; the fourth intermediate feature information is then input into the batch normalization layer, and the range of the eigenvalues ​​corresponding to the fourth intermediate feature information is converted to obtain the fifth intermediate feature information; the fifth intermediate feature information is input into the ReLU function layer, and the eigenvalues ​​corresponding to the fifth intermediate feature information are numerically converted using the ReLU function to obtain the sixth intermediate feature information; the sixth intermediate feature information is input into the upsampling layer, and the upsampling layer is used to upsample the sixth intermediate feature information to obtain the seventh intermediate feature information; finally, the seventh intermediate feature information is input into the sigmoid activation function layer, and the sigmoid activation function is used to perform feature processing on the seventh intermediate feature information, and the first probability map of the target grid belonging to the lane line category and the second probability map of the road boundary line category are output. Among them, the probability intervals corresponding to the first probability map and the second probability map are both (0, 1). According to the first probability map, the first probability of the target grid belonging to the lane line category is obtained, and according to the second probability map, the second probability of the target grid belonging to the road boundary line category is obtained.

[0139] In this way, the trained target neural network has reliable prediction accuracy. By using the trained target neural network, the probability that the target grid belongs to each category of traffic markings can be accurately determined.

[0140] In one possible manner, the method for recognizing traffic markings provided in the embodiments of the present disclosure can also be directly executed using a trained target neural network, that is, the trained target neural network can be used to execute the above S101 to S103 and the following S104, and the trained target neural network can be used to directly output the marking recognition result.

[0141] S104: Determine a traffic marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each target grid belongs.

[0142] Here, the marking recognition result is used to indicate the marking information of at least one traffic marking corresponding to the road point cloud data. Specifically, the marking recognition result can indicate the marking information of each lane line corresponding to the road point cloud data and / or the marking information of each road boundary line corresponding to the road point cloud data. The marking information may include the marking position and / or marking category of the traffic marking.

[0143] Exemplarily, after obtaining the probability of the traffic marking category to which each target grid belongs, when the traffic marking category includes only one, the probability corresponding to the target grid will also include only one, and then the marking grid with a probability greater than the preset probability can be screened out from the multiple target grids. Based on the position information of each screened marking grid in the world coordinate system, the connected marking grids are determined, and then a traffic marking can be determined based on the coordinates of the point cloud points in the connected marking grids, and the marking position of the traffic marking can be determined based on the coordinates of the point cloud points in the connected marking grids. In this way, based on the probabilities corresponding to each target grid, each traffic marking corresponding to the road point cloud data and the marking information can be determined, that is, the marking recognition result is obtained.

[0144] Exemplarily, after obtaining the probabilities of the traffic marking categories to which each target grid belongs, when the traffic marking categories include multiple ones, the probability corresponding to the target grid will also include the probabilities corresponding to the traffic marking categories. For example, when the traffic marking categories include lane line categories and road boundary line categories, the probability corresponding to the target grid may include a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category. When the traffic marking categories include multiple ones, for each traffic marking category, the category probability corresponding to the traffic marking category can be screened out from the probability corresponding to each target grid. For each target grid, it can be determined whether the category probability corresponding to the target grid is greater than the preset probability. If so, the traffic marking category indicated by the category probability is used as the traffic marking category to which the target grid belongs. If not, it is determined that the target grid does not belong to the traffic marking category. Based on this, the traffic marking category to which each target grid belongs can be determined.

[0145] Afterwards, for each target grid belonging to the same traffic marking category, the target grids with connectivity can be determined based on the position information of each target grid in the world coordinate system, and then a traffic marking can be determined based on the coordinates of the point cloud points in each target grid with connectivity. The marking position of the traffic marking is determined based on the coordinates of the point cloud points in each target grid corresponding to the traffic marking; at the same time, the traffic marking category associated with the category probability of each target grid corresponding to the traffic marking can be used as the marking category of the traffic marking. In this way, traffic markings of various categories corresponding to the road point cloud data and the marking information of each traffic marking can be obtained.

[0146] In this way, since the road point cloud data photographed by the laser radar has morphological stability, the road point cloud data obtained can be used to identify the traffic marking category, which can improve the continuity of the identified traffic markings. By rasterizing the road point cloud data and then processing the local point cloud data corresponding to each target grid obtained after rasterization, the probability that each target grid belongs to the traffic marking category can be determined. Based on the probability corresponding to each target grid, each target grid belonging to the traffic marking category can be accurately determined from multiple target grids. Then, using each target grid belonging to the traffic marking category, each point cloud point in the road point cloud data belonging to the traffic marking category can be accurately determined, and then each category of traffic markings, that is, the marking recognition result, can be accurately obtained.

[0147] In one embodiment, with respect to S104, when the traffic marking line category includes a lane line category and a road boundary line category, and the probability of the traffic marking line category to which the target grid belongs includes a first probability and a second probability, S104 may be implemented according to the following steps:

[0148] S104-1: For each target grid, determine whether there is a target probability greater than a preset threshold in the first probability and the second probability of the target grid.

[0149] Here, the preset threshold is a preset minimum probability value, and the lane line category and the road boundary line category can correspond to the same preset threshold. The preset threshold can be set based on experience and is not specifically limited here. When the first probability is greater than the preset threshold, it can be determined that the target grid has a greater probability of belonging to the traffic marking category associated with the first probability; similarly, when the second probability is greater than the preset threshold, it can be determined that the target grid has a greater probability of belonging to the traffic marking category associated with the second probability.

[0150] The target probability is a probability between the first probability and the second probability that is greater than a preset threshold.

[0151] In specific implementation, for each target grid, the first probability and the second probability of the target grid can be compared with the preset thresholds respectively, so as to determine whether there is a target probability in the first probability and the second probability. If yes, the following S104-1 can be executed. If not, that is, the first probability and the second probability of the target grid are not greater than the preset threshold, it can be determined that the target grid belongs to the background grid, that is, it can be determined that each point cloud point in the local point cloud data contained in the target grid is a point cloud point of the background category.

[0152] In one embodiment, the preset threshold may also include two, specifically a first preset threshold corresponding to the lane line category and a second preset threshold corresponding to the road boundary line category.

[0153] With respect to the step of "determining whether there is a target probability greater than a preset threshold value in the first probability and the second probability of the target grid" in the above S104-1, the first probability can be compared with the first preset threshold value. If the first probability is greater than the first preset threshold value, it is determined that the first probability belongs to the target probability; otherwise, it is determined that the first probability does not belong to the target probability. At the same time, the second probability can be compared with the second preset threshold value. If the second probability is greater than the second preset threshold value, it is determined that the second probability belongs to the target probability; otherwise, it can be determined that the second probability does not belong to the target probability.

[0154] In this way, by setting different preset thresholds for different road marking categories, and then using the preset thresholds corresponding to each road marking category to screen the target probability, the rationality and accuracy of the determined target probability can be improved.

[0155] In addition, when the first probability is not greater than the first preset threshold and the second probability is not greater than the second preset threshold, it can be determined that the target grid belongs to the background grid, that is, it can be determined that each point cloud point in the local point cloud data contained in the target grid is a point cloud point of the background category.

[0156] S104-2: When there is a target probability, determine the recognition result corresponding to the target grid according to the traffic line category associated with the target probability.

[0157] Here, when the target probability is the first probability, the traffic marking category associated with the target probability is the lane line category; when the target probability is the second probability, the traffic marking category associated with the target probability is the road boundary line category. The recognition result is used to indicate the traffic marking category to which the target grid belongs.

[0158] In specific implementation, the recognition result corresponding to the target grid can be determined according to the traffic marking category associated with the target probability. For example, when the traffic marking category associated with the target probability is the lane line category, the recognition result corresponding to the target grid is: the target grid belongs to the traffic marking category, that is, each point cloud point in the local point cloud data contained in the target grid is a point cloud point of the traffic marking category. For another example, when the traffic marking category associated with the target probability is the road boundary line category, the recognition result corresponding to the target grid is: the target grid belongs to the road boundary line category, that is, each point cloud point in the local point cloud data contained in the target grid is a point cloud point of the road boundary line category.

[0159] In one embodiment, in the process of using the same preset threshold corresponding to the lane line category and the road boundary line category to filter the target probability, or using the first preset threshold and the second preset threshold to filter the target probability, there is a situation where the target probability includes both the first probability and the second probability. In this case, for the above S104-2, the maximum probability can be filtered out from the first probability and the second probability included in the target probability. Then, the traffic marking category associated with the maximum probability is used as the recognition result of the target grid. That is, the traffic marking category associated with the maximum probability is used as the traffic marking category to which the target grid belongs. In this way, the uniqueness of the obtained target probability can be guaranteed, and the rationality of the obtained recognition result can be improved.

[0160] S104-3: Based on the recognition results of each target grid, determine the line marking recognition result of the road point cloud data.

[0161] In specific implementation, the target grids belonging to the same traffic marking category can be determined according to the recognition results corresponding to the target grids. Then, the target grids with connectivity can be determined according to the coordinates of the point cloud points in the local point cloud data respectively contained in the target grids belonging to the same traffic marking category, and a traffic marking with the traffic marking category can be determined according to the coordinates of the point cloud points in the target grids with connectivity. In addition, the marking position of each traffic marking can be determined based on the coordinates of the point cloud points in the target grids corresponding to each traffic marking.

[0162] Then, for each traffic marking, the traffic marking category and the marking position of the traffic marking can be used as the marking information of the traffic marking. Finally, the marking recognition result of the road point cloud data can be determined according to the marking information of each traffic marking.

[0163] In one embodiment, the road point cloud data may be collected by a driving device. Specifically, the road point cloud data may be collected by a laser radar installed on the driving device. After obtaining the line marking recognition result of the road point cloud data, the driving device may be controlled to travel based on the line marking information of at least one traffic line marking indicated by the line marking recognition result, wherein the line marking information includes the line marking position and / or the line marking category.

[0164] Here, the driving device may include any device that can travel on the road, such as an autonomous vehicle, a manually driven vehicle, a robot, etc. The marking position is used to characterize the position of the traffic marking in the world coordinate system. The marking category is the traffic marking category to which the traffic marking belongs, for example, the lane line category and the road boundary line category. The marking information may include but is not limited to the marking position and / or the marking category, for example, it may also include the distance of the traffic marking from the driving device, the angle between the direction of the traffic marking and the driving direction of the driving device, etc.

[0165] For example, after obtaining the line marking recognition result, each traffic line marking indicated by the line marking recognition result, as well as the line marking position and line marking type of each traffic line marking can be determined. According to the line marking position and line marking type of each traffic line marking, a safe driving area is determined, and the driving device is controlled to drive in the safe driving area.

[0166] For another example, when it is determined based on the position of the driving device and the position of the road boundary line marking that the distance between the driving device and the road boundary line is less than a preset distance, an alarm prompt may be given, such as a voice alarm prompt, a buzzer alarm prompt, and the like.

[0167] In this way, by controlling the driving of the traveling device through the determined line marking position and / or line marking type, it is possible to ensure that the traveling device drives in a reasonable area and improve the driving safety of the traveling device.

[0168] In addition, it can be seen from the above embodiment that the above S103 can be executed using the trained target neural network. Therefore, the embodiment of the present disclosure also provides a method for training a neural network to be trained, such as Figure 2 As shown, a flowchart of a method for training a neural network to be trained provided by an embodiment of the present disclosure may include the following steps:

[0169] S201: Obtain sample point cloud data.

[0170] Here, the sample point cloud data may be road point cloud data collected by any laser radar. The sample point cloud data may be a sample point cloud vector set, and the sample point cloud vector set includes point cloud information of multiple sample point cloud points. The point cloud information may include the coordinates of the sample point cloud points in a three-dimensional world coordinate system, the color information of the sample point cloud points, the reflection intensity information, the distance information, etc.

[0171] S202: Rasterize and divide the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determine the annotation label information of each sample grid.

[0172] Here, the label information may specifically be a label value corresponding to the sample grid. When the sample grid belongs to the lane line category, the label information may be a first label value corresponding to the lane line category; when the sample grid belongs to the road boundary line category, the label information may be a second label value corresponding to the road boundary line category; when the sample grid belongs to the background category, the label information may be a third label value corresponding to the background category.

[0173] In specific implementation, the number of divided sample grids and the sample grid where each sample point cloud point is located can be determined according to the preset grid size and the coordinates of each sample point cloud point in the sample point cloud data in the world coordinate system, and each sample point cloud point located in the same sample grid is used as the local sample point cloud data contained in the sample grid. At the same time, for each sample grid, the annotation label information corresponding to the sample grid can be predetermined.

[0174] In one embodiment, the label information of each sample grid may be determined according to the following steps:

[0175] Step 1: Generate a sample top view based on the local sample point cloud data contained in each sample grid; wherein each sample grid corresponds to a pixel point in the sample top view.

[0176] Exemplarily, after the sample point cloud data is rasterized and divided to obtain each sample grid, each sample grid can be projected to obtain a sample top view. The number of pixels in the sample top view is the number of sample grids, and one sample grid corresponds to one pixel in the sample top view. The pixel information of the pixel can be determined based on the point cloud information of each sample point cloud point in the local sample point cloud data contained in the sample grid. For example, pixel 1 corresponds to sample grid 1, and the pixel information of pixel 1 can be determined based on the point cloud information of each sample point cloud point in the local sample point cloud data contained in sample grid 1.

[0177] Step 2: Based on the pixel information of each pixel point in the sample top view, determine the annotation label information of the sample grid matching each pixel point.

[0178] Here, the sample grid that matches the pixel point is the sample grid corresponding to the pixel point.

[0179] In specific implementation, after obtaining the sample top view, for each pixel in the sample top view, the semantic label of the pixel can be determined based on the pixel information of the pixel, wherein the semantic label can include a lane label, a road boundary label and a background label.

[0180] The lane line label indicates the lane line category, the road boundary line label indicates the road boundary line category, and the background label indicates the background category. Different semantic labels correspond to different label values. Specifically, the lane line label corresponds to the first label value, the road boundary line label corresponds to the second label value, and the background label corresponds to the third label value.

[0181] Afterwards, the label information of the sample grids matching each pixel can be determined based on the label value corresponding to the semantic label of each pixel. For example, for any pixel, the label value corresponding to the semantic label of the pixel can be used as the label information of the sample grid corresponding to the pixel.

[0182] In one embodiment, the above step 2 may be implemented according to the following steps:

[0183] S1: Based on the pixel information of each pixel point in the sample top view, determine the annotation label information corresponding to each pixel point in the sample top view.

[0184] In a specific implementation, for each pixel, the semantic label of the pixel can be determined according to the pixel information of the pixel in a manual labeling manner, and then the label value corresponding to the semantic label of the pixel can be used as the label information of the pixel. The label information of the pixel is the label information corresponding to the pixel.

[0185] S2: For a target pixel whose label information indicates that the target pixel is a traffic line category, based on a preset expansion width, the label information of the neighboring pixels of the target pixel is adjusted to the label information of the target pixel.

[0186] Here, the traffic label category may include a lane line category and a road boundary line category. The preset extension width may be determined according to the number of pixels located on the left and right sides of the target pixel point that need to be used. For example, for any target pixel point, if it is necessary to adjust the label information of a pixel point adjacent to the left and right sides of the target pixel point (that is, a pixel point adjacent to the left side of the target pixel point and a pixel point adjacent to the right side of the target pixel point), the preset extension width may be 3 pixels wide.

[0187] In a specific implementation, after obtaining the label information of each pixel, each target pixel belonging to the traffic marking category can be determined according to the label information of each pixel. Then, for each target pixel, the two adjacent pixels located on the left and right sides of the target pixel and adjacent to the target pixel can be determined using a preset expansion width. Afterwards, the label information of the two adjacent pixels can be adjusted to the label information of the target pixel. Here, if the label information of the adjacent pixels of the target pixel is the same as the label information of the target pixel, the label information of the adjacent pixels may not be adjusted.

[0188] S3: Determine the label information of the sample grid matching each pixel point based on the adjusted label information of the adjacent pixels in the sample top view and the unadjusted label information of other pixels except the adjacent pixels.

[0189] In a specific implementation, the adjusted label information corresponding to each adjacent pixel point with adjusted label information in the sample top view can be used as the label information of the sample grid that matches each adjacent pixel point with adjusted label information. At the same time, the unadjusted label information of other pixel points in the sample top view, except for each adjacent pixel point with adjusted label information, can be used as the label information of the sample grid that matches other pixel points. Among them, the other pixel points may include the pixel points with unadjusted label information.

[0190] In a possible implementation, after obtaining the sample overhead view, each lane line and each road boundary line in the sample overhead view can be determined based on the semantic label of each pixel point. Afterwards, for any lane line or any road boundary line, the corresponding width can be 1 pixel wide. Afterwards, the preset expansion width can be used to expand the width of each lane line and each road boundary line. That is, for each pixel point in each lane line, the adjacent pixel points on the left and right sides of the pixel point can be used as added pixel points in the lane line, so as to achieve the width expansion of each lane line and obtain the expanded lane line. Afterwards, the annotation label information of the sample grid corresponding to each pixel point in the expanded lane line can be set to the first label value corresponding to the lane line category.

[0191] For each pixel point in each road boundary line, the adjacent pixel points on the left and right sides of the pixel point can be used as additional pixel points in the road boundary line, so as to expand the width of each road boundary line and obtain the expanded road boundary line. Afterwards, the annotation label information of the sample grid corresponding to each pixel point in the expanded road boundary line can be set to the second label value corresponding to the road boundary line category.

[0192] The label information of the sample grids corresponding to the pixel points that are not located on the expanded lane line and the expanded road boundary line are set to the third label value corresponding to the background category.

[0193] S203: Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, and generating a prediction probability that the sample grid belongs to the traffic marking category.

[0194] Here, the neural network to be trained is the target neural network to be trained. The predicted probability is the probability output by the neural network to be trained that the sample grid belongs to the traffic marking category.

[0195] In specific implementation, the local sample point cloud data contained in each sample grid can be input into the neural network to be trained in turn, and the local sample point cloud data can be processed by the neural network to be trained to output the predicted probability that the sample grid belongs to the traffic marking category.

[0196] S204: Based on the predicted probability of the traffic line category to which each sample grid belongs and the annotation label information of each sample grid, the neural network to be trained is iteratively trained until a training cutoff condition is met to obtain a target neural network.

[0197] Here, the training cutoff condition may include that the number of iterative training rounds reaches a preset number of rounds, and / or the prediction accuracy of the trained neural network reaches a preset accuracy.

[0198] In specific implementation, the prediction loss of the neural network to be trained can be determined based on the predicted probability that the sample grid belongs to the traffic marking category and the annotation label information of the sample grid. The prediction loss is used to iteratively train the neural network to be trained until the training cutoff condition is met to obtain the target neural network.

[0199] In one embodiment, the prediction probability may include a first prediction probability that the sample grid belongs to a lane line category, and a second prediction probability that the sample grid belongs to a road boundary line category.

[0200] In specific implementation, the above S204 can be implemented according to the following steps:

[0201] When the labeled label information of the sample grid indicates that the sample grid does not belong to the background, that is, when the labeled label information of the sample grid indicates that the sample grid does not belong to the background category, the first loss can be determined based on the first predicted probability of the sample grid, the second predicted probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label.

[0202] In a specific implementation, the first loss may include a first sub-loss and a second sub-loss. The step of determining the first loss may be implemented according to the following sub-steps:

[0203] Sub-step 1: Determine, from the first prediction probability and the second prediction probability of the sample grid, the target prediction probability that matches the traffic line category indicated by the annotated label information of the sample grid.

[0204] Among them, when the traffic marking category indicated by the labeled label information is the lane line category, the target prediction probability is the first prediction probability corresponding to the sample grid, and the label value corresponding to the labeled label information is the first label value; when the traffic marking category indicated by the labeled label information is the road boundary line category, the target prediction probability is the second prediction probability corresponding to the sample grid, and the label value corresponding to the labeled label information is the second label value.

[0205] Exemplarily, for each sample grid, the traffic line category to which the sample grid actually belongs can be determined based on the annotated label information of the sample grid. Then, the target prediction probability matching the traffic line category to which the sample grid actually belongs is determined from the first prediction probability and the second prediction probability of the sample grid.

[0206] Sub-step 2: Determine the first sub-loss based on the target prediction probability and the labeled label information of the sample grid.

[0207] In specific implementation, a binary cross entropy function can be used to determine the first sub-loss based on the target prediction probability and the label value corresponding to the annotated label information.

[0208] Sub-step three: determining a second sub-loss based on the other prediction probabilities except the target prediction probability in the first prediction probability and the second prediction probability, and the annotated label information corresponding to the background label.

[0209] Here, the background label is the background category, and the annotated label information corresponding to the background label is the third label value. Specifically, the second sub-loss can be determined based on the other predicted probabilities and the third label value corresponding to the annotated label information of the background label using a binary cross entropy function.

[0210] Exemplarily, when the traffic marking category indicated by the label information of the sample grid is a lane line category, the target prediction probability is the first prediction probability, and the other prediction probabilities are the second prediction probabilities. Afterwards, the first sub-loss can be determined based on the first prediction probability and the first label value using a binary cross entropy function. At the same time, the second sub-loss can be determined based on the second prediction probability and the third label value using a binary cross entropy function.

[0211] Exemplarily, when the traffic marking category indicated by the label information of the sample grid is a road boundary category, the target prediction probability is the second prediction probability, and the other prediction probabilities are the first prediction probabilities. Afterwards, the first sub-loss can be determined based on the second prediction probability and the second label value using a binary cross entropy function. At the same time, the second sub-loss can be determined based on the first prediction probability and the third label value using a binary cross entropy function.

[0212] Finally, the obtained first sub-loss and second sub-loss can be used as the first loss.

[0213] When the annotated label information of the sample grid indicates that the sample grid belongs to the background, that is, when the annotated label information of the sample grid indicates that the sample grid belongs to the background category, the second loss can be determined based on the first prediction probability, the second prediction probability and the annotated label information corresponding to the background label.

[0214] Here, the second loss may include a third sub-loss and a fourth sub-loss. In specific implementation, the third sub-loss may be determined based on the first predicted probability and the third label value corresponding to the sample grid using a binary cross entropy function; and the fourth sub-loss may be determined based on the second predicted probability and the third label value corresponding to the sample grid using a binary cross entropy function. The third sub-loss and the fourth sub-loss are used as the second loss.

[0215] Furthermore, based on at least one of the first loss and the second loss, the neural network to be trained can be iteratively trained until a training cutoff condition is met to obtain a target neural network.

[0216] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0217] Based on the same inventive concept, the embodiment of the present disclosure also provides a traffic marking recognition device corresponding to the traffic marking recognition method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned traffic marking recognition method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0218] like Figure 3 FIG. 1 is a schematic diagram of a traffic marking recognition device provided by an embodiment of the present disclosure, comprising:

[0219] An acquisition module 301 is used to acquire road point cloud data;

[0220] A division module 302 is used to perform raster division on the road point cloud data to obtain local point cloud data contained in at least one target grid;

[0221] A first determination module 303, configured to determine the probability that the target grid belongs to a traffic marking category based on the local point cloud data contained in each target grid;

[0222] The second determination module 304 is used to determine the traffic marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each of the target grids belongs.

[0223] In a possible implementation, the traffic marking category includes a lane line category and a road boundary line category; the first determination module 303, when determining the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid, is used to extract features from the point cloud information of each point cloud point in the local point cloud data contained in each target grid, so as to generate road feature information of the target grid;

[0224] Based on the road feature information, a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category are determined.

[0225] In a possible implementation manner, the second determination module 304, when determining the line marking recognition result of the road point cloud data based on the probability of the traffic line marking category to which each of the target grids belongs, is used to determine, for each of the target grids, whether there is a target probability greater than a preset threshold in the first probability and the second probability of the target grid;

[0226] In the case where the target probability exists, determining the recognition result of the target grid according to the traffic line category associated with the target probability;

[0227] Based on the recognition results of each of the target grids, a line marking recognition result of the road point cloud data is determined.

[0228] In a possible implementation manner, the second determination module 304, when determining the recognition result of the target grid according to the traffic marking category associated with the target probability, is used to determine the maximum probability of the first probability and the second probability when the target probability includes the first probability and the second probability;

[0229] The traffic marking category associated with the maximum probability is used as the recognition result of the target grid.

[0230] In a possible implementation manner, the road point cloud data is collected by a driving device, and the device further includes:

[0231] The control module 305 is used to control the driving device to drive based on the marking information of at least one traffic marking indicated by the marking recognition result after determining the marking recognition result of the road point cloud data, wherein the marking information includes the marking position and / or marking category.

[0232] In a possible implementation, the first determination module 303, when determining the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid, is used to use a trained target neural network to determine the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid.

[0233] In a possible implementation, the device further includes:

[0234] The training module 306 is used to train the target neural network according to the following steps:

[0235] Get sample point cloud data;

[0236] Rasterizing the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determining the annotation label information of each sample grid;

[0237] Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, and generating a predicted probability that the sample grid belongs to the traffic marking category;

[0238] Based on the predicted probability of the traffic line category to which each of the sample grids belongs and the annotated label information of each of the sample grids, the neural network to be trained is iteratively trained until a training cutoff condition is met to obtain the target neural network.

[0239] In a possible implementation, the prediction probability includes a first prediction probability that the sample grid belongs to a lane line category, and a second prediction probability that the sample grid belongs to a road boundary line category;

[0240] The training module 306, when iteratively training the neural network to be trained based on the predicted probability of the traffic line category to which each of the sample grids belongs and the labeled label information of each of the sample grids until a training cutoff condition is satisfied and the target neural network is obtained, is used to determine a first loss based on the first predicted probability of the sample grid, the second predicted probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label when the labeled label information of the sample grid indicates that the sample grid does not belong to the background;

[0241] When the annotated label information of the sample grid indicates that the sample grid belongs to the background, determining a second loss based on the first predicted probability, the second predicted probability and the annotated label information corresponding to the background label;

[0242] Based on at least one of the first loss and the second loss, the neural network to be trained is iteratively trained until a training cutoff condition is met, so as to obtain the target neural network.

[0243] In a possible implementation manner, the first loss includes a first sub-loss and a second sub-loss;

[0244] The training module 306 is used to determine, from the first prediction probability and the second prediction probability of the sample grid, a target prediction probability matching the traffic line category indicated by the labeled label information of the sample grid when determining the first loss based on the first prediction probability of the sample grid, the second prediction probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label;

[0245] Determining a first sub-loss based on the target prediction probability and the labeled label information of the sample grid;

[0246] A second sub-loss is determined based on other prediction probabilities except the target prediction probability in the first prediction probability and the second prediction probability, and the annotated label information corresponding to the background label.

[0247] In a possible implementation, the training module 306, when determining the annotation label information of each sample grid, is used to generate a sample top view based on the local sample point cloud data respectively contained in each sample grid; wherein each sample grid corresponds to a pixel point in the sample top view;

[0248] Based on the pixel information of each pixel point in the sample top view, the annotation label information of the sample grid matching each pixel point is determined.

[0249] In a possible implementation, the training module 306, when determining the annotation label information of the sample grid matching each pixel point based on the pixel information of each pixel point in the sample top view, is used to determine the annotation label information corresponding to each pixel point in the sample top view based on the pixel information of each pixel point in the sample top view;

[0250] For a target pixel point indicated by the annotated label information as a traffic line category, based on a preset expansion width, the annotated label information of the neighboring pixel points of the target pixel point is adjusted to the annotated label information of the target pixel point;

[0251] Based on the adjusted label information of the adjacent pixels in the sample top view and the unadjusted label information of other pixels except the adjacent pixels, the label information of the sample grid matching each pixel is determined.

[0252] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0253] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 4 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application, including:

[0254] Processor 41, memory 42 and bus 43. The memory 42 stores machine-readable instructions executable by the processor 41, and the processor 41 is used to execute the machine-readable instructions stored in the memory 42. When the machine-readable instructions are executed by the processor 41, the processor 41 executes the following steps: S101: Acquire road point cloud data; S102: Rasterize the road point cloud data to obtain local point cloud data contained in at least one target grid; S103: Based on the local point cloud data contained in each target grid, determine the probability that the target grid belongs to the traffic marking category; S104: Based on the probability of the traffic marking category to which each target grid belongs, determine the marking recognition result of the road point cloud data.

[0255] The above-mentioned memory 42 includes internal memory 421 and external memory 422; the memory 421 here is also called internal memory, which is used to temporarily store the calculation data in the processor 41, as well as the data exchanged with the external memory 422 such as the hard disk. The processor 41 exchanges data with the external memory 422 through the internal memory 421. When the computer device is running, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes the execution instructions mentioned in the above-mentioned method embodiment.

[0256] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the traffic marking recognition method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0257] The computer program product of the method for identifying traffic markings provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the method for identifying traffic markings described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0258] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0259] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0260] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0261] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0262] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0263] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0264] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for identifying traffic markings, It is characterized in that include: Obtain road point cloud data; Rasterizing the road point cloud data to obtain local point cloud data contained in at least one three-dimensional target grid; Using the trained target neural network, based on the local point cloud data contained in each target grid, determine the probability that the target grid belongs to the traffic marking category; Based on the probability of the traffic marking category to which each of the target grids belongs, the marking recognition result of the road point cloud data is determined; the target neural network is trained according to the following steps: Acquire sample point cloud data; perform rasterization on the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determine the annotation label information of each sample grid; Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, generating a prediction probability that the sample grid belongs to the traffic marking category; the prediction probability includes a first prediction probability that the sample grid belongs to the lane line category, and a second prediction probability that the sample grid belongs to the road boundary line category; When the labeled label information of the sample grid indicates that the sample grid does not belong to the background, a first loss is determined based on the first predicted probability of the sample grid, the second predicted probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label; when the labeled label information of the sample grid indicates that the sample grid belongs to the background, a second loss is determined based on the first predicted probability, the second predicted probability, and the labeled label information corresponding to the background label; Based on the first loss and the second loss, the neural network to be trained is iteratively trained until a training cutoff condition is met, thereby obtaining the target neural network.

2. The method according to claim 1, It is characterized in that The traffic marking line category includes a lane line category and a road boundary line category; the determining, based on the local point cloud data contained in each target grid, the probability that the target grid belongs to the traffic marking line category includes: For the local point cloud data contained in each of the target grids, feature extraction is performed on the point cloud information of each point cloud point in the local point cloud data to generate road feature information of the target grid; Based on the road feature information, a first probability that the target grid belongs to the lane line category and a second probability that the target grid belongs to the road boundary line category are determined.

3. The method according to claim 2, It is characterized in that The determining the road marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each of the target grids belongs includes: For each of the target grids, determining whether there is a target probability greater than a preset threshold value in the first probability and the second probability of the target grid; In the case where the target probability exists, determining the recognition result of the target grid according to the traffic line category associated with the target probability; Based on the recognition results of each of the target grids, a line marking recognition result of the road point cloud data is determined.

4. The method according to claim 3, It is characterized in that The step of determining the recognition result of the target grid according to the traffic marking category associated with the target probability includes: In a case where the target probability includes the first probability and the second probability, determining a maximum probability of the first probability and the second probability; The traffic marking category associated with the maximum probability is used as the recognition result of the target grid.

5. The method according to any one of claims 1 to 4, It is characterized in that The road point cloud data is collected by a driving device, and after determining the line marking recognition result of the road point cloud data, the method further includes: The driving device is controlled to travel based on the line marking information of at least one traffic line marking indicated by the line marking recognition result, wherein the line marking information includes a line marking position and / or a line marking category.

6. The method according to claim 1, It is characterized in that The first loss includes a first sub-loss and a second sub-loss; The determining a first loss based on the first prediction probability of the sample grid, the second prediction probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label includes: Determining, from the first predicted probability and the second predicted probability of the sample grid, a target predicted probability that matches the traffic line category indicated by the annotated label information of the sample grid; Determining a first sub-loss based on the target prediction probability and the labeled label information of the sample grid; A second sub-loss is determined based on other prediction probabilities except the target prediction probability in the first prediction probability and the second prediction probability, and the annotated label information corresponding to the background label.

7. The method according to claim 1, It is characterized in that The step of determining the label information of each sample grid includes: Generate a sample top view based on the local sample point cloud data respectively contained in each of the sample grids; wherein each of the sample grids corresponds to a pixel point in the sample top view; Based on the pixel information of each pixel point in the sample top view, the annotation label information of the sample grid matching each pixel point is determined.

8. The method according to claim 7, It is characterized in that The step of determining the label information of the sample grid matching each pixel point based on the pixel information of each pixel point in the sample top view includes: Determine, based on pixel information of each pixel point in the sample top view, annotation label information corresponding to each pixel point in the sample top view; For a target pixel point indicated by the annotated label information as a traffic line category, based on a preset expansion width, the annotated label information of the neighboring pixel points of the target pixel point is adjusted to the annotated label information of the target pixel point; Based on the adjusted label information of the adjacent pixels in the sample top view and the unadjusted label information of other pixels except the adjacent pixels, the label information of the sample grid matching each pixel is determined.

9. A traffic marking recognition device, It is characterized in that include: Acquisition module, used to acquire road point cloud data; A division module, used for performing raster division on the road point cloud data to obtain local point cloud data contained in at least one three-dimensional target grid; A first determination module is used to determine the probability that the target grid belongs to the traffic marking category based on the local point cloud data contained in each target grid by using the trained target neural network; A second determination module is used to determine the marking recognition result of the road point cloud data based on the probability of the traffic marking category to which each of the target grids belongs; the device also includes: The training module is used to train the target neural network according to the following steps: Acquire sample point cloud data; perform rasterization on the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determine the annotation label information of each sample grid; Inputting the local sample point cloud data contained in each sample grid into the neural network to be trained, generating a prediction probability that the sample grid belongs to the traffic marking category; the prediction probability includes a first prediction probability that the sample grid belongs to the lane line category, and a second prediction probability that the sample grid belongs to the road boundary line category; When the labeled label information of the sample grid indicates that the sample grid does not belong to the background, a first loss is determined based on the first predicted probability of the sample grid, the second predicted probability, the labeled label information of the sample grid, and the labeled label information corresponding to the background label; when the labeled label information of the sample grid indicates that the sample grid belongs to the background, a second loss is determined based on the first predicted probability, the second predicted probability, and the labeled label information corresponding to the background label; Based on the first loss and the second loss, the neural network to be trained is iteratively trained until a training cutoff condition is met, thereby obtaining the target neural network.

10. A computer device, It is characterized in that include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the traffic marking recognition method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the traffic marking recognition method according to any one of claims 1 to 8.

Images