Lane line recognition method, device and computer-readable storage medium

Abstract

The present application discloses a lane line recognition method, device and computer-readable storage medium, the method comprising: based on a lane line recognition model, detecting a road image to be detected to obtain detection data; processing the detection data to obtain position information of each lane line in the road image to be detected and first multivariate attribute information corresponding to each lane line, the first multivariate attribute information including multiple sub-attribute information; the lane line recognition model is trained based on a multi-label dataset, the multi-label dataset includes a sample image, and multivariate attribute labels corresponding to the lane lines in the sample image, the multivariate attribute labels are obtained by decoupling and reconstructing the full-content category labels corresponding to the lane lines in the sample image, and the categories of the multivariate attribute labels are smaller than the categories of the full-content category labels. In the above manner, the present application can improve the accuracy of lane line recognition, and at the same time improve the richness of the types of recognized lane lines.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lane line recognition method, device, and computer-readable storage medium. Background Art

[0002] Currently, assisted driving and autonomous driving are gradually coming into the public eye. Lane recognition technology, as a foundational technology in these fields, predicts the position and category information of different lane lines on the road surface, providing support for subsequent upstream designs in the autonomous driving field, such as autonomous navigation, deviation warning, planning, or control. However, existing lane recognition methods are often prone to confusion when predicting similar categories that are easily coupled, resulting in low lane recognition accuracy. Summary of the Invention

[0003] The present application provides a lane line recognition method, device and computer-readable storage medium, which can improve the accuracy of lane line recognition and increase the richness of the types of recognized lane lines.

[0004] In order to solve the above technical problems, the technical solution adopted in this application is: to provide a lane line recognition method, the method comprising: based on a lane line recognition model, detecting a road image to be detected to obtain detection data; processing the detection data to obtain the position information of each lane line in the road image to be detected and the first multivariate attribute information corresponding to each lane line, the first multivariate attribute information including multiple sub-attribute information; wherein, the lane line recognition model is trained based on a multi-label dataset, the multi-label dataset includes a sample image and a multivariate attribute label corresponding to the lane line in the sample image, the multivariate attribute label is obtained by decoupling and reconstructing the full-content category label corresponding to the lane line in the sample image, and the category of the multivariate attribute label is smaller than the category of the full-content category label.

[0005] In order to solve the above technical problems, another technical solution adopted in this application is: to provide a lane line recognition device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the lane line recognition method in the above technical solution.

[0006] In order to solve the above technical problems, another technical solution adopted in this application is: providing a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the lane line recognition method in the above technical solution.

[0007] Through the above scheme, the beneficial effects of the present application are as follows: when training the lane line recognition model, the present application adopts a multi-label dataset, which includes sample images and multi-attribute labels corresponding to the lane lines in the sample images; the full-content category labels corresponding to the lane lines in the sample images are decoupled and reconstructed to obtain multi-attribute labels, and then the decoupled multi-attribute labels are used to train the lane line recognition model, which solves the problem that similar attribute categories may not be recognized (i.e., category prediction coupling) when training with full-content category labels. By decoupling and reconstructing, each sub-structure attribute in the full-content category label can be extracted to obtain each sub-structure attribute of the lane line, providing richer and more comprehensive lane line category information. , thereby improving the accuracy of lane line recognition. Moreover, since the multi-attribute labels used have fewer categories than the full-content category labels, the number of output channels of the lane recognition model can be reduced, the spatial complexity of the lane recognition model and the time complexity of the prediction can be reduced, thereby improving the efficiency of lane line recognition. When the lane line recognition model is actually applied, the lane line recognition model is used to detect the road image to be detected to obtain detection data. By processing the detection data, the position information of each lane line in the road image to be detected and the first multi-attribute information corresponding to each lane line can be obtained. In this way, not only the position information of the lane line can be identified, but also the multi-attribute information of the lane line can be accurately obtained, and the lane line recognition is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0009] Figure 1 This is a flow chart of an embodiment of a lane line recognition method provided by the present application;

[0010] Figure 2 This is a flow chart of an embodiment of a training method for a lane recognition model provided by the present application;

[0011] Figure 3 This is a flowchart of step 22 provided by this application;

[0012] Figure 4 This is a flow chart of extracting the second substructure attribute tag set provided by this application;

[0013] Figure 5 This is a flow chart of extracting multiple attribute labels provided by this application;

[0014] Figure 6 This is a flowchart of step 23 provided by this application;

[0015] Figure 7 This is a flow chart of another embodiment of the lane recognition method provided by the present application;

[0016] Figure 8 This is a structural diagram of an embodiment of a lane line recognition device provided by the present application;

[0017] Figure 9 It is a structural diagram of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0018] The present application will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only intended to illustrate the present application and are not intended to limit the scope of the present application. Similarly, the following examples are only some examples of the present application and not all examples. All other examples obtained by those of ordinary skill in the art without creative work are intended to fall within the scope of protection of this application.

[0019] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and should not be understood as indicating or suggesting relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units that are inherent to these processes, methods, products or devices.

[0021] See also Figure 1 , Figure 1 : This is a flow chart of an embodiment of a lane line recognition method provided by the present application, which includes:

[0022] Step 11: Detect the road image to be detected based on the lane line recognition model to obtain detection data.

[0023] The road image to be detected contains lane lines to be detected. A lane line recognition model can be used to detect the road image to be detected to obtain corresponding detection data. Specifically, the detection data may include a confidence map of the lane lines or an attribute prediction probability map, etc. The confidence map of the lane lines may be the confidence (i.e., probability) of each pixel point in the road image to be detected on the lane line, and the attribute prediction probability map may be the confidence (i.e., probability) of each pixel point in the road image to be detected for various sub-attributes.

[0024] Furthermore, lane markings can correspond to multiple attribute types, such as color or number of lanes. Each attribute type corresponds to a substructure attribute, and each substructure attribute can contain multiple sub-attributes. For example, if the attribute type is color, the substructure attribute can be {white, yellow}, with {white} and {yellow} being two sub-attributes. It is understandable that, for example, if lane markings correspond to K attribute types, i.e., K substructure attributes, the attribute prediction probability map can contain K probability values ​​corresponding to each pixel.

[0025] Step 12: Process the detection data to obtain the position information of each lane line in the road image to be detected and the first multivariate attribute information corresponding to each lane line.

[0026] By processing the detection data, the position information of each lane line in the road image to be detected and the first multivariate attribute information corresponding to each lane line can be obtained. The first multivariate attribute information includes multiple sub-attribute information. For example, if the lane line is identified as a single white solid line by the lane line recognition model, the obtained first multivariate attribute information corresponding to the lane line includes three sub-attribute information: "white", "single" and "solid".

[0027] The lane line recognition model is trained based on a multi-label dataset. The trained lane line recognition model can be used to detect the road image to be detected to obtain the position information of each lane line and the first multivariate attribute information corresponding to each lane line. The multi-label dataset includes sample images, lane position labels corresponding to the lane lines in the sample images, and multivariate attribute labels. The multivariate attribute labels are obtained by decoupling and reconstructing the full content category labels corresponding to the lane lines in the sample images.

[0028] Specifically, the full-content category label may include the category label corresponding to the combination of all sub-structure attributes of the lane line. By decoupling and reconstructing the full-content category label, the individual sub-structure attributes in the full-content category label can be extracted to obtain a multi-attribute label corresponding to each sub-structure attribute. For example, taking the lane line in the sample image as a single white solid line as an example, its corresponding full-content category label is {"single white solid line"}. The full-content category label can be decoupled and reconstructed to extract each sub-structure attribute in the full-content category label to obtain a multi-attribute label {"single", "white", "solid"}.

[0029] Understandably, the categories of multi-attribute labels are smaller than the categories of full-content category labels. For example, the lane lines are divided into eight categories: "single white dashed line", "single white solid line", "single yellow dashed line", "single yellow solid line", "double white dashed line", "double white solid line", "double yellow dashed line", and "double yellow solid line". The original full-content category label is a multi-type label containing the above eight categories. After decoupling and reconstruction, the sub-structure attributes can be extracted to obtain a multi-attribute label containing three sub-structure attributes: "single / double", "white / yellow", and "virtual / real". Specifically, the number of output channels of the lane line recognition model can be selected according to the number of sub-structure attributes of the lane line, for example: the number of sub-structure attributes For example, if the lane line recognition model has K as its output channel, the number of output channels can be K+1 (i.e., K substructure attribute channels and 1 lane line confidence channel). By decoupling and reconstructing the full content category labels, as in the example above, the original 8-channel classification task can be converted into a 3-channel classification task, thereby reducing the number of output channels of the lane line recognition model and thus reducing the spatial complexity of the lane line recognition model. Moreover, when using the full content category labels for lane line recognition, it is easy to cause confusion between similar categories, such as "single white dashed line" and "single white solid line". By decoupling and reconstructing, each substructure attribute in the full content category label is extracted, which can make lane line recognition more accurate.

[0030] The solution provided in this embodiment can use a multi-label dataset to train a lane recognition model. The multi-label dataset includes sample images and multi-attribute labels corresponding to the lane lines in the sample images. The multi-attribute labels are obtained by decoupling and reconstructing the full-content category labels corresponding to the lane lines in the sample images. The lane recognition model is then trained using the decoupled multi-attribute labels, which solves the problem of category prediction coupling that may arise when training with full-content category labels. Through decoupling and reconstruction, each substructure attribute in the full-content category label is extracted to obtain the substructure attributes of the lane line, providing richer and more comprehensive lane category information while improving the accuracy of lane recognition. Moreover, since the multi-attribute labels used have fewer categories than the full-content category labels, the number of output channels of the lane recognition model can be reduced, reducing the spatial complexity of the lane recognition model and the time complexity of the prediction, further improving recognition efficiency. Therefore, by improving the classification of attributes, the recognition efficiency and accuracy of the lane recognition model can be improved.

[0031] See also Figure 2 , Figure 2 : is a flow chart of an embodiment of a method for training a lane recognition model provided by the present application, the method comprising:

[0032] Step 21: Get training data.

[0033] The training data includes multiple sample images, lane position labels corresponding to the lane lines in the sample images, and full-content category labels. The sample images are images containing lane lines used to train the lane line recognition model. The lane line position labels may include lane line key points, lane line markings, or lane line areas (and area masks). The area mask may be an image containing the values ​​"0" and "1". The value "1" is used to indicate that the corresponding pixel in the image is a pixel on the lane line, and the value "0" is used to indicate that the corresponding pixel in the image is not a pixel on the lane line (for example, it can be ordinary ground, a car, or a building, etc.). The full-content category label of the lane line may include the category label corresponding to the combination of all sub-structure attributes of the lane line (for example, "single white solid line").

[0034] Step 22: Decouple and reconstruct all content category labels to obtain multi-attribute labels.

[0035] All full-content category labels corresponding to all sample images can be decoupled and reconstructed to obtain the multi-attribute labels corresponding to each full-content category label, such as Figure 3 As shown, the steps of decoupling and reconstructing all content category labels to obtain multi-attribute labels include steps 31 to 36:

[0036] Step 31: Get all category label sets.

[0037] The full-category label set includes full-content category labels for all categories. The attribute types of lane lines include but are not limited to color, number, or solid / dashed. In this embodiment, the full-category label set includes the full-content category labels of eight categories: "single white dashed line", "single white solid line", "single yellow dashed line", "single yellow solid line", "double white dashed line", "double white solid line", "double yellow dashed line", and "double yellow solid line" as an example for illustration.

[0038] Step 32: Select a full-content category label from the full-category label set, and perform extraction processing on the full-content category label to obtain the current decoupled label.

[0039] Select a full-content category label from the full-category label set, extract each sub-structure attribute, and thus obtain the current decoupled label; when performing extraction processing on the combination of sub-structure attributes in the full-content category label, split the combination of sub-structure attributes in the full-content category label, and extract all the sub-structure attributes in the full-content category label. The full-content category label can be understood as a string. The process of decoupling (i.e., the above-mentioned extraction processing) the full-content category label is to extract each character unit in the string. For example, if the full-content category label is {"single white solid line"}, its corresponding sub-structure attributes are three, namely "single", "white", and "solid"; specifically, according to the number of sub-structure attributes of the lane line, obtain the current decoupled label containing the corresponding number of sub-structure attributes. The order of selecting the full-content category label from the full-category label set is not limited.

[0040] Step 33: Determine whether the current decoupled label meets the first preset condition.

[0041] The sub-structure attributes in the full-content category label can be extracted sequentially. During the process of extracting the full-content category label, there may be situations where the decoupling is incomplete, such as extracting {"single white solid"} or {"single white"}. At this time, determine whether the current decoupled label meets the first preset condition to determine whether the current decoupled label is completely decoupled; specifically, it can be determined whether the length of the current decoupled label is the preset length. If the length of the current decoupled label is the preset length, it is determined that the current decoupled label meets the first preset condition; where the preset length can be 1. When the length of the current decoupled label generated after decoupling is 1, it can be explained that the decoupling is complete at this time. For example, if the current decoupled label is {"single"}, it means the extraction is successful.

[0042] When the current decoupled label does not meet the first preset condition, it means that the current decoupled label is not the smallest substructure attribute. At this time, the full content category label is updated to the current decoupled label, and the step of extracting the full content category label is returned to continue extracting the full content category label until the current decoupled label meets the first preset condition. After one extraction process, the current decoupled label obtained is "double white real", where "double white real" is a combination of three substructure attributes and is not the smallest substructure attribute. At this time, the extraction process of "double white real" can be continued to extract "double", which is The smallest substructure attribute, and then the remaining "white and real" are extracted to obtain "white" and "real". When all current decoupling tags meet the first preset condition, it can be understood that after each extraction, the obtained smallest substructure attribute can be added to the current decoupling tag to replace the non-minimum substructure attribute extracted before, thereby obtaining the final current decoupling tag {"double", "white", "real"}; specifically, there is no limitation on the order of extracting and processing the substructure attributes in the full content category tag, and the substructure attribute "white" can be extracted first, and then the remaining "double and real" can be extracted.

[0043] Step 34: If the current decoupling tag meets the first preset condition, a first substructure attribute tag set is constructed based on the current decoupling tag.

[0044] When the current decoupled tag meets the first preset condition, a first substructure attribute tag set can be constructed based on the current decoupled tag, and a first substructure attribute tag set containing multiple sub-attribute tags can be constructed based on all current coupled tags extracted from the full content category tags of all categories in the full category tag set, such as: {"double", "white", "real", "single", "yellow", "virtual"}. The following embodiments are all illustrated using the above-mentioned first substructure attribute tag set as an example.

[0045] Step 35: Perform matching processing on the first substructure attribute tag set to obtain a second substructure attribute tag set.

[0046] The sub-attribute tags belonging to the same attribute type in the first sub-structure attribute tag set can be matched to obtain the second sub-structure attribute tag; specifically, the sub-attribute tags in the first sub-structure attribute tag set can be matched with the preset mapping table to determine whether there are other sub-attribute tags corresponding to the sub-attribute tags in the preset mapping table; if there are other sub-attribute tags corresponding to the sub-attribute tags in the preset mapping table, the sub-attribute tags and the other sub-attribute tags are combined to form a second sub-structure attribute tag set.

[0047] The preset mapping table may include all sub-attribute tags under each attribute type. The first sub-structure attribute tag set {"double", "white", "real", "single", "yellow", "virtual"} corresponds to 6 sub-attribute tags. For example, the preset mapping table includes sub-attribute tags such as "white" and "yellow" in the color attribute type. At this time, based on the preset mapping table, the sub-attribute tags "white" and "yellow" representing the color in the first sub-structure attribute tag set can be matched. Similarly, the sub-attribute tags "single" and "double" representing the number of items can be matched, and the sub-attribute tags "real" and "virtual" representing the real and virtual can be matched, thereby obtaining the second sub-structure attribute tag set {"single, double", "white, yellow", "real, virtual"}.

[0048] like Figure 4 As shown, Figure 4 In order to extract and process all category labels, a schematic diagram of the second substructure attribute label set is obtained. The full content category label also contains special sub-attribute labels, such as waiting line or stop line. At this time, the special sub-attribute label itself is a substructure attribute. Therefore, when extracting and processing the full content category label, a special identifier can be used as the substructure attribute identifier of the special sub-attribute and added to the first substructure attribute label set. For example, the waiting line can be identified by "left / right", "left" can be used to indicate that the lane line is a waiting line, and "right" can be used to indicate that the lane line is not a waiting line. The stop line can be identified by "yes / no", "yes" can be used to indicate that the lane line is a stop line, and "right" can be used to indicate that the lane line is not a stop line. The corresponding second substructure attribute label set can include {"single, double", "white, yellow", "real, virtual", "left, right", "yes, no"}.

[0049] In a specific embodiment, during the process of extracting and processing all category tags, it can also be determined whether the full category tag set meets the second preset condition. When the full category tag set meets the second preset condition, the first substructure attribute tag set is matched to obtain the second substructure attribute tag set; when the full category tag set does not meet the second preset condition, the step of selecting a full content category tag from the full category tag set is returned until the full category tag set meets the second preset condition.

[0050] Specifically, when the full content category tag is extracted and processed to obtain the current decoupled tag, the obtained current decoupled tag is added to the first substructure attribute tag set, and the full content category tag corresponding to the current decoupled tag is removed from the full category tag set. By removing the extracted full content category tag from the full category tag set, repeated processing and missed processing of the full content category tag can be effectively avoided; then it is determined whether the full category tag set is empty. If the full category tag set is empty, it is determined that the full category tag set meets the second preset condition, indicating that all full content category tags in the full category tag set have been extracted and processed, and subsequent matching processing is performed; if the full category tag set is not empty, it means that there are still full content category tags that have not been extracted, and it is necessary to return to the step of selecting a full content category tag from the full category tag set, and continue to extract and process the remaining full content category tags until the full category tag set is empty.

[0051] Step 36: Reconstruct all content category labels based on the second substructure attribute label set to obtain multi-attribute labels.

[0052] The second substructure attribute tag set includes multiple sub-attribute tags. The full content category tag can be reconstructed based on the second substructure attribute tag set to obtain a multi-attribute tag. Specifically, the full content category tag can be compared with the sub-attribute tags in the second substructure attribute tag set to obtain a sub-attribute tag that matches the full content category tag. Then, all sub-attribute tags that match the full content category tag can be combined into a multi-attribute tag.

[0053] like Figure 5 As shown, Figure 5 Taking the full content category label "double white solid line" in the example, after extraction processing, the second sub-structure attribute label set {"single, double", "white, yellow", "real, virtual"} can be obtained, and then the sub-structure attribute labels "single, double", "white, yellow" and "real, virtual" in the second sub-structure attribute label set are compared with the full content category label "double white solid line" respectively, and the sub-attribute labels "double", "white" and "real" that match the full content category label can be obtained. Then the matched sub-attribute labels are combined to obtain the multi-attribute label {"double", "white", "real"}.

[0054] In a specific embodiment, the multivariate attribute label may be encoded data, and the multivariate attribute label obtained by the above decoupling and reconstruction may be encoded to obtain corresponding encoded data. The encoded data corresponding to the second substructure attribute label set {"double, single", "white, yellow", "real, virtual"} may be set to {"1, 0", "1, 0", "1, 0"}, so that the multivariate attribute label is encoded accordingly according to the encoding correspondence. For example: the multivariate attribute label {"double", "white", "real"} is encoded as {1, 1, 1}, the multivariate attribute label {"single", "yellow", "virtual"} is encoded as {0, 0, 0}, and the multivariate attribute label {"single", "white", "virtual"} is encoded as {0, 1, 0}.

[0055] Specifically, when the above-mentioned full content category label is a special sub-attribute label (stop line or wait line, etc.), after decoupling and reconstruction, the sub-attribute labels corresponding to other sub-structure attributes in the multi-attribute label can be directly represented by "empty", thereby representing the missing label. For example: when the full content category label is "stop line", decoupling and reconstruction can be performed to obtain the multi-attribute label {"empty", "empty", "empty", "yes"}, and "empty" can be correspondingly encoded as "-1", and "yes, no" can be correspondingly encoded as "1, 0", then the encoded multi-attribute label can be "-1, -1, -1, 1"; when the full category label is "double white solid line", decoupling and reconstruction can be performed to obtain the multi-attribute label {"double", "white", "solid", "no"}, and the corresponding encoding can be {1, 1, 1, 0, 0}. It can be understood that the specific correspondence between the encoding data and the sub-attribute label can be set according to the actual situation and is not limited here.

[0056] Through the above decoupling and reconstruction processing, the 8-channel classification task of "single white dashed line", "single white solid line", "single yellow dashed line", "single yellow solid line", "double white dashed line", "double white solid line", "double yellow dashed line", and "double yellow solid line" can be converted into a 3-channel classification task of "010", "011", "000", "001", "110", "111", "100", and "101". The number of channels after the decoupling and reconstruction processing is the same as the number of substructure attributes of the lane line.

[0057] By decoupling and reconstructing the full-content category labels corresponding to the sample images in the training data, on the one hand, the decoupling operation is performed on similar confusing categories such as single yellow dashed lines and single white dashed lines, which solves the category prediction coupling problem when outputting full categories of lane lines and improves the prediction performance of similar confusing categories. On the other hand, because multiple substructure attributes are used to fully describe the original full-content category labels of lane lines, and the corresponding number of model output channels can be selected according to the number of decoupled substructure attributes, the number of output channels of the lane line recognition model can be reduced compared to the full-content category output channels, thereby reducing the spatial complexity of the model.

[0058] It can be understood that the decoupled reconstruction operation of the full content category label of the lane line can be applied to lane line recognition models under different frameworks based on segmentation, key points or object detection, and has universal applicability.

[0059] Step 23: Use the multi-label dataset to train the lane line recognition model to obtain a trained lane line recognition model.

[0060] After obtaining the multivariate attribute label of each full-content category label, the multivariate attribute label can be added to the multi-label dataset to obtain a multi-label dataset containing multivariate attribute labels of different categories. The multi-label dataset is then used to train the lane recognition model to obtain a trained lane recognition model. The specific steps for training the lane recognition model using the multi-label dataset are as follows: Figure 6 As shown:

[0061] Step 61: Select a sample image from the multi-label dataset and input it into the lane recognition model to obtain training output data.

[0062] The training output data includes a first lane line confidence map and a first attribute prediction probability map. The first lane line confidence map includes the lane line probability value of each pixel point in the sample image. The lane line probability value is the probability value of the pixel point in the sample image being a pixel point on the lane line. The first attribute prediction probability map includes the attribute probability value of the pixel point in the sample image. The attribute probability value is the probability value of the attribute of the pixel point in the sample image belonging to each sub-attribute label. For example: the sample image contains M*N pixels, then the lane line recognition model can obtain the probability value of the M*N pixels corresponding to the pixel points on the lane line. Taking the K-seed structure attribute of the lane line in the sample image as an example, the probability value of the K-seed structure attribute corresponding to each of the M*N pixels can be obtained.

[0063] Step 62: Calculate the current loss value based on the training output data, lane position labels, and multivariate attribute labels.

[0064] The activation function can be used to process the first lane line confidence map and the first attribute prediction probability map respectively to obtain multiple activation maps, and then the current loss value is calculated based on the activation maps, the lane line labels of the sample images, and the multivariate attribute labels of the sample images.

[0065] The multiple activation maps include a lane line activation map corresponding to the position of the lane line and a first preset number of attribute activation maps corresponding to the attributes of the lane line. The activation map can be three-dimensional data. Taking the sample image size of H*W as an example, a three-dimensional activation map of size H*W*(K+1) can be obtained through the activation function, wherein the first preset number is the number of attribute types of lane lines contained in all sample images. When K types of lane line attribute types (corresponding to substructure attributes) are included, the number of attribute activation maps corresponds to K. The lane line recognition model can output feature data of (K+1) channels (K attribute activation maps and 1 lane line confidence map); specifically, the sigmoid activation function can be used to activate the first lane line confidence map and the first attribute prediction probability map to obtain multiple activation maps. It can be understood that the activation function includes but is not limited to the sigmoid activation function, which is not limited here.

[0066] Calculate the loss between the lane loss label and the lane activation map to get the first loss, then calculate the loss between the multi-attribute label and the attribute activation map to get the second loss, and then calculate the current loss value based on the first and second losses.

[0067] Specifically, the current loss value may include: lane line confidence cross entropy loss, loss of all sub-attribute labels, and smoothing term loss; the cross entropy loss function can be used to calculate the loss between the lane line loss label and the lane line activation map, thereby obtaining the first loss (i.e., lane line confidence cross entropy loss); in order to obtain the second loss, the loss between each sub-attribute label and the corresponding attribute activation map can be first calculated to obtain the sub-loss (i.e., the loss of the sub-attribute label), and then the average of the attribute probability values ​​of each pixel in the lane line area (i.e., the area where the lane line is located) corresponding to all sub-attribute labels is calculated; the square of the difference between the attribute probability value of each pixel in the lane line area and the average value is calculated to obtain the first value; all first values ​​corresponding to the lane line area are accumulated to obtain the second value; based on the second value and the number of pixels in the lane line area, a smoothing value (i.e., smoothing term loss) is calculated; the smoothing value and the corresponding sub-loss are weighted to obtain a third value; all third values ​​are superimposed to obtain the second loss, and the current loss value can be calculated by the following calculation formula:

[0068]

[0069] In formula (1), L total Indicates the current loss value, L ce represents the cross entropy loss, L confidence Represents the first loss, L label represents the sub-loss of each sub-attribute label; L smooth Represents the smoothed value. R mIndicates the lane line area corresponding to the mth sub-attribute label, Indicates the lane line area R corresponding to the mth sub-attribute label m The probability value of the j-th pixel in ; Represents the lane line area R corresponding to the mth sub-attribute label m The average value of the attribute probability values ​​of all pixels in |R m | represents the lane line area R corresponding to the mth sub-attribute label m The number of pixels in the lane. λ is a hyperparameter whose value is generally 1. It is 0 if and only if the sub-attribute label is missing. For example, if the multi-attribute label is "empty, empty, empty, yes", that is, the lane line is "stop line", then "empty" means that the sub-attribute label is missing, and the corresponding λ is 0.

[0070] Step 63: Determine whether the lane line recognition model meets the training termination condition.

[0071] The training termination conditions may include: loss convergence, that is, the difference between the previous loss and the current loss value is less than the set value; determining whether the current loss value is less than the preset loss, the preset loss is a preset loss threshold. If the current loss value is less than the preset loss, it is determined that the preset stop condition is met; the number of training times reaches the set value (for example: training 10,000 times); or the accuracy obtained when testing using the test set reaches the set condition (for example: exceeding the preset accuracy), etc.

[0072] Step 64: If the lane line recognition model meets the training termination condition, the successfully trained lane line recognition model is output.

[0073] When the current training result of the lane line recognition model meets the training termination condition, it means that the lane line recognition model is successfully trained, and the successfully trained lane line recognition model is output.

[0074] Step 65: If the lane line recognition model does not meet the training termination conditions, the model parameters of the lane line recognition model are adjusted based on the current loss value.

[0075] If the lane recognition model does not meet the training termination criteria, further training is required. The model parameters of the lane recognition model can be adjusted, and the process returns to the step of selecting a sample image from the multi-label dataset and inputting it into the lane recognition model until the lane recognition model meets the training termination criteria.

[0076] The method of calculating the current loss value in the above embodiment can be applied to a lane line recognition model based on a segmentation or key point framework. In other embodiments, the lane line recognition model can also be a lane line recognition model based on a target detection framework. In this case, the training output data output by the lane line recognition model may include the position information of at least one lane line, the confidence of the target box corresponding to each lane line, and the second multivariate attribute information corresponding to each target box. The multi-label dataset also includes the target box label corresponding to the lane line in the sample image. The second multivariate attribute information includes the first sub-attribute information. Taking the number of lane line target boxes as m and the number of substructure attributes as k as an example, the lane line recognition model based on the target detection framework outputs a two-dimensional m*(k+5) feature vector, where the "5" in (k+5) includes the following five data: the confidence value of the lane line target box, the horizontal coordinate of the center point of the lane line target box, the vertical coordinate of the center point of the target box, the width of the target box, and the height of the target box.

[0077] The current loss value may include the mean square error (MSE) loss of the lane line target box coordinate regression, the confidence loss of the lane line target box, and the cross entropy loss of all substructure attributes of the lane line target box; specifically, in order to obtain the current loss value, the loss between the lane line label and the position information can be calculated to obtain the third loss (i.e., the lane line target box coordinate regression loss); the loss between the multivariate attribute label and the second multivariate attribute information is calculated to obtain the fourth loss; and the current loss value is calculated based on the third loss and the fourth loss.

[0078] Furthermore, the losses between all sub-attribute labels and the corresponding first sub-attribute information can be weighted and summed to obtain the fourth loss (cross entropy loss of each sub-structure attribute of the lane line frame). Then, the loss between the target frame and the target frame label is calculated to obtain the fifth loss (i.e., the confidence loss of the lane line frame). The fifth loss, the third loss, and the fourth loss are weighted and summed to obtain the current loss value. That is, the current loss value is calculated using the following calculation formula:

[0079]

[0080] In formula (2), L mse , L ce Represents mean square error loss and cross entropy loss respectively; L coordinate Indicates the third loss; L confidence Indicates the fifth loss; L label represents the fourth loss, and the hyperparameter λ is 1, and is 0 if and only if the sub-attribute label is missing.

[0081] The solution adopted in this embodiment improves the prediction performance of similar confusion categories and can reduce the spatial complexity of lane line recognition model parameters and the time complexity of prediction; at the same time, since the lane line recognition model is trained using multi-attribute labels, not only the position information of the lane line can be determined, but also the various substructure attributes of the lane line can be predicted, thereby improving the accuracy of lane line recognition, and at the same time, the richness of the types of identified lane lines can be increased, providing more accurate and comprehensive lane line category information; in addition, the decoupling reconstruction operation adopted in this embodiment can be applied to different lane line recognition models, and corresponding loss functions can be set to participate in the training of different lane line recognition models, which has better applicability.

[0082] See also Figure 7 , Figure 7 : is a flow chart of another embodiment of the lane line recognition method provided by the present application, the method comprising:

[0083] Step 71: Detect the road image to be detected based on the lane line recognition model to obtain detection data.

[0084] The lane line recognition model can be the lane line recognition model obtained in the above embodiment, and the detection data includes a second lane line confidence map and a second attribute prediction probability map. The second lane line confidence map includes the lane line probability value of each pixel point in the road image to be detected, and the second attribute prediction probability map includes the attribute probability value of each pixel point.

[0085] Step 72: Based on the second lane line confidence map, determine all positions of lane lines in the road image to be detected.

[0086] The positions of the pixels in the second lane line confidence map where the lane line probability value is greater than the first preset probability value are determined as all the positions of the lane lines. The first preset probability value can be set according to the actual situation. For example, if the first preset probability value is 0.8, the pixels in the second lane line confidence map where the lane line probability value is greater than 0.8 are considered to be pixels on the lane line. All the pixels are polled in turn, so as to determine all the positions of the pixels containing the lane lines. It can be understood that the road image to be detected may contain multiple lane lines, and all the positions of all lane lines can be obtained based on the second lane line confidence map.

[0087] Step 73: Based on all the positions of the lane lines, determine the position information of each lane line in the road image to be detected.

[0088] All positions of lane lines can be clustered to obtain the position information of lane lines.

[0089] Step 74: Determine the first multivariate attribute information of each lane line based on the position information of each lane line in the road image to be detected and the second attribute prediction probability map.

[0090] The position information of each lane line can be matched with the second attribute prediction probability map to obtain corresponding attribute detection information, and then the attribute detection information is screened to obtain the first multivariate attribute information of each lane line; specifically, the attribute detection information may include multiple second sub-attribute information, and the position information of each lane line can be matched with the attribute probability value of each pixel point in each lane position area to obtain multiple second sub-attribute information corresponding to each pixel point in each lane position area and the attribute probability value corresponding to each second sub-attribute information, and then the maximum value of the attribute probability value in the second sub-attribute information is calculated, and the second sub-attribute information corresponding to the maximum value is determined as the third sub-attribute information, and then the third sub-attribute information is voted to obtain the first multivariate attribute information.

[0091] When the second sub-attribute information contains more than two attribute values, the above method can be used to select the second sub-attribute information corresponding to the maximum attribute probability value as the third sub-attribute information; when the second sub-attribute information contains only two attribute values, for example: {"odd, even", "white, yellow", "real, virtual"}, the position information of each lane line can be matched with the second attribute prediction probability map to obtain the attribute probability value corresponding to each second sub-attribute information in each pixel point, and then the attribute probability value can be compared with the preset threshold value to determine the third sub-attribute information of the pixel point. For example: the attribute probability value of the second sub-attribute information "odd, even" is 0.8, the attribute probability value of the second sub-attribute information "white, yellow" is 0.2, and the attribute probability value of the second sub-attribute information "real, imaginary" is 0.6. They are compared with the preset threshold value 0.5 respectively. When the attribute probability value is less than the preset threshold value, the third sub-attribute information is judged to be "0", and when the attribute probability value is less than the preset threshold value, the third sub-attribute information is judged to be "1". Among them, "0" and "1" are coded data. For example, "white" corresponds to the coded data "1", and "yellow" corresponds to the coded data "0". When the attribute probability value of the second sub-attribute information "white, yellow" is 0.2, the third sub-attribute information can be obtained as "0", that is, "yellow".

[0092] After obtaining the third sub-attribute information corresponding to each pixel point in all lane line areas, voting is performed on the third sub-attribute information to obtain the first multivariate attribute information; specifically, the voting process may include soft voting and hard voting. Soft voting may be averaging the attribute probability values ​​of all third sub-attribute information in the lane line position area, and taking the third sub-attribute information with the largest average value as the first multivariate attribute information; hard voting may follow the principle of minority obeys majority. If the third sub-attribute information of 18 out of 20 pixels is A, then the first multivariate attribute information of this lane line is A; the voting processing method includes but is not limited to the above-mentioned soft voting and hard voting. In other embodiments, the voting processing method may also be weighted voting, etc., which is not limited here.

[0093] It can be understood that the first multivariate attribute information can be encoded data, and the first multivariate attribute information can be decoded to obtain the decoded first multivariate attribute information. Taking the encoded data corresponding to {"double, single", "white, yellow", "real, virtual"} as {"1, 0", "1, 0", "1, 0"} as an example, if the first multivariate attribute information of a lane line is {1, 1, 0}, then the first multivariate attribute information can be decoded at this time, and the decoded first multivariate attribute information is {double, white, virtual}, which means that the lane line is a white double-dashed line.

[0094] This embodiment can use the lane line recognition model to determine the multi-dimensional attribute information of each lane line for the road image to be detected, and can not only identify the position information of the lane line, but also accurately and comprehensively obtain the attribute information of each lane line.

[0095] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a lane line recognition device provided in the present application. The lane line recognition device 80 includes a memory 81 and a processor 82 connected to each other. The memory 81 is used to store a computer program. When the computer program is executed by the processor 82, it is used to implement the lane line recognition method in the above embodiment.

[0096] See also Figure 9 , Figure 9 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 90 is used to store a computer program 91. When the computer program 91 is executed by the processor, it is used to implement the lane line recognition method in the above embodiment.

[0097] The computer-readable storage medium 90 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program codes.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0099] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0101] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A lane line recognition method, characterized in that: include: Based on the lane line recognition model, the road image to be detected is detected to obtain detection data; Processing the detection data to obtain position information of each lane line in the road image to be detected and first multi-element attribute information corresponding to each lane line, wherein the first multi-element attribute information includes multiple sub-attribute information; The lane recognition model is trained based on a multi-label dataset, which includes sample images and multi-attribute labels corresponding to lane lines in the sample images. The multi-attribute labels are obtained by decoupling and reconstructing the full-content category labels corresponding to the lane lines in the sample images. The categories of the multi-attribute labels are smaller than the categories of the full-content category labels. The step of decoupling and reconstructing the full content category labels to obtain the multi-attribute labels includes: Obtaining a full category tag set, wherein the full category tag set includes all content category tags of all categories; Selecting a full-content category label from the full-category label set, extracting the full-content category label to obtain a current decoupled label; Determining whether the current decoupling tag meets a first preset condition, wherein if the length of the current decoupling tag is a preset length, determining that the current decoupling tag meets the first preset condition; If not, updating the full content category label to the current decoupled label, and returning to the step of extracting the full content category label until the current decoupled label meets the first preset condition; If yes, constructing a first substructure attribute tag set based on the current decoupling tag; Performing matching processing on the first substructure attribute tag set to obtain a second substructure attribute tag set; The full content category label is reconstructed based on the second substructure attribute label set to obtain the multi-attribute label.

2. The lane line recognition method according to claim 1, characterized in that: The training method of the lane line recognition model includes: Acquire training data, the training data including the plurality of sample images, lane position labels corresponding to lane lines in the sample images, and full content category labels; Decoupling and reconstructing the full content category labels to obtain the multi-attribute labels; The lane line recognition model is trained using the multi-label dataset to obtain a trained lane line recognition model.

3. The lane line recognition method according to claim 2, characterized in that: The step of using the multi-label dataset to train the lane recognition model includes: Selecting one of the sample images from the multi-label dataset and inputting it into the lane recognition model to obtain training output data; Calculating a current loss value based on the training output data, the lane line position label, and the multivariate attribute label; Determining whether the lane line recognition model meets the training termination condition; If not, the model parameters of the lane line recognition model are adjusted based on the current loss value, and the step of selecting a sample image from the multi-label dataset and inputting it into the lane line recognition model is returned to until the lane line recognition model meets the training termination condition.

4. The lane line recognition method according to claim 1, characterized in that: The method further comprises: Adding the current decoupled label to the first substructure attribute label set, and removing the current decoupled label from the full category label set; Determining whether the full-category tag set satisfies a second preset condition, wherein if the full-category tag set is empty, determining that the full-category tag set satisfies the second preset condition; If so, performing matching processing on the first substructure attribute tag set to obtain the second substructure attribute tag set, including: matching sub-attribute tags belonging to the same attribute type in the first substructure attribute tag set to obtain the second substructure attribute tag; If not, the process returns to the step of selecting a full-content category tag from the full-category tag set until the full-category tag set meets the second preset condition.

5. The lane line recognition method according to claim 4, characterized in that: The method comprises: Determining whether the length of the current decoupling tag is a preset length, and if so, determining that the current decoupling tag meets the first preset condition; It is determined whether the full-category tag set is empty, and if so, whether the full-category tag set satisfies the second preset condition.

6. The lane line recognition method according to claim 1, characterized in that: The second substructure attribute tag set includes a plurality of sub-attribute tags, and the step of reconstructing the full content category tag based on the second substructure attribute tag set to obtain the multi-attribute tag includes: Comparing the full content category tag with the sub-attribute tags in the second sub-structure attribute tag set to obtain a sub-attribute tag that matches the full content category tag; All sub-attribute tags that match the full content category tag are combined into the multi-attribute tag.

7. The lane line recognition method according to claim 4, characterized in that: The first substructure attribute tag set includes a plurality of sub-attribute tags, and the step of performing matching processing on the first substructure attribute tag set to obtain the second substructure attribute tag set includes: Matching the sub-attribute tags in the first sub-structure attribute tag set with a preset mapping table, and determining whether there are other sub-attribute tags corresponding to the sub-attribute tags in the preset mapping table; If so, the sub-attribute tag and the other sub-attribute tags are combined into the second sub-structure attribute tag set.

8. The lane line recognition method according to claim 3, characterized in that: The training output data includes a first lane line confidence map and a first attribute prediction probability map, the first lane line confidence map includes a lane line probability value for each pixel in the sample image, the lane line probability value is a probability value of a pixel in the sample image being a pixel on the lane line, the first attribute prediction probability map includes attribute probability values ​​for the pixels in the sample image, the attribute probability value is a probability value of the attribute of the pixel in the sample image belonging to each sub-attribute label, and the step of calculating the current loss value based on the training output data, the lane line position label, and the multivariate attribute label includes: Using an activation function to process the first lane confidence map and the first attribute prediction probability map respectively to obtain multiple activation maps; The current loss value is calculated based on the activation map, the lane line label of the sample image, and the multivariate attribute label of the sample image.

9. The lane line recognition method according to claim 8, characterized in that: The multiple activation maps include a lane line activation map corresponding to the position of the lane line and a first preset number of attribute activation maps corresponding to the attributes of the lane line. The step of calculating the current loss value based on the activation maps, the lane line labels of the sample image, and the multivariate attribute labels of the sample image includes: Calculating the loss between the lane line loss label and the lane line activation map to obtain a first loss; Calculating a loss between the multivariate attribute label and the attribute activation map to obtain a second loss; The current loss value is calculated based on the first loss and the second loss.

10. The lane line recognition method according to claim 9, characterized in that: The method comprises: Calculate the loss between each of the sub-attribute labels and the corresponding attribute activation map to obtain a sub-loss; Calculate the average of the attribute probability values ​​of each pixel in the lane line area corresponding to all the sub-attribute labels, where the lane line area is the area where the lane line is located; Calculating the square of the difference between the attribute probability value of each pixel point in the lane line area and the average value to obtain a first value; Accumulating all first values ​​corresponding to the lane line area to obtain a second value; Calculating a smoothing value based on the second value and the number of pixels in the lane line area; Weighting the smoothed value and the corresponding sub-loss to obtain a third value; All the third values ​​are superimposed to obtain the second loss.

11. The lane line recognition method according to claim 3, characterized in that: The training output data includes position information of at least one lane line, a confidence value of a target box corresponding to each lane line, and second multivariate attribute information corresponding to each target box, wherein the second multivariate attribute information includes a confidence value of the target box, a horizontal coordinate of a center point of the target box, a vertical coordinate of a center point of the target box, a width of the target box, and a height of the target box; and the step of calculating the current loss value based on the training output data, the lane line position label, and the multivariate attribute label includes: Calculating a loss between the lane line label and the position information to obtain a third loss; calculating a loss between the multivariate attribute label and the second multivariate attribute information to obtain a fourth loss; The current loss value is calculated based on the third loss and the fourth loss.

12. The lane line recognition method according to claim 11, characterized in that: The multi-label dataset further includes a target box label corresponding to a lane line in the sample image, the second multivariate attribute information includes first sub-attribute information, and the step of calculating the current loss value based on the third loss and the fourth loss includes: Performing a weighted summation on the losses between all sub-attribute labels and the corresponding first sub-attribute information to obtain the fourth loss; Calculating the loss between the target box and the target box label to obtain a fifth loss; A weighted sum is performed on the fifth loss, the third loss, and the fourth loss to obtain the current loss value.

13. The lane line recognition method according to claim 1, characterized in that: The detection data includes a second lane line confidence map and a second attribute prediction probability map. The step of processing the detection data to obtain position information of each lane line in the road image to be detected and first multivariate attribute information corresponding to each lane line includes: Determining all positions of lane lines in the road image to be detected based on the second lane line confidence map; Determining the position information of each lane line in the road image to be detected based on all positions of the lane lines; Based on the position information of each lane line in the road image to be detected and the second attribute prediction probability map, the first multivariate attribute information of each lane line is determined.

14. The lane line recognition method according to claim 13, characterized in that: The second lane line confidence map includes a lane line probability value of each pixel in the road image to be detected, and the second attribute prediction probability map includes an attribute probability value of each pixel; Determine the positions of pixels in the second lane line confidence map where the lane line probability value is greater than the first preset probability value as all positions of the lane line; Clustering the positions of all lane lines to obtain position information of each lane line; Matching the position information of each lane line with the second attribute prediction probability map to obtain corresponding attribute detection information; The attribute detection information is screened to obtain first multivariate attribute information of each lane line.

15. The lane line recognition method according to claim 14, characterized in that: The attribute detection information includes a plurality of second sub-attribute information. The step of filtering the attribute detection information to obtain the first multi-attribute information of the lane line includes: Calculating a maximum value of the attribute probability values ​​in the second sub-attribute information, and determining the second sub-attribute information corresponding to the maximum value as the third sub-attribute information; Voting is performed on the third sub-attribute information to obtain the first multi-attribute information.

16. The lane line recognition method according to claim 1, characterized in that: The first multi-attribute information and the multi-attribute label are both encoded data, and the method further includes: The first multi-dimensional attribute information is decoded to obtain decoded first multi-dimensional attribute information.

17. A lane line recognition device, characterized in that: The invention comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the lane line recognition method according to any one of claims 1 to 16.

18. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the lane line recognition method according to any one of claims 1 to 16.

Images