Road marking recognition method and device, road marking recognition model training method, electronic equipment and program product

Abstract

The present application discloses a method for identifying line markings, comprising: inputting an image related to line markings into a line marking recognition model based on the YOLO image segmentation algorithm for line marking recognition processing, thereby identifying line marking areas in the image. The method specifically comprises: extracting features from the input image and outputting multiple feature maps through multiple feature channels; adding a partial attention mechanism to the multiple feature channels, thereby applying attention processing to portions of the multiple feature maps and not applying attention processing to other portions of the multiple feature maps; sampling processing the multiple feature maps that have undergone partial attention processing, and outputting the sampled feature maps to multiple segmentation heads; and using the multiple segmentation heads to separately process the sampled feature maps to segment and extract line marking areas from the multiple feature maps. The present application also relates to a method for training a line marking recognition model, a line marking recognition model, an electronic device, and a program product.

Description

Technical Field

[0001] The present application relates to the field of highway traffic, and in particular to a road marking recognition method and device, electronic equipment, and program products. Background Art

[0002] Traffic markings are key traffic control devices applied to the road surface, designed to channel traffic flow, guide vehicle alignment, and ensure traffic safety. These markings have multiple functions: defining lane boundaries, regulating and managing driving behavior, guiding drivers' sightlines, indicating and warning of road conditions ahead, and clarifying right-of-way.

[0003] Road marking recognition has a wide range of applications in the field of highway transportation. In particular, accurate and rapid identification of road markings is a key step in assessing their visibility under different lighting conditions during the testing of their retroreflectivity.

[0004] In view of this, the industry is eager to improve the recognition accuracy and efficiency of road marking recognition solutions.

[0005] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the Invention

[0006] Therefore, the embodiments of the present application intend to provide a marking recognition solution with high recognition accuracy and recognition efficiency.

[0007] In a first aspect, a method for identifying a marking line is provided, comprising:

[0008] An image related to markings is input into a marking recognition model based on the YOLO image segmentation algorithm for marking recognition processing, thereby identifying the marking area in the image. The marking recognition model based on the YOLO image segmentation algorithm includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes.

[0009] The marking line recognition process includes:

[0010] Extract features from the input image using the backbone network, and output multiple feature maps through multiple feature channels, where the multiple feature channels correspond to the multiple segmentation heads;

[0011] Adding an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps and not applying attention processing to other portions of the plurality of feature maps;

[0012] Using the neck network to perform sampling processing on the multiple feature maps processed by the partial attention, and outputting the multiple feature maps processed by the sampling processing to the multiple segmentation heads accordingly;

[0013] The plurality of segmentation heads are used to process the plurality of feature maps that have been sampled, respectively, to segment and extract the marked line areas in the plurality of feature maps.

[0014] Optionally, the plurality of segmentation heads include a large-size segmentation head, a plurality of intermediate-size segmentation heads and a small-size segmentation head.

[0015] The plurality of channels include a first characteristic channel corresponding to a large-size segmentation head, a plurality of second characteristic channels corresponding to a plurality of intermediate-size segmentation heads, and a third characteristic channel corresponding to a small-size segmentation head;

[0016] The outputting of multiple feature maps through multiple feature channels includes:

[0017] Outputting the large-size feature map through the first feature channel,

[0018] Outputting the plurality of intermediate-size feature maps respectively through the plurality of second feature channels,

[0019] Outputting the small-size feature map through the third feature channel;

[0020] The applying an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps and not applying attention processing to other portions of the plurality of feature maps, includes:

[0021] Adding an attention mechanism to the first feature channel and the third channel so as to apply attention processing to the large-size feature map and the small-size feature map, and not adding an attention mechanism to the second channel so as not to apply attention processing to the intermediate-size feature map;

[0022] The step of outputting the plurality of feature maps processed by sampling to the plurality of segmentation heads includes:

[0023] The sampled large-scale feature map is output to the large-scale segmentation head.

[0024] Outputting the sampled medium-sized feature maps to the plurality of second segmentation heads respectively.

[0025] The sampled small-size feature map is output to the small-size segmentation head.

[0026] Optionally, the attention mechanism includes an encoder for embedding spatial coordinate information of the marking area into the feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

[0027] Optionally, the backbone network includes multiple feature extraction convolution modules corresponding to the multiple feature channels respectively, at least some of the feature extraction convolution modules corresponding to the feature channels without adding the attention mechanism include deformable convolution extraction layers, and the feature extraction convolution modules corresponding to the feature channels with adding the attention mechanism include fixed convolution extraction layers.

[0028] Optionally, a ratio of a large-size feature map corresponding to the large-size segmentation head to a small-size feature map corresponding to the small-size segmentation head is greater than or equal to 8.

[0029] Optionally, before performing the marking line recognition process, the method further includes:

[0030] Inputting a training image containing annotated markings into a line recognition model framework to be trained to obtain a trained line recognition model, wherein the training image has true values ​​associated with the annotated markings, the true values ​​including true frames of the annotated markings;

[0031] The training includes iteratively executing the following steps until a preset training completion condition is reached:

[0032] Inputting the training samples into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line;

[0033] Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculating the loss value between the predicted value and the true value includes:

[0034] Determine the intersection-over-union ratio of the predicted box and the true box,

[0035] Determine the distance between the key points of the predicted box and the corresponding key points of the real box according to a given plurality of box key points, wherein the plurality of box key points include diagonal points.

[0036] Determining a key point intersection-to-union loss value according to the area intersection-to-union ratio and the distance;

[0037] The parameters of the line recognition model framework are reversely updated based on the loss value.

[0038] In a second aspect, a method for training a line marking recognition model is provided, comprising: inputting a training image containing annotated lines into a line marking recognition model framework to be trained to obtain a trained line marking recognition model;

[0039] The line marking recognition model framework to be trained is a line marking recognition model framework based on the YOLO image segmentation algorithm, which includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes.

[0040] The training image has true values ​​associated with the marked lines, and the true values ​​include true frames of the marked lines;

[0041] The training includes iteratively executing the following steps until a preset training completion condition is reached:

[0042] Inputting the training samples into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line;

[0043] Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculating the loss value between the predicted value and the true value includes:

[0044] Determine the intersection-over-union ratio of the predicted box and the true box,

[0045] Determine the distance between the key points of the predicted box and the corresponding key points of the real box according to a given plurality of box key points, wherein the plurality of box key points include the diagonal points of the box.

[0046] Determining a key point intersection-to-union loss value according to the area intersection-to-union ratio and the distance;

[0047] Based on the loss value, the parameters of the line recognition model framework are reversely updated.

[0048] In a third aspect, a marking line recognition model is provided, wherein the marking line recognition model is trained by the marking line recognition model training method according to the second aspect.

[0049] According to a fourth aspect, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to implement the method according to the first and second aspects when running the computer program.

[0050] According to a fifth aspect, a program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first and second aspects is implemented.

[0051] The line marking recognition method provided in the embodiment of the present application performs feature extraction on an input image and outputs multiple feature maps through multiple feature channels; adds a partial attention mechanism to the multiple feature channels, thereby applying attention processing to parts of the multiple feature maps and not applying attention processing to other parts of the multiple feature maps; performs sampling processing on the multiple feature maps that have been partially attended to, and outputs the multiple feature maps that have been sampled to multiple segmentation heads accordingly; utilizes multiple segmentation heads to process the multiple feature maps that have been sampled respectively to segment and extract the line marking areas in the multiple feature maps, which can accurately and quickly identify the line markings, especially in the process of detecting the line marking retroreflection coefficient or other scenarios where line markings need to be identified.

[0052] The optional features and other effects of the embodiments of the present application are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings, and the same or similar reference numerals in the drawings represent the same or similar elements, wherein:

[0054] Figure 1 An exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0055] Figure 2 A second exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0056] Figure 3 A third exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0057] Figure 4 A schematic structural diagram of a marking line identification model according to a marking line identification method that can be used to implement an embodiment of the present application is shown;

[0058] Figure 5 Another schematic structural diagram of a marking line identification model that can be used to implement the marking line recognition method according to an embodiment of the present application is shown;

[0059] Figure 6 Shown Figure 4 The schematic structure diagram of the feature extraction convolution module of the line recognition model shown;

[0060] Figure 7 A fourth exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0061] Figure 8 A fifth exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0062] Figure 9 An exemplary flow chart showing a method for training a marking line recognition model according to a marking line recognition method according to an embodiment of the present application; and

[0063] Figure 10 A structural diagram of an electronic device that can be used to implement the marking line recognition method and the marking line recognition model training method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0065] As used herein, the term "including" and its variations denote open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" denotes "and / or". The term "based on" denotes "based at least in part on". The terms "an example embodiment" and "an embodiment" denote "at least one example embodiment". The term "another embodiment" denotes "at least one other embodiment". To facilitate understanding of this specification, the sequential terms "first", "second", etc. are used herein to distinguish different elements / items / objects and do not denote the order or importance of different elements / items / objects. In particular, method steps expressed with the terms "first", "second", etc. are not intended to indicate the order in which the methods are executed; when an embodiment contains elements / items / objects expressed in a later order, the elements / items / objects expressed in the earlier order with the terms "first", "second", etc. are not necessarily essential technical features of the embodiment.

[0066] As mentioned above, it is currently desired to provide a road marking recognition solution with high recognition accuracy and recognition efficiency.

[0067] Therefore, in some embodiments of the present application, a marking line recognition method is provided. The method can perform marking line recognition based on a marking line recognition model of a YOLO image segmentation algorithm, thereby identifying a marking line area in the image.

[0068] In the embodiments of the present application, YOLO (You Only Look Once) is an image processing algorithm that can be used for computer vision tasks. It can quickly perform real-time object detection and segmentation on images. In a preferred embodiment of the present application, a line recognition model based on the improved YOLOv8n-seg image segmentation algorithm can be used for processing.

[0069] like Figures 2 to 4 As shown, the marking line recognition model based on the YOLO image segmentation algorithm includes a backbone network (Backbone network) 220, a neck network (Neck network) 230 and multiple segmentation heads (Segment Head) 240 for processing feature maps of different sizes.

[0070] Accordingly, in the embodiment of the present application, the marking line recognition method includes: inputting an image 610, which is an image related to the marking line, into a marking line recognition model based on the YOLO image segmentation algorithm for marking line recognition processing, thereby identifying the marking line area in the image. More specifically, in conjunction with reference to Figure 1 and Figures 4 to 6 The input image 410 is processed in sequence by the backbone network 420, the neck network 230 and multiple segmentation heads 441, 442, 443 and 444.

[0071] like Figure 4 and Figure 5 As shown, the backbone network 420 can be used to extract multiple feature maps from the input image 410 and output the multiple feature maps to the neck network 430 through multiple feature channels. Figure 4 and Figure 5 As shown, the backbone network 420 includes multiple feature extraction convolution modules 421, 422, 423, and 424 corresponding to multiple feature channels. Figure 5 Specifically, in addition to the multiple feature extraction convolution modules 421, 422, 423, and 424 of the multiple feature channels, the backbone network 420 may further include additional layers or modules, which are not described in detail here.

[0072] like Figure 4 and Figure 5 As shown, the neck network 430 may include a bottom-up and top-down sampling structure, including a plurality of necessary neural network layers, such as various layers with sampling functions and required splicing layers.

[0073] like Figure 4 and Figure 5 As shown, after the neck network 430 is sampled and processed, it can output the sampled and processed feature maps to the corresponding segmentation heads 441 to 444. Figure 5 In the embodiment specifically shown, each segmentation head uses a dual loss function to jointly perform detection and segmentation, namely, a prediction box loss function (Bbox Loss) and a target classification loss function (ClsLoss).

[0074] In the embodiments of the present application, the (improved) YOLO-based image segmentation algorithm refers to a YOLO image segmentation algorithm that uses any novel feature of the embodiments of the present application to optimize line marking recognition, and in particular, refers to a YOLOv8n-seg image segmentation algorithm that uses any novel feature of the embodiments of the present application to optimize line marking recognition. It is conceivable that the features of the embodiments of the present application for line marking recognition processing can be incorporated into or used to improve other end-to-end image segmentation algorithm frameworks, and this application or subsequent applications may cover such novel combinations and improvements.

[0075] In a specific example, the road marking detection model can be used to configure an existing road marking detection device to obtain multiple road images, and the images can be divided into a training set, a test set, and a validation set in a certain ratio, such as 8:1:1, to verify the recognition results.

[0076] Before the road marking recognition process or as a preprocessing step, the images used as the training set can also be annotated. In one example, an image annotation tool (Labelme image labeling software) can be used to annotate the road marking images in the training set. To accurately obtain the boundary coordinates and categories of road markings during deep learning, Labelme uses irregular bounding box masks to mark the targets during annotation. The target road marking categories are then created and named, and different color masks are used to mark different categories of targets.

[0077] In some embodiments of the present application, Figure 1 As shown, the processing using the marking line recognition model based on the YOLO image segmentation algorithm may specifically include steps S100 to S140:

[0078] S100 (not marked): Inputting an image related to marking lines into a marking line recognition model based on the YOLO image segmentation algorithm to perform marking line recognition processing, thereby marking a marking line area in the image.

[0079] In the embodiment of the present application, the images related to the road markings include but are not limited to the road marking images obtained by the road marking detection device, the monitoring images related to the road markings obtained by road monitoring, and the video frames related to the road markings obtained by the vehicle camera, but the present application is not limited thereto.

[0080] S110: Use the backbone network to extract features from the input image and output multiple feature maps through multiple feature channels.

[0081] In the embodiment of the present application, for example, in the above step S110, the multiple feature channels correspond to the multiple segmentation heads.

[0082] S120: Add an attention mechanism to the multiple feature channel parts, thereby applying attention processing to parts of the multiple feature maps, and not applying attention processing to other parts of the multiple feature maps.

[0083] S130: Using the neck network to sample the multiple feature maps that have been partially attention processed, and output the multiple feature maps that have been sampled to multiple segmentation heads.

[0084] S140: using a plurality of segmentation heads to respectively process the plurality of feature maps that have been sampled, so as to segment and extract the marked line areas in the plurality of feature maps.

[0085] Combined with reference Figures 4 to 6 The plurality of segmentation heads may include a large-size segmentation head 441 , a plurality (eg, two) of intermediate-size segmentation heads 442 , 443 , and a small-size segmentation head 444 .

[0086] In this preferred embodiment, a small-size segmentation head, which may also be called a small target segmentation head, is added to the line recognition model based on the improved YOLO image segmentation algorithm.

[0087] Therefore, if Figure 2 As shown, in the above step S110, outputting multiple feature maps through multiple feature channels may specifically include:

[0088] S111: Output the large-size feature map through the first feature channel. S112: Output multiple intermediate-size feature maps through multiple second feature channels. S113: Output the small-size feature map through the third feature channel.

[0089] Further, if Figures 4 to 6 As shown, the multiple channels also include a first characteristic channel corresponding to the large-size segmentation head, a plurality of second characteristic channels corresponding to the multiple intermediate-size segmentation heads, and a third characteristic channel corresponding to the small-size segmentation head.

[0090] Therefore, if Figure 3As shown, in the above step S130, the outputting of the plurality of feature maps processed by sampling to the plurality of segmentation heads may specifically include:

[0091] S131: Output the sampled large-scale feature map to the large-scale segmentation head. S132: Output the sampled multiple intermediate-scale feature maps to the multiple intermediate segmentation heads. S133: Output the sampled small-scale feature map to the small-scale segmentation head.

[0092] In the road marking recognition model based on the improved YOLO image segmentation algorithm of the embodiments of the present application, after processing by the backbone network (Backbone network) and the neck network (Neck network), not only a large-scale feature map and several intermediate-scale feature maps are output to the corresponding segmentation head for instance segmentation, but also, specifically for the specific scenario of road marking recognition, a small-scale feature map is output to the corresponding small-scale segmentation head (small target segmentation head) to segment small target road markings. By way of explanation and not limitation, the pixel areas occupied by road markings captured by a camera at a low angle vary greatly in scale in the image. Conventional image segmentation algorithms that have not been specifically modified for road marking recognition have weak instance detection capabilities for small-scale targets, and therefore are prone to missed detection of road markings that are far away from the camera.

[0093] In a preferred embodiment of the present application, the ratio of the large-scale feature map corresponding to the large-scale segmentation head to the small-scale feature map corresponding to the small-scale segmentation head is greater than or equal to 8. In one example, an input image of size 640*640 can be downsampled by 20 times (corresponding to the first feature channel of the large-scale segmentation head), 40 times, 80 times and 160 times (corresponding to the third feature channel of the small-scale segmentation head) to obtain feature maps of sizes 32*32, 16*16, 8*8 and 4*4, which are respectively used for segmentation by multiple feature heads. By using a segmentation head for a small target in the line recognition model of the improved YOLO image segmentation algorithm (as well as the corresponding backbone network and neck network structure, as further described below), the instance detection capability of small target road markings is effectively improved.

[0094] Combined with reference Figures 1 to 3 and Figures 4 to 6 In step S120, an attention mechanism, such as a channel attention mechanism (Coordinate Attention, CA for short), is added to the multiple feature channel parts, so that attention processing is applied to parts of the multiple feature maps, and no attention processing is applied to other parts of the multiple feature maps.

[0095] In some embodiments of this application, reference Figures 4 to 6 , the above step S120 may correspondingly include:

[0096] S121 (unmarked): Add an attention mechanism to the first feature channel and the third channel so as to apply attention processing to the large-size feature map and the small-size feature map, and do not add an attention mechanism to the second channel so as not to apply attention processing to the intermediate-size feature map.

[0097] In other words, in this embodiment, the channel attention mechanism 460 is added only to the first and third feature channels corresponding to the large-size segmentation head and the small-size segmentation head, while the channel attention mechanism is not added to the second feature channel corresponding to the intermediate-size segmentation head.

[0098] In a further embodiment of the present application, when the attention mechanism is applied to multiple feature channel portions, the feature maps corresponding to the feature channels to which the attention mechanism is applied are not adjacent in size. In other words, in this embodiment, the channel attention mechanism 460 is applied only to the first and third feature channels corresponding to the large-scale feature maps and the small-scale feature maps, while the channel attention mechanism is not applied to the second feature channel corresponding to the intermediate-sized feature map.

[0099] In some embodiments of the present application, the attention mechanism includes an encoder for embedding spatial coordinate information of the marking area into the feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

[0100] In the embodiment of the present application, the channel attention mechanism is a method of increasing the importance of the channel and the attention information by adding weights to the feature maps of each channel with the attention mechanism. The inventors have made the following surprising discovery through research, that is, adding a partial channel attention mechanism, such as adding a channel attention mechanism non-adjacently, and especially adding a channel attention mechanism only for the maximum size and minimum size feature maps, can avoid the interference of various interference factors on the road to the target information of the road marking to the greatest extent. As an explanation and not limitation, through the backbone network bottom-up (such as Figure 4 and Figure 5 ) Extract deeper features layer by layer. The feature map at the top contains the richest feature information and the most ambiguous position information, while the feature map at the bottom is the opposite. By applying the channel attention mechanism to the feature channels at both ends, the clearest position information and feature information can be extracted and fused to achieve higher accuracy. In addition, according to the embodiments of the present application, the partially added channel attention mechanism, such as adding the channel attention mechanism non-adjacently, especially adding the channel attention mechanism only to the maximum and minimum size feature maps, can also be combined with further preferred features to achieve further beneficial effects of optimizing line marking recognition.

[0101] Continue to refer Figures 4 to 6As mentioned above, the backbone network 420 includes multiple feature extraction convolution modules 421, 422, 423, and 424 corresponding to multiple feature channels respectively. In a preferred embodiment of the present application, at least some, preferably all, of the feature extraction convolution modules corresponding to the feature channels without the addition of the attention mechanism may include a deformable convolution extraction layer, and the feature extraction convolution modules corresponding to the feature channels with the addition of the attention mechanism may include a fixed convolution extraction layer. Figure 5 Specifically, it is shown that the feature extraction convolution modules 421 and 424 corresponding to the first and third feature channels can include C2F (Coarse to Fine) fixed convolution extraction layers; and the feature extraction convolution modules 422 and 423 corresponding to the second feature channel to which the attention mechanism is not added can include deformable convolution extraction layers 4221 and 4231.

[0102] In a preferred embodiment of the present application, the deformable convolution extraction layer may include a DCNv3 deformable convolution layer. Figure 6 , showing that the deformable convolution extraction layer 4221 includes a DCNv3 deformable convolution functional layer. In the embodiments of the present application, it will be understood that the convolution extraction layer can be broadly interpreted and may include a functional convolution (sub) layer with extraction functions, and may also include additional (sub) layers such as batch layers, bottleneck layers, and splicing layers as needed. The embodiments of the present application do not limit the number and structure of additional (sub) layers in the convolution extraction layer.

[0103] The inventors noted that when capturing road markings, the road markings can experience significant shape distortion due to maneuvers like turns and lane changes by the vehicle collecting them, and their angles in the image can also change in unknown ways. Through research, the inventors made the surprising discovery that by including variable convolutional extraction layers in the feature extraction convolution modules corresponding to at least some feature channels without an attention mechanism, these distorted road marking features can be effectively and accurately extracted.

[0104] As mentioned above, if Figure 5 As shown, each segmentation head uses a dual loss function to jointly perform detection and segmentation, namely the box loss function (Bbox Loss) and the target classification loss function (Cls Loss). In the embodiment of the present application, the box loss function (Bbox Loss) is further improved to optimize the recognition of the marking line.

[0105] In a further embodiment of the present application, Figure 7 As shown, before performing the marking line recognition process, the relevant method may optionally include a training step, specifically including:

[0106] S700 (not marked): inputting the training image containing the marked lines into the line recognition model framework to be trained for training to obtain a trained line recognition model.

[0107] In some embodiments of the present application, the training image carries true values ​​associated with the labeled lines, and the true values ​​include true boxes of the labeled lines.

[0108] In some embodiments of the present application, the training includes iteratively executing the following steps until a preset training completion condition is met:

[0109] S710: Input the training sample into the line marking recognition model framework to obtain the prediction value related to the marked line marking.

[0110] In some embodiments of the present application, for example, in the above step S710, the predicted value includes a predicted box marked with a marking line.

[0111] S720: Calculate the loss value between the predicted value and the true value based on a given loss function.

[0112] In some embodiments of the present application, for example, in the above step S720, the given loss function includes a keypoint intersection-over-union (MPDIOU) loss function.

[0113] In some embodiments of the present application, Figure 8 As shown, calculating the loss value between the predicted value and the true value may include the following steps: S721: Determining the area intersection-over-union ratio of the predicted box and the true box. S722: Determining the distance between the box key points of the predicted box and the corresponding box key points of the true box based on a given plurality of box key points. S723: Determining the key point intersection-over-union loss value based on the area intersection-over-union ratio and the distance.

[0114] S730: Reversely update the parameters of the line recognition model framework based on the loss value.

[0115] In the preferred embodiment, the plurality of frame key points include diagonal points. The selection of the frame diagonal points (frame diagonal lines) located on the frame diagonal lines is determined based on the orientation and / or shape distortion of the marked marking lines.

[0116] In some known solutions, the box loss function Box_Loss = DFL_Loss + CIOU_Loss is used to measure the overlap between the predicted box and the true box. DFL_Loss is a deep feature loss function, and CIOU_Loss is a complete intersection-over-union loss function, which uses the distance ratio between the "true box" and the "predicted box" to measure the overlap between the predicted box and the true box. However, after research, the inventors found that this loss function does not effectively improve the loss value during line recognition (training).

[0117] In this regard, an embodiment of the present application proposes to use a key point intersection-over-union (MPDIOU) loss function, and preferably, the diagonal points of the box are used as key points of the key point intersection-over-union (MPDIOU) loss function.

[0118] In a further embodiment of the present application, for example in the above-mentioned step S721, the selection of the frame diagonal points (frame diagonal points) located on the frame diagonal is determined according to the direction and / or shape distortion of the marked marking. In one example, for a marking that turns right, the upper right and lower left frame diagonal points can be used as key points. In one example, for a marking that turns left, the upper left and lower right frame diagonal points can be used as key points. In one example, for a straight marking in an image acquired when turning right (and thus causing shape distortion), the upper right and lower left predicted frame diagonal points can be used as key points. In one example, for a straight marking in an image acquired when turning left (and thus causing shape distortion), the upper left and lower right predicted frame diagonal points can be used as key points. In one example, for a straight marking in an image acquired when turning left (and thus causing shape distortion), the upper left and lower right predicted frame diagonal points can be used as key points.

[0119] By way of explanation and not limitation, for road marking recognition, when the predicted box and the real box have the same aspect ratio but different width and height values, it is difficult to improve the loss value; the embodiment of the present application utilizes the key point intersection-over-union (MPDIOU) loss function, and uses the diagonal points of the predicted box as key points, which can accurately segment road markings caused by interference such as occlusion by vehicles and pedestrians; further, the embodiment of the present application also determines the direction of the diagonal points (frame diagonals) of the predicted box based on the orientation and / or shape distortion of the marked markings, thereby further improving the ability to accurately detect road markings.

[0120] In one embodiment, the accuracy, recall, and precision of road marking detection and segmentation in an image validation set were analyzed. The experimental results showed that the model based on this embodiment not only meets the requirements for end-to-end real-time road marking recognition, but also achieved 98.4% accuracy, 94.7% recall, and 96.7% precision in detection categories; and 98.3% accuracy, 94.2% recall, and 96.4% precision in segmentation. This model achieves accurate detection and segmentation of road markings.

[0121] The line marking recognition method provided in the embodiment of the present application performs feature extraction on an input image and outputs multiple feature maps through multiple feature channels; adds a partial attention mechanism to the multiple feature channels, thereby applying attention processing to parts of the multiple feature maps and not applying attention processing to other parts of the multiple feature maps; performs sampling processing on the multiple feature maps that have been partially attended to, and outputs the multiple feature maps that have been sampled to multiple segmentation heads accordingly; utilizes multiple segmentation heads to process the multiple feature maps that have been sampled respectively to segment and extract the line marking areas in the multiple feature maps, which can achieve accurate and rapid recognition of line markings, especially in the process of detecting the retroreflection coefficient of line markings or other scenarios where line markings need to be identified, such as unmanned driving scenarios.

[0122] Accordingly, in some embodiments of the present application, Figure 9 As shown, a method for training a marking line recognition model is also provided, which may include the following steps S900 to S930:

[0123] S900 (not marked): inputting the training image containing the marked lines into the line recognition model framework to be trained for training, so as to obtain a trained line recognition model.

[0124] In an embodiment of the present application, the line marking recognition model framework to be trained is a line marking recognition model framework based on the YOLO image segmentation algorithm, which includes a backbone network (Backbone network), a neck network (Neck network) and multiple segmentation heads (Segment Head) for processing feature maps of different sizes.

[0125] In some embodiments of the present application, the training image carries true values ​​associated with the labeled lines, and the true values ​​include true boxes of the labeled lines.

[0126] In the embodiments of this application, Figure 9 As shown, the training includes iteratively executing the following steps until a preset training completion condition is reached:

[0127] S910: Input the training sample into the line marking recognition model framework to obtain the prediction value related to the marked line marking.

[0128] In an embodiment of the present application, for example, in the above step S910, the predicted value includes a predicted box marked with a marking line.

[0129] S920: Calculate the loss value between the predicted value and the true value based on a given loss function.

[0130] In an embodiment of the present application, for example, in the above-mentioned step S910, the given loss function may include a key point intersection-over-union (MPDIOU) loss function. In an embodiment of the present application, the calculation of the loss value between the predicted value and the true value may specifically include the following steps A1 to A3: A1: Determine the area intersection-over-union ratio of the predicted box and the true box. A2: Based on a given plurality of box key points, determine the distance between the box key points of the predicted box and the corresponding box key points of the true box. In an embodiment of the present application, for example, in the above-mentioned step A2, the multiple predicted box key points include box diagonal points. In a further embodiment of the present application, for example, in the above-mentioned step A2, the selection of the box diagonal points (box diagonal lines) located on the box diagonal is determined according to the direction and / or shape distortion of the marked marking line. A3: Determine the key point intersection-over-union loss value based on the area intersection-over-union ratio and the distance.

[0131] S930: Based on the loss value, reversely update the parameters of the line recognition model framework.

[0132] The specific description of the above steps S900 to S930 and their sub-steps of this application can refer to the description of steps S800 to S830 and their sub-steps in the aforementioned embodiments of this application, and will not be repeated here.

[0133] In some embodiments of the present application, a marking line recognition model is further provided, and the marking line recognition model is trained by the marking line recognition model training method described in the above embodiments of the present application.

[0134] In an embodiment of the present application, the line marking recognition method and the line marking recognition model training method can be implemented by an electronic device that integrates relevant functional modules or components. Figure 10 A schematic diagram of an electronic device that can be used to implement the above-mentioned method and device of the embodiment of the present application is shown.

[0135] like Figure 10As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1002 or the programs and / or data loaded from the storage part 1008 into the random access memory (RAM) 1003. The processor 1001 may include a central processing unit (CPU), and may be a multi-core processor, or may include multiple processors. In some embodiments, the processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processor 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0136] The processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, it can implement the steps or functions of the method for determining a given spacing based on the marking retroreflection evaluation model described in the above embodiments.

[0137] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 10 Only some components are shown schematically, which does not mean that the computer system 1000 only includes Figure 10 Components shown.

[0138] In some embodiments, the computer 1000 refers to a mobile terminal, including a mobile phone, a vehicle-mounted terminal, etc. Taking a mobile phone as an example, the electronic device 1000 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0139] The systems, devices, modules, or units described in the above embodiments may be implemented by the computer or its associated components. The computer may be, for example, a mobile terminal, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a tablet computer, or a combination thereof.

[0140] Although not shown, in an embodiment of the present application, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to execute any method of the embodiment of the present application when executed.

[0141] Although not shown, in an embodiment of the present application, a program product is provided, including a computer program, wherein the computer program is configured to execute any method of the embodiments of the present application when executed.

[0142] Storage media in embodiments of the present application include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0143] The methods, programs, systems, and apparatuses of the embodiments of the present application may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0144] Unless explicitly stated, the actions or steps of the methods, procedures, and embodiments of the present application do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] In this document, multiple embodiments of the present application are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the various embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present application, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0146] While the exemplary systems and methods of the present application have been specifically shown and described with reference to the above-described embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present application as defined in the appended claims.

Claims

1. A marking line recognition method, characterized in that: include: An image related to marking lines is input into a marking line recognition model based on the YOLO image segmentation algorithm for marking line recognition processing, thereby identifying the marking line area in the image. The marking line recognition model based on the YOLO image segmentation algorithm includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes. The marking line recognition process includes: Extract features from the input image using the backbone network, and output multiple feature maps through multiple feature channels, where the multiple feature channels correspond to the multiple segmentation heads; Adding an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps and not applying attention processing to other portions of the plurality of feature maps; Using the neck network to perform sampling processing on the multiple feature maps processed by the partial attention, and outputting the multiple feature maps processed by the sampling processing to the multiple segmentation heads accordingly; Using the multiple segmentation heads to process the multiple feature maps that have been sampled, respectively, to segment and extract the marked line areas in the multiple feature maps; The plurality of segmentation heads include a large-size segmentation head, a plurality of intermediate-size segmentation heads and a small-size segmentation head. The plurality of channels include a first characteristic channel corresponding to a large-size segmentation head, a plurality of second characteristic channels corresponding to a plurality of intermediate-size segmentation heads, and a third characteristic channel corresponding to a small-size segmentation head; The outputting of multiple feature maps through multiple feature channels includes: Output a large-scale feature map through the first feature channel, Outputting a plurality of intermediate-size feature maps respectively through the plurality of second feature channels, Outputting a small-size feature map through the third feature channel; The adding of an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps and not applying attention processing to other portions of the plurality of feature maps, includes: Adding an attention mechanism to the first feature channel and the third feature channel so as to apply attention processing to the large-size feature map and the small-size feature map, and not adding an attention mechanism to the second feature channel so as not to apply attention processing to the intermediate-size feature map; The step of outputting the plurality of feature maps processed by sampling to the plurality of segmentation heads includes: The sampled large-scale feature map is output to the large-scale segmentation head. Output the sampled and processed multiple medium-sized feature maps to the multiple intermediate-sized segmentation heads respectively. The sampled small-size feature map is output to the small-size segmentation head.

2. The marking line recognition method according to claim 1, characterized in that: The attention mechanism includes an encoder for embedding spatial coordinate information of a marking area into a feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

3. The marking line recognition method according to claim 1, characterized in that: The backbone network includes multiple feature extraction convolution modules corresponding to the multiple feature channels respectively, at least some of the feature extraction convolution modules corresponding to the feature channels without adding the attention mechanism include deformable convolution extraction layers, and the feature extraction convolution modules corresponding to the feature channels with adding the attention mechanism include fixed convolution extraction layers.

4. The marking line recognition method according to claim 1, characterized in that: A ratio of a large-size feature map corresponding to the large-size segmentation head to a small-size feature map corresponding to the small-size segmentation head is greater than or equal to 8.

5. The method for recognizing a line marking according to any one of claims 1 to 4, further comprising: Inputting a training image containing annotated markings into a line recognition model framework to be trained to obtain a trained line recognition model, wherein the training image has true values ​​associated with the annotated markings, the true values ​​including true frames of the annotated markings; The training includes iteratively executing the following steps until a preset training completion condition is reached: Inputting the training samples into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line; Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculating of the loss value between the predicted value and the true value includes: Determine the area intersection-over-union ratio of the predicted box and the real box, and determine the distance between the box key points of the predicted box and the corresponding box key points of the real box based on a given plurality of box key points, wherein the plurality of box key points include diagonal points, Determining a key point intersection-to-union loss value according to the area intersection-to-union ratio and the distance; The parameters of the line recognition model framework are reversely updated based on the loss value.

6. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein the processor is configured to implement the method according to any one of claims 1 to 5 when running the computer program.

7. A program product comprising a computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images