Marking line identification method and device, marking line identification model training method, electronic equipment and program product

Abstract

The invention discloses a marking line identification method, which comprises the following steps: inputting an image related to a marking line into a marking line identification model based on a YOLO image segmentation algorithm to carry out marking line identification processing so as to mark a marking line area in the image, and specifically comprises the following steps: carrying out feature extraction on the input image, and outputting a plurality of feature maps through a plurality of feature channels; adding a partial attention mechanism to the plurality of feature channels so as to apply attention processing to parts of the plurality of feature maps and not apply attention processing to other parts of the plurality of feature maps; performing sampling processing on the plurality of feature maps subjected to partial attention processing, and correspondingly outputting the plurality of feature maps subjected to sampling processing to a plurality of segmentation heads; and respectively processing the plurality of feature maps subjected to sampling processing by using a plurality of segmentation heads so as to segment and extract marking areas in the plurality of feature maps. The invention also relates to a marked line identification model training method, a marked line identification model, electronic equipment and a program product.

Description

Technical Field

[0001] The present application relates to the field of highway traffic, and in particular to a road marking recognition method and device, electronic equipment and program products. Background Art

[0002] Traffic markings are key traffic control facilities painted on the road to channel traffic flow, guide vehicle alignment, and ensure traffic safety. These markings have multiple functions of defining lane boundaries, regulating and managing driver behavior, guiding driver vision, indicating and warning road conditions ahead, and clarifying the right of way.

[0003] Road marking recognition has a wide range of applications in the field of highway traffic. In particular, in the process of testing the retroreflection coefficient of road markings, accurate and rapid identification of road markings is a key link in evaluating the visibility of road markings under different lighting conditions.

[0004] In view of this, the industry is eager to improve the recognition accuracy and efficiency of road marking recognition solutions.

[0005] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0006] Therefore, the embodiments of the present application intend to provide a marking line recognition solution with high recognition accuracy and recognition efficiency.

[0007] In a first aspect, a method for identifying a marking line is provided, comprising:

[0008] Inputting an image related to the marking lines into a marking line recognition model based on the YOLO image segmentation algorithm for marking line recognition processing, thereby identifying the marking line area in the image, wherein the marking line recognition model based on the YOLO image segmentation algorithm includes a backbone network, a neck network, and a plurality of segment heads for processing feature maps of different sizes.

[0009] The marking line recognition process includes:

[0010] Using the backbone network to extract features from an input image, and outputting multiple feature maps through multiple feature channels, wherein the multiple feature channels correspond to the multiple segmentation heads;

[0011] Adding an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps, and not applying attention processing to other portions of the plurality of feature maps;

[0012] Using the neck network to perform sampling processing on the multiple feature maps processed by partial attention, and outputting the multiple feature maps processed by sampling to the multiple segmentation heads accordingly;

[0013] The plurality of segmentation heads are used to process the plurality of feature maps that have been sampled and processed respectively, so as to segment and extract the marked line areas in the plurality of feature maps.

[0014] Optionally, the plurality of segmentation heads include a large-size segmentation head, a plurality of intermediate-size segmentation heads and a small-size segmentation head.

[0015] The plurality of channels include a first characteristic channel corresponding to a large-size segmentation head, a plurality of second characteristic channels corresponding to a plurality of intermediate-size segmentation heads, and a third characteristic channel corresponding to a small-size segmentation head;

[0016] Wherein, outputting multiple feature maps through multiple feature channels includes:

[0017] Outputting the large-size feature map through the first feature channel,

[0018] Outputting the plurality of intermediate-size feature maps respectively through the plurality of second feature channels,

[0019] Outputting the small-size feature map through the third feature channel;

[0020] The applying an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps, and not applying attention processing to other portions of the plurality of feature maps, includes:

[0021] Adding an attention mechanism to the first feature channel and the third channel so as to apply attention processing to the large-size feature map and the small-size feature map, and not adding an attention mechanism to the second channel so as not to apply attention processing to the intermediate-size feature map;

[0022] The step of outputting the plurality of feature maps processed by sampling to the plurality of segmentation heads includes:

[0023] The sampled large-scale feature map is output to the large-scale segmentation head.

[0024] Outputting the sampled medium-sized feature maps to the plurality of second segmentation heads respectively.

[0025] The sampled small-size feature map is output to the small-size segmentation head.

[0026] Optionally, the attention mechanism includes an encoder for embedding spatial coordinate information of the marking area into the feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

[0027] Optionally, the backbone network includes multiple feature extraction convolution modules corresponding to the multiple feature channels respectively, at least some of the feature extraction convolution modules corresponding to the feature channels without adding the attention mechanism include deformable convolution extraction layers, and the feature extraction convolution modules corresponding to the feature channels with adding the attention mechanism include fixed convolution extraction layers.

[0028] Optionally, a ratio of a large-size feature map corresponding to the large-size segmentation head to a small-size feature map corresponding to the small-size segmentation head is greater than or equal to 8.

[0029] Optionally, before performing the marking line recognition process, the method further includes:

[0030] Inputting a training image containing annotated markings into a marking recognition model framework to be trained to obtain the trained marking recognition model, wherein the training image carries true values ​​associated with the annotated markings, and the true values ​​include true frames of the annotated markings;

[0031] The training includes iteratively executing the following steps until a preset training completion condition is reached:

[0032] Inputting the training sample into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line;

[0033] Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculation of the loss value between the predicted value and the true value includes:

[0034] Determine the intersection-over-union ratio of the predicted box and the true box,

[0035] According to a given plurality of box key points, determining the distance between the box key points of the predicted box and the corresponding box key points of the real box, wherein the plurality of box key points include diagonal points,

[0036] Determine a key point intersection-and-union ratio loss value according to the area intersection-and-union ratio and the distance;

[0037] The parameters of the marking line recognition model framework are reversely updated based on the loss value.

[0038] In a second aspect, a method for training a marking line recognition model is provided, comprising: inputting a training image containing annotated marking lines into a marking line recognition model framework to be trained to obtain a trained marking line recognition model;

[0039] The line marking recognition model framework to be trained is a line marking recognition model framework based on the YOLO image segmentation algorithm, which includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes.

[0040] The training image has true values ​​associated with the marked lines, and the true values ​​include true frames of the marked lines;

[0041] The training includes iteratively executing the following steps until a preset training completion condition is reached:

[0042] Inputting the training sample into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line;

[0043] Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculation of the loss value between the predicted value and the true value includes:

[0044] Determine the intersection-over-union ratio of the predicted box and the true box,

[0045] According to a given plurality of box key points, determining the distance between the box key points of the predicted box and the corresponding box key points of the real box, wherein the plurality of box key points include box diagonal points,

[0046] Determine a key point intersection-and-union ratio loss value according to the area intersection-and-union ratio and the distance;

[0047] Based on the loss value, the parameters of the marking line recognition model framework are reversely updated.

[0048] In a third aspect, a marking line recognition model is provided, wherein the marking line recognition model is trained by the marking line recognition model training method according to the second aspect.

[0049] According to a fourth aspect, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to implement the method according to the first aspect and the second aspect when running the computer program.

[0050] According to a fifth aspect, a program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect and the second aspect is implemented.

[0051] The marking line recognition method provided in the embodiment of the present application performs feature extraction on an input image and outputs multiple feature maps through multiple feature channels; adds a partial attention mechanism to the multiple feature channels, thereby applying attention processing to parts of the multiple feature maps, and not applying attention processing to other parts of the multiple feature maps; performs sampling processing on the multiple feature maps that have been partially attended to, and outputs the multiple feature maps that have been sampled to multiple segmentation heads accordingly; utilizes multiple segmentation heads to process the multiple feature maps that have been sampled, respectively, to segment and extract the marking line areas in the multiple feature maps, which can accurately and quickly identify the marking line, especially in the process of detecting the retroreflection coefficient of the marking line or in other scenarios where the marking line needs to be identified.

[0052] The optional features and other effects of the embodiments of the present application are partially described below and partially understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Hereinafter, embodiments of the present application will be described in detail in conjunction with the accompanying drawings. The elements shown are not limited to the proportions shown in the drawings and the same or similar reference numerals in the drawings represent the same or similar elements, wherein:

[0054] Figure 1 An exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0055] Figure 2 A second exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0056] Figure 3 A third exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0057] Figure 4 A schematic structural diagram of a marking line identification model according to a marking line identification method that can be used to implement an embodiment of the present application is shown;

[0058] Figure 5 Another schematic structural diagram of a marking line identification model that can be used to implement the marking line identification method according to an embodiment of the present application is shown;

[0059] Figure 6 Shows Figure 4 Schematic diagram of the structure of the feature extraction convolution module of the marking line recognition model shown;

[0060] Figure 7 A fourth exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0061] Figure 8 A fifth exemplary flow chart of a marking line recognition method according to an embodiment of the present application is shown;

[0062] Fig. 9 An exemplary flow chart showing a marking line recognition model training method of a marking line recognition method according to an embodiment of the present application; and

[0063] Fig.10 A structural diagram of an electronic device that can be used to implement a marking line recognition method and a marking line recognition model training method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0065] The term "including" and its variations used in this document represent open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one other embodiment". In order to facilitate the understanding of this specification, the sequential terms "first", "second", etc. are used in this document to distinguish different elements / items / objects, and do not indicate the order or importance of different elements / items / objects. In particular, the method steps represented by the terms "first", "second", etc. are not used to indicate the order of method execution; when an embodiment contains elements / items / objects represented in a later order, the elements / items / objects represented by the terms "first", "second", etc. in the previous order are not necessarily essential technical features of the embodiment.

[0066] As mentioned above, it is currently desired to provide a marking line recognition solution with higher recognition accuracy and recognition efficiency.

[0067] Therefore, in some embodiments of the present application, a marking line recognition method is provided. The method can perform marking line recognition based on a marking line recognition model of a YOLO image segmentation algorithm, thereby identifying a marking line area in the image.

[0068] In the embodiment of the present application, the YOLO (You Only Look Once) is an image processing algorithm that can be used for computer vision tasks, which can quickly perform real-time object detection and segmentation on images. In the preferred embodiment of the present application, a line recognition model based on an improved YOLOv8n-seg image segmentation algorithm can be used for processing.

[0069] like Figures 2 to 4 As shown, the marking line recognition model based on the YOLO image segmentation algorithm includes a backbone network (Backbone network) 220, a neck network (Neck network) 230 and a plurality of segmentation heads (Segment Head) 240 for processing feature maps of different sizes.

[0070] Accordingly, in the embodiment of the present application, the marking line recognition method includes: inputting an image 610, which is an image related to the marking line, into a marking line recognition model based on the YOLO image segmentation algorithm for marking line recognition processing, thereby identifying the marking line area in the image. More specifically, in conjunction with reference Figure 1 and Figures 4 to 6 The input image 410 is processed in sequence by the backbone network 420, the neck network 230 and multiple segmentation heads 441, 442, 443, and 444.

[0071] like Figure 4 and Figure 5 As shown, the backbone network 420 can be used to extract multiple feature maps from the input image 410, and output the multiple feature maps to the neck network 430 through multiple feature channels. Figure 4 and Figure 5 As shown, the backbone network 420 includes a plurality of feature extraction convolution modules 421, 422, 423, 424 corresponding to a plurality of feature channels. Figure 5 Specifically, in addition to the multiple feature extraction convolution modules 421, 422, 423, and 424 of the above-mentioned multiple feature channels, the backbone network 420 may also include additional layers or modules, which are not described in detail here.

[0072] like Figure 4 and Figure 5 As shown, the neck network 430 may include a bottom-up and top-down sampling structure, including a plurality of necessary neural network layers, such as various layers with sampling functions and required concatenation layers.

[0073] like Figure 4 and Figure 5 As shown, after the sampling process, the neck network 430 can output the sampled feature map to the corresponding segmentation heads 441 to 444. Figure 5 In the specific illustrated embodiment, each segmentation head uses a dual loss function to jointly perform detection and segmentation, namely, a prediction box loss function (Bbox Loss) and a target classification loss function (ClsLoss).

[0074] In the embodiment of the present application, the (improved) YOLO-based image segmentation algorithm refers to a YOLO image segmentation algorithm that uses any novel feature of the embodiment of the present application to optimize the marking line recognition, and in particular, refers to a YOLOv8n-seg image segmentation algorithm that uses any novel feature of the embodiment of the present application to optimize the marking line recognition. It can be imagined that the features of the embodiment of the present application for marking line recognition processing can be combined with or used to improve other end-to-end image segmentation algorithm frameworks, and this application or subsequent applications can cover such novel combinations and improvements.

[0075] In a specific example, the existing road marking detection device can be configured by the road marking detection model to obtain multiple road images, and the recognition results can be verified by dividing them into training set, test set and verification set in a certain ratio, such as 8:1:1.

[0076] Before the road marking recognition process or as a preprocessing step, the image used as the training set may also be annotated. In one example, an image annotation tool (Labelme image labeling software) may be used to annotate the road marking images in the training set. In order to accurately obtain the boundary coordinate points and categories of road markings during deep learning, an irregular bounding box mask is used in the Labelme software to mark the target during annotation, and then the category of the target road marking is created and named, and different color masks are used to mark targets of different categories.

[0077] In some embodiments of the present application, Figure 1 As shown, the processing using the marking line recognition model based on the YOLO image segmentation algorithm may specifically include steps S100 to S140:

[0078] S100 (not marked): inputting an image related to marking lines into a marking line recognition model based on the YOLO image segmentation algorithm to perform marking line recognition processing, thereby marking a marking line area in the image.

[0079] In the embodiments of the present application, the images related to the road markings include but are not limited to the road marking images obtained by the road marking detection device, the monitoring images related to the road markings obtained by the road monitoring, and the video frames related to the road markings obtained by the vehicle camera, but the present application is not limited thereto.

[0080] S110: Extract features of the input image using the backbone network, and output multiple feature maps through multiple feature channels.

[0081] In the embodiment of the present application, for example, in the above step S110, the multiple feature channels correspond to the multiple segmentation heads.

[0082] S120: Add an attention mechanism to the multiple feature channel parts, so as to apply attention processing to the parts of the multiple feature maps, and do not apply attention processing to other parts of the multiple feature maps.

[0083] S130: Using the neck network to perform sampling processing on the multiple feature maps that have been partially processed by attention, and outputting the multiple feature maps that have been sampled to the multiple segmentation heads accordingly.

[0084] S140: using a plurality of segmentation heads to respectively process the plurality of feature maps that have been sampled, so as to segment and extract the marked line areas in the plurality of feature maps.

[0085] Combined with reference Figures 4 to 6 The plurality of segmentation heads may include a large-size segmentation head 441 , a plurality of (eg, two) intermediate-size segmentation heads 442 , 443 , and a small-size segmentation head 444 .

[0086] In this preferred embodiment, a small-size segmentation head is added to the marking line recognition model based on the improved YOLO image segmentation algorithm, which can also be called a small target segmentation head.

[0087] Therefore, if Figure 2 As shown, in the above step S110, outputting multiple feature maps through multiple feature channels may specifically include:

[0088] S111: Output the large-size feature map through the first feature channel. S112: Output multiple intermediate-size feature maps through multiple second feature channels respectively. S113: Output the small-size feature map through the third feature channel.

[0089] Furthermore, if Figures 4 to 6 As shown, the plurality of channels also include a first characteristic channel corresponding to a large-size segmentation head, a plurality of second characteristic channels corresponding to a plurality of intermediate-size segmentation heads, and a third characteristic channel corresponding to a small-size segmentation head.

[0090] Therefore, if Figure 3As shown, in the above step S130, the outputting the plurality of feature maps processed by sampling to the plurality of segmentation heads may specifically include:

[0091] S131: Output the sampled large-size feature map to the large-size segmentation head. S132: Output the sampled multiple intermediate-size feature maps to the multiple intermediate segmentation heads. S133: Output the sampled small-size feature map to the small-size segmentation head.

[0092] In the road marking recognition model based on the improved YOLO image segmentation algorithm in the embodiment of the present application, after being processed by the backbone network (Backbone network) and the neck network (Neck network), not only a large-scale feature map and several intermediate-scale feature maps are output to the corresponding segmentation head for instance segmentation, but also a small-scale feature map is output to the corresponding small-scale segmentation head (small target segmentation head) to segment small target markings, especially for the specific scene of road marking recognition. As an explanation but not a limitation, the pixel area occupied by road markings captured by a camera at a small angle in the picture has a large scale difference, and the conventional image segmentation algorithm that has not been specifically modified for road marking recognition has a weak instance detection capability for small-scale targets, so it is easy to miss the detection of road markings that are far away from the camera.

[0093] In a preferred embodiment of the present application, the ratio of the large-size feature map corresponding to the large-size segmentation head to the small-size feature map corresponding to the small-size segmentation head is greater than or equal to 8. In one example, the input image size of 640*640 can be downsampled by 20 times (corresponding to the first feature channel of the large-size segmentation head), 40 times, 80 times and 160 times (corresponding to the third feature channel of the small-size segmentation head) to obtain feature maps of sizes 32*32, 16*16, 8*8 and 4*4, which are respectively used for segmentation by multiple feature heads. By using a segmentation head for a small target in the line recognition model of the improved YOLO image segmentation algorithm (and the corresponding backbone network and neck network structure, as further described below), the instance detection capability of small target road markings is effectively improved.

[0094] Combined with reference Figures 1 to 3 and Figures 4 to 6 In step S120, an attention mechanism, such as a channel attention mechanism (Coordinate Attention, CA for short), is added to the multiple feature channel parts, so that attention processing is applied to parts of the multiple feature maps, and no attention processing is applied to other parts of the multiple feature maps.

[0095] In some embodiments of the present application, in combination with reference Figures 4 to 6 , the above step S120 may correspondingly include:

[0096] S121 (unmarked): adding an attention mechanism to the first feature channel and the third channel so as to apply attention processing to the large-size feature map and the small-size feature map, and not adding an attention mechanism to the second channel so as not to apply attention processing to the intermediate-size feature map.

[0097] In other words, in this embodiment, the channel attention mechanism 460 is added only to the first and third feature channels corresponding to the large-size segmentation head and the small-size segmentation head, while the channel attention mechanism is not added to the second feature channel corresponding to the intermediate-size segmentation head.

[0098] In a further embodiment of the present application, when the attention mechanism is applied to multiple feature channel parts, the feature map sizes processed by the segmentation head corresponding to the feature channels to which the attention mechanism is added are not adjacent. In other words, in this embodiment, the channel attention mechanism 460 is added only to the first and third feature channels corresponding to the large-size feature map and the small-size feature map, while the channel attention mechanism is not added to the second feature channel corresponding to the intermediate-size feature map.

[0099] In some embodiments of the present application, the attention mechanism includes an encoder for embedding spatial coordinate information of the marking area into the feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

[0100] In the embodiment of the present application, the channel attention mechanism is a method of increasing the importance of the channel and the attention information by applying weights to the feature maps of each channel with the attention mechanism added. The inventors have made the following surprising discovery through research, that is, adding a partial channel attention mechanism, such as adding a channel attention mechanism non-adjacently, and especially adding a channel attention mechanism only to the feature maps of the maximum size and the minimum size, can avoid various interference factors on the road from interfering with the target information of the road markings to the greatest extent. As an explanation but not limitation, through the backbone network from bottom to top (such as Figure 4 and Figure 5 ) Extract deeper features layer by layer. The feature map at the top contains the richest feature information and the most ambiguous position information, while the feature map at the bottom is the opposite. By applying the channel attention mechanism to the feature channels at both ends, the clearest position information and feature information can be extracted and fused to obtain higher accuracy. In addition, the partially added channel attention mechanism according to the embodiment of the present application, such as adding the channel attention mechanism non-adjacently, especially adding the channel attention mechanism only for the maximum size and minimum size feature maps, can also be combined with further preferred features to obtain further beneficial effects of optimizing the recognition of marking lines.

[0101] Continue to refer Figures 4 to 6As mentioned above, the backbone network 420 includes a plurality of feature extraction convolution modules 421, 422, 423, and 424 corresponding to a plurality of feature channels. In a preferred embodiment of the present application, at least some, preferably all, of the feature extraction convolution modules corresponding to the feature channels without adding the attention mechanism may include a deformable convolution extraction layer, and the feature extraction convolution modules corresponding to the feature channels with adding the attention mechanism include a fixed convolution extraction layer. Figure 5 Specifically, it is shown that the feature extraction convolution modules 421 and 424 corresponding to the first and third feature channels can include C2F (Coarse to Fine) fixed convolution extraction layers; and the feature extraction convolution modules 422 and 423 corresponding to the second feature channel to which the attention mechanism is not added can include deformable convolution extraction layers 4221 and 4231.

[0102] In a preferred embodiment of the present application, the deformable convolution extraction layer may include a DCNv3 deformable convolution layer. Figure 6 , showing that the deformable convolution extraction layer 4221 has a DCNv3 deformable convolution functional layer. In the embodiment of the present application, it will be understood that the convolution extraction layer can be interpreted broadly, which may include a functional convolution (sub) layer with an extraction function, and may also include additional (sub) layers such as a batch layer, a bottleneck layer, and a splicing layer as needed. The embodiment of the present application does not limit the different numbers and structures of additional (sub) layers in the convolution extraction layer.

[0103] The inventors noticed that when collecting road markings, the road markings may be severely distorted in shape due to the turning and lane changing of the road marking collection vehicle, and the angle of the road markings in the image may also change unknownly. The inventors made the following surprising discovery through research: by making the feature extraction convolution module corresponding to at least some feature channels without adding the attention mechanism include a variable convolution extraction layer, the features of these distorted road markings can be effectively and accurately extracted.

[0104] As mentioned above, if Figure 5 As shown, each segmentation head uses a dual loss function to jointly perform detection and segmentation, namely, a box loss function (Bbox Loss) and an object classification loss function (Cls Loss). In the embodiment of the present application, the box loss function (Bbox Loss) is further improved to optimize the recognition of the marking line.

[0105] In a further embodiment of the present application, Figure 7 As shown, before performing the marking line recognition process, the relevant method may optionally include a training step, specifically including:

[0106] S700 (not marked): inputting the training image containing the annotated marking lines into the marking line recognition model framework to be trained to obtain a trained marking line recognition model.

[0107] In some embodiments of the present application, the training image carries true values ​​associated with the labeled lines, and the true values ​​include true boxes of the labeled lines.

[0108] In some embodiments of the present application, the training includes iteratively executing the following steps until a preset training completion condition is reached:

[0109] S710: Input the training samples into the marking line recognition model framework to obtain the prediction value related to the marked marking line.

[0110] In some embodiments of the present application, for example, in the above step S710, the predicted value includes a predicted box with marked lines.

[0111] S720: Based on a given loss function, calculate the loss value between the predicted value and the true value.

[0112] In some embodiments of the present application, for example, in the above step S720, the given loss function includes a key point intersection-over-union (MPDIOU) loss function.

[0113] In some embodiments of the present application, Figure 8 As shown, the calculation of the loss value between the predicted value and the true value may include the following steps: S721: Determine the area intersection and union ratio of the predicted box and the true box. S722: Determine the distance between the box key points of the predicted box and the corresponding box key points of the true box according to the given multiple box key points. S723: Determine the key point intersection and union loss value according to the area intersection and union ratio and the distance.

[0114] S730: Reversely update the parameters of the line recognition model framework based on the loss value.

[0115] In the preferred embodiment, the plurality of frame key points include diagonal points. The selection of frame diagonal points (frame diagonal lines) located on the frame diagonal lines is determined based on the orientation and / or shape distortion of the marked marking lines.

[0116] In some known solutions, the border loss function Box_Loss = DFL_Loss + CIOU_Loss is used to measure the overlap between the predicted box and the true box. DFL_Loss is a deep feature loss function, and CIOU_Loss is a complete intersection-over-union loss function, which uses the distance ratio between the "true box" and the "predicted box" to measure the overlap between the predicted box and the true box. However, after research, the inventor found that this loss function cannot improve the loss value well during line recognition (training).

[0117] In this regard, an embodiment of the present application proposes to use a key point intersection-over-union (MPDIOU) loss function, and preferably, the diagonal points of the box are used as key points of the key point intersection-over-union (MPDIOU) loss function.

[0118] In a further embodiment of the present application, for example in the above-mentioned step S721, the selection of the frame diagonal points (frame diagonal lines) located on the frame diagonal lines is determined according to the direction and / or shape distortion of the marked marking lines. In one example, for marking lines turning right. The upper right and lower left frame diagonal points can be used as key points. In one example, for marking lines turning left, the upper left and lower right frame diagonal points can be used as key points. In one example, for straight marking lines in an image acquired when turning right (and thus causing shape distortion), the upper right and lower left predicted frame diagonal points can be used as key points. In one example, for straight marking lines in an image acquired when turning left (and thus causing shape distortion), the upper left and lower right predicted frame diagonal points can be used as key points.

[0119] By way of explanation and not limitation, for road marking recognition, when the predicted box and the real box have the same aspect ratio but different width and height values, it is difficult to improve the loss value; the embodiment of the present application utilizes the key point intersection-over-union (MPDIOU) loss function, and uses the diagonal points of the predicted box as key points, which can accurately segment road markings caused by interference such as occlusion by vehicles and pedestrians; further, the embodiment of the present application also determines the direction of the diagonal points of the predicted box (box diagonal) based on the orientation and / or shape distortion of the marked line markings, thereby further improving the ability to accurately detect line markings.

[0120] In one embodiment, the accuracy, recall rate and precision of road marking detection and segmentation in the image verification set are analyzed. The test results show that the model according to the embodiment of the present application can not only meet the end-to-end real-time road marking recognition, but also achieves an accuracy rate of 98.4%, a recall rate of 94.7% and a precision of 96.7% in the detection category; the accuracy rate in segmentation reaches 98.3%, the recall rate reaches 94.2% and the precision reaches 96.4%. The model realizes accurate detection and segmentation of road markings.

[0121] The method for identifying line markings provided in the embodiments of the present application performs feature extraction on an input image and outputs multiple feature maps through multiple feature channels; adds a partial attention mechanism to the multiple feature channels, thereby applying attention processing to parts of the multiple feature maps and not applying attention processing to other parts of the multiple feature maps; performs sampling processing on the multiple feature maps that have been partially processed with attention, and outputs the multiple feature maps that have been sampled to multiple segmentation heads accordingly; and uses multiple segmentation heads to process the multiple feature maps that have been sampled, respectively, to segment and extract the line marking areas in the multiple feature maps, which can achieve accurate and rapid identification of line markings, especially in the process of detecting the line marking retroreflection coefficient or other scenarios where line markings need to be identified, such as unmanned driving and the like.

[0122] Accordingly, in some embodiments of the present application, Fig. 9 As shown, a marking line recognition model training method is also provided, which may include the following steps S900 to S930:

[0123] S900 (not marked): inputting the training image containing the marked lines into the line recognition model framework to be trained for training, so as to obtain a trained line recognition model.

[0124] In an embodiment of the present application, the line recognition model framework to be trained is a line recognition model framework based on the YOLO image segmentation algorithm, which includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes.

[0125] In some embodiments of the present application, the training image carries true values ​​associated with the labeled lines, and the true values ​​include true boxes of the labeled lines.

[0126] In the embodiments of the present application, Fig. 9 As shown, the training includes iteratively executing the following steps until a preset training completion condition is reached:

[0127] S910: Input the training sample into the marking line recognition model framework to obtain the prediction value related to the marked marking line.

[0128] In the embodiment of the present application, for example, in the above step S910, the predicted value includes a predicted box with annotated marking lines.

[0129] S920: Based on a given loss function, calculate the loss value between the predicted value and the true value.

[0130] In an embodiment of the present application, for example, in the above step S910, the given loss function may include a key point intersection-over-union (MPDIOU) loss function. In an embodiment of the present application, the calculation of the loss value between the predicted value and the true value may specifically include the following steps A1 to A3: A1: Determine the area intersection-over-union ratio of the predicted box and the true box. A2: Determine the distance between the box key points of the predicted box and the corresponding box key points of the true box based on a given plurality of box key points. In an embodiment of the present application, for example, in the above step A2, the plurality of predicted box key points include box diagonal points. In a further embodiment of the present application, for example, in the above step A2, the selection of the box diagonal points (box diagonal lines) located on the box diagonal is determined according to the direction and / or shape distortion of the marked marking line. A3: Determine the key point intersection-over-union loss value based on the area intersection-over-union ratio and the distance.

[0131] S930: Based on the loss value, reversely update the parameters of the marking line recognition model framework.

[0132] The specific description of the above steps S900 to S930 and their sub-steps of the present application can refer to the description of steps S800 to S830 and their sub-steps in the aforementioned embodiments of the present application, and will not be repeated here.

[0133] In some embodiments of the present application, a marking line recognition model is further provided, and the marking line recognition model is trained by the marking line recognition model training method described in the above embodiments of the present application.

[0134] In the embodiments of the present application, the marking line recognition method and the marking line recognition model training method can be implemented by an electronic device integrating relevant functional modules or components. Fig.10 A schematic diagram of an electronic device that can be used to implement the above method and device of the embodiment of the present application is shown.

[0135] like Fig.10As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1002 or the programs and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 may include a central processing unit (CPU), and may be a multi-core processor, or may include multiple processors. In some embodiments, processor 1001 may include a general main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of electronic device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0136] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the steps or functions of the given spacing determination method based on the marking line retroreflection evaluation model described in the above embodiments can be realized.

[0137] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Fig.10 Only some components are schematically shown in the figure, which does not mean that the computer system 1000 only includes Fig.10 Components shown.

[0138] In some embodiments, the computer 1000 refers to a mobile terminal, including a mobile phone, a car terminal, etc. Taking a mobile phone as an example, the electronic device 1000 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by the computer or its associated components. The computer may be, for example, a mobile terminal, a personal computer, a notebook computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a tablet computer, or a combination thereof.

[0140] Although not shown, in an embodiment of the present application, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute any method of the embodiments of the present application when executed.

[0141] Although not shown, in an embodiment of the present application, a program product is provided, including a computer program, wherein the computer program is configured to execute any method of the embodiments of the present application when executed.

[0142] The storage media in the embodiments of the present application include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0143] The methods, programs, systems, devices, etc. of the embodiments of the present application may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0144] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present application do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] In this article, multiple embodiments of the present application are described, but for the sake of simplicity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the various embodiments may be omitted. In this article, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to be applicable to at least one embodiment or example according to the present application, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0146] The exemplary systems and methods of the present application have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present application as defined in the appended claims.

Claims

1. A marking line recognition method, characterized in that: include: Inputting an image related to the marking lines into a marking line recognition model based on the YOLO image segmentation algorithm for marking line recognition processing, thereby identifying the marking line area in the image, wherein the marking line recognition model based on the YOLO image segmentation algorithm includes a backbone network, a neck network, and a plurality of segment heads for processing feature maps of different sizes. The marking line recognition process includes: Using the backbone network to extract features from an input image, and outputting multiple feature maps through multiple feature channels, wherein the multiple feature channels correspond to the multiple segmentation heads; Adding an attention mechanism to the plurality of feature channel portions, thereby applying attention processing to a portion of the plurality of feature maps, and not applying attention processing to other portions of the plurality of feature maps; Using the neck network to perform sampling processing on the multiple feature maps processed by partial attention, and outputting the multiple feature maps processed by sampling to the multiple segmentation heads accordingly; The plurality of segmentation heads are used to process the plurality of feature maps that have been sampled and processed respectively, so as to segment and extract the marked line areas in the plurality of feature maps.

2. The marking line recognition method according to claim 1, characterized in that: The plurality of segmentation heads include a large-size segmentation head, a plurality of intermediate-size segmentation heads and a small-size segmentation head. The plurality of channels include a first characteristic channel corresponding to a large-size segmentation head, a plurality of second characteristic channels corresponding to a plurality of intermediate-size segmentation heads, and a third characteristic channel corresponding to a small-size segmentation head; Wherein, outputting multiple feature maps through multiple feature channels includes: Outputting the large-size feature map through the first feature channel, Outputting the plurality of intermediate-size feature maps respectively through the plurality of second feature channels, Outputting the small-size feature map through the third feature channel; The adding of the attention mechanism to the plurality of feature channel parts, thereby applying attention processing to a part of the plurality of feature maps and not applying attention processing to other parts of the plurality of feature maps, includes: Adding an attention mechanism to the first feature channel and the third channel so as to apply attention processing to the large-size feature map and the small-size feature map, and not adding an attention mechanism to the second channel so as not to apply attention processing to the intermediate-size feature map; The step of outputting the plurality of feature maps processed by sampling to the plurality of segmentation heads includes: The sampled large-scale feature map is output to the large-scale segmentation head. Outputting the sampled medium-sized feature maps to the plurality of second segmentation heads respectively. The sampled small-size feature map is output to the small-size segmentation head.

3. The marking line recognition method according to claim 1, characterized in that: The attention mechanism includes an encoder for embedding spatial coordinate information of a marking area into a feature and a coordinate attention generation module for generating coordinate attention based on the embedded spatial coordinate information of the marking area.

4. The marking line recognition method according to claim 1, characterized in that: The backbone network includes multiple feature extraction convolution modules corresponding to the multiple feature channels respectively, at least some of the feature extraction convolution modules corresponding to the feature channels without adding the attention mechanism include deformable convolution extraction layers, and the feature extraction convolution modules corresponding to the feature channels with adding the attention mechanism include fixed convolution extraction layers.

5. The marking line recognition method according to claim 2, characterized in that: A ratio of a large-size feature map corresponding to the large-size segmentation head to a small-size feature map corresponding to the small-size segmentation head is greater than or equal to 8.

6. The method for identifying a line marking according to any one of claims 1 to 5, before performing the line marking identification process, the method further comprises: Inputting a training image containing annotated markings into a marking recognition model framework to be trained to obtain the trained marking recognition model, wherein the training image carries true values ​​associated with the annotated markings, and the true values ​​include true frames of the annotated markings; The training includes iteratively executing the following steps until a preset training completion condition is reached: Inputting the training sample into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line; Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculation of the loss value between the predicted value and the true value includes: Determine the intersection-and-union ratio of the predicted box and the real box, and determine the distance between the box key points of the predicted box and the corresponding box key points of the real box according to a given plurality of box key points, wherein the plurality of box key points include diagonal points, Determine a key point intersection-and-union ratio loss value according to the area intersection-and-union ratio and the distance; The parameters of the marking line recognition model framework are reversely updated based on the loss value.

7. A marking line recognition model training method, characterized in that: include: Input the training image containing the annotated markings into the marking recognition model framework to be trained to obtain a trained marking recognition model. The line marking recognition model framework to be trained is a line marking recognition model framework based on the YOLO image segmentation algorithm, which includes a backbone network, a neck network, and multiple segmentation heads for processing feature maps of different sizes. The training image has true values ​​associated with the marked lines, and the true values ​​include true frames of the marked lines; The training includes iteratively executing the following steps until a preset training completion condition is reached: Inputting the training sample into the line recognition model framework to obtain a prediction value related to the marked line, wherein the prediction value includes a prediction box of the marked line; Based on a given loss function, a loss value between a predicted value and a true value is calculated, wherein the given loss function includes a key point intersection-over-union (MPDIOU) loss function, and the calculation of the loss value between the predicted value and the true value includes: Determine the intersection-over-union ratio of the predicted box and the true box, According to a given plurality of prediction box key points, determining the distance between the box key points of the prediction box and the corresponding box key points of the real box, wherein the plurality of prediction box key points include box diagonal points, Determine a key point intersection-and-union ratio loss value according to the area intersection-and-union ratio and the distance; Based on the loss value, the parameters of the marking line recognition model framework are reversely updated.

8. A marking line recognition model, characterized in that: The marking line recognition model is obtained by training the marking line recognition model training method according to claim 7.

9. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein the processor is configured to implement the method according to any one of claims 1 to 7 when running the computer program.

10. A program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images