Lane line detection method, storage medium and program product

Abstract

The invention discloses a lane line detection method, a storage medium and a program product, and belongs to the technical field of image processing. The method comprises the following steps: performing lane line feature extraction on each frame of picture in a target picture set through a feature extraction network to obtain a first lane line feature of each frame of picture in the target picture set; executing a first fusion task based on the first lane line feature through a feature fusion network to determine a second lane line feature of each frame of picture in the target picture set, and executing a second fusion task based on the second lane line feature to determine a third lane line feature of the to-be-detected picture; and determining a lane line detection result of the to-be-detected picture based on the third lane line feature through an attention network and a lane line detection head which are connected in sequence. The lane line detection accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a lane line detection method, storage medium and program product. Background Art

[0002] Lane line detection technology is an important detection technology for realizing autonomous driving. It can detect and mark the lane lines on the road, thereby guiding the vehicle to drive in the correct area, providing a basis for the automatic cruising, vehicle holding and vehicle overtaking of autonomous vehicles, and contributing to the safe driving of the car.

[0003] The current lane line detection technology is based on the current frame to obtain the lane line detection result of the current frame. Once a frame has partial occlusion or reflection, it will directly affect the overall lane line detection result, resulting in low lane line detection accuracy. Summary of the invention

[0004] The embodiments of the present application provide a lane line detection method, storage medium and program product, which can improve the detection accuracy of lane lines.

[0005] In a first aspect, an embodiment of the present application provides a lane line detection method, the method comprising:

[0006] Input a target picture set corresponding to the picture to be detected into a feature extraction network, and extract lane line features from each frame of the picture in the target picture set through the feature extraction network to obtain the first lane line features of each frame of the picture in the target picture set; wherein the target picture set includes the picture to be detected and historical pictures collected earlier than the picture to be detected;

[0007] Input the first lane line feature into a feature fusion network, and determine the second lane line feature of each frame image in the target image set by performing a first fusion task based on the first lane line feature through the feature fusion network, and determine the third lane line feature of the image to be detected by performing a second fusion task based on the second lane line feature; wherein the second lane line feature of each frame image in the target image set is determined based on the first lane line feature of the image and the first lane line feature of a historical image in the target image set that was collected earlier than the image; and the third lane line feature is determined based on the second lane line feature of the image to be detected and the second lane line feature of a historical image in the target image set that was collected earlier than the image to be detected;

[0008] Through the attention network and the lane line detection head connected in sequence, the lane line detection result of the image to be detected is determined based on the third lane line feature.

[0009] In a second aspect, an embodiment of the present application provides a lane line detection device, the device comprising:

[0010] A first acquisition module is used to input a target picture set corresponding to the picture to be detected into a feature extraction network, and extract lane line features from each frame of the picture in the target picture set through the feature extraction network to obtain a first lane line feature of each frame of the picture in the target picture set; wherein the target picture set includes the picture to be detected and historical pictures collected earlier than the picture to be detected;

[0011] A second acquisition module is used to input the first lane line feature into a feature fusion network, and determine the second lane line feature of each frame picture in the target picture set by performing a first fusion task based on the first lane line feature through the feature fusion network, and determine the third lane line feature of the picture to be detected by performing a second fusion task based on the second lane line feature; wherein the second lane line feature of each frame picture in the target picture set is determined based on the first lane line feature of the picture and the first lane line feature of a historical picture in the target picture set whose acquisition time is earlier than that of the picture; the third lane line feature is determined based on the second lane line feature of the picture to be detected and the second lane line feature of a historical picture in the target picture set whose acquisition time is earlier than that of the picture to be detected;

[0012] The third acquisition module is used to determine the lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence.

[0013] In a third aspect, an embodiment of the present application provides a lane line detection device, the device comprising:

[0014] a processor and a memory storing computer program instructions;

[0015] When the processor executes the computer program instructions, the lane line detection method as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the lane line detection method described in the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the lane line detection method as described in the first aspect.

[0018] In an embodiment of the present application, when lane line detection is performed on a picture to be detected, a target picture set corresponding to the picture to be detected can be first obtained, which includes the picture to be detected and its corresponding historical picture; then, lane line features are extracted for each frame picture in the target picture set through a backbone feature network to obtain a first lane line feature of each frame picture (which can be regarded as a temporal feature); thereafter, a feature fusion model can be used to first perform a first fusion task based on the first lane line feature to determine a second lane line feature of each frame picture in the target picture set, and then a second fusion task is performed based on the second lane line feature to determine a third lane line feature of the picture to be detected, wherein the second lane line feature of each frame picture in the target picture set is determined based on its own first lane line feature and the first lane line feature of its corresponding historical picture, and the third lane line feature of the picture to be detected is determined based on its own second lane line feature and the second lane line feature of its corresponding historical picture; then, through the attention network and the lane line detection head connected in sequence, the lane line detection result of the picture to be detected is determined based on the third lane line feature. It can be seen that in the embodiment of the present application, the lane line detection result of the image to be detected is determined based on the lane line features obtained by secondary fusion of the temporal features. In this way, the lane line features of the image to be detected can be effectively compensated by the lane line features of the historical images. Moreover, since the historical images corresponding to each frame of the target image set are different, the influence weight of the lane line features of the single-frame historical image on the lane line features of the image to be detected can be reduced through secondary fusion. In this way, even if some images in the target image set are occluded or reflected, the lane line misses caused by occlusion or reflection can be reduced, thereby improving the detection accuracy of the lane lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is one of the schematic diagrams of lane line detection provided in the embodiment of the present application;

[0021] Figure 2 is a flow chart of a lane line detection method provided in an embodiment of the present application;

[0022] Figure 3 This is the second schematic diagram of lane line detection provided in the embodiment of the present application;

[0023] Figure 4a yes Figure 3 Schematic diagram of the working principle of the feature fusion network;

[0024] Figure 4b yes Figure 3 Schematic diagram of the working principle of the encoder;

[0025] Figure 4c yes Figure 3 Schematic diagram of the working principle of the decoder;

[0026] Figure 5 is one of the schematic diagrams of the lane line detection results provided in the embodiment of the present application;

[0027] Figure 6 This is a second schematic diagram of the lane detection result provided in an embodiment of the present application;

[0028] Figure 7 is a structural diagram of a lane line detection device provided in an embodiment of the present application;

[0029] Figure 8 It is a structural diagram of the lane detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0032] For ease of understanding, some contents involved in the embodiments of the present application are described below:

[0033] When performing lane line detection, you can use Figure 1 The lane detection method shown.

[0034] for Figure 1 The lane line detection method shown uses a single frame image for lane line detection. Specifically, the lane line features of the input image are extracted through the Backbone network, and then the features are fused through the Neck network. The lane line detection result is output through the lane line detection head. The lane line detected has poor accuracy and is prone to lane line misses.

[0035] Based on this, an embodiment of the present application provides a new lane line detection method, which can not only effectively compensate the lane line features of the image to be detected through the lane line features of the historical image, but also reduce the influence weight of the lane line features of the single-frame historical image on the lane line features of the image to be detected, thereby improving the accuracy of lane line detection.

[0036] The lane line detection method provided in the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0037] The lane line detection method provided in the embodiment of the present application can be applied to terminals, servers, service platforms, cloud, distributed systems, Internet of Things, Internet of Vehicles systems, etc. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.

[0038] See also Figure 2 , Figure 2 : is a flow chart of the lane line detection method provided in the embodiment of the present application. Figure 2 As shown, the lane line detection method may include the following steps:

[0039] Step 201: input a target picture set corresponding to a picture to be detected into a feature extraction network, and extract lane line features from each frame of the picture in the target picture set through the feature extraction network to obtain a first lane line feature of each frame of the picture in the target picture set; wherein the target picture set includes the picture to be detected and historical pictures whose acquisition time is earlier than that of the picture to be detected.

[0040] In an embodiment of the present application, a target image set corresponding to the image to be detected may be used to determine the lane line detection result of the image to be detected.

[0041] When the vehicle is performing automatic driving, the forward-looking camera can be used to collect road condition pictures according to a preset picture collection frequency, so as to detect lane lines through road condition pictures, and provide a basis for automatic driving. The pictures in the target picture set corresponding to the picture to be detected can all be road condition pictures collected by the forward-looking camera. The picture to be detected can be understood as the road condition picture currently collected by the forward-looking camera. In addition to the picture to be detected, the target picture set corresponding to the picture to be detected further includes historical pictures whose collection time is earlier than the picture to be detected. In some embodiments, the target picture set may include historical pictures of a preset frame number corresponding to the picture to be detected. These historical pictures are road condition pictures collected historically by the forward-looking camera, and their collection time is continuous with the collection time of the picture to be detected.

[0042] Furthermore, the acquisition perspectives of each frame of the target image set may be the same or different, and may be set based on actual needs, and the embodiments of the present application do not limit this. It can be understood that occlusion or reflection may only exist due to a specific perspective. Therefore, if the target image set includes images acquired from different acquisition perspectives, that is, lane line detection is performed using images from different acquisition perspectives, it can reduce lane line misses caused by occlusion or reflection to a certain extent, thereby improving the accuracy of lane line detection.

[0043] In specific implementation, in order to obtain the lane line detection result of the image to be detected, the target image set corresponding to the image to be detected can be obtained first, and then the target image set can be input into the feature extraction network, so as to extract lane line features from each frame of the target image set through the feature extraction network, and obtain the first lane line feature of each frame of the target image set. It can be understood that the input of the feature extraction network is the target image set, and the output is the first lane line feature of each frame of the target image set.

[0044] Since the acquisition time of each frame of the target image set has a temporal relationship, the first lane line feature of each frame of the image can be regarded as the temporal feature of the image to be detected.

[0045] The embodiments of the present application do not limit the network results of the feature extraction network. In some embodiments, the feature extraction network may be a Backbone network, but is not limited thereto.

[0046] Step 302: The first lane line feature is input into a feature fusion network, and the feature fusion network performs a first fusion task based on the first lane line feature to determine a second lane line feature of each frame image in the target image set, and performs a second fusion task based on the second lane line feature to determine a third lane line feature of the image to be detected; wherein the second lane line feature of each frame image in the target image set is determined based on the first lane line feature of the image and the first lane line feature of a historical image in the target image set that was collected earlier than the image; and the third lane line feature is determined based on the second lane line feature of the image to be detected and the second lane line feature of a historical image in the target image set that was collected earlier than the image to be detected.

[0047] After obtaining the first lane line features of each frame image in the target image set, these first lane line features can be input into a feature fusion network to obtain the third lane line features of the image to be detected through the feature fusion network.

[0048] In an embodiment of the present application, the feature fusion network can perform a secondary fusion of the temporal features. Specifically, the first lane line features can be firstly fused by executing a first fusion task to obtain the second lane line features of each frame image in the target image set, and then the second lane line features can be fused by executing a second fusion task.

[0049] In specific implementation, for the first fusion task, the current frame fusion feature can be obtained by fusing the current frame feature with the historical frame feature, wherein the current frame feature is the first lane line feature of the current frame, and the current frame can be any frame in the target picture set; the historical frame feature can be understood as the first lane line feature of the first historical frame of the current frame, and the first historical frame can include some or all pictures in the target picture set whose acquisition time is before the current frame, that is, the first historical frame can include some or all historical frames of the current frame; the current frame fusion feature can be understood as the second lane line feature of the current frame.

[0050] It is worth noting that the number of frames L1 of the first historical frame can be preset. If there is no historical frame for the current frame, the current frame feature can be directly determined as the current frame fusion feature, that is, the historical frame feature can be regarded as 0; if the number of historical frames of the current frame is less than or equal to L1, the current frame fusion feature can be obtained by fusing the current frame feature and the first lane line features of all historical frames of the current frame; if the number of historical frames of the current frame is greater than L1, the L1 historical frames whose acquisition time is closest to the current frame can be selected from the historical frames of the current frame, and the current frame fusion feature can be obtained by fusing the first lane line features of these L1 historical frames.

[0051] For the second fusion task, the final fusion feature of the current frame can be obtained by fusing the fusion feature of the current frame with the fusion feature of the historical frame, wherein the fusion feature of the historical frame can be understood as the second lane line feature of the second historical frame, and the second historical frame may include some or all of the images in the target image set whose acquisition time is before the current frame, that is, the second historical frame may include some or all of the historical frames of the current frame; the final fusion feature of the current frame can be understood as the third lane line feature of the current frame.

[0052] It is worth noting that the number of frames L2 of the second historical frame can be preset. If there is no historical frame for the current frame, the current frame fusion feature can be directly determined as the final fusion feature of the current frame, that is, the historical frame fusion feature can be regarded as 0; if the number of historical frames of the current frame is less than or equal to L2, the final fusion feature of the current frame can be obtained by fusing the current frame fusion feature and the second lane line features of all historical frames of the current frame; if the number of historical frames of the current frame is greater than L2, the L2 historical frames whose acquisition time is closest to the current frame can be selected from the historical frames of the current frame, and the current frame fusion feature can be obtained by fusing the second lane line features of these L2 historical frames.

[0053] The above current frame may also be referred to as a current picture, and the historical frame may also be referred to as a historical picture.

[0054] It should be noted that each frame of the target image set corresponds to a first lane line feature and a second lane line feature, and since the first lane line feature corresponding to each frame of the image is obtained by extracting the lane line features of the image itself through the feature extraction network; the second lane line feature corresponding to each frame of the image is obtained by feature fusion based on the first lane line features of the image and its corresponding historical images, and different images correspond to different historical images. Based on this, it can be understood that the first lane line features and second lane line features corresponding to different images are different.

[0055] In addition, it can be seen from the above content that the second lane line feature and the third lane line feature are both obtained by performing lane line feature fusion. Therefore, the second lane line feature and the third lane line feature can both be regarded as fused lane line features.

[0056] The feature fusion network obtains the third lane line feature by performing secondary fusion of the temporal features. The lane line features of the historical images can be used to effectively compensate for the lane line features of the image to be detected. In addition, since different images correspond to different historical images, the influence weight of the lane line features of a single-frame historical image on the lane line features of the image to be detected can be reduced through secondary fusion. In this way, even if some images in the target image set are occluded or reflected, the lane line misses caused by occlusion or reflection can be reduced, thereby improving the detection accuracy of the lane lines.

[0057] In some embodiments, the feature fusion network may include two memory buffer modules: a fusion feature memory module (Feature Fusion) and a historical feature memory module (Feature Memory). Therefore, the feature fusion network may also be referred to as Feature Fusion Feature Memory. Feature Fusion may be used to store the second lane line features, and Feature Memory may be used to store the first lane line features. Furthermore, the storage order of the second lane line features of each frame image in Feature Fusion may be consistent with the acquisition order of each frame image. The storage order of the first lane line features of each frame image in Feature Memory may be consistent with the acquisition order of each frame image.

[0058] Step 303: Determine a lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence.

[0059] In specific implementation, after obtaining the third lane line feature of the image to be detected, the third lane line feature can be firstly fused with the lane line feature through the attention network to obtain the target lane line feature of the image to be detected that can be used for lane line detection. After obtaining the target lane line feature of the image to be detected, it can be input into the lane line detection head to obtain the lane line detection result of the image to be detected through the lane line detection head.

[0060] The present application embodiment does not limit the specific network structure of the attention network. In some embodiments, the attention network can be Figure 1 Neck network shown. In other embodiments, the network architecture of the attention network can be a Transformer architecture. In these embodiments, lane line feature fusion is performed by means of Transformer, so that the third feature sequence of the current image can retain important information and filter out non-important information after passing through the Transformer. For example, if some lane lines are occluded at a certain moment during driving, since there is no occlusion before and after this moment, the features before and after are fused with the current features, and the features occluded at this moment can be effectively compensated by the Transformer, thereby solving the missed call problem caused by occlusion. In this way, through the excellent concurrency and long-distance modeling capabilities of the Transformer, the accuracy of acquiring the target lane line features of the image to be detected can be further improved, thereby improving the lane line detection accuracy of the image to be detected.

[0061] The embodiment of the present application does not limit the structure and output of the lane line detection head, which can be flexibly set based on actual needs. In some embodiments, the lane line detection head can be Figure 1The Head shown can directly output lane line detection results based on the target lane line features.

[0062] In some other embodiments, determining the lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence includes:

[0063] Processing the third lane line feature through an attention network to obtain a target lane line feature;

[0064] Performing lane line detection on the target lane line feature by the lane line detection head to obtain lane line parameters of the image to be detected; the lane line parameters include at least one of the normalized length of each virtual lane line, a lane line probability value, an initial coordinate point, an angle, and a coordinate offset compensation;

[0065] The lane line detection result of the image to be detected is determined according to the lane line parameters.

[0066] In these embodiments, the output of the lane line detection head can be defined as lane line parameters. Specifically, the lane line parameters can include the normalized length (length) of each virtual lane line, the lane line probability value (probability), the initial coordinate point (x, y), the angle (angle), and at least one of the coordinate offset compensation (offset).

[0067] After obtaining the lane line parameters output by the lane line detection head, the lane line detection result of the image to be detected is further determined based on the lane line parameters. In this way, compared with directly outputting the lane line detection result through the lane line detection head, the lane line detection accuracy can be further improved.

[0068] It should be noted that, in some embodiments, the feature extraction network, feature fusion network, attention network and lane detection head in the embodiments of the present application may all be independent models; in other embodiments, they may all be networks in the lane detection model, which may be set according to actual needs, and the embodiments of the present application do not limit this. For the latter, that is, the lane detection model may include the above-mentioned feature extraction network, feature fusion network, attention network and lane detection head. In specific implementation, the lane result of the image to be detected can be obtained by inputting the target image set corresponding to the image to be detected into the lane detection model.

[0069] The lane line detection method of this embodiment, when performing lane line detection on the image to be detected, can first obtain a target image set corresponding to the image to be detected, which includes the image to be detected and its corresponding historical image; then, the backbone feature network is used to extract lane line features of each frame image in the target image set to obtain the first lane line features of each frame image (which can be regarded as a time series feature); thereafter, the feature fusion model can be used to first perform a first fusion task based on the first lane line features to determine the second lane line features of each frame image in the target image set, and then perform a second fusion task based on the second lane line features to determine the third lane line features of the image to be detected, wherein the second lane line features of each frame image in the target image set are determined based on their own first lane line features and the first lane line features of their corresponding historical images, and the third lane line features of the image to be detected are determined based on their own second lane line features and the second lane line features of their corresponding historical images; then, the lane line detection result of the image to be detected is determined based on the third lane line features through the attention network and the lane line detection head connected in sequence. It can be seen that in the embodiment of the present application, the lane line detection result of the image to be detected is determined based on the lane line features obtained by secondary fusion of the temporal features. In this way, the lane line features of the image to be detected can be effectively compensated by the lane line features of the historical images. Moreover, since the historical images corresponding to each frame of the target image set are different, the influence weight of the lane line features of the single-frame historical image on the lane line features of the image to be detected can be reduced through secondary fusion. In this way, even if some images in the target image set are occluded or reflected, the lane line misses caused by occlusion or reflection can be reduced, thereby improving the detection accuracy of the lane lines.

[0070] In some embodiments, the first fusion task includes: performing lane line feature fusion on the first lane line feature of the image to be detected and the historical lane line feature to obtain a second lane line feature of the image to be detected; wherein the historical lane line feature includes the first lane line feature of each frame in the first historical image; the first historical image includes L1 frame historical images in the target image set whose acquisition time is earlier than that of the image to be detected, where L1 is a positive integer;

[0071] and / or

[0072] The second fusion task includes: performing lane line feature fusion on the second lane line features of the picture to be detected and the historical fused lane line features to obtain the third lane line features of the picture to be detected; wherein the historical fused lane line features include the second lane line features corresponding to each frame in the second historical picture; the second historical picture includes L2 frames of historical pictures in the target picture set whose acquisition time is earlier than that of the picture to be detected, and L2 is a positive integer.

[0073] In specific implementation, in the process of executing the first fusion task, in order to obtain the second lane line features of the image to be detected, the historical lane line features of the image to be detected can be obtained first, wherein the historical lane line features of the image to be detected include the first lane line features of the L1 frame historical image whose acquisition time is earlier than that of the image to be detected, and L1 can also be called seq_len1. Then, by fusing the first lane line features of the image to be detected and the historical lane line features, the second lane line features of the image to be detected are obtained.

[0074] In the process of executing the second fusion task, in order to obtain the third lane line feature of the image to be detected, the historical fused lane line feature of the image to be detected can be obtained first, wherein the historical fused lane line feature of the image to be detected includes the second lane line feature of the L2 frame historical image whose acquisition time is earlier than that of the image to be detected, and L2 can also be called seq_len2. Then, the third lane line feature of the image to be detected is obtained by fusing the second lane line feature of the image to be detected and the historical fused lane line feature.

[0075] When the feature fusion network stores the first lane line features through Feature Memory, the first lane line features and historical lane line features of the image to be detected can be obtained from Feature Memory. When the feature fusion network stores the second lane line features through Feature Fusion, the second lane line features and historical fused lane line features of the image to be detected can be obtained from Feature Fusion.

[0076] It should be noted that the above embodiment describes the acquisition of the second lane line features and the third lane line features of the image to be detected, but it can be understood that the acquisition of the second lane line features and the third lane line features of other images can be implemented by the same logic as the above embodiment. To avoid repetition, it will not be described again.

[0077] In practical applications, seq_len1 and seq_len2 can be determined based on any one factor or combination of image acquisition frequency and vehicle speed, but are not limited to. When the image device frequency is 25fps or 30fps, that is, 25 frames or 30 frames per second, seq_len1 and seq_len2 can be set to 64 or 128. 64 is equivalent to 2 seconds of multi-image memory, which is sufficient for the fusion effect of the previous and next frames, that is, it can ensure the accuracy of lane line detection. If it is for longer memory, seq_len1 and seq_len2 can be set larger.

[0078] By obtaining the third lane line features of the image to be detected in the above manner, the third lane line features of the image to be detected can not only integrate the lane line features of the historical image corresponding to the image to be detected, but also reduce the influence weight of the lane line features of the single-frame historical image on the lane line features of the image to be detected. In this way, the acquisition accuracy of the lane line features of the image to be detected can be improved, thereby improving the detection accuracy of the lane lines.

[0079] The embodiment of the present application does not limit the specific fusion method of the feature fusion network for lane line features. In some embodiments, the feature fusion network can achieve the fusion of lane line features by directly splicing lane line features.

[0080] In other embodiments, when the feature fusion network is fusing lane features, it can further perform position encoding on the lane features, and then perform lane feature fusion in combination with the position features obtained by the position encoding, so that the lane feature accuracy can be further improved, thereby improving the lane detection accuracy. The specific description is as follows:

[0081] In some implementations, performing lane line feature fusion on the first lane line feature of the image to be detected and the historical lane line feature to obtain the second lane line feature of the image to be detected may include:

[0082] For the first lane line feature of the image to be detected and each lane line feature in the historical lane line features, position encoding the lane line feature to obtain a position feature corresponding to the lane line feature, and adding the lane line feature to the position feature corresponding to the lane line feature to obtain an embedded feature corresponding to the lane line feature;

[0083] A weighted sum is performed on the first lane line feature of the image to be detected and the embedded features corresponding to each lane line feature in the historical lane line features to obtain a second lane line feature of the image to be detected.

[0084] In a specific implementation, for the first lane line feature of the image to be detected and each lane line feature in the historical lane line features, first, position encoding can be performed on them respectively to obtain their corresponding position encodings. The embodiment of the present application does not limit the position encoding method of the lane line features. In some embodiments, the position encoding can be implemented using sine and cosine functions. In a specific implementation, the lane line features can be first separated into multiple rows and columns. For the values ​​on the even columns, the sine function can be used for calculation; for the values ​​on the odd columns, the cosine function can be used for calculation.

[0085] Afterwards, for each lane line feature, it can be added to its own position feature to obtain its corresponding embedded feature. Then the embedded features of each lane line feature are weighted summed to obtain the second lane line feature of the image to be detected. It is worth noting that the weights of the embedded features of each lane line feature can be equal or unequal, which can be set based on actual needs, and the embodiment of the present application does not limit this.

[0086] In these embodiments, in the process of acquiring the second lane line feature, the position feature of the first lane line feature is introduced, so that the lane line feature precision can be further improved, thereby improving the lane line detection accuracy.

[0087] In some implementations, performing lane line feature fusion on the second lane line feature of the image to be detected and the historical fused lane line feature to obtain a third lane line feature of the image to be detected includes:

[0088] For the second lane line feature of the image to be detected and each fused lane line feature in the historical fused lane line feature, position encoding the fused lane line feature obtains a position feature corresponding to the fused lane line feature, and adds the fused lane line feature to the position feature corresponding to the fused lane line feature to obtain an embedded feature corresponding to the fused lane line feature;

[0089] A weighted sum is performed on the second lane line feature corresponding to the image to be detected and the embedded features corresponding to each fused lane line feature in the historical fused lane line feature to obtain a third lane line feature of the image to be detected.

[0090] In specific implementation, for the second lane line feature of the image to be detected and each fused lane line feature in the historical fused lane line feature, you can first perform position encoding on them to obtain their corresponding position encodings and then add them to their own position features to obtain their corresponding embedded features. Then, the embedded features of each fused lane line feature are weighted summed to obtain the third lane line feature of the image to be detected. The specific implementation logic of this process is similar to the acquisition logic of the second lane line feature in the above implementation. For details, please refer to the above related description, which will not be repeated here.

[0091] In these embodiments, in the process of acquiring the third lane line feature, the position feature of the second lane line feature is introduced, so that the lane line feature precision can be further improved, thereby improving the lane line detection accuracy.

[0092] In some embodiments, the attention network includes an encoder and a decoder; the determining the lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence includes:

[0093] Inputting the third lane line feature into the attention network, encoding the third lane line feature through the encoder to obtain an encoded feature, and processing the encoded feature and the query feature corresponding to the image to be detected through the decoder to obtain a target lane line feature; the query feature is obtained by linearly transforming the second lane line feature of the image to be detected;

[0094] The lane line detection head performs lane line detection on the target lane line feature to obtain a lane line detection result of the image to be detected.

[0095] In these embodiments, the attention network includes an encoder and a decoder. After obtaining the third lane line feature of the image to be detected, the third lane line feature can be linearly transformed to obtain the input feature of the encoder, and then encoded by the encoder to obtain the encoded feature.

[0096] Afterwards, the decoder can process the encoded features output by the encoder and the query features corresponding to the image to be detected to obtain the target lane line features. Specifically, the input features of the decoder are determined based on the encoded features output by the encoder and the query features corresponding to the image to be detected, and the output features of the decoder are the target lane line features.

[0097] The query feature corresponding to the image to be detected can be obtained by changing the second lane line feature of the image to be detected. In specific implementation, if the number of lane lines a is preset, that is, assuming that there are a virtual lane lines, a is a positive integer, then the second lane line feature of the image to be detected can be subjected to a different linear changes to obtain the query feature (Query) corresponding to the a virtual lane lines one by one.

[0098] In practical applications, the input features of the encoder and decoder may include: query (Q), key (V), and value (K).

[0099] By acquiring the target lane line features corresponding to the image to be detected through the above-mentioned attention network, the target lane line features of the image to be detected can better express the correlation between lane line features between different images, thereby further improving the accuracy of acquiring lane line features and further improving the accuracy of lane line detection.

[0100] Further, the decoder includes a first multi-head attention layer and a second multi-head attention layer; the decoder processes the encoding feature and the query feature corresponding to the image to be detected to obtain the target lane line feature, including:

[0101] The query feature, the target position feature and the mask feature are processed by the first multi-head attention layer to obtain an attention feature; wherein the target position feature is generated based on the second lane line feature of the image to be detected, and the mask feature is generated based on the second lane line feature of each frame image in the target image set;

[0102] The encoding feature and the attention feature are processed by the second multi-head attention layer to obtain the target lane line feature.

[0103] In some embodiments, the first multi-head attention layer can be referred to as a masked multi-head attention layer (MaskedMultiHead Attention), and the second multi-head attention layer can be referred to as a multi-head attention layer (Multi-HeadAttention).

[0104] In these embodiments, the input features of the first multi-head attention layer of the decoder can be generated based on the query features corresponding to the image to be detected and the position features of the second lane line features of the image to be detected. In specific implementation, if the number of lane lines a is preset, that is, assuming that there are a virtual lane lines, a is a positive integer, then the second lane line features of the image to be detected can be subjected to a different linear changes to obtain query features (Query) corresponding to the a virtual lane lines one by one; for each query feature, it can be added to the position features of the second lane line features of the image to be detected to obtain the reference features corresponding to each query feature, that is, to obtain the reference features of the a virtual lane lines; then, the reference features of the a virtual lane lines can be linearly transformed to obtain a group of Q, K and V corresponding to the a virtual lane lines one by one, and then the a group of Q, K and V are combined to obtain the second input feature.

[0105] For example, assuming that there are 36 virtual lane lines, 36 different linear transformations can be performed on the second lane line features of the image to be detected, and 36 queries corresponding to the 36 virtual lane lines are obtained: Query1 to Query36. Then, Query1 is added to the second lane line features of the image to be detected to obtain reference feature 1, and Query2 is added to the second lane line features of the image to be detected to obtain reference feature 2. And so on, 36 reference features corresponding to the 36 virtual lane lines are obtained: reference feature 1 to reference feature 36. Then, the 36 reference features are linearly transformed respectively to obtain 36 groups of Q, K, and V corresponding to the 36 virtual lane lines. These 36 groups of Q, K, and V constitute the input features of Masked MultiHeadAttention.

[0106] The Q, K and V groups corresponding to each virtual lane line are used to determine the target lane line features of a to-be-detected image, and the target lane line features are used to determine whether the virtual lane line is a valid lane line, that is, whether the virtual lane line actually exists. It can be understood that the number of target lane line features of the to-be-detected image is equal to the number of virtual lane lines, that is, the number of target lane line features of the to-be-detected image is a.

[0107] The mask features of the first multi-head attention layer of the decoder can be generated based on the second lane line features of each frame in the target image set. In specific implementation, for each frame in the target image set, a mask sequence can be generated based on the corresponding second lane line features, and then these mask sequences can be concatenated to obtain the mask features.

[0108] In specific implementation, the first multi-head attention layer of the decoder can use the input features determined based on the query features and the second lane line features of the picture to be detected, as well as the mask features of the picture to be detected, to obtain the attention features of the picture to be detected, that is, the output of the first multi-head attention layer of the decoder is the attention features of the picture to be detected.

[0109] For the input features of the second multi-head attention layer of the decoder, its K and V can be obtained through the encoding features output by the linear transform encoder, and its Q can be obtained through the attention features output by the first multi-head attention layer of the linear transform decoder.

[0110] In specific implementation, the second multi-head attention layer of the decoder can use the input features determined based on the encoding features and the attention features to obtain the target lane line features of the image to be detected, that is, the output of the second multi-head attention layer of the decoder is the target lane line features of the image to be detected.

[0111] By obtaining the input features of the encoder, the input features of the first and second multi-head attention layers of the decoder, and the mask features of the first multi-head attention layer in the above manner, the target lane line features of the image to be detected obtained through the attention network can better express the correlation between lane line features between different images, thereby further improving the accuracy of lane line feature acquisition and further improving the accuracy of lane line detection.

[0112] In some embodiments, before the query feature, the target position feature, and the mask feature are processed by the first multi-head attention layer to obtain the attention feature, the method further includes:

[0113] For each frame of the picture in the target picture set, determine the number x of third historical pictures corresponding to the picture; wherein the third historical pictures include historical pictures in the target picture set whose acquisition time is earlier than that of the picture, and x is a natural number;

[0114] Performing a mask operation on the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture; wherein the mask operation is determined based on a relationship between x+1 and m; m is the number of elements included in the mask sequence, and m is an integer greater than 1;

[0115] The mask sequences corresponding to the frames of the target image set are concatenated to obtain the mask features.

[0116] In these embodiments, for determining the mask sequence corresponding to each frame of the target picture set, in a specific implementation, the number x of the third historical pictures corresponding to the picture can be first obtained. The third historical pictures are all pictures in the target picture set whose acquisition time is before the acquisition time of the picture. Therefore, x is the total number of historical pictures of the current picture. Then, x of the picture is added by 1, and (x+1) represents the total number of the current picture and its corresponding historical pictures. Afterwards, (x+1) is compared with the preset number of elements m included in the mask sequence, that is, the relationship between the number of total features corresponding to the current picture and m is compared to determine the mask operation of the picture, wherein the total features corresponding to the current picture include the second lane line features of the current picture itself and the second lane line features of the historical pictures of the current picture. In this way, the acquisition accuracy of the mask features can be improved.

[0117] Further, performing a mask operation on the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture includes:

[0118] When x+1 is less than m, concatenate the second lane line features of the picture and the third historical picture, and m-(x+1) zeros, to obtain a mask sequence corresponding to the picture;

[0119] When x+1 is equal to m, concatenate the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture;

[0120] When x+1 is greater than m, the second lane line features of the picture and the fourth historical picture are spliced ​​to obtain a mask sequence corresponding to the picture; wherein the fourth historical picture is the m-1 frame picture in the third historical picture whose acquisition time is closest to the acquisition time of the picture.

[0121] x+1 is less than m, that is, the total number of features corresponding to the current image is less than the preset number of elements m included in the mask sequence. Therefore, in order to make the number of elements in the generated mask sequence reach m, m-(x+1) 0s can be added to the total features corresponding to the current image, and the mask sequence corresponding to the current image can be obtained by splicing, where m-(x+1) represents the difference between m and (x+1). In specific implementation, m-(x+1) 0s can be spliced ​​before the total features corresponding to the current image.

[0122] x+1 is equal to m, that is, the total number of features corresponding to the current image is equal to the preset number of elements m included in the mask sequence. In this case, the mask sequence with m elements can be obtained directly by concatenating the total features corresponding to the current image.

[0123] x+1 is greater than m, that is, the total number of features corresponding to the current image is greater than the preset number of elements m included in the mask sequence. Therefore, in order to make the number of elements in the generated mask sequence reach m, m-1 historical images closest to the acquisition time of the current image can be selected from the historical images of the current image, and then the second lane line features of the current image and the second lane line features of these m-1 historical images can be spliced ​​to obtain a mask sequence with m elements.

[0124] For example, it is assumed that the target picture set includes 20 frames of pictures and m=10. Then:

[0125] For the 9th frame, since the number of its corresponding total features is 9, which is less than 10, its corresponding total features and 1 0 can be concatenated to obtain the mask sequence corresponding to the 9th frame.

[0126] For the 10th frame image, since the number of its corresponding total features is 10, which is equal to 10, its corresponding total features can be concatenated to obtain the mask sequence corresponding to the 9th frame image.

[0127] For the 11th frame, since the total number of features corresponding to it is 11, which is greater than 10, the second lane line features of the 2nd to 11th frames can be spliced ​​to obtain the mask sequence corresponding to the 9th frame.

[0128] By means of the above method, the number of elements included in the generated mask sequence can meet the preset requirements, thus improving the generation accuracy of the mask sequence.

[0129] It can be understood that the input of the lane line detection head is the target lane line features of the image to be detected based on each virtual lane line. In the case where the lane line detection head outputs lane line parameters, the output of the lane line detection head may include lane line parameters of each virtual lane line, wherein the lane line parameters of each virtual lane line are determined by the lane line detection head through the target lane line features of the image to be detected based on the virtual lane line.

[0130] In some embodiments, the lane line parameters include the normalized length (length) of each virtual lane line, the lane line probability value (probability), the initial coordinate point (x, y), the angle (angle) and the coordinate offset compensation (offset);

[0131] The determining, according to the lane line parameter, a lane line detection result of the image to be detected includes:

[0132] Determine the virtual lane line whose lane line probability value is greater than a threshold as a valid lane line;

[0133] Multiplying the normalized length of the effective lane line by the height of the image to be detected to obtain a target length of the effective lane line;

[0134] Determine a first coordinate point of the valid lane line according to the initial coordinate point of the valid lane line, the angle and the coordinate offset compensation;

[0135] Deleting the coordinate points of the first coordinate points that exceed the target length and the range of the image to be detected, to obtain the second coordinate points of the valid lane line;

[0136] The lane line detection result of the image to be detected is determined according to the target length of the effective lane line and the second coordinate point.

[0137] In these embodiments, for each virtual lane line, the lane line detection head may output its corresponding length, probability, (x, y), angle, and offset respectively.

[0138] Afterwards, for each query, its corresponding probability can be compared with the threshold to determine whether it is a valid lane line. If its corresponding probability is greater than the threshold, it can be determined to be a valid lane line, otherwise, it can be determined to be an invalid lane line.

[0139] For invalid lane lines, their lane line parameters can be deleted. For valid lane lines, their lane line parameters can be used to mark the lane lines on the image to be detected. In this way, the lane line detection results of the image to be detected can be obtained.

[0140] In specific implementation, for valid lane lines:

[0141] On the one hand, the corresponding length can be multiplied by the height of the image to be detected to obtain the target length of the effective lane line. The target length of the effective lane line can be understood as the length of the effective lane line on the image to be detected.

[0142] On the other hand, the first coordinate point of the effective lane line can be determined using its corresponding (x, y), angle and offset. After that, the first coordinate point can be filtered using the target length of the effective lane line and the coordinate range of the image to be detected to obtain the second coordinate point of the effective lane line. The second coordinate point of the effective lane line can be understood as the coordinate point of the effective lane line on the image to be detected.

[0143] Afterwards, the target length and the second coordinate point corresponding to each valid lane line can be used to mark the valid lane line on the image to be detected, and the lane line detection result of the image to be detected can be obtained.

[0144] By obtaining the lane line detection results through the above lane line parameters, the acquisition accuracy of the lane line detection results can be improved, thereby improving the accuracy of lane line detection.

[0145] It should be noted that the various optional implementation modes introduced in the embodiments of the present application can be implemented in combination with each other or separately if they do not conflict with each other, and the embodiments of the present application are not limited to this.

[0146] The lane line detection method of the embodiment of the present application can solve the problem of lane line prediction loss caused by uncontrollable factors such as occlusion or reflection in one or several frames, affecting the accuracy of lane lines and affecting the decision-making of autonomous driving vehicles.

[0147] The lane line detection method of the embodiment of the present application includes the following contents:

[0148] (1) Based on temporal features, a method for fusing current frame features with historical frame features and a method for fusing historical frame fusion features with current frame fusion features are proposed.

[0149] (2) Based on the fused features, the lane line corresponding to the query is generated through the transformer method, and the lane line length lenth, the probability of whether it is a lane line, the coordinate point xy, the angle angle, and the offset offset of the corresponding coordinate are used for modeling. The threshold is used to output high-quality lane line results.

[0150] Compared with the prior art, the lane line detection method of the embodiment of the present application has at least the following beneficial effects:

[0151] (1) Through the fusion of temporal features, some prior knowledge can be provided for the features of the current frame, making the recognition effect more accurate and the recognition results of the previous and next frames more stable.

[0152] (2) For some scenes with frame occlusion, reflection, large color difference in lighting conditions, etc., the problem can be well solved through timing.

[0153] (3) The existing technology only recognizes the current frame results, which is irrelevant to the historical results. It has weak generalization ability and may cause missed calls. However, this solution integrates historical features, has stronger generalization ability, and effectively avoids the occurrence of missed calls.

[0154] (4) This solution uses transformer modeling. Due to its excellent concurrency and long-distance modeling capabilities, it has better recognition effects than existing technologies.

[0155] For ease of understanding, a specific embodiment is used as an example for illustration:

[0156] In a specific embodiment, the lane line detection method can be found in Figure 3 .exist Figure 3 In this paper, the feature extraction network is the Backbone network, the feature fusion network is the Feature Fusion Feature Memory, and the attention network is the Transformer architecture. The working principle of the feature fusion network can be found in Figure 4a As described above, the attention network can include an encoder (Transformer Encoder) and a decoder (Transformer Decoder). The working principle of the encoder can be found in Figure 4b As described above, the working principle of the encoder can be found in Figure 4c and the aforementioned related description.

[0157] In addition, Figure 3 In , the output of the lane detection head is the following lane parameters: length, probability, (x, y), angle and offset.

[0158] Understandably, Figure 3 The networks in the figure are only examples, and the specific forms of the networks are not limited thereby.

[0159] for Figure 3 The lane line detection method shown in the figure uses the transformer method to model the lane lines in the overall framework.

[0160] After the introduction of time series features, two major memory buffer modules are formed [Feature fusion memory module Feature fusion, historical feature memory Feature Memory module], that is, Figure 4a The pink and purple areas in the framework diagram are represented by different sequence lengths [i.e., seq_len1, seq_len2].

[0161] In this way, the fusion features of the current frame are obtained through the current frame features and the historical features, and the final fusion features of the current frame are obtained through the current frame fusion features and the historical fusion features, which can better capture the key features of the previous and next frames.

[0162] More specific operation: The operation method of position encoding here is to perform position encoding operation on each frame feature or fusion feature, that is, BNHW becomes BN*HW and then reshapes to BNHW after position encoding operation. Then add the corresponding features at element level to get the final feature. In the transformer encoder, there are position encoding 1 and position encoding 2 because they have different functions. Position encoding 1 is the position encoding of historical features. Position encoding 2 is the position encoding of historical fusion features.

[0163] Encoder module: The encoder layer is stacked with 6 layers, and the number of layers here can be customized. The Q, K, and V entering the selfattention are the Q, K, and V formed by the fusion features after 3 conv1x1 convolutions. Then the normal Attention operation is performed.

[0164] Decoder module:

[0165] The decoder layer is stacked in 6 layers, and the number of layers here can be customized.

[0166] The query of the decoder is the number of lane lines that are preset. For example, if there are 36 lane lines, there are 36 queries corresponding to 36 virtual lane lines. During the model training process, the prediction results of the 36 virtual lane lines are matched with the actual true value (GT) in a bipartite graph matching loss value, thereby making the model converge.

[0167] Masked MultiHead Attention: Assuming that the historical fusion feature has seq len (corresponding to the aforementioned m), the mask sequence corresponding to the image frame can be obtained in the following way: For image frames that are initially less than seqlen, the missing ones need to be excluded from the calculation through mask. For example, seq len 10, if the number of images at the beginning is less than seq len 10, then the image frames with less than 10 missing frames need to be masked; if the number of images exceeds seq len 10, such as 20, then the features of the first 10 frames need to be masked, and only the most recent 10 frames are retained.

[0168] The function of MultiHead Attention is to decode the encoded features output by the encoder, and the outputs are K and V respectively. The lane line Query is obtained after position Encoding 2 and Masked MultiHead Attention operations to obtain Q, and then participates in the Attention operation of Q, K, V.

[0169] Head output:

[0170] Each query, i.e. the length of each lane line, is a normalized value [0 to 1];

[0171] Whether probability is the lane line probability value;

[0172] x, y, angle each line point and angle;

[0173] Offset: offset compensation of each line point position;

[0174] Finally, the valid lane lines are filtered by checking whether the probability is greater than the threshold. The length of the lane line can be obtained by multiplying the length by the height H of the image.

[0175] The coordinate points of each line are obtained through x, y, angle, and offset. The coordinate points that exceed the lane line length and exceed the image range are removed to obtain the final lane line recognition result.

[0176] pass Figure 3 Lane line detection can improve the accuracy of lane line detection. For easy understanding, please refer to Figure 5 and Figure 6 ,in, Figure 5 To adopt Figure 1 The lane line detection result obtained by the lane line detection method is Figure 6 To adopt Figure 3 The lane line detection results obtained by the lane line detection method are compared with Figure 5 , Figure 6 The lane line detection result adds green lane lines.

[0177] Based on the lane line detection method provided in the above embodiment, the present application also provides a specific implementation of a lane line detection device. Please refer to the following embodiment.

[0178] See also Figure 7 , the lane line detection device provided in the embodiment of the present application may include:

[0179] The first acquisition module 701 is used to input the target picture set corresponding to the picture to be detected into the feature extraction network, and extract the lane line feature of each frame picture in the target picture set through the feature extraction network to obtain the first lane line feature of each frame picture in the target picture set; wherein the target picture set includes the picture to be detected and the historical pictures collected earlier than the picture to be detected;

[0180] A second acquisition module 702 is used to input the first lane line feature into a feature fusion network, and the feature fusion network performs a first fusion task based on the first lane line feature to determine the second lane line feature of each frame image in the target image set, and performs a second fusion task based on the second lane line feature to determine the third lane line feature of the image to be detected; wherein the second lane line feature of each frame image in the target image set is determined based on the first lane line feature of the image, and the first lane line feature of a historical image in the target image set that was collected earlier than the image; the third lane line feature is determined based on the second lane line feature of the image to be detected, and the second lane line feature of a historical image in the target image set that was collected earlier than the image to be detected;

[0181] The third acquisition module 703 is used to determine the lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence.

[0182] The lane line detection device provided in the embodiment of the present application can implement each process implemented by the lane line detection device in the method embodiment, and will not be described again here to avoid repetition.

[0183] Figure 8 A schematic diagram of the hardware structure of lane line detection provided in an embodiment of the present application is shown.

[0184] The lane line detection device may include a processor 801 and a memory 802 storing computer program instructions.

[0185] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0186] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory.

[0187] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0188] The processor 801 implements any one of the lane line detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .

[0189] In one example, the lane line detection device may further include a communication interface 808 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 808 are connected via a bus 810 and communicate with each other.

[0190] The communication interface 808 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0191] Bus 810 includes hardware, software, or both, coupling components of the lane detection device to each other. By way of example and not limitation,

[0192] The bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnection, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnection, a Low Pin Count (LPC) bus, a Memory Bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, bus 810 may include one or more buses. Although the present application describes and illustrates a specific bus, the present application considers any suitable bus or interconnection.

[0193] In addition, in combination with the insulation resistance detection method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any insulation resistance detection method in the above embodiment is implemented.

[0194] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0195] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM, floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0196] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0197] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0198] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A lane line detection method, characterized in that: include: Input a target picture set corresponding to the picture to be detected into a feature extraction network, and extract lane line features from each frame of the picture in the target picture set through the feature extraction network to obtain the first lane line features of each frame of the picture in the target picture set; wherein the target picture set includes the picture to be detected and historical pictures collected earlier than the picture to be detected; Input the first lane line feature into a feature fusion network, and determine the second lane line feature of each frame picture in the target picture set by performing a first fusion task based on the first lane line feature through the feature fusion network, and determine the third lane line feature of the picture to be detected by performing a second fusion task based on the second lane line feature; wherein the second lane line feature of each frame picture in the target picture set is determined based on the first lane line feature of the picture and the first lane line feature of a historical picture in the target picture set that was collected earlier than the picture; the third lane line feature is determined based on the second lane line feature of the picture to be detected and the second lane line feature of a historical picture in the target picture set that was collected earlier than the picture to be detected; Through the attention network and the lane line detection head connected in sequence, the lane line detection result of the image to be detected is determined based on the third lane line feature.

2. The method according to claim 1, characterized in that The first fusion task includes: performing lane line feature fusion on the first lane line feature of the image to be detected and the historical lane line feature to obtain a second lane line feature of the image to be detected; wherein the historical lane line feature includes the first lane line feature of each frame in the first historical image; the first historical image includes L1 frame historical images in the target image set whose acquisition time is earlier than that of the image to be detected, where L1 is a positive integer; and / or The second fusion task includes: performing lane line feature fusion on the second lane line features of the picture to be detected and the historical fused lane line features to obtain the third lane line features of the picture to be detected; wherein the historical fused lane line features include the second lane line features corresponding to each frame in the second historical picture; the second historical picture includes L2 frames of historical pictures in the target picture set whose acquisition time is earlier than that of the picture to be detected, and L2 is a positive integer.

3. The method according to claim 2, characterized in that The performing lane line feature fusion on the first lane line feature of the image to be detected and the historical lane line feature to obtain the second lane line feature of the image to be detected includes: For the first lane line feature of the image to be detected and each lane line feature in the historical lane line features, position encoding the lane line feature to obtain a position feature corresponding to the lane line feature, and adding the lane line feature to the position feature corresponding to the lane line feature to obtain an embedded feature corresponding to the lane line feature; A weighted sum is performed on the first lane line feature of the image to be detected and the embedded features corresponding to each lane line feature in the historical lane line features to obtain a second lane line feature of the image to be detected.

4. The method according to claim 2, characterized in that: The performing lane line feature fusion on the second lane line feature of the image to be detected and the historical fused lane line feature to obtain the third lane line feature of the image to be detected includes: For the second lane line feature of the image to be detected and each fused lane line feature in the historical fused lane line feature, position encoding the fused lane line feature obtains a position feature corresponding to the fused lane line feature, and adds the fused lane line feature to the position feature corresponding to the fused lane line feature to obtain an embedded feature corresponding to the fused lane line feature; A weighted sum is performed on the second lane line feature corresponding to the image to be detected and the embedded features corresponding to each fused lane line feature in the historical fused lane line feature to obtain a third lane line feature of the image to be detected.

5. The method according to claim 1, characterized in that The attention network includes an encoder and a decoder; the lane line detection result of the to-be-detected image is determined based on the third lane line feature through the attention network and the lane line detection head connected in sequence, including: Inputting the third lane line feature into the attention network, encoding the third lane line feature through the encoder to obtain an encoded feature, and processing the encoded feature and the query feature corresponding to the image to be detected through the decoder to obtain a target lane line feature; the query feature is obtained by linearly transforming the second lane line feature of the image to be detected; The lane line detection head performs lane line detection on the target lane line feature to obtain a lane line detection result of the image to be detected.

6. The method according to claim 5, characterized in that The decoder includes a first multi-head attention layer and a second multi-head attention layer; the decoder processes the encoding feature and the query feature corresponding to the image to be detected to obtain the target lane line feature, including: The query feature, the target position feature and the mask feature are processed by the first multi-head attention layer to obtain an attention feature; wherein the target position feature is generated based on the second lane line feature of the image to be detected, and the mask feature is generated based on the second lane line feature of each frame image in the target image set; The encoding feature and the attention feature are processed by the second multi-head attention layer to obtain the target lane line feature.

7. The method according to claim 6, characterized in that Before the query feature, the target position feature and the mask feature are processed by the first multi-head attention layer to obtain the attention feature, the method further includes: For each frame of the picture in the target picture set, determine the number x of third historical pictures corresponding to the picture; wherein the third historical pictures include historical pictures in the target picture set whose acquisition time is earlier than that of the picture, and x is a natural number; Performing a mask operation on the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture; wherein the mask operation is determined based on a relationship between x+1 and m; m is the number of elements included in the mask sequence, and m is an integer greater than 1; The mask sequences corresponding to the frames of the target image set are concatenated to obtain the mask features.

8. The method according to claim 7, characterized in that The performing a mask operation on the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture includes: When x+1 is less than m, concatenate the second lane line features of the picture and the third historical picture, and m-(x+1) zeros, to obtain a mask sequence corresponding to the picture; When x+1 is equal to m, concatenate the second lane line features of the picture and the third historical picture to obtain a mask sequence corresponding to the picture; When x+1 is greater than m, the second lane line features of the picture and the fourth historical picture are spliced ​​to obtain a mask sequence corresponding to the picture; wherein the fourth historical picture is the m-1 frame picture in the third historical picture whose acquisition time is closest to the acquisition time of the picture.

9. The method according to claim 1, characterized in that: The determining the lane line detection result of the image to be detected based on the third lane line feature through the attention network and the lane line detection head connected in sequence includes: Processing the third lane line feature through an attention network to obtain a target lane line feature; Performing lane line detection on the target lane line feature by the lane line detection head to obtain lane line parameters of the image to be detected; the lane line parameters include at least one of the normalized length of each virtual lane line, a lane line probability value, an initial coordinate point, an angle, and a coordinate offset compensation; The lane line detection result of the image to be detected is determined according to the lane line parameters.

10. The method according to claim 9, characterized in that The lane line parameters include the normalized length of each virtual lane line, the lane line probability value, the initial coordinate point, the angle and the coordinate offset compensation; The determining, according to the lane line parameter, a lane line detection result of the image to be detected includes: Determine the virtual lane line whose lane line probability value is greater than a threshold as a valid lane line; Multiplying the normalized length of the effective lane line by the height of the image to be detected to obtain a target length of the effective lane line; Determine a first coordinate point of the valid lane line according to the initial coordinate point of the valid lane line, the angle and the coordinate offset compensation; Deleting the coordinate points of the first coordinate points that exceed the target length and the range of the image to be detected, to obtain the second coordinate points of the valid lane line; The lane line detection result of the image to be detected is determined according to the target length of the effective lane line and the second coordinate point.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the lane line detection method according to any one of claims 1 to 10 is implemented.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the lane line detection method as described in any one of claims 1 to 10.

Images