Event stream data lane line extraction and detection method based on momentum distillation

Abstract

The invention provides an event stream data lane line extraction and detection method based on momentum distillation, and the method comprises the steps: 1, carrying out the preprocessing of sparse event stream data, and determining an event stream representation mode; 2, building a multi-scale event flow feature extractor and a regression module as basic models; 3, copying the basic model as a momentum model; the event flow representation mode obtained in the first step is input into a basic model and a momentum model at the same time, two lane line extraction results are obtained, and the result obtained by the momentum model serves as a pseudo label to supervise training of the basic model together with a real label; only the parameters of the basic model network are updated; 4, updating parameters of the momentum model by adopting an index moving average strategy; and 5, repeating the step 3 and the step 4 until the total loss is converged. According to the invention, the problems of misjudgment, sparsity of event stream data and particularity of lane line shapes are solved.

Description

Technical Field

[0001] The present invention provides a method for extracting and detecting lane lines from event stream data based on momentum distillation, and belongs to the technical field of autonomous driving. Background Art

[0002] The application of artificial intelligence in the field of autonomous driving has recently attracted widespread attention from academia and industry. As an important component of autonomous driving systems, advanced driver assistance systems (ADAS) require vehicles to respond quickly and accurately to environmental changes. Lane extraction is a fundamental aspect of ADAS. It involves identifying and tracking lane lines on the road to ensure that the vehicle remains in the designated lane. This function is essential for maintaining road safety, improving driving comfort, and realizing advanced functions such as lane keeping, adaptive cruise control, and automatic navigation. Early lane detection was performed on frame-based cameras using traditional methods or machine learning. However, under high-speed driving or extreme lighting conditions (too dark and overexposed), traditional cameras cannot capture key information to determine whether a lane exists and where it is.

[0003] Event cameras mimic human vision. Similar to the human eye, instead of focusing on an entire frame of an image, event cameras capture pixel-level intensity differences, and whenever the brightness of a pixel changes by more than a preset threshold, an event is generated that contains a timestamp, location, and direction of change (increase or decrease in brightness). This mechanism gives event cameras significant advantages in many aspects. Event cameras have an extremely high dynamic range and can adapt to changes in environments from extremely dark to extremely bright. Secondly, since only intensity change signals are recorded, extremely low latency and high temporal resolution can be achieved, which is essential for application scenarios that require fast reactions, such as environmental perception of autonomous vehicles. In addition, compared with traditional cameras, event cameras consume less power in most cases. The application areas of event cameras include but are not limited to autonomous driving, robotic vision, and augmented reality and virtual reality technologies. In autonomous driving, it can provide accurate environmental perception to help vehicles better understand their surroundings. In robotic vision, event cameras can help robots navigate and identify objects.

[0004] In recent years, the rapid development of deep learning has significantly advanced lane detection technology. Deep learning models have achieved remarkable success in improving the accuracy and robustness of lane detection systems. These models are trained on large datasets containing various driving environments, which enables them to learn complex patterns and make accurate predictions even in challenging situations such as occlusion, shadows, and lane line wear. As the complexity of the network gradually increases, the corresponding evaluation indicators will also become higher and higher. However, in the development of this task, the methods proposed by various scholars only focus on how to extract better deep features from event data, ignoring a critical problem in event stream data, that is, the misjudgment of lane lines caused by the lack of color information. Due to the lack of color information in event stream data, objects with similar shapes and textures are easily identified as the same object, such as lane lines and road edges. The two are difficult to effectively distinguish in event stream data, and the feature maps after the feature extractor are also extremely similar. The corresponding algorithm cannot distinguish them, which will lead to misjudgment of lane lines, which may lead to danger in autonomous driving. Lane detection serves ADAS, which provides a solid guarantee for users' safe and reliable driving. Without considering safety, simply improving evaluation indicators is meaningless, whether in academia or industry.

[0005] Event cameras can obtain event stream data with high frame rate and high dynamic range, which is more suitable for downstream tasks in autonomous driving scenarios than traditional cameras. However, due to the lack of information such as color, different objects with similar shapes and textures may be mistakenly identified as the same object, which may provide wrong judgment signals in autonomous driving. In addition, event stream data is relatively sparse, and standard convolution cannot extract effective features well. Summary of the invention

[0006] The present invention provides a lane line extraction and detection method for event stream data based on momentum distillation. The momentum model adopted in the visual language large model ALBEF is referred to and used in the lane line detection task of event stream data to solve the above-mentioned misjudgment problem. In the decoder stage, variable convolution is used to dynamically and adaptively perform feature regression to solve the sparsity of event stream data and the particularity of lane line shape.

[0007] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] A lane line extraction and detection method for event stream data based on momentum distillation, the lane line extraction method specifically comprises the following steps:

[0009] Step 1: Preprocess the sparse event stream data and determine the event stream representation method.

[0010] Step 2: Build a multi-scale event stream feature extractor and regression module, which is called the basic model.

[0011] Step 3: Copy the basic model in step 2 as the momentum model. Input the event stream representation obtained in step 1 into both the basic model and the momentum model in step 2 to obtain two lane line extraction results. The results obtained by the momentum model are used as pseudo labels to supervise the training of the basic model together with the real labels. In this step, only the parameters of the basic model network are updated.

[0012] Step 4: Use the exponential moving average (EMA) strategy to update the parameters of the momentum model.

[0013] Step 5: Repeat steps 3 and 4 until the total loss converges.

[0014] Specifically, the representation method mentioned in step 1 ignores the time information contained in the event stream and represents the sparse event stream with timestamps as a one-dimensional grayscale value image. The event stream is compressed along the time dimension and compressed at a fixed time interval Δt, where Δt = 30ms, and Δt represents the time span of the event stream corresponding to a single image. The grayscale value representation method of the event stream data is G∈R C×H×W , C is the number of channels, H and W are its height and width respectively, where C = 3. H = 800, W = 1280. The real label of the lane line is L∈R C×H×W , where C = 1, the values ​​of H and W correspond to the representation of event stream data, and the annotations in the label correspond to the original data pixel by pixel.

[0015] The basic model described in step 2 consists of two parts: a multi-scale event stream feature extractor and a feature regression module based on an encoder-decoder structure. The multi-scale feature extractor structure used uses dilated convolution to obtain features with receptive fields of various sizes, including global and local features, which is conducive to the extraction of lane line features.

[0016] Specifically, the initial module of the multi-scale event stream feature extractor uses pooling operations in parallel with standard convolutions. This structure is repeated three times as the initial feature extraction module. The subsequent multi-scale feature modules use five convolutions with different dilation factors to extract multi-scale features. The dilation factors are 1, 4, 8, 16, and 32. The feature layers with different receptive fields are obtained respectively, i.e., F0∈R 256×h×w , F1∈R 256×h×w , F2∈R 256×h×w , F3∈R 256×h×w , F4∈R 256×h×wFrom F0 to F4, the receptive field gradually increases. Finally, F0, F1, F2, F3 and F4 are connected in the channel dimension through the concat function to form the output of the multi-scale event stream feature extractor, as shown in the following formula:

[0017] F=concat(F0,F1,F2,F3,F4) (1)

[0018] In order to improve the model detection capability and obtain more detailed and accurate classification results, the proposed method uses an encoder-decoder as the basic architecture. After extracting the multi-scale features of the input event stream representation method, the decoder obtains the lane line pixel-by-pixel probability map; at the same time, a lane line branch network is connected to the output features of the encoder; at this point, the basic model structure has been completed. During training, it outputs a single-channel lane line detection probability map, and calculates the cross loss between it and the true label. As shown in the following formula:

[0019]

[0020] Where N is the total number of pixels in the true label, M is the total number of lane line categories, when different categories of lane lines are not distinguished, M = 2, when different categories of lane lines need to be distinguished, M = 5. m,n is the probability that the corresponding pixel in the true label belongs to category m. For the true label, this probability value is either 0 or 1, which is called a hard label and contains limited mutual information. Indicates the probability that the predicted lane line extraction result belongs to class m.

[0021] The above loss can enable the model to have a preliminary lane line detection effect in the training stage, but considering that objects with similar shapes and textures are difficult to distinguish in event stream data, this method adopts the momentum distillation method in the large visual language model ALBEF. In ALBEF, the data used in the training stage is a large number of image-text pairs crawled from the Internet, but the text and image are not very matched. For example, the text does not summarize the content of the image very well or the description in the text does not appear in the image. The momentum model can solve the problem of chaotic data sources. It combines the methods of knowledge distillation and transfer learning. In addition to using the real label, the model is copied as a momentum model, and the predicted value output by the momentum model is used as a pseudo label. and the true label y m,n Supervise the training of the base model together.

[0022] The pseudo-label is not like a hard label. It has a non-zero probability value for each category, which greatly increases the mutual information and enables the model to acquire richer knowledge during the training phase. The predicted value of the basic model The similarity loss is calculated by the L2 norm, as shown in the following formula:

[0023]

[0024] The meanings of N and M are the same as those of L sup The corresponding parameters in have the same meaning. The predicted value output by the momentum model has a non-zero value in each category and has more mutual information than the real label, which can also be called a soft label. The structure and parameters of the momentum model are exactly the same as those of the basic model. However, in the training stage, in order to make the pseudo-label have a certain degree of guidance, the momentum model needs to be preheated, that is, trained several rounds in advance, while the parameters of the basic model are randomly initialized.

[0025] The total loss in step 3 is the weighted sum of the two losses mentioned above, as shown in the following formula:

[0026] L total =λ1L sup +λ2L consistency (4)

[0028] Where L sup and L consistency As described in formula (2) and formula (3). λ1 and λ2 are two hyperparameters that control L sup and L consistency The impact on model parameter update during the training phase. consistency If the hyperparameter λ2 is set too large, the prediction result will be too close to the prediction value of the momentum model and deviate from the true label. Even if the misjudgment of the lane line can be avoided, the relevant evaluation index will be too low. Therefore, in this method, λ1 and λ2 are set in a ratio of 5:1.

[0029] The exponential moving average described in step 4 is used to update the parameters of the momentum model. Specifically, the total loss of formula (4) is used to update the parameters of the basic model, while the parameters of the momentum model are updated according to the exponential moving average strategy (EMA) using the parameters of the momentum model of the previous round and the basic model of this round. As shown in the following formula:

[0030] θ' t =αθ' t-1 +(1-α)θ t (5)

[0032] where θ' t and θ tIt refers to the parameters of the momentum model and the basic model in the tth round. α is the smoothing coefficient hyperparameter. In this method, it is set to 0.99 to avoid the model parameters being updated too quickly, resulting in unstable detection performance.

[0033] The key points of the present invention are:

[0034] 1. The momentum distillation method is applied to the lane detection task of event stream data, which improves the mutual information of the supervision signal in the model training stage and solves the common misjudgment problem of existing algorithms.

[0035] 2. In terms of feature extractor, the use of multi-scale feature extraction strategy not only solves the sparsity problem of event stream data, but also solves the particularity of lane line shape.

[0036] 3. In the decoder regression stage, considering the large span of the feature map output by the encoder, variable convolution is used to solve the problem that the feature extraction position cannot cover the lane line well. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of an event stream representation method and corresponding real labels provided in an embodiment;

[0038] Figure 2 A schematic diagram of an encoder-decoder structure is adopted as the basic model of the embodiment;

[0039] Figure 3 Figure 2 is a visual effect diagram of the detection example. DETAILED DESCRIPTION

[0040] The specific technical solution of the present invention is explained in conjunction with embodiments.

[0041] The event stream data lane line extraction and detection method of this embodiment includes:

[0042] Step 1: Preprocess the sparse event stream data and determine the event stream representation method:

[0043] In an optional embodiment, the sparse event stream with timestamps is represented as a one-dimensional grayscale image. The event stream is compressed along the time dimension and compressed at a fixed time interval Δt, where Δt = 30ms. The grayscale representation of the event stream data is G∈R C×H×W , C is the number of channels, H and W are its height and width respectively, where C = 3, H = 800, W = 1280. The real label of the lane line is L∈R C×H×W , where C = 1, the values ​​of H and W correspond to the representation of event stream data, and the annotations in the label correspond to the original data pixel by pixel.

[0044] like Figure 1 The event stream representation method and the corresponding real label diagram provided by the embodiment of the present invention are shown. In this embodiment, the timestamp information of the event stream is not considered. There is some noise in the event stream data, and filtering operation can be used to remove the noise, but some detailed information may also be lost. Therefore, this method uses the original noisy event stream data.

[0045] Step 2: Build a basic model, including a multi-scale event stream feature extractor and a regression module;

[0046] like Figure 2 As shown in the figure, the basic model of this embodiment adopts an encoder-decoder structure. The encoder adopts a multi-scale feature extractor to obtain feature maps of multiple receptive fields in the event stream; its initial module uses pooling operations in parallel with standard convolutions, and this structure is repeated three times as the initial feature extraction module. The subsequent multi-scale feature module uses five convolutions with different expansion factors to extract multi-scale features. The expansion factors are 1, 4, 8, 16 and 32 respectively. The feature layers of different receptive fields are obtained respectively, that is, F0∈R 256×h×w , F1∈R 256×h×w , F2∈R 256×h×w , F3∈R 256×h×w , F4∈R 256×h×w From F0 to F4, the receptive field gradually increases. Finally, F0, F1, F2, F3 and F4 are concatenated in the channel dimension through the concat function.

[0047] In order to improve the model detection capability and obtain more detailed and accurate classification results, the proposed method uses an encoder-decoder as the basic architecture. After extracting the multi-scale features of the input event stream representation method, the decoder obtains the lane line pixel-by-pixel probability map; at the same time, a lane line branch network is connected to the output features of the encoder; at this point, the basic model structure has been completed. During training, it outputs a single-channel lane line detection probability map, and the cross loss is calculated between it and the true label

[0048] Step 3: Copy the basic model in step 2 as the momentum model. Input the event stream representation obtained in step 1 into both the basic model and the momentum model in step 2 to obtain two lane line extraction results. The results obtained by the momentum model are used as pseudo labels to supervise the training of the basic model together with the real labels. In this step, only the parameters of the basic model network are updated.

[0049] like Figure 2As shown in the figure, there is a two-way interaction between the momentum model and the base model. Specifically, the output of the momentum model supervises the training of the base model, and the parameters of the base model are used to update the parameters of the momentum model. Momentum distillation combines the methods of knowledge distillation and transfer learning. In addition to using the real labels, a copy of the model is made as the momentum model. The predicted values ​​output by the momentum model are used as pseudo labels to supervise the training of the base model together with the real labels.

[0050] The pseudo-label is not like a hard label. It has a non-zero probability value for each category, which greatly increases the mutual information and enables the model to acquire richer knowledge during the training phase. The predicted value of the basic model The similarity loss is calculated by L2 norm. The total loss is obtained by weighted summing the above two losses. The weight can be determined according to the specific situation. In this embodiment, the loss between the predicted value and the true label is dominant.

[0051] Step 4: Use the exponential moving average (EMA) strategy to update the parameters of the momentum model.

[0052] The exponential moving average described in step 4 is used to update the parameters of the momentum model. Specifically, the total loss of formula (4) is used to update the parameters of the basic model, while the parameters of the momentum model are updated according to the exponential moving average strategy (EMA) using the parameters of the momentum model of the previous round and the basic model of this round.

[0053] In this embodiment, the momentum model parameters of the tth round are the weighted sum of the parameters of the momentum model of the t-1th round and the basic model of the tth round. Specifically, the weight of the momentum model of the t-1th round is set to 0.99 to smooth the update of the momentum model parameters.

[0054] Experimental setup:

[0055] The training dataset uses the public event flow lane detection dataset DET. The CAVE dataset consists of 5424 event-based sensor images with a resolution of 1280×800, which is the highest among all DVS datasets. These images include complex traffic scenes and various lane types. All images of DET are annotated in a multi-class segmentation format. The fully annotated DET image contains 17103 lane instances, each of which is manually labeled pixel by pixel. During the model training phase, the DET dataset is divided into training set, validation set, and test set in a ratio of 3:1:2. Both this method and the comparative method have conducted a large number of experiments on the DET dataset, including instance segmentation and semantic segmentation.

[0056] Compared with the best detection method currently available:

[0057] Results of lane line detection on the event stream data DET dataset. The DET dataset contains two types of labels, instance segmentation and semantic segmentation. Specifically, semantic segmentation does not distinguish between different types of lane lines, but only distinguishes between lane lines and backgrounds. On the contrary, instance segmentation considers different types of lane lines, which includes four different types of lane lines and backgrounds.

[0058] Table 1 shows the average results of OA, F1 and IOU of the method provided by the present invention and all the comparison methods on the DET dataset. It can be observed that the method of the present invention has better performance than the comparison methods. The important point is that other methods will have more or less lane line misjudgment phenomenon, but the proposed method does not have misjudgment phenomenon. The compared methods all belong to deep learning networks and only use real labels as supervision signals, so they do not have rich mutual information, which makes the model unable to recognize different objects with similar shapes and textures. Obviously, the proposed method is superior to other algorithms, both in terms of objective evaluation indicators and subjective visual effects.

[0059] Table 1 Detection results of the proposed method and the comparison method on the DET dataset

[0060]

[0061] Figure 3 Several detection examples in DET are shown, and the best method currently available, namely LDNet, is selected for comparison. Figure 3 It can be seen that although LDNet can detect lane lines well, some parts that should not be lane lines, such as the edge of the road, are misjudged as lane lines by LDNet because their shape and texture are extremely similar to those of lane lines. However, the proposed method is not confused by the similarity of shape and texture, and effectively separates the edge of the road from the lane line without causing any misjudgment.

[0062] Specifically, in the first example, LDNet did not detect the lane line (the area circled by yellow dashed circle), while the proposed method detected it well. In the second example, LDNet misjudged the lane line as the lane line (the area circled by red dashed circle), while the proposed method did not misjudge it. From these examples, we can see that the addition of momentum distillation makes the proposed method more robust.

Claims

1. A lane line extraction and detection method for event stream data based on momentum distillation, characterized in that: The following steps are involved: Step 1: Preprocess the sparse event stream data and determine the event stream representation method; Step 2: Build a multi-scale event stream feature extractor and a regression module, which is called the basic model; Step 3: Copy the basic model in step 2 as the momentum model; input the event stream representation obtained in step 1 into the basic model and momentum model in step 2 at the same time to obtain two lane line extraction results. The results obtained by the momentum model are used as pseudo labels and supervise the training of the basic model together with the real labels; In this step, only the parameters of the basic model network are updated; Step 4: Use the exponential moving average strategy to update the parameters of the momentum model; Step 5: Repeat steps 3 and 4 until the total loss converges.

2. According to the method for lane line extraction and detection of event stream data based on momentum distillation in claim 1, it is characterized in that: The representation method described in step 1 ignores the time information contained in the event stream and represents the sparse event stream with timestamps as a one-dimensional grayscale value image; the event stream is compressed along the time dimension and compressed at a fixed time interval Δt, where Δt = 30ms, Δt represents the time span of the event stream corresponding to a single image; the grayscale value representation method of the event stream data is G∈R C×H×W , C is the number of channels, H and W are its height and width respectively, where C = 3; H = 800, W = 1280; the real label of the lane line is L∈R C×H×W , where C = 1, the values ​​of H and W correspond to the representation of event stream data, and the annotations in the label correspond to the original data pixel by pixel.

3. The method for lane line extraction and detection based on event stream data based on momentum distillation according to claim 2, characterized in that: The basic model described in step 2 consists of two parts, namely a multi-scale event stream feature extractor and a feature regression module based on an encoder-decoder structure; the multi-scale feature extractor structure used uses dilated convolution to obtain features with receptive fields of various sizes, including global and local features.

4. The method for lane line extraction and detection based on event stream data based on momentum distillation according to claim 3 is characterized in that: The initial module of the multi-scale event stream feature extractor uses pooling operation in parallel with standard convolution, repeated three times, as the initial feature extraction module; the subsequent multi-scale feature module uses five convolutions with different expansion factors to extract multi-scale features, and the expansion factors are 1, 4, 8, 16 and 32 respectively; the feature layers with different receptive fields are obtained respectively, that is, F0∈R 256 ×h×w , F1∈R 256×h×w , F2∈R 256×h×w , F3∈R 256×h×w , F4∈R 256×h×w ; From F0 to F4, the receptive field gradually increases; finally, F0, F1, F2, F3 and F4 are connected in the channel dimension through the concat function to form the output of the multi-scale event stream feature extractor, as shown in the following formula: F=concat(F0,F1,F2,F3,F4) (1) The encoder-decoder is used as the basic architecture. After extracting the multi-scale features of the input event stream representation, the decoder obtains the lane line pixel-by-pixel probability map. At the same time, a lane line branch network is connected to the output features of the encoder. At this point, the basic model structure has been completed. During training, it outputs a single-channel lane line detection probability map, and the cross loss is calculated between it and the true label; as shown in the following formula: Where N is the total number of pixels in the true label, M is the total number of lane line categories, when different categories of lane lines are not distinguished, M = 2, when different categories of lane lines need to be distinguished, M = 5; y m,n is the probability that the corresponding pixel in the true label belongs to category m. For the true label, this probability value is either 0 or 1, which is called a hard label and contains limited mutual information. Indicates the probability that the predicted lane line extraction result belongs to class m; The momentum distillation method in the visual language model ALBEF is adopted. In ALBEF, the data used in the training phase is a large number of image-text pairs crawled from the Internet. The momentum model is used to solve the problem of chaotic data sources, integrating knowledge distillation and transfer learning methods. In addition to using the real label, a copy of the model is used as a momentum model, and the predicted value output by the momentum model is used as a pseudo label. and the true label y m,n Supervise the training of the base model together; Pseudo Labels The predicted value of the basic model The similarity loss is calculated by the L2 norm, as shown in the following formula: The meanings of N and M are the same as those of L sup The corresponding parameters in have the same meaning. It is the predicted value output by the momentum model. It has non-zero values ​​in each category and has more mutual information than the true label. It is also called a soft label. During the training phase, in order to make the pseudo-labels more instructive, the momentum model is preheated, that is, trained for several rounds in advance, while the parameters of the basic model are randomly initialized.

5. The method for lane line extraction and detection based on event stream data based on momentum distillation according to claim 4, characterized in that: The total loss L in step 3 total The weighted sum of the two losses is as follows: L total =λ1L sup +λ2L consistency (4) Where L sup and L consistency As described in formula (2) and formula (3); λ1 and λ2 are two hyperparameters.

6. The method for lane line extraction and detection based on event stream data based on momentum distillation according to claim 1, characterized in that: In step 4, the exponential moving average strategy is adopted to update the parameters of the momentum model. Specifically, the total loss of formula (4) is used to update the parameters of the basic model, while the parameters of the momentum model are updated according to the exponential moving average strategy (EMA) using the parameters of the momentum model of the previous round and the basic model of this round; as shown in the following formula: I will t =θ' t-1 +(1-α)θ t (5) where θ' t and θ t It refers to the parameters of the momentum model and the basic model in the tth round, and α is the smoothing coefficient hyperparameter.

Images