ViT-based full convolution lane line model training and deployment method

Abstract

The invention relates to a ViT-based full convolution lane line model training and deployment method, which comprises a model training stage and a model deployment stage, and is characterized in that the model training stage comprises the following steps: S1, obtaining real data; s2, carrying out back propagation by adopting an AdamW optimizer; s3, data enhancement; s4, a backbone is designed; s5, making a training strategy; the model deployment stage comprises the following steps: S6, pre-processing optimization; S7, copying data to model time consumption optimization by adopting a shared memory access mode; S8, model reasoning optimization; S9, post-processing optimization;

Description

Technical Field

[0001] The present invention relates to the field of automobile autonomous driving technology, and in particular to a ViT-based full convolution lane line model training and deployment method. Background Art

[0002] The lane line model refers to the use of deep visual neural network model technology. The process includes preprocessing of lane line images such as cropping, resizing, and normalization. The preprocessed images are input into the model for feature extraction. Finally, the features extracted by the model are post-processed to obtain the final lane line detection result, and the lane line coordinates and lane line type are output.

[0003] With the continuous development of autonomous driving technology, the requirements for lane line detection accuracy are becoming higher and higher, and more and more diversified. For example, for the detection of different types of lane lines, the existing lane line detection technology condlane uses ResNet as the network skeleton, including BatchNormalization normalization method, ReLU activation function and other neural network layers, adopts traditional training strategy, and uses Adam back propagation optimizer. The accuracy index on the CULane dataset, the training method of ViT visual Transformers can significantly improve the accuracy of the Transformers-based deep vision model, but the deployment of Transformers in the industry is not perfect and the accuracy is poor. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by this patent application is how to provide a ViT-based fully convolutional lane line model training and deployment method with high accuracy, strong stability and wide applicability.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A ViT-based full convolution lane line model training and deployment method includes a model training phase and a model deployment phase. The model training phase includes the following steps: S1: obtaining real data; S2: using AdamW optimizer for back propagation; S3: data enhancement; S4: designing backbone; S5: formulating training strategy; the model deployment phase includes the following steps: S6: pre-processing optimization S7: using shared memory access to copy data to the model time-consuming optimization S8: model reasoning optimization S9: post-processing optimization S10: model input size optimization

[0007] As an optimization, when obtaining real data in step S1, a pinhole camera is used to obtain it, and after saving it in a bag package, a frame extraction tool is used to extract a frame every two seconds, and dedistortion is completed according to the camera's internal parameters, a total of 600,000 frames are obtained, and data annotation is performed.

[0008] As an optimization, in step S2, the AdamW optimizer adds weight decay regularization to balance the effects of gradients and regularization at different times. In AdamW, weight decay is not directly subtracted from parameter updates, but is used as part of parameter updates to maintain the consistency of adaptive learning rates and ensure that the learning rates of different parameters remain consistent. At the same time, the adjusted weight decay helps the algorithm converge faster and improves the stability of the training process. The weight update formula of the AdamW optimizer is as follows:

[0009]

[0010] As an optimization, in step S3, data enhancement is performed using a variety of enhancement techniques, including but not limited to Mixup, Cutmix, RandAugment, and RandomErasing data enhancement techniques.

[0011] As an optimization, in step S4, ConvNeXt is used for the skeleton, and condlanenet_head is improved for the head, so that the lane line model of this patent can detect the lane lines at the same time, including using a combination of a starting point head, an offset distance head, and a segmentation head. Each head uses a combination of convolution, activation layer, and convolution, which can detect common lane line types. Due to the existence of the segmentation head, special linearity can also be detected, including but not limited to merging lines and separating lines.

[0012] As an optimization, the training strategy in step S5 adopts batch_size=4, and trains 300 epochs on 4 GPUs on the labeled 60W dataset until the loss value, F1 Score, precision, recall and other evaluation indicators converge.

[0013] As an optimization, in step S6, the pre-processing includes resize, transpose, and normalization operators; the transpose and normalization operators are written into the model using the parallel acceleration capability of AI Core.

[0014] As an optimization, in step S8, model inference optimization mainly includes quantization and operator tuning. For quantization, the amct tool is used to quantize the float32 onnx model to the int8 onnx model, and then the atc tool is used to convert the int8 onnx model into the om model required by the board. For operator tuning, the AOE tool is used. The AOE optimizer fuses operators on an entire computational graph, divides the fused graph into operator granularity, and generates different operator tuning strategies for each fused operator subgraph, thereby achieving optimal operator performance.

[0015] As an optimization, the post-processing optimization in step S9 is to write the time-consuming maxpooling, exp, sigmoid, reshape, permute, where, and relu6 parallel operators into the model, making full use of the parallel acceleration capabilities of the ai core hardware to reduce the time consumption, and introduce the eigen matrix calculation library instead of vector to store and calculate the values ​​of matrices with large data volumes, thereby reducing the time consumption of the entire post-processing.

[0016] The present invention has the following effects:

[0017] The lane line detection method of this patent is based on ViT in terms of training. It applies some training strategies of Transformers to the fully convolutional network. By adopting new optimizers, data enhancement schemes, and regularization methods, the head is modified to realize multi-category lane line detection models. By writing time-consuming operators in post-processing into the model, the inference time is greatly reduced, so that the lane line detection model of this patent is successfully deployed on edge computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a method flow chart of a ViT-based fully convolutional lane line model training and deployment method described in the present invention.

[0019] Figure 2 The netron.app tool is used in the present invention to visualize the writing model operators as follows.

[0020] Figure 3 It is the time consumed in running at the board end in the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "upper, lower" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description. Unless otherwise stated, these directional words do not indicate or imply that the devices or components referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present invention; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0022] like Figure 1-3 As shown, a ViT-based full convolution lane line model training and deployment method includes a model training phase and a model deployment phase. The model training phase contains the following steps: S1: obtaining real data; S2: using the AdamW optimizer for back propagation; S3: data enhancement; S4: designing the backbone; S5: formulating a training strategy; the model deployment phase contains the following steps: S6: pre-processing optimization S7: using shared memory access to copy data to the model to optimize the time consumption S8: model inference optimization S9: post-processing optimization S10: model input size optimization

[0023] In this embodiment, when obtaining real data in step S1, a pinhole camera is used to obtain it, and after saving it in a bag package, a frame extraction tool is used to extract one frame every two seconds, and dedistortion is completed according to the camera's internal parameters, a total of 600,000 frames, and data annotation is performed.

[0024] In this embodiment, in step S2, the AdamW optimizer adds weight decay regularization to balance the effects of gradient and regularization at different times. In AdamW, weight decay is not directly subtracted from the parameter update, but is used as part of the parameter update to maintain the consistency of the adaptive learning rate and ensure that the learning rates of different parameters remain consistent. At the same time, the adjusted weight decay helps the algorithm converge faster and improves the stability of the training process. The weight update formula of the AdamW optimizer is as follows:

[0025]

[0026] Specifically, AdamW is an improved version based on the Adam optimizer. The main difference between the two is that AdamW adds weight decay regularization, while the Adam optimizer does not. In Adam, weight decay is directly added to the gradient; while in AdamW, regularization is directly reflected in the attenuation of weights. Therefore, AdamW is ahead of Adam in terms of convergence speed. In addition, AdamW has better performance and stability when training deep neural networks because it directly adds the gradient of the regularization term to the back-propagation formula, eliminating the step of manually adding the regularization term to the loss function, thereby improving computational efficiency.

[0027] In this embodiment, in step S3, data enhancement is performed using a variety of enhancement technologies, including but not limited to Mixup, Cutmix, RandAugment, and RandomErasing data enhancement technologies.

[0028] In this embodiment, in step S4, ConvNeXt is used in the skeleton, and condlanenet_head is improved in the head, so that the lane line model of this patent can simultaneously detect the type of lane line, including using a combination of a starting point head, an offset distance head, and a segmentation head. Each head uses a combination of convolution, activation layer, and convolution, which can detect common lane line types. Due to the existence of the segmentation head, special linearity can also be detected, including but not limited to merging lines and separating lines.

[0029] Specifically, the number of lane line types has increased to 13, including no lane line, other line types, solid line, long dashed line, short dashed line, separating line, merging line, double solid line, double dashed line, fishbone solid line and fishbone dashed line.

[0030] In this embodiment, the training strategy in step S5 adopts batch_size=4, and trains 300 epochs on 4 GPUs on a 60W labeled data set; until the loss value, F1 Score, precision, recall and other evaluation indicators converge.

[0031] In this embodiment, in step S6, the pre-processing includes resize, transpose, and normalization operators; the transpose and normalization operators are written into the model using the parallel acceleration capability of AI Core.

[0032] Specifically, the resize operation on the vpc hardware takes 2 ms, so no optimization is needed. The transpose and normalization operations using C++ take 165 ms, so the transpose and normalization operators are written into the model, and the parallel acceleration capability of the AI ​​Core is used to reduce the pre-processing time by only 2 ms, reducing the time by 80 times.

[0033] In this embodiment, in step S8, model reasoning optimization mainly includes quantization and operator tuning; in terms of quantization, the amct tool is used to quantize the float32 onnx model into the int8 onnx model, and then the atc tool is used to convert the int8 onnx model into the om model required by the board; in terms of operator tuning, the AOE tool is used, and the AOE optimizer fuses operators on an entire computational graph, divides the fused graph into operator granularity, and generates different operator tuning strategies for each fused operator subgraph, thereby achieving optimal operator performance.

[0034] Specifically, the time consumption of each operator in model reasoning is analyzed using the msprof tool as shown in Table 1.

[0035]

[0036] Table 1

[0037] In this embodiment, the post-processing optimization in step S9 is to write the time-consuming maxpooling, exp, sigmoid, reshape, permute, where, and relu6 parallel operators into the model, fully utilize the parallel acceleration capability of the ai core hardware, reduce the time consumption, introduce the eigen matrix calculation library instead of vector, store and calculate the values ​​of matrices with large data volumes, and reduce the time consumption of the entire post-processing.

[0038] Specifically, the maxpooling, exp, sigmoid, reshape, permute, where, and relu6 parallel operators are written into the model, making full use of the parallel acceleration capabilities of the ai core hardware, reducing the time consumption from 762.6ms to 90.94ms, of which the maxpooling operator is reduced from 400ms to 2ms. Finally, the eigen matrix calculation library is introduced to replace vector to store and calculate the values ​​of matrices with large data volumes, optimizing the entire post-processing time to 4ms. At the same time, although a large number of operators are written into the model, the final model inference time is only 34ms.

[0039] Specifically, the Maxpooling layer in the post-processing takes about 400ms if executed on the CPU. This lane model writes these neural network layers into the ONNX model and then converts them into the OM model for inference. Finally, the msprof tool finds that when executed on AICore, Maxpoopling takes only 2ms, which is about 200 times faster. This solution can make full use of the parallel computing power of AI Core, thereby greatly reducing the inference time of the entire lane detection model. The same operation also applies to sigmoid, relu6 and other functions in post-processing.

[0040] Specifically, the existing model input size is 576x1024, and the model inference time is 34ms. In order to further reduce the model time, the adaptability of convolution to the model input size is used to reduce the model input size to 504x896. When the model lane line accuracy still meets the requirements, the model inference time is reduced to 27ms, reducing the inference time by 20%.

[0041] The lane line detection method of this patent is based on ViT in terms of training. It applies some training strategies of Transformers to the fully convolutional network. By adopting new optimizers, data enhancement schemes, and regularization methods, the head is modified to realize multi-category lane line detection models. By writing time-consuming operators in post-processing into the model, the inference time is greatly reduced, so that the lane line detection model of this patent is successfully deployed on edge computing devices.

[0042] Finally, it should be noted that those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A ViT-based fully convolutional lane line model training and deployment method, characterized in that: It includes the model training phase and the model deployment phase. The model training phase contains the following steps: S1: Get real data; S2: Back propagation using AdamW optimizer; S3: Data augmentation; S4: Design backbone; S5: Develop training strategies; The model deployment phase consists of the following steps: S6: Pre-processing optimization S7: Optimize the time consumption of copying data to the model using shared memory access S8: Model Inference Optimization S9: Post-processing optimization S10: Model input size optimization.

2. According to the ViT-based fully convolutional lane line model training and deployment method of claim 1, it is characterized in that: When obtaining real data in step S1, a pinhole camera is used to obtain it, and after saving it in a bag package, a frame extraction tool is used to extract a frame every two seconds, and dedistortion is completed according to the camera's internal parameters, a total of 600,000 frames are obtained, and data annotation is performed.

3. According to the ViT-based full convolutional lane line model training and deployment method of claim 2, it is characterized in that: In step S2, the AdamW optimizer adds weight decay regularization to balance the effects of gradient and regularization at different times. In AdamW, weight decay is not directly subtracted from the parameter update, but is used as part of the parameter update to maintain the consistency of the adaptive learning rate and ensure that the learning rates of different parameters remain consistent. At the same time, the adjusted weight decay helps the algorithm converge faster and improves the stability of the training process. The weight update formula of the AdamW optimizer is as follows:

4. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 3, characterized in that: In step S3, data enhancement is performed using a variety of enhancement technologies, including but not limited to Mixup, Cutmix, RandAugment, and RandomErasing data enhancement technologies.

5. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 4, characterized in that: In step S4, ConvNeXt is used for the skeleton, and condlanenet_head is improved for the head, so that the lane line model of this patent can detect the lane lines at the same time, including the combination of the starting point head, the offset distance head, and the segmentation head. Each head uses a combination of convolution, activation layer, and convolution. Common lane line types can be detected. Due to the existence of the segmentation head, special linearity can also be detected, including but not limited to merging lines and separating lines.

6. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 5, characterized in that: The training strategy in step S5 uses batch_size=4, and trains 300 epochs on 4 GPUs on a 60W labeled dataset until the loss value, F1 Score, precision, and recall evaluation indicators converge.

7. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 6, characterized in that: In step S6, the pre-processing includes resize, transpose, and normalization operators; the transpose and normalization operators are written into the model using the parallel acceleration capability of AI Core.

8. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 6, characterized in that: In step S8, model inference optimization mainly includes quantization and operator tuning. For quantization, the amct tool is used to quantize the float32 onnx model into the int8 onnx model, and then the atc tool is used to convert the int8 onnx model into the om model required by the board. For operator tuning, the AOE tool is used. The AOE optimizer fuses operators on an entire computational graph, divides the fused graph into operator granularity, and generates different operator tuning strategies for each fused operator subgraph, thereby achieving optimal operator performance.

9. The method for training and deploying a fully convolutional lane line model based on ViT according to claim 6, characterized in that: The post-processing optimization in step S9 is to write the time-consuming maxpooling, exp, sigmoid, reshape, permute, where, and relu6 parallel operators into the model, fully utilize the parallel acceleration capabilities of the ai core hardware, introduce the eigen matrix calculation library instead of vector, store and calculate the values ​​of matrices with large data volumes, and reduce the time consumption of the entire post-processing.

Images