Method, device and equipment for detecting driving behavior quality of driver and medium

Abstract

This application relates to a method, apparatus, device, and medium for detecting the quality of a driver's driving behavior. It acquires a front-view image of a vehicle and natural language commands, and based on a pre-trained visual-language model, obtains the detection results of the driver's driving behavior corresponding to the front-view image. This application can identify the driver's driving behavior corresponding to the front-view image of the vehicle, thereby obtaining detection results reflecting the driver's driving behavior. Front-view image data showing non-compliant driving behavior can be filtered out, cleaning low-quality driving data at the data source. This prevents low-quality driving data from entering the training set of the end-to-end autonomous driving model and polluting the training set, thereby improving the purity and quality of the training dataset. This ensures that the end-to-end autonomous driving model can gradually learn how to accurately identify whether driving behavior complies with regulations, reducing subsequent training costs and improving the safety of autonomous driving strategies.

Description

Technical Field

[0001] This application relates to the fields of autonomous driving data governance and artificial intelligence, and in particular to a method, device, equipment and medium for detecting the quality of a driver's driving behavior. Background Technology

[0002] Despite significant progress in end-to-end models for autonomous driving, the sheer volume of data required remains a major obstacle. Existing datasets (such as nuScenes, Waymo, and BLVD) contain only a few thousand hours of driving data, far from meeting the demand for millions of hours. Furthermore, the data exhibits a long-tail distribution, with corner cases (such as irregularly shaped vehicles, extreme weather, and construction zones) contributing over 30% of accident risk despite their extremely low frequency.

[0003] It is evident that the quality of the training dataset affects the decision-making of the end-to-end autonomous driving model; a low-quality training dataset can cause the end-to-end autonomous driving model to learn incorrect or dangerous strategies. Summary of the Invention

[0004] This application provides a method, device, equipment, and medium for detecting the quality of a driver's driving behavior. It can identify the driver's driving behavior corresponding to the forward view image of the vehicle, thereby obtaining detection results that reflect the driver's driving behavior. It cleans up low-quality driving data from the source, thereby improving the purity and quality of the training dataset for the end-to-end autonomous driving model.

[0005] In a first aspect, embodiments of this application provide a method for detecting the quality of a driver's driving behavior, comprising: Obtain the front view image from the vehicle; Natural language instructions are obtained, and based on a pre-trained visual language model, the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle are obtained.

[0006] In conjunction with the first aspect, in one implementation, the driving behavior quality detection result includes a response to the natural language instruction and a reason for obtaining the response.

[0007] In conjunction with the first aspect, in one implementation, the response is represented by an assignment, wherein when the driver's driving behavior is compliant, a first value is assigned, and when the driver's driving behavior is non-compliant, a second value is assigned.

[0008] In conjunction with the first aspect, in one implementation, the compliance type includes at least one of not crossing a solid line, not failing to yield to pedestrians, and the vehicle being driven in the center of the lane. The types of violations include at least one of the following: crossing a solid line, failing to yield to pedestrians, and driving on the left or right side of the road.

[0009] In conjunction with the first aspect, in one implementation, the driving behavior quality detection results are screened based on the responses to the driving behavior quality detection results.

[0010] In conjunction with the first aspect, in one implementation, if the answer is a first numerical value, then the vehicle front view image is retained; If the answer is the second value, then the vehicle's front view image is discarded.

[0011] In conjunction with the first aspect, in one embodiment, after obtaining the driving behavior quality detection result, the method further includes: The self-view images that reflect the driver's compliant driving behavior as determined by the driving behavior quality detection results are selected and used as the end-to-end training dataset for training the end-to-end autonomous driving model.

[0012] In conjunction with the first aspect, in one implementation, the visual language model includes: A visual encoder is used to encode the front view image of the vehicle to obtain an image encoding vector; A text encoder, used to encode the natural language instructions to obtain a text encoding vector; A large language model is used to take the image encoding vector and text encoding vector as input and output the driving behavior quality detection result of the driver corresponding to the front view image of the vehicle.

[0013] In conjunction with the first aspect, in one implementation, the training process of the visual language model includes: Obtain the model training dataset, which includes image-text pairs; The large language model is trained using the model training dataset as input to obtain the visual language model.

[0014] In conjunction with the first aspect, in one implementation, obtaining the model training dataset includes: Collect driving image data and use a data annotation model to pre-annotate driving behavior norm types to obtain a pre-annotated dataset; The pre-labeled dataset is manually re-labeled to obtain the model training dataset.

[0015] In conjunction with the first aspect, in one implementation, the pre-labeled dataset is manually re-labeled to obtain a model training dataset, including: The pre-labeled dataset is then manually re-labeled as the result to be verified. The pre-labeled dataset is sampled to obtain the annotation results of the sampled dataset being manually re-annotated by experts, which are used as the ground truth results; Based on the results to be verified and the ground truth results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset.

[0016] In conjunction with the first aspect, in one implementation, the accuracy of the first manual re-annotation is evaluated based on the result to be verified and the ground truth result to obtain the model training dataset, including: Obtain the percentage of driving image data in the sample dataset whose verification results are the same as the true results; If the proportion is greater than the accuracy threshold, the pre-annotated dataset that has been manually re-annotated by the first person will be used as the model training dataset. Otherwise, the pre-labeled dataset is re-labeled manually for the first time.

[0017] In conjunction with the first aspect, in one implementation, the model training dataset is divided into multiple training subsets according to the type of driving behavior norms; The prompt word templates for each of the training subsets are the same, but the content of the prompt words varies depending on the type of driving behavior norm.

[0018] In conjunction with the first aspect, in one implementation, the driving behavior norm type includes at least one of the following: whether the vehicle crosses a solid line, whether the vehicle fails to yield to pedestrians, and whether the vehicle is driving in the center of the lane.

[0019] Secondly, embodiments of this application provide a method for training an end-to-end autonomous driving model, comprising: Obtain an end-to-end training dataset, wherein the end-to-end training dataset is obtained by filtering the driver's driving behavior quality detection images of the vehicle using any of the above-described methods. The end-to-end autonomous driving model is trained using the end-to-end training dataset to obtain a trained end-to-end autonomous driving model.

[0020] Thirdly, embodiments of this application provide a driver's driving behavior quality detection device, the device comprising: Front view image acquisition module, which is used to acquire front view images of the vehicle; The detection module is used to acquire natural language instructions and, based on a pre-trained visual language model, obtain the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle.

[0021] Fourthly, embodiments of this application provide a driver's driving behavior quality detection device, the driver's driving behavior quality detection device including a processor, a memory, and a driver's driving behavior quality detection program stored in the memory and executable by the processor, wherein when the driver's driving behavior quality detection program is executed by the processor, it implements the steps of the driver's driving behavior quality detection method as described in any of the preceding claims.

[0022] Fifthly, embodiments of this application provide a computer-readable storage medium storing a driver's driving behavior quality detection program, wherein when the driver's driving behavior quality detection program is executed by a processor, it implements the steps of the driver's driving behavior quality detection method as described in any of the preceding claims.

[0023] The beneficial effects of the technical solution provided in this application include: This application provides a method, apparatus, device, and medium for detecting the quality of a driver's driving behavior. In this application, by combining natural language instructions and using a pre-trained visual language model, the driving behavior of the driver corresponding to the front view image of the vehicle can be identified, thereby obtaining detection results reflecting the driving behavior of the driver of the vehicle.

[0024] For autonomous vehicle driver's forward view image data with irregular driving behavior, it can be directly filtered out. Low-quality driving data is cleaned from the data source to prevent this low-quality driving data from entering the data training set of the end-to-end autonomous driving model and thus polluting the data training set. This improves the purity and quality of the training dataset, ensuring that the end-to-end autonomous driving model can gradually learn how to accurately identify whether driving behavior complies with regulations, reducing subsequent training costs and improving the safety of autonomous driving strategies. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating an embodiment of the driver's driving behavior quality detection method of this application; Figure 2 This is a schematic diagram of the functional modules of the visual language model in this application; Figure 3 This is a schematic diagram of the functional modules of an embodiment of the driver's driving behavior quality detection device of this application; Figure 4This is a schematic diagram of the hardware structure of the driver's driving behavior quality detection device involved in the embodiments of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In recent years, data-driven end-to-end (E2E) autonomous driving has become the mainstream development direction in academia and industry. Its core idea is to integrate the traditional four-level architecture of autonomous driving—perception, prediction, planning, and control—into a single neural network, utilizing large-scale behavior cloning or reinforcement learning (RL) to directly output control commands (such as steering, throttle, and braking) from the raw inputs of sensors. Representative solutions include: NVIDIA's PilotNet, which directly outputs a single-view image as input and a steering wheel angle as output; Wayve's LINGO-1 and GAIA-1, which use a multimodal Transformer to uniformly encode images, text, and actions to achieve interpretable E2E driving; Tesla FSD v12, which completely removed 300,000 lines of C++ rule code, adopting an Occupancy Network + end-to-end planning network, relying on millions of hours of real-world driving data; and the "BEV + Transformer + E2E" solution from leading domestic new energy vehicle companies, which uniformly converts surround-view images into BEV features and directly outputs the trajectory.

[0029] Since the quality of the training dataset affects the decision-making of end-to-end autonomous driving models, improving the quality of the training dataset in related technologies mainly involves improving the accuracy of the annotation results during the training dataset annotation stage.

[0030] However, the above annotation scheme does not improve the quality of the training dataset from the data source. For example, it fails to effectively address the question of "whether the driver's behavior is good or bad." Drivers frequently checking their phones, yawning, or engaging in lengthy conversations with their passengers can lead to unsafe driving behavior, such as crossing lane lines or deviating from the lane.

[0031] The driver exhibited aggressive driving behavior, such as weaving lanes, rapid acceleration, and rapid deceleration.

[0032] When a driver takes their hands off the steering wheel for an extended period of time, the vehicle becomes unstable.

[0033] During driving, a driver's lack of concentration or personal driving habits may cause the vehicle to drive on the edge of the lane or run a red light.

[0034] Even if the above behaviors can improve the accuracy of the annotation results during the annotation stage, they will still pollute the training dataset, causing the end-to-end autonomous driving model to learn incorrect or dangerous policies.

[0035] Therefore, there is an urgent need for an automated data cleaning technology that requires no additional hardware, is scalable, interpretable, and directly addresses "driver behavior quality".

[0036] Based on this, the technical solution of this application is proposed.

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0038] In a first aspect, embodiments of this application provide a method for detecting the quality of a driver's driving behavior.

[0039] In one embodiment, reference is made to Figure 1 , Figure 1 This is a schematic flowchart illustrating an embodiment of the driver's driving behavior quality detection method of this application. Figure 1 As shown, the methods for detecting the quality of a driver's driving behavior include: 101: Obtain the front view image from the vehicle.

[0040] Understandably, while the driver is driving the car, the camera installed on the car can collect real-time images of the front view of the current driving scene, which can then be used as input for the visual language model.

[0041] Understandably, the forward-view camera can clearly and completely capture the driver's actions while driving, including whether the vehicle crosses solid lines, whether it fails to yield to pedestrians, and whether it is driving in the center of the lane.

[0042] 102: Obtain natural language instructions and, based on a pre-trained visual language model, obtain the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle.

[0043] The visual language model is pre-trained using a model training dataset, enabling it to process images and text.

[0044] Specifically, in the data processing stage, the visual language model can encode the input front view image of the vehicle to obtain an image encoding vector. The visual language model can also encode the input text information or natural language instructions to obtain a text encoding vector.

[0045] In the perception reasoning stage, the visual language model takes the image encoding vector and text encoding vector as input and outputs the driving behavior quality detection result of the driver corresponding to the front view image of the vehicle.

[0046] In other words, the visual language model has multiple modules that can process and reason about the input text information or natural language instructions and the vehicle's front view image to obtain the driver's driving behavior quality detection result corresponding to the vehicle's front view image.

[0047] It is understood that the visual language model outputs the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle based on the preset driving behavior norm type.

[0048] For example, the preset driving behavior norms include whether the vehicle crosses a solid line, whether the vehicle fails to yield to pedestrians, and whether the vehicle is driving in the center of the lane. It is understandable that the types of driving behavior norms mentioned above are not limited to the few listed.

[0049] Accordingly, the aforementioned natural language instructions are prompts about preset driving behavior norms. Based on these natural language instructions, the visual language model can extract and recognize features from the vehicle's forward view image to obtain the corresponding driver's driving behavior quality detection results.

[0050] The aforementioned natural language instructions can be input from the user to the visual language model.

[0051] For example, in a specific embodiment, with the specific task of determining whether "the vehicle has crossed a solid line," the aforementioned natural language instruction can be described as " The current image is from a car's forward-facing camera. Did your car cross a solid line? Please output your answer in JSON format, including the key: score, reason; where score represents the confidence level of the driving behavior of crossing the solid line, with a value range of [0, 1], where 0 means very likely that the car did not cross the solid line, and 1 means very certain that the car crossed the solid line; reason provides an explanation for the score.

[0052] For example, in a specific embodiment, with the specific task of determining whether "the vehicle failed to yield to pedestrians," the aforementioned natural language instruction can be described as " The current image is from a car's forward-facing camera. Is your vehicle failing to yield to pedestrians? Please output your answer in JSON format, including the key: score, reason; where score represents the confidence level of failing to yield to pedestrians, with a value range of [0, 1], where 0 means it is very likely that the vehicle did not fail to yield, and 1 means it is very certain that the vehicle did not yield; reason provides the explanation for the score.

[0053] For example, in a specific embodiment, with the specific task of determining "whether the vehicle's driving position has shifted laterally," the aforementioned natural language instruction can be described as " The current image is from a car's forward-facing camera. Based on the lane markings on the left and right sides of the vehicle, please determine whether the vehicle is driving in the center of the lane, or is driving to the left or right. Please output your answer in JSON format, including the key: score, reason; where score represents the confidence level of the vehicle's lateral deviation driving behavior, with a value range of [0, 1], where 0 indicates that the vehicle is driving in the center of the lane, and 1 indicates that the vehicle is very certain to be driving to the left or right; reason provides an explanation for the score.

[0054] As can be seen, in this embodiment, by combining natural language instructions and using a pre-trained visual language model, the driving behavior of the driver corresponding to the front view image of the vehicle can be identified, thereby obtaining detection results reflecting the driving behavior of the driver of the vehicle.

[0055] For autonomous vehicle driver's forward view image data with irregular driving behavior, it can be directly filtered out. Low-quality driving data is cleaned from the data source to prevent this low-quality driving data from entering the data training set of the end-to-end autonomous driving model and thus polluting the data training set. This improves the purity and quality of the training dataset, ensuring that the end-to-end autonomous driving model can gradually learn how to accurately identify whether driving behavior complies with regulations, reducing subsequent training costs and improving the safety of autonomous driving strategies.

[0056] Furthermore, in one embodiment, the driving behavior quality detection result includes a response to the natural language instruction and a reason for obtaining the response.

[0057] For example, the response reflecting the driving behavior quality test results can be in the following format: "score,reason".

[0058] For example, score∈{0,1}.

[0059] Here, "score" refers to the response to the natural language instruction, and "reason" refers to the reason for obtaining the response.

[0060] In other words, this application can assign values ​​to the answers, where the first value is assigned when the driver's driving behavior is compliant, such as 0, and the second value is assigned when the driver's driving behavior is non-compliant, such as 1.

[0061] For example, in a specific embodiment, the specific task is to determine whether the vehicle is crossing a solid line. Corresponding to the aforementioned natural language instruction, the driving behavior quality detection result is: "{score: 1, reason: "A single solid lane line appears in the central area of ​​the vehicle image, which is very likely to be driving over the solid line"}.

[0062] For example, in a specific embodiment, the specific task is to determine whether the vehicle has failed to yield to pedestrians. Corresponding to the aforementioned natural language instruction, the driving behavior quality detection result is: "{score: 1, reason: "There is a pedestrian crossing in front of the vehicle. There are pedestrians waiting on the pedestrian crossing. The vehicle has approached the pedestrian crossing without stopping to yield to the pedestrians."}

[0063] For example, in a specific embodiment, the specific task is to determine whether the vehicle's driving position is laterally deviated. Corresponding to the aforementioned natural language instruction, the resulting driving behavior quality detection result is: "{score: 1, reason: "The difference between the left and right lane lines and the center of the image is too large. The distance between the right lane line and the center of the image is significantly smaller than the distance between the left lane line and the center of the image. Therefore, the vehicle is driving close to the right side of the lane."}

[0064] Using the aforementioned fixed format for responses allows for two advantages. First, assigning a credibility score to the response makes it easier to eliminate or retain the current vehicle's forward-view image based on actual needs. Second, providing explanations for the response facilitates further user assessment and assists in rapid decision-making by either humans or the system.

[0065] Furthermore, in one embodiment, the compliance type includes at least one of not crossing a solid line, not yielding to pedestrians, and the vehicle being driven in the center of the lane; the violation type includes at least one of crossing a solid line, not yielding to pedestrians, and the vehicle being driven on the left or right.

[0066] Crossing a solid line can be understood as a vehicle crossing the solid line in the middle of the road (such as a continuous white or yellow line) to change lanes, which is considered a dangerous behavior. For example, crossing the line to change lanes on a highway can easily lead to oncoming collisions.

[0067] Failure to yield to pedestrians can be understood as a driver failing to stop and yield to pedestrians while they are crossing the road at a crosswalk. For example, a vehicle may accelerate through a pedestrian's path even after the pedestrian has reached the middle of the road.

[0068] Driving on the left or right can be understood as the vehicle not being in the center of the lane, but rather closely following the left or right lane lines. For example, if a vehicle is driving close to the left edge (left) or the right edge (right) of the road, this is considered lane departure.

[0069] Understandably, by using natural language instructions to describe the above-mentioned violation or compliance types, textual features of the natural language instructions can be clearly extracted, making it easier to infer from the collected vehicle front view images.

[0070] Furthermore, in one embodiment, the driving behavior quality detection results are filtered based on the responses. If the response is a first value, such as 0, the driving behavior quality detection result is considered compliant, and the vehicle's forward view image is retained. If the response is a second value, such as 1, the driving behavior quality detection result is considered non-compliant, and the vehicle's forward view image is discarded.

[0071] For example, in a specific embodiment, the driving behavior quality detection result is: "{score: 1, reason:"A single solid lane line appears in the central area of ​​the vehicle image, which is very likely that the vehicle is driving over the solid line"}.

[0072] Since score=1, it indicates that it belongs to the preset violation type, so it can be filtered out directly.

[0073] This embodiment directly filters based on the answers, automatically discarding violation types (such as "driving across a solid line") and automatically retaining compliant types (such as "yielding to pedestrians"). No manual review is required, achieving automated data cleaning and ensuring that only compliant driving behaviors enter the training dataset of the end-to-end autonomous driving model, eliminating low-quality data pollution and enhancing system safety and reliability.

[0074] Furthermore, in one embodiment, the end-to-end autonomous driving model is trained using the front view image of the vehicle, which reflects the driver's compliant driving behavior, as an end-to-end training dataset.

[0075] In this embodiment, the aforementioned compliance detection can be used to select self-view images with high reliability of driving behavior quality detection results and compliant driving behavior as training datasets, and then train the end-to-end autonomous driving model to eliminate low-quality noise, providing a clean and high-quality training foundation for the end-to-end autonomous driving model, avoiding "bad data" pollution, and accelerating the learning of safe driving behavior.

[0076] Furthermore, in one embodiment, see... Figure 2 As shown, the visual language model includes a visual encoder, a text encoder, and a large language model. The visual encoder is used to encode the front view image of the vehicle to obtain an image encoding vector; the text encoder is used to encode the natural language instructions to obtain a text encoding vector; and the large language model is used to take the image encoding vector and the text encoding vector as input and output the driving behavior quality detection result of the driver corresponding to the front view image of the vehicle.

[0077] The visual language model provided in this example is built on a visual encoder, a text encoder, and a large language model. The visual encoder is responsible for extracting high-level features from the input image and analyzing key information in the image, such as the vehicle's trajectory and the position of lane markings.

[0078] Large Language Models (LLMs) are used to understand natural language instructions related to the problem and combine visual features to perform cross-modal reasoning and judgment.

[0079] The training scheme for the visual language model in this application is based on a combination of reinforcement learning and traditional supervised learning, and adopts an architecture in which the visual encoder and the large language model LLM work together.

[0080] The training process of the visual language model includes: 201: Obtain the model training dataset, which includes image-text pairs.

[0081] The model training dataset in this embodiment is a multimodal dataset, consisting of images and corresponding text descriptions.

[0082] In addition to the standard scenario understanding task, the above text description also includes questions about the driving regulations of the vehicle, including but not limited to: whether the vehicle crosses a solid line, whether the vehicle fails to yield to pedestrians, and whether the vehicle is driving in the center of the lane.

[0083] 202: Using the model training dataset as input, train the large language model to obtain the visual language model.

[0084] Understandably, during model training, the model needs to be guided to output corresponding values ​​for the input image-text pairs based on prompts. As this is a specific task, and model generalization is not considered, this application uses a fixed prompt template. The prompt will be adjusted accordingly based on different quality assessment sub-tasks.

[0085] The visual language model provided in this embodiment integrates a visual encoder, a text encoder, and an LLM (Local Language Model). Based on multimodal training data (including descriptions of regulatory issues such as crossing solid lines and yielding to pedestrians), it achieves accurate detection through supervised and reinforcement learning. It automatically outputs structured results "score, reason," retaining only highly credible and compliant data, eliminating manual review costs, and ensuring that the model only learns safe driving behaviors.

[0086] Within the framework of reinforcement learning, a reward mechanism is introduced to guide the visual language model in learning how to judge the quality of driving behavior based on visual and linguistic information. Specifically, a positive reward is given when the model makes a correct classification, and a negative penalty is given if the classification is incorrect. This mechanism allows the model to continuously adjust its judgment strategy during training, thereby improving accuracy and stability.

[0087] In the traditional supervised learning part, the visual language model is trained through a binary classification task, aiming to distinguish between "illegal" and "compliant" driving behaviors. During this process, the cross-entropy loss function is used to calculate the error between the model output and the true label, further optimizing the model's parameters. By minimizing the cross-entropy loss, the model can gradually learn how to accurately identify whether driving behavior complies with regulations.

[0088] Leveraging the advantages of reinforcement learning, visual language models not only rely on manually labeled "violation / compliance" tags for training but also self-adjust through interaction with the environment. The reward signals from reinforcement learning encourage the model to adaptively adjust to constantly changing driving scenarios in real-world applications, improving its long-term evaluation capabilities. For example, in different road environments, the model can dynamically adjust its judgment criteria based on changes in traffic rules and road markings.

[0089] The training process comprises two phases: supervised learning and reinforcement learning. In the initial phase, the visual language model quickly masters basic behavior classification capabilities through supervised learning. In the subsequent phase, reinforcement learning algorithms help the model continuously optimize its quality assessment criteria through long-term feedback and reward mechanisms, thereby achieving a more accurate assessment of "driver behavior quality."

[0090] Furthermore, in one embodiment, obtaining the model training dataset includes: 301: Collect driving image data and use a data annotation model to pre-annotate driving behavior norm types to obtain a pre-annotated dataset.

[0091] It is understandable that when collecting driving image data, including driving data of one's own vehicle as well as other driving data, it is not necessary to strictly limit it to driving data of one's own vehicle. The purpose of using driving data of one's own vehicle and other vehicles is to enable the trained model to accurately identify illegal or compliant driving behaviors.

[0092] Understandably, this data covers a variety of real-world driving scenarios, including urban roads, highways, and rural roads. The data has been sampled, duplicate entries have been removed, and invalid data has been deleted.

[0093] In addition, these data are pre-annotated using existing data annotation models, and the results of the pre-annotation serve as a reference for subsequent manual annotation.

[0094] 302: Manually re-annotate the pre-annotated dataset to obtain the model training dataset.

[0095] Pre-annotation utilizes AI models to quickly generate initial annotations, significantly reducing manual costs. Subsequent manual annotation is performed by professional data annotation companies. The manual annotation process can be verified based on the pre-annotated results, further accelerating the manual annotation process. The data covers multiple scenarios including urban areas, highways, and rural areas, improving model generalization and ensuring efficient and accurate training data. This provides a high-quality, clean training set for autonomous driving systems, eliminating the risk of low-quality data contamination.

[0096] Furthermore, in one embodiment, the pre-labeled dataset is manually re-labeled to obtain a model training dataset, including: 401: Perform a first manual re-annotation on the pre-annotated dataset, which will be used as the result to be verified.

[0097] In step 401, the pre-labeled dataset is manually labeled by personnel from a data labeling company.

[0098] 402: Sample the pre-labeled dataset and obtain the annotation results of the sampled dataset being manually re-annotated by experts, which are used as the ground truth results.

[0099] In step 402, samples are drawn from the pre-labeled dataset. The drawn sample dataset is labeled by experts, and the standard result is used as the true value result.

[0100] 403: Based on the results to be verified and the ground truth results, evaluate the accuracy of manual annotation to obtain the model training dataset.

[0101] In step 403, the accuracy of the first manual re-annotation is evaluated based on the verification results and ground truth results of the sample dataset.

[0102] This embodiment employs expert sampling for calibration and annotation. Small samples are annotated as true values ​​by experts, and the accuracy is evaluated by comparing the results with those from annotation companies. Data is retained only when the accuracy is high, avoiding full manual review, significantly reducing annotation costs, ensuring high reliability of training data, eliminating low-quality noise, and improving the quality of training data.

[0103] Specifically, based on the results to be verified and the ground truth results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset, including: 501: Obtain the percentage of driving image data in the sample dataset whose verification results are the same as the true results.

[0104] 502: If the proportion is greater than the accuracy threshold, the pre-labeled dataset that has been manually re-annotated by the first person shall be used as the model training dataset; otherwise, the pre-labeled dataset shall be manually re-annotated by the first person.

[0105] The accurate threshold can be preset according to actual needs.

[0106] In one specific embodiment, the pre-labeled dataset is first manually re-labeled by personnel from an annotation company to obtain the results to be verified. Then, 10,000 data points are extracted from the pre-labeled dataset and manually re-labeled by experts to obtain the ground truth results. If the accuracy threshold is preset to 90%, the number of data points whose verification results match the corresponding ground truth results is counted. For example, if 9,500 data points match the ground truth results, the percentage is 95%, which is higher than 90%, indicating that the results of the first manual re-labeling by the annotation company are reliable and can be used as a model training dataset. If only 8,500 data points match the ground truth results, the percentage is 85%, which is lower than 90%, indicating that the results of the first manual re-labeling by the annotation company are unreliable and need to be manually re-labeled by the annotation company again.

[0107] This verification process can greatly improve the speed of verifying manually labeled data while ensuring the quality of data labeling.

[0108] Furthermore, in some embodiments, the model training dataset can be divided into multiple training subsets according to the driving behavior norm type; the prompt word templates corresponding to each training subset are the same, and the prompt word content is different based on the different driving behavior norm types.

[0109] Secondly, embodiments of this application also provide a method for training an end-to-end autonomous driving model, comprising: Obtain an end-to-end training dataset, wherein the end-to-end training dataset is obtained by filtering the driver's driving behavior quality detection image of the vehicle as described above; The end-to-end autonomous driving model is trained using the end-to-end training dataset to obtain a trained end-to-end autonomous driving model.

[0110] In this embodiment, the driver's driving behavior quality detection method can be used to screen the vehicle's front view images. The vehicle's front view images with high reliability of driving behavior quality detection results and compliant driving behavior can be selected as training datasets and then used to train the end-to-end autonomous driving model. This eliminates low-quality noise, provides a clean and high-quality training foundation for the end-to-end autonomous driving model, avoids "bad data" pollution, and accelerates the learning of safe driving behavior.

[0111] The aforementioned end-to-end autonomous driving model can be an existing model, such as ViT (VisionTransformer) and uniAD.

[0112] Thirdly, embodiments of this application also provide a driver's driving behavior quality detection device.

[0113] In one embodiment, reference is made to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the driver's driving behavior quality detection device of this application. Figure 3 As shown, the driver's driving behavior quality detection device includes: The front view image acquisition module is used to acquire front view images of the vehicle.

[0114] The detection module is used to acquire natural language instructions and, based on a pre-trained visual language model, obtain the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle.

[0115] This embodiment aims to provide an automated driving data evaluation and filtering method for end-to-end autonomous driving systems, which can accurately assess whether the quality of the current driving data meets human preferences.

[0116] Furthermore, in one embodiment, the detection module is also used to filter the driving behavior quality detection results based on the responses to the driving behavior quality detection results. If the response is a first value, the vehicle's forward view image is retained; if the response is a second value, the vehicle's forward view image is discarded.

[0117] Furthermore, in one embodiment, the detection module is also used to train the end-to-end autonomous driving model using the self-view image of the vehicle reflecting the driver's compliant driving behavior as an end-to-end training dataset.

[0118] The functions of each module in the aforementioned driver's driving behavior quality detection device correspond to the steps in the aforementioned driver's driving behavior quality detection method embodiment, and their functions and implementation processes will not be described in detail here.

[0119] Fourthly, embodiments of this application provide a driver's driving behavior quality detection device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0120] Reference Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of a driver behavior quality detection device involved in an embodiment of this application. In this embodiment, the driver behavior quality detection device may include a processor, a memory, a communication interface, and a communication bus.

[0121] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0122] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the driver behavior quality detection device, as well as interfaces used for interconnecting the driver behavior quality detection device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0123] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0124] The processor can be a general-purpose processor, which can call the driver's driving behavior quality detection program stored in the memory and execute the driver's driving behavior quality detection method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the driver's driving behavior quality detection program is called can refer to the various embodiments of the driver's driving behavior quality detection method of this application, and will not be repeated here.

[0125] Those skilled in the art will understand that Figure 4The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0126] Fifthly, embodiments of this application also provide a computer-readable storage medium.

[0127] The present application provides a computer-readable storage medium storing a driver's driving behavior quality detection program, wherein when the driver's driving behavior quality detection program is executed by a processor, it implements the steps of the driver's driving behavior quality detection method described above.

[0128] The method implemented when the driver's driving behavior quality detection procedure is executed can be referred to in the various embodiments of the driver's driving behavior quality detection method of this application, and will not be repeated here.

[0129] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0130] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0131] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0132] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0133] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0135] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting the quality of a driver's driving behavior, characterized in that, It includes: The vehicle front view image is obtained by the driver of the vehicle in real time through a camera installed on the vehicle during the driving process. Natural language instructions are obtained, and based on a pre-trained visual language model, the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle are obtained. The self-view images that reflect the driver's compliant driving behavior as the driving behavior quality detection results are selected and used as the end-to-end training dataset for training the end-to-end autonomous driving model. The visual language model includes: A visual encoder is used to encode the front view image of the vehicle to obtain an image encoding vector; A text encoder, used to encode the natural language instructions to obtain a text encoding vector; A large language model, which takes the image encoding vector and text encoding vector as input and outputs the driving behavior quality detection result of the driver corresponding to the front view image of the vehicle; The training process of the visual language model includes: Obtain the model training dataset, which includes image-text pairs; The large language model is trained using the model training dataset as input to obtain the visual language model; Obtaining the model training dataset includes: Collect driving image data and use a data annotation model to pre-annotate driving behavior norm types to obtain a pre-annotated dataset; The pre-labeled dataset is manually re-labeled to obtain the model training dataset; The pre-labeled dataset is manually re-labeled to obtain the model training dataset, including: The pre-labeled dataset is then manually re-labeled as the result to be verified. The pre-labeled dataset is sampled to obtain the annotation results of the sampled dataset being manually re-annotated by experts, which are used as the ground truth results; Based on the results to be verified and the true results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset. Based on the results to be verified and the ground truth results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset, including: Obtain the percentage of driving image data in the sample dataset whose verification results are the same as the true results; If the proportion is greater than the accuracy threshold, the pre-annotated dataset that has been re-annotated by the first human will be used as the model training dataset. Otherwise, the pre-labeled dataset is re-labeled manually for the first time; The model training dataset is divided into multiple training subsets according to the driving behavior norm type; the prompt word templates corresponding to each training subset are the same, but the prompt word content is different based on the different driving behavior norm types; The driving behavior quality detection results include the response to the natural language command and the reason for obtaining the response; The response is represented by a value, where the driver's driving behavior is compliant and is assigned a first value, and the driver's driving behavior is non-compliant and is assigned a second value.

2. The method for detecting the quality of a driver's driving behavior as described in claim 1, characterized in that: The compliance types include at least one of the following: not crossing solid lines, not failing to yield to pedestrians, and the vehicle being driven in the center of the lane. The types of violations include at least one of the following: crossing a solid line, failing to yield to pedestrians, and driving on the left or right side of the road.

3. The method for detecting the quality of a driver's driving behavior as described in claim 1, characterized in that: Based on the responses to the driving behavior quality test results, the driving behavior quality test results are filtered.

4. The method for detecting the quality of a driver's driving behavior as described in claim 3, characterized in that: If the answer is the first value, then the front view image of the vehicle is retained; If the answer is the second value, then the vehicle's front view image is discarded.

5. The method for detecting the quality of a driver's driving behavior as described in claim 1, characterized in that: The driving behavior norms include at least one of the following: whether the vehicle crosses a solid line, whether the vehicle fails to yield to pedestrians, and whether the vehicle is driving in the center of the lane.

6. A training method for an end-to-end autonomous driving model, characterized in that, It includes: Obtain an end-to-end training dataset, wherein the end-to-end training dataset is obtained by filtering the driver's driving behavior quality detection method as described in any one of claims 1 to 5 from the front view images of the vehicle. The end-to-end autonomous driving model is trained using the end-to-end training dataset to obtain a trained end-to-end autonomous driving model.

7. A device for detecting the quality of a driver's driving behavior, characterized in that, The device includes: A front view image acquisition module is used to acquire the front view image of the vehicle. The front view image of the vehicle is obtained in real time by the driver of the vehicle through a camera installed on the vehicle during the driving process. The detection module is used to acquire natural language instructions and, based on a pre-trained visual language model, acquire the driving behavior quality detection results of the driver corresponding to the front view image of the vehicle. The self-view images that reflect the driver's compliant driving behavior as the driving behavior quality detection results are selected and used as the end-to-end training dataset for training the end-to-end autonomous driving model. The visual language model includes: A visual encoder is used to encode the front view image of the vehicle to obtain an image encoding vector; A text encoder, used to encode the natural language instructions to obtain a text encoding vector; A large language model, which takes the image encoding vector and text encoding vector as input and outputs the driving behavior quality detection result of the driver corresponding to the front view image of the vehicle; The training process of the visual language model includes: Obtain the model training dataset, which includes image-text pairs; The large language model is trained using the model training dataset as input to obtain the visual language model; Obtaining the model training dataset includes: Collect driving image data and use a data annotation model to pre-annotate driving behavior norm types to obtain a pre-annotated dataset; The pre-labeled dataset is manually re-labeled to obtain the model training dataset; The pre-labeled dataset is manually re-labeled to obtain the model training dataset, including: The pre-labeled dataset is then manually re-labeled as the result to be verified. The pre-labeled dataset is sampled to obtain the annotation results of the sampled dataset being manually re-annotated by experts, which are used as the ground truth results; Based on the results to be verified and the true results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset. Based on the results to be verified and the ground truth results, the accuracy of the first manual re-annotation is evaluated to obtain the model training dataset, including: Obtain the percentage of driving image data in the sample dataset whose verification results are the same as the true results; If the proportion is greater than the accuracy threshold, the pre-annotated dataset that has been re-annotated by the first human will be used as the model training dataset. Otherwise, the pre-labeled dataset is re-labeled manually for the first time; The model training dataset is divided into multiple training subsets according to the driving behavior norm type; the prompt word templates corresponding to each training subset are the same, but the prompt word content is different based on the different driving behavior norm types; The driving behavior quality detection results include the response to the natural language command and the reason for obtaining the response; The response is represented by a value, where the driver's driving behavior is compliant and is assigned a first value, and the driver's driving behavior is non-compliant and is assigned a second value.

8. A driver's driving behavior quality detection device, characterized in that, The driver's driving behavior quality detection device includes a processor, a memory, and a driver's driving behavior quality detection program stored in the memory and executable by the processor, wherein when the driver's driving behavior quality detection program is executed by the processor, it implements the steps of the driver's driving behavior quality detection method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a driver's driving behavior quality detection program, wherein when the driver's driving behavior quality detection program is executed by a processor, it implements the steps of the driver's driving behavior quality detection method as described in any one of claims 1 to 5.

Images