Auxiliary driving method of camera based on deep learning model
Through the camera-assisted driving method based on deep learning model, the abnormal factors are identified and distinguished, and the problems of unstable optical data acquisition and misjudgment of fatigue driving monitoring data in traditional automobile driving assistance structures are solved, achieving higher intelligent driving experience and safety sex.
Patent Information
- Application Number
- CN202410997573.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
AI Technical Summary
The driving assistance structure of traditional cars has problems such as unstable optical data acquisition and misjudgment of fatigue driving monitoring data in the demand for intelligent driving, especially due to data acquisition deviations and external interference caused by the driver's position, which is difficult to meet the needs of intelligent driving.
The camera-assisted driving method based on deep learning models is adopted to train and deploy deep learning models, identify and distinguish abnormal factors, obtain image feature data, and use it for automatic operation and model improvement, gradually optimize the recognition accuracy and eliminate external interference.
It improves the intelligent driving experience, enhances the accuracy of identification of abnormal behaviors and light interference, reduces the impact of external interference on the system, and improves driving safety and stability of intelligent driving.
Smart Images

Figure CN118968463A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automobile technology, and in particular to an assisted driving method using a camera based on a deep learning model. Background Art
[0002] With the development of intelligent driving technology, the driving assistance structure on traditional cars can no longer meet people's requirements for easy driving.
[0003] Traditional anti-glare interior rearview mirrors usually have forward-facing ambient light sensors and rear-facing glare sensors. Although the ambient light in the scene where the vehicle is located can be collected through the forward-facing ambient light sensor and the interference light from the rear vehicle headlights can be collected through the rear-facing glare sensor, the collection process is easily affected by external light sources and structures such as the rear windshield. In addition, if there is a passenger in the middle of the back seat, it will also affect the collection of light data and the automatic adjustment of anti-glare, thereby affecting driving, making it difficult to meet the existing automotive intelligent driving requirements. In addition, although the existing interior rearview mirrors also integrate functions such as fatigue driving monitoring on the basis of realizing anti-glare adjustment, it is convenient for the driver to observe the situation behind the car and can also realize the collection of the driver's facial status data, and judge whether the driver is driving fatigued by the change of facial status data, so as to issue an early warning. Although the above structure has increased the function of the interior rearview mirror, making it more intelligent, however, for the needs of automobile intelligent driving, when detecting driving fatigue, since the driver is located on the left side of the vehicle, the interior rearview mirror is usually set in the middle of the vehicle, resulting in a misaligned arrangement between the driver and the interior rearview mirror, resulting in the collected data being non-ordinary frontal facial data. In addition, it is difficult to correctly judge the driver's behavior when only facial data is collected, which affects intelligent driving. Summary of the invention
[0004] In view of the above-mentioned problems existing in the prior art, the present invention aims to provide an assisted driving method of a camera based on a deep learning model. By training a deep learning model through images collected in normal and abnormal conditions in the early stage, the abnormal factors can be preferentially identified and the levels can be distinguished. The model is then deployed to the analysis device to analyze the collected images, obtain image feature data, and provide a data basis for automatic operation and model improvement in subsequent applications. Since the model used can continuously perform deep learning, it can continuously optimize the recognition accuracy, gradually avoid interference problems in daily use, and enhance the intelligent driving experience.
[0005] The specific technical solutions are as follows: The camera-assisted driving method based on the deep learning model has the following characteristics, including: two processes: model training and model deployment; Model training includes the following steps: Step S11, collecting and storing abnormal images of several driving scenes; Step S12, collecting and storing normal images of several driving scenes; Step S13, marking abnormal factors in the abnormal image and storing them; Step S14, training an initial model based on abnormal images and normal images with abnormal factors marked, setting the optimal recognition of abnormal factors and performing grade differentiation, and outputting a training data model; Step S15, using a preset independent test data set to test the training data model to evaluate whether the current training data model meets the usage requirements; Step S16, converting the training data model evaluated to meet the usage requirements into an open neural network exchange format; Model deployment includes the following steps: Step S21, deploying the training data model obtained in step S16 to the model analysis device, and according to the driving images collected and transmitted back by the camera in real time, preferentially identifying abnormal factors, and performing grade differentiation to obtain image feature data; Step S22, using the obtained image feature data as the data basis for automatic operation during the application process and the data basis for model improvement.
[0006] In the above-mentioned camera-assisted driving method based on a deep learning model, the initial model is an open source Yolov8 model.
[0007] In the above-mentioned camera-assisted driving method based on a deep learning model, the abnormal image is a glare image, including a rear-view image with backlight and strong light.
[0008] In the above-mentioned camera-assisted driving method based on a deep learning model, the abnormal factors are the description of the object and the overexposed area.
[0009] In the above-mentioned camera-assisted driving method based on a deep learning model, the levels are divided into glare intensity grading.
[0010] In the above-mentioned camera-assisted driving method based on a deep learning model, the preset independent test data set includes glare images with backlight and strong light, as well as normal images.
[0011] In the above-mentioned camera-assisted driving method based on a deep learning model, the abnormal image is an abnormal driving image, including images with abnormal behavior and abnormal posture.
[0012] In the above-mentioned camera-assisted driving method based on a deep learning model, the abnormal factors are abnormal behavior types and abnormal behavior human body postures.
[0013] In the above-mentioned camera-assisted driving method based on a deep learning model, the level distinction is the amplitude level distinction of abnormal behavior.
[0014] In the above-mentioned camera-assisted driving method based on a deep learning model, in step S21, an intervention measure threshold is preset, and the obtained image feature data is compared with the preset intervention measure threshold to determine whether intervention is required.
[0015] The positive effects of the above technical solution are: The above-mentioned camera-assisted driving method based on deep learning model includes two processes: model training and model deployment. By collecting abnormal images and normal images and marking abnormal factors, the initial model is trained according to the abnormal images and normal images with marked abnormal factors, and the abnormal factors are preferentially identified and graded, and the training data model is output. The output training data model is then evaluated through a preset independent test data set, and the training data model that meets the usage requirements is converted into an open neural network exchange format and deployed in the model parsing device. According to the images collected and returned by the camera in real time, the abnormal factors are preferentially identified and graded to obtain image feature data. The image feature data provides a data basis for automated operations and model improvements, and can continue deep learning, continuously optimize recognition accuracy, gradually eliminate external interference during daily use, and enhance the intelligent driving experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of an embodiment of a camera-assisted driving method based on a deep learning model of the present invention; Figure 2 A schematic diagram of a flow chart when an abnormal image is a glare image in a preferred embodiment of the present invention; Figure 3 It is a flow chart of a preferred embodiment of the present invention when the abnormal image is an abnormal driving image. DETAILED DESCRIPTION
[0017] In order to make the technical means, creative features, objectives and effects of the present invention easier to understand, the following embodiments are combined with the attached Figure 1 To Attachment Figure 3 The technical solution provided by the present invention is described in detail, but the following content is not intended to limit the present invention.
[0018] Figure 1 FIG. 1 is a flow chart of an embodiment of the camera-assisted driving method based on a deep learning model of the present invention. Figure 1As shown, the assisted driving method of the camera based on the deep learning model provided in this embodiment includes: two processes: model training and model deployment. The deep learning model is first trained, and then the trained model is deployed to the model parsing device. The acquired data is used to provide a data basis for automatic operation and a data basis for model improvement. Through continuous deep learning, the interference factors that occur in the actual driving process are gradually avoided, the recognition accuracy is improved, and thus the intelligent driving experience is improved.
[0019] Specifically, model training includes the following steps: Step S11, collecting and storing abnormal images of several driving scenes. At this time, sufficient abnormal images of multiple driving scenes need to be collected; Step S12, collect normal images of several driving scenes and store them. Similarly, it is necessary to collect a sufficient number of normal images of multiple driving scenes. Preferably, a normal image is simultaneously collected in each driving scene of an abnormal image, ensuring that each abnormal image can find a reference in the subsequent training.
[0020] Step S13, marking abnormal factors in the abnormal image and storing them; Step S14, training an initial model based on abnormal images and normal images with abnormal factors marked, setting the priority of identifying abnormal factors and distinguishing their levels, and outputting a training data model; Step S15, using a preset independent test data set to test the training data model to evaluate whether the current training data model meets the usage requirements; Step S16, converting the training data model evaluated to meet the usage requirements into an open neural network exchange format; Specifically, model deployment includes the following steps: Step S21, deploying the training data model obtained in step S16 to the model analysis device, and according to the driving images collected and transmitted back by the camera in real time, preferentially identifying abnormal factors, and performing grade differentiation to obtain image feature data; Step S22, using the obtained image feature data as the data basis for automatic operation during the application process and the data basis for model improvement.
[0021] More specifically, the initial model used in model training is the open source Yolov8 model, which is easy to obtain and can be used directly. It is widely used and the technology is mature.
[0022] Figure 2 FIG. 1 is a flow chart of a preferred embodiment of the present invention when the abnormal image is a glare image. As a preferred implementation mode, Figure 2As shown in FIG. 1 , when the abnormal image of the driving scene collected is a glare image, the glare image includes a rear view image with backlight and strong light. The model training process includes the following steps: Step S111, collecting and storing a sufficient number of glare images of multiple driving scenes.
[0023] Step S121, collecting and storing sufficient normal images of multiple driving scenes under normal light.
[0024] Step S131, annotate the collected glare image, annotate the abnormal factors in the glare image, and store them. At this time, the abnormal factors include the description of the object in the glare image and the overexposed area, which provides conditions for preferentially identifying the abnormal factors and distinguishing the levels in subsequent training.
[0025] Step S141 , training an initial model based on the glare images and normal images with abnormal factors marked, setting the priority of identifying abnormal factors and distinguishing glare levels, and outputting the training data model after the training is completed.
[0026] Step S151, use a preset independent test data set to test the training data model to evaluate whether the current training data model meets the usage requirements. It is worth noting that the preset independent test data includes several glare images with backlight, strong light, etc. and several normal images under normal light. The output results are used to determine whether the output training data model is sufficient to meet subsequent usage requirements.
[0027] Step S161, converting the training data model that is evaluated to meet the usage requirements into the Open Neural Network Exchange Format (ONNX) so that it can be applied to different hardware devices, providing conditions for popularization of applications.
[0028] In addition, when the abnormal image is a glare image, the model deployment process includes the following steps: Step S211, deploying the training data model obtained in step S161 to the model analysis device, and preferentially identifying abnormal factors including glare characteristics according to the driving images collected and transmitted back by the camera in real time, and distinguishing the glare intensity levels to obtain image feature data; Step S221, using the obtained image feature data as the data basis of the algorithm of the automatic anti-glare device of the automobile rearview mirror.
[0029] Figure 3 FIG. 1 is a flow chart of a preferred embodiment of the present invention when the abnormal image is an abnormal driving image. As another preferred implementation manner, Figure 3 As shown, when the abnormal image of the driving scene collected is an abnormal driving image, at this time, the abnormal driving image includes an image with abnormal behavior and abnormal posture. The model training process includes the following steps: Step S112, collect a sufficient number of abnormal driving behavior images of multiple driving scenes as abnormal driving images, and store them. At this time, the abnormal driving images include abnormal behavior types and abnormal behavior human body postures, avoiding the defect of only focusing the monitoring object on the face and making it impossible to correctly judge the driver's behavior, thereby ensuring the effectiveness and accuracy of the monitoring.
[0030] Step S122, collect and store a sufficient number of normal images of normal behavior types and human postures of multiple driving scenes. Similarly, a normal image is synchronously collected in the driving scene of each abnormal driving image, ensuring that each abnormal driving image can find a reference basis in subsequent training.
[0031] Step S132, annotating the collected abnormal driving images, annotating the abnormal factors in the abnormal driving images, and storing them. At this time, the abnormal factors in the abnormal driving images include abnormal behavior types and abnormal behavior human body postures, and the abnormal behavior types include yawning, rapid eye opening and closing and other behavior types, and the abnormal behavior human body postures include making phone calls, smoking, the associated postures of the eyes and forearms, the associated postures of the head and shoulders when dozing off and other human body postures. Abnormal factors are annotated from two aspects of behavior type and human body posture, increasing the number of judgment factor types in abnormal situations and improving judgment accuracy.
[0032] Step S142, training an initial model based on abnormal driving images and normal images with abnormal factors marked, setting the priority of identifying abnormal factors and distinguishing the magnitude of abnormal behaviors, and outputting a training data model; Step S152, use a preset independent test data set to test the training data model to evaluate whether the current training data model meets the usage requirements. It is worth noting that the preset independent test data set includes a number of abnormal driving images with abnormal behaviors and normal images under normal behaviors. The output results are used to determine whether the output training data model is sufficient to meet subsequent usage requirements.
[0033] Step S162, converting the training data model that is evaluated to meet the usage requirements into an open neural network exchange format (Open Neural Network Exchange, abbreviated as ONNX) so that it can be applied to different hardware devices, providing conditions for popularization of applications.
[0034] In addition, its model deployment process includes the following steps: Step S212: deploy the training data model obtained in step S162 to the model analysis device, and prioritize the identification of abnormal factors and the level of abnormal behavior according to the driving images collected and transmitted back by the camera in real time, and obtain image feature data. In addition, the model analysis device is connected to the controller installed with the driver monitoring system (Driver Monitoring System, abbreviated as DMS). At this time, the driver monitoring system has a preset intervention measure threshold, and the obtained image feature data is compared with the preset intervention measure threshold to determine whether intervention is required.
[0035] Step S222, based on the reference of the intervention measure threshold in step S212, the obtained image feature data is used as the data basis for subsequent improvement of the relevant model of the driver monitoring system.
[0036] The camera-assisted driving method based on a deep learning model provided in this embodiment includes two processes: model training and model deployment. By collecting abnormal images and normal images and marking abnormal factors, an initial model is trained according to the abnormal images and normal images with abnormal factors marked, and a training data model that prioritizes identifying abnormal factors and distinguishing their levels is output. The output training data model is then evaluated through a preset independent test data set and deployed in a model parsing device. According to the driving images collected and returned by the camera in real time, abnormal factors are prioritized and distinguished in levels to obtain image feature data. The image feature data provides a data basis for automated operations and model improvements, and can continue deep learning, continuously optimize recognition accuracy, gradually eliminate external interference during daily use, and enhance the intelligent driving experience.
[0037] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.
Claims
1. A camera-assisted driving method based on a deep learning model, characterized in that: include: There are two processes: model training and model deployment; and, Model training includes the following steps: Step S11, collecting and storing abnormal images of several driving scenes; Step S12, collecting and storing normal images of several driving scenes; Step S13, marking abnormal factors in the abnormal image and storing them; Step S14, training an initial model based on abnormal images and normal images with abnormal factors marked, setting the optimal recognition of abnormal factors and performing grade differentiation, and outputting a training data model; Step S15, using a preset independent test data set to test the training data model to evaluate whether the current training data model meets the usage requirements; Step S16, converting the training data model evaluated to meet the usage requirements into an open neural network exchange format; Model deployment includes the following steps: Step S21, deploying the training data model obtained in step S16 to the model analysis device, and according to the driving images collected and transmitted back by the camera in real time, preferentially identifying abnormal factors, and performing grade differentiation to obtain image feature data; Step S22, using the obtained image feature data as the data basis for automatic operation during the application process and the data basis for model improvement.
2. The camera-assisted driving method based on a deep learning model according to claim 1, characterized in that: The initial model is the open source Yolov8 model.
3. The camera-assisted driving method based on a deep learning model according to claim 1, characterized in that: The abnormal image is a glare image, including a rear-view image with backlight and strong light.
4. The camera-assisted driving method based on a deep learning model according to claim 3, characterized in that: The anomaly factors are the description of the object and the overexposed area.
5. The camera-assisted driving method based on a deep learning model according to claim 4, characterized in that: The grades are differentiated into glare intensity classes.
6. The camera-assisted driving method based on a deep learning model according to claim 5, characterized in that: The preset independent test data set includes glare images with backlighting, strong light, and normal images.
7. The camera-assisted driving method based on a deep learning model according to claim 1, characterized in that: The abnormal images are abnormal driving images, including images with abnormal behaviors and abnormal postures.
8. The camera-assisted driving method based on a deep learning model according to claim 7, characterized in that: The abnormal factors are abnormal behavior type and abnormal behavior human body posture.
9. The camera-assisted driving method based on a deep learning model according to claim 8, characterized in that: The level distinction is the magnitude level distinction of the abnormal behavior.
10. The camera-assisted driving method based on a deep learning model according to claim 1, characterized in that: In step S21, an intervention measure threshold is preset, and the obtained image feature data is compared with the preset intervention measure threshold to determine whether intervention is required.
Citation Information
Patent Citations
Deep-learning-based driver's fatigue detection method and apparatus
CN106446811A
Driver abnormal behavior detection method and device, storage medium and equipment
CN111860254A
Fatigue driving detection method based on deep learning and facial features
CN115393830A
Fatigue driving detection method and system and machine readable storage medium
CN117132967A
Driver fatigue driving detection and early warning method based on deep learning
CN117253220A
