Driving assistance method, device and equipment based on driver state and storage medium

By collecting and analyzing driver multimodal information, combining artificial intelligence algorithms and driving behavior knowledge base, personalized driving suggestions are provided, which solves the problems of inaccurate driver status assessment and poor intuitiveness of suggestions in the existing technology, and improves driving Accuracy and practicality of auxiliary systems.

CN118907119APending Publication Date: 2024-11-08GAC HONDA AUTOMOBILE CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411182944.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing driving assistance system relies on a single information indicator to determine the driver's status, which has limitations and false alarm rates, making it difficult to comprehensively and accurately evaluate the driver's status, and the driving suggestions provided are poor intuition and low in practicality, making it difficult to meet user needs.

Method used

By collecting multimodal information of the driver, including voice information, facial images, eye movements, head posture, hand and foot movement information, using artificial intelligence algorithms to analyze driving status, obtain fatigue, concentration, The degree of mood swings and driving skills level, search driving suggestions based on the pre-established driving behavior knowledge base, and determine the information display mode based on interactive habit information to provide personalized driving suggestions.

Benefits of technology

It improves the accuracy of driving status recognition, provides practical and personalized driving suggestions, improves users' driving experience, and enhances driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118907119A_ABST
    Figure CN118907119A_ABST
Patent Text Reader

Abstract

The invention discloses a driving assistance method and device based on a driver state, equipment and a storage medium. The method comprises the steps that multi-modal information and interaction habit information of a driver are collected; processing the multi-modal information, and analyzing the driving state of the driver through an artificial intelligence algorithm to obtain a driving state evaluation index corresponding to the driver; retrieving associated driving suggestions in a pre-established driving behavior knowledge base according to the driving state evaluation index; and determining an information display mode corresponding to the driver according to the interaction habit information, and outputting the driving suggestion to the driver based on the information display mode. According to the method, the accuracy of driving state recognition can be improved, the driving suggestions are pushed by using a personalized information display mode, the practicability is high, and the driving experience of the user can be improved. The method can be widely applied to the technical field of vehicles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a driver status-based driving assistance method, device, equipment and storage medium. Background Art

[0002] With the rapid development of automobile intelligence and networking technology, driving safety has become one of the focuses of the automotive industry. In order to effectively prevent traffic accidents and protect the lives of drivers and passengers, real-time monitoring of the driver's driving status has become particularly important.

[0003] In related technologies, most driving assistance systems rely on a single information indicator to judge the driver's state, such as monitoring the frequency of the driver's eyelid opening and closing to determine whether there is fatigue driving, and monitoring the driver's control of the steering wheel to determine whether there is distraction. This implementation method has certain limitations and false alarm rates, and it is difficult to fully and accurately assess the driver's actual state. Moreover, when assisting the driver in driving the vehicle, the driving suggestions provided often have poor intuitiveness and low practicality, which makes it difficult to meet the actual needs of users.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present application is to provide a driving assistance method, device, equipment and storage medium based on driver status.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] On the one hand, an embodiment of the present application provides a driving assistance method based on a driver state, the method comprising:

[0009] Collecting multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information;

[0010] Processing the multimodal information, analyzing the driver's driving state through an artificial intelligence algorithm, and obtaining a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level;

[0011] Retrieving relevant driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index;

[0012] An information display mode corresponding to the driver is determined according to the interaction habit information, and the driving suggestion is output to the driver based on the information display mode.

[0013] In addition, the driving assistance method based on the driver state according to the above embodiment of the present application may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present application, the collecting of the driver's multimodal information includes:

[0015] The multimodal information of the driver is collected through sensors installed in the vehicle;

[0016] Wherein, the sensor includes a camera, an infrared sensor, an accelerometer and a gyroscope.

[0017] Furthermore, in one embodiment of the present application, the multimodal information is processed, and the driving state of the driver is analyzed by an artificial intelligence algorithm to obtain a driving state evaluation index corresponding to the driver, including:

[0018] Preprocessing the multimodal information;

[0019] Extracting feature data corresponding to the preprocessed multimodal information;

[0020] According to the characteristic data, the driving state of the driver is analyzed and evaluated based on a pre-built evaluation model to obtain a driving state evaluation index corresponding to the driver.

[0021] Further, in one embodiment of the present application, the preprocessing of the multimodal information includes at least one of the following:

[0022] Performing data cleaning processing on the multimodal information;

[0023] Alternatively, performing data standardization and alignment processing on the multimodal information;

[0024] Alternatively, image enhancement processing is performed on the multimodal information.

[0025] Further, in one embodiment of the present application, determining the information display mode corresponding to the driver according to the interaction habit information, and outputting the driving advice to the driver based on the information display mode includes:

[0026] Determining, according to the interaction habit information, a first frequency of the driver using the vehicle-mounted display screen and a second frequency of the driver using the voice prompt system;

[0027] If the first frequency is greater than or equal to the second frequency, the driving advice is displayed through the vehicle-mounted display screen; or, if the first frequency is less than the second frequency, the driving advice is broadcast through the voice prompt system.

[0028] Furthermore, in one embodiment of the present application, the method further includes:

[0029] If the driver is determined to be a novice driver based on the driver's license information or driving skill level, adjusting the vehicle to enter a novice driver guidance mode;

[0030] Among them, the novice driver guidance mode includes functions of simulation training, real-time feedback, and guidance prompts.

[0031] Furthermore, in one embodiment of the present application, the method further includes:

[0032] Judging whether there are any driving safety hazards according to the driving status evaluation indicators;

[0033] If it is determined that there is a driving safety hazard, the vehicle's corresponding emergency intervention mechanism will be triggered.

[0034] On the other hand, an embodiment of the present application provides a driving assistance device based on a driver state, the device comprising:

[0035] A collection unit, used to collect multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information;

[0036] A processing unit, configured to process the multimodal information, analyze the driving state of the driver through an artificial intelligence algorithm, and obtain a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level;

[0037] A search unit, configured to retrieve associated driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index;

[0038] A display unit is used to determine the information display mode corresponding to the driver according to the interaction habit information, and output the driving suggestion to the driver based on the information display mode.

[0039] On the other hand, an embodiment of the present application provides a computer device, including:

[0040] at least one processor;

[0041] at least one memory for storing at least one program;

[0042] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned driving assistance method based on the driver status.

[0043] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned driving assistance method based on the driver status.

[0044] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0045] The embodiment of the present application discloses a driving assistance method based on driver status, which collects multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's facial image, eye movement information, head posture information, hand and foot movement information; the multimodal information is processed, and the driving status of the driver is analyzed by an artificial intelligence algorithm to obtain the corresponding driving status evaluation index of the driver; wherein the driving status evaluation index includes fatigue level, concentration, emotional fluctuation level and driving skill level; according to the driving status evaluation index, the associated driving suggestions are retrieved in a pre-established driving behavior knowledge base; according to the interaction habit information, the information display mode corresponding to the driver is determined, and the driving suggestions are output to the driver based on the information display mode. The method determines the driving status of the driver through multimodal information fusion perception, which can improve the accuracy of driving status recognition; retrieves the associated driving suggestions according to the corresponding driving status evaluation index of the driver, and uses a personalized information display mode to push the driving suggestions, which is highly practical and can improve the user's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of an implementation environment of a driving assistance method based on a driver state provided in an embodiment of the present application;

[0048] Figure 2A schematic diagram of a process flow of a driving assistance method based on a driver state provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a driving assistance device based on driver status provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, some nouns involved in this application are analyzed:

[0056] 1) Artificial intelligence is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] 2) Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0058] 3) Deep learning, a branch of machine learning, is a machine learning method based on artificial neural networks. Compared with traditional machine learning methods, deep learning learns the feature representation of data through a multi-level neural network model, thereby achieving the learning and modeling of complex patterns and relationships. Deep learning has a wide range of applications in various fields, including computer vision, natural language processing, speech recognition, recommendation systems, etc. In the field of computer vision, deep learning can be used for tasks such as image classification, object detection, and image generation; in the field of natural language processing, deep learning can be used for tasks such as text classification, machine translation, and sentiment analysis.

[0059] With the rapid development of automobile intelligence and networking technology, driving safety has become one of the focuses of the automotive industry. In order to effectively prevent traffic accidents and protect the lives of drivers and passengers, real-time monitoring of the driver's driving status has become particularly important.

[0060] In related technologies, most driving assistance systems rely on a single information indicator to judge the driver's state, such as monitoring the frequency of the driver's eyelid opening and closing to determine whether there is fatigue driving, and monitoring the driver's control of the steering wheel to determine whether there is distraction. This implementation method has certain limitations and false alarm rates, and it is difficult to fully and accurately assess the driver's actual state. Moreover, when assisting the driver in driving the vehicle, the driving suggestions provided often have poor intuitiveness and low practicality, which makes it difficult to meet the actual needs of users.

[0061] In view of this, a driving assistance method based on driver status is provided in an embodiment of the present application. The method collects multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's facial image, eye movement information, head posture information, hand and foot movement information; the multimodal information is processed, and the driving status of the driver is analyzed by an artificial intelligence algorithm to obtain the corresponding driving status evaluation index of the driver; wherein the driving status evaluation index includes fatigue level, concentration level, emotional fluctuation level and driving skill level; according to the driving status evaluation index, the associated driving suggestions are retrieved from a pre-established driving behavior knowledge base; according to the interaction habit information, the information display mode corresponding to the driver is determined, and the driving suggestions are output to the driver based on the information display mode. The method determines the driving status of the driver through multimodal information fusion perception, which can improve the accuracy of driving status recognition; retrieves the associated driving suggestions according to the corresponding driving status evaluation index of the driver, and uses a personalized information display mode to push the driving suggestions, which is highly practical and can improve the user's driving experience.

[0062] Next, the implementation environment of the driving assistance method based on the driver's state provided in the embodiment of the present application is first introduced. Figure 1 , Figure 1 A schematic diagram of an implementation environment of a driver state-based driving assistance method is provided, wherein the software and hardware components of the implementation environment mainly include a terminal device 110 and a server 120, and the terminal device 110 is in communication connection with the server 120. The driver state-based driving assistance method can be configured on the terminal device 110 side, or the driver state-based driving assistance can be implemented through the interaction between the terminal device 110 and the server 120.

[0063] Specifically, in the embodiment of the present application, the terminal device 110 may include but is not limited to any one or more of a smart watch, a smart phone, a computer, a personal digital assistant (PDA), an intelligent voice interaction device, a smart home appliance or a vehicle-mounted terminal. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal device 110 and the server 120 may establish a communication connection through a wireless network or a wired network, which uses standard communication technology and / or protocols. The network may be set to the Internet or any other network, such as but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network.

[0064] Of course, it is understandable that Figure 1 The implementation environment in the embodiment of the present application is only an optional application scenario of the driving assistance method based on the driver's state. The actual application is not fixed. Figure 1 The hardware and software environment shown below. Figure 1 The implementation environment shown is used to explain in detail the driver status-based driving assistance method provided in the embodiments of the present application.

[0065] Please refer to Figure 2 , Figure 2 is a flowchart of a driving assistance method based on driver status provided by an embodiment of the present application, referring to Figure 2 , a driving assistance method based on driver status provided in this application includes but is not limited to:

[0066] Step 210: Collecting multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, and hand and foot movement information;

[0067] Step 220: Process the multimodal information, analyze the driver's driving state through an artificial intelligence algorithm, and obtain a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level;

[0068] Step 230: Retrieve relevant driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index;

[0069] Step 240: Determine an information display mode corresponding to the driver according to the interaction habit information, and output the driving suggestion to the driver based on the information display mode.

[0070] In an embodiment of the present application, a driving assistance method based on driver status is provided, which determines the driver's driving status through multimodal information fusion perception, thereby improving the accuracy of driving status recognition; retrieves related driving suggestions based on the driver's corresponding driving status evaluation index, and uses a personalized information display mode to push driving suggestions, which is highly practical and can improve the user's driving experience.

[0071] The driving assistance method provided in the embodiment of the present application includes the collection, processing and analysis of multimodal information, and then the process of generating driving suggestions and personalized display. Specifically, when performing driving assistance, first, the driver's multimodal information and interaction habit information can be collected. Here, the multimodal information may include the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information, and the information can be obtained by sensors installed in the vehicle. These sensors may include but are not limited to cameras, infrared sensors, accelerometers and gyroscopes. Specifically, for example, the in-vehicle camera can be used to capture the driver's facial expression changes to obtain a facial image; the driver's gaze direction and blinking frequency can be obtained through infrared eye tracking technology to obtain eye movement information; the camera and posture estimation algorithm can be used to obtain the position and angle of the driver's head to obtain head posture information; the camera can be combined with a deep learning model to identify the changes in hand and foot movements to obtain hand and foot movement information.

[0072] Of course, it can be understood that the above embodiments are only used to exemplify the application of collecting multimodal information in the embodiments of the present application, and are not meant to limit the actual implementation process.

[0073] In the embodiment of the present application, in addition to multimodal information, the driver's interaction habit information can also be collected. The interaction habit information can be used to characterize which type of interaction mode the driver is accustomed to using. For example, if the driver often uses the vehicle display screen, it means that his interaction habit is more inclined to the screen display mode; if the driver often uses voice commands, it means that his interaction habit is more inclined to the voice interaction mode. Specifically, in the embodiment of the present application, the driver's interaction habit information can be obtained based on the analysis of the driver's historical operation records.

[0074] After obtaining the multimodal information, it can be processed and the driver's driving status can be analyzed by an artificial intelligence algorithm to obtain the driver's corresponding driving status evaluation index. It can be understood that in the embodiment of the present application, multimodal information refers to data collected from different angles or in different ways. By analyzing the driver's driving status through multimodal information, multiple dimensions can be integrated, and the results obtained are relatively more accurate, which is convenient for giving more practical driving suggestions later.

[0075] The application of artificial intelligence algorithms in the embodiments of the present application mainly includes deep learning models, computer vision technology, natural language processing technology, etc. These algorithms can extract useful information from the above multimodal information and further analyze the driver's state. For example, the use of deep learning models can identify facial expressions, head postures, etc. to determine whether the driver is in a state of fatigue; computer vision technology can detect the driver's eye movements (such as blinking frequency) by analyzing video frames to evaluate the concentration of attention; natural language processing technology can be used to analyze the driver's voice characteristics, such as changes in speech speed and tone, to determine the degree of emotional fluctuation. In the embodiment of the present application, according to the driver's eye closure time, head tilt angle and other characteristics, the fatigue state can be evaluated and the index data related to the degree of fatigue can be obtained; by analyzing the driver's gaze direction, blinking frequency and other behaviors, it can be determined whether the driver is fully focused on driving, and the index data related to the concentration of attention can be obtained; combined with the changes in voice characteristics and facial expressions, the emotional stability of the driver can be evaluated, and the index data related to the degree of emotional fluctuation can be obtained; by comprehensively considering the driver's operating habits (such as braking stability, steering stability, etc.) and other behavior patterns, the driver's driving skill level can be determined, thereby obtaining the corresponding driving state evaluation index of the driver.

[0076] In the embodiment of the present application, a driving behavior knowledge base can be established in advance. The driving behavior knowledge base is a database containing rich driving behavior rules and knowledge, covering safe driving suggestions, wrong operation correction measures and good driving habit cultivation methods in various driving scenarios. The driving behavior knowledge base supports dynamic updates and can continuously learn and optimize the accuracy and practicality of driving suggestions. Specifically, in the embodiment of the present application, expert suggestions, historical accident case analysis results, etc. can be collected to build a driving behavior knowledge base containing various driving scenarios and their corresponding safety suggestions. After obtaining the driving state evaluation index, the associated driving suggestions can be detected according to the driving state evaluation index. Then, the information display mode corresponding to the driver can be determined according to the interactive habit information, that is, the information display mode is determined according to the driver's habitual preferences, and the most suitable method (such as voice broadcast, screen display, etc.) is selected to provide personalized driving suggestions to the driver. For example, if the driver prefers to listen rather than watch, the driving suggestions can be conveyed by voice broadcast.

[0077] For example, the method in the embodiments of the present application can be applied in the following scenarios:

[0078] Scenario 1: Fatigue driving detection

[0079] When it is detected that the driver's blinking frequency increases abnormally, the eye closure time prolongs, and the head nods frequently, it is judged as a fatigue driving state. Driving suggestions can include immediate suggestions and long-term suggestions, such as immediate suggestions: "Please pay attention to rest. It is recommended to stop and rest for 15 minutes every two hours to avoid fatigue driving"; long-term suggestions: "It is recommended to adjust your work and rest time to ensure adequate sleep. Use refreshing drinks or listen to music to stay awake"; and, in the embodiment of the present application, emergency measures can be configured: if fatigue is continuously detected and driving behavior is abnormal (such as deviating from the lane), the system automatically triggers an alarm and the vehicle's automatic driving system takes over.

[0080] Scenario 2: Distracted Driving Detection

[0081] Through facial feature analysis, it was found that the driver's eyes were away from the road ahead for a long time, his head was facing the co-pilot or central control area, and his hands frequently operated non-driving related devices (such as mobile phones), which was judged as distracted driving.

[0082] Driving advice can include instant reminders, safety advice and educational guidance, such as instant reminders: "Please pay attention to concentrate and avoid distracted driving. Please focus your eyes and attention on the road ahead"; safety advice: "It is recommended to use voice assistants or hands-free calling functions to handle non-emergency matters to ensure driving safety"; educational guidance: "Distracted driving can easily lead to accidents. Please pay attention to driving safety and protect your own and others' lives."

[0083] Scenario 3: Emotional Fluctuation Detection

[0084] Through facial expression analysis and voice recognition technology, the system detects that the driver is emotionally agitated (such as angry, anxious), has tense facial expressions, and has a rising voice tone, and judges it as an emotional state. Driving suggestions can include emotional regulation, suspension of driving, and psychological support, such as emotional regulation: "Please stay calm, take a deep breath, and try to calm down. Keeping a calm mind while driving will help safe driving"; suspension of driving: "If your emotions are difficult to control, it is recommended to find a safe place to stop, take a short rest, and continue driving after your emotions are stable"; psychological support: "If you face long-term driving pressure or emotional fluctuations, it is recommended to seek psychological counseling or professional help."

[0085] In some embodiments, the processing of the multimodal information, analyzing the driver's driving state through an artificial intelligence algorithm, and obtaining a driving state evaluation index corresponding to the driver include:

[0086] Preprocessing the multimodal information;

[0087] Extracting feature data corresponding to the preprocessed multimodal information;

[0088] According to the characteristic data, the driving state of the driver is analyzed and evaluated based on a pre-built evaluation model to obtain a driving state evaluation index corresponding to the driver.

[0089] In an embodiment of the present application, when processing multimodal information, it can be preprocessed first, for example, data cleaning can be performed on the multimodal information, and the data cleaning process can remove noise, such as using a filter to remove Gaussian noise in the image. In some embodiments, data standardization and alignment can also be performed on the multimodal information. Data standardization can convert the data to the same scale. If data of different modes are collected at the same time, it is necessary to ensure that they are aligned on the time axis to facilitate the analysis of the correlation between them. In some embodiments, image enhancement processing can also be performed on the multimodal information, such as increasing the diversity of the data set by rotation, flipping, scaling, brightness adjustment, etc.

[0090] In the embodiment of the present application, feature data corresponding to the pre-processed multimodal information is extracted, and based on the feature data, a pre-built evaluation model can be used to analyze and evaluate the driver's driving state to obtain a driving state evaluation index corresponding to the driver. The present application does not limit the specific structure and training algorithm of the evaluation model here.

[0091] In some embodiments, determining the information display mode corresponding to the driver according to the interaction habit information, and outputting the driving advice to the driver based on the information display mode includes:

[0092] Determining, according to the interaction habit information, a first frequency of the driver using the vehicle-mounted display screen and a second frequency of the driver using the voice prompt system;

[0093] If the first frequency is greater than or equal to the second frequency, the driving advice is displayed through the vehicle-mounted display screen; or, if the first frequency is less than the second frequency, the driving advice is broadcast through the voice prompt system.

[0094] In the embodiment of the present application, when determining the information display mode corresponding to the driver, the first frequency of the driver using the vehicle display screen and the second frequency of the driver using the voice prompt system can be determined according to the interaction habit information, and then the first frequency and the second frequency can be compared. If the first frequency is greater than or equal to the second frequency, the driving advice can be displayed through the vehicle display screen. Conversely, if the first frequency is less than the second frequency, the driving advice can be broadcasted through the voice prompt system.

[0095] In some embodiments, the method further comprises:

[0096] If the driver is determined to be a novice driver based on the driver's license information or driving skill level, adjusting the vehicle to enter a novice driver guidance mode;

[0097] Among them, the novice driver guidance mode includes functions of simulation training, real-time feedback, and guidance prompts.

[0098] In the embodiments of the present application, a novice driver guidance mode can also be set. Specifically, the driver's license information or driving skill level can be used to determine whether the driver is a novice driver. If so, simulation training, real-time feedback, and guidance prompts can be used to help novice drivers quickly master driving skills, deeply understand traffic rules, and develop good driving habits. Specifically, such as basic operating instructions: "Please make sure to turn on the turn signal in advance, observe the rearview mirror and blind spots, and confirm that it is safe before changing lanes or merging." Simulation training: "It is recommended to use the on-board simulation training function to strengthen driving skills practice under complex road conditions." Safety tips: "Keep a safe distance between vehicles, avoid sudden acceleration and braking, and control the speed steadily." This application does not impose any restrictions on this.

[0099] In some embodiments, the method further comprises:

[0100] Judging whether there are any driving safety hazards according to the driving status evaluation indicators;

[0101] If it is determined that there is a driving safety hazard, the vehicle's corresponding emergency intervention mechanism will be triggered.

[0102] In the embodiment of the present application, it is also possible to determine whether there are driving safety hazards based on the driving status evaluation index. If a driving safety hazard is found (such as excessive driver fatigue, serious distraction, or an imminent collision, etc.), the system can automatically trigger an emergency intervention mechanism, such as issuing a strong alarm, or the vehicle's own intelligent driving system temporarily taking over, etc., to ensure driving safety.

[0103] The following describes a driving assistance device based on driver status according to an embodiment of the present application with reference to the accompanying drawings.

[0104] Reference Figure 3 , the driving assistance device based on the driver's state proposed in the embodiment of the present application includes:

[0105] The collecting unit 310 is used to collect the multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information;

[0106] The processing unit 320 is used to process the multimodal information, analyze the driving state of the driver through an artificial intelligence algorithm, and obtain a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level;

[0107] A search unit 330, configured to retrieve associated driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index;

[0108] The display unit 340 is used to determine the information display mode corresponding to the driver according to the interaction habit information, and output the driving suggestion to the driver based on the information display mode.

[0109] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] Reference Figure 4 , an embodiment of the present application provides a computer device, including:

[0111] at least one processor 410;

[0112] At least one memory 420, used to store at least one program;

[0113] When at least one program is executed by at least one processor 410, at least one processor 410 implements Figure 2A driving assistance method based on driver status is shown.

[0114] Similarly, the contents of the above method embodiments are all applicable to the present computer device embodiment. The functions specifically implemented by the present computer device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] The embodiment of the present application also provides a computer-readable storage medium, in which a program executable by the processor 410 is stored. When the program executable by the processor 410 is executed by the processor 410, it is used to execute the above-mentioned driving assistance method based on the driver status.

[0116] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 2 An embodiment of a driving assistance method based on driver status is shown.

[0117] It is understandable that if Figure 2 The contents of the embodiment of the driving assistance method based on the driver's state shown in the figure are applicable to the embodiment of the computer-readable storage medium. The functions specifically implemented by the embodiment of the computer-readable storage medium are similar to those in the embodiment of the driving assistance method based on the driver's state shown in the figure. Figure 2 The embodiment of the driving assistance method based on the driver state shown in FIG. 1 is the same as that shown in FIG. 1 , and the beneficial effects achieved are the same as those of FIG. Figure 2 The beneficial effects achieved by the embodiment of the driving assistance method based on the driver's state shown are also the same.

[0118] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0119] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical system and / or software module, or one or more functions and / or features can be implemented in a separate physical system or software module. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the system disclosed herein, the actual implementation of the module will be understood within the conventional techniques of engineers. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0120] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, system or apparatus and execute the instructions), or in conjunction with such instruction execution system, system or apparatus. For the purpose of this specification, "computer-readable medium" can be any system that can contain, store, communicate, propagate or transport a program for use by an instruction execution system, system or apparatus, or in conjunction with such instruction execution system, system or apparatus.

[0122] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic systems), a portable computer disk case (magnetic systems), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic system, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0123] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0125] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0126] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "another embodiment" or "certain embodiments" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0128] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A driving assistance method based on driver status, characterized in that: The method comprises: Collecting multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information; Processing the multimodal information, analyzing the driver's driving state through an artificial intelligence algorithm, and obtaining a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level; Retrieving relevant driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index; An information display mode corresponding to the driver is determined according to the interaction habit information, and the driving suggestion is output to the driver based on the information display mode.

2. The driving assistance method based on driver status according to claim 1, characterized in that: The collecting of the driver's multimodal information includes: The multimodal information of the driver is collected through sensors installed in the vehicle; Wherein, the sensor includes a camera, an infrared sensor, an accelerometer and a gyroscope.

3. The driving assistance method based on driver status according to claim 1, characterized in that: The processing of the multimodal information, analyzing the driving state of the driver by an artificial intelligence algorithm, and obtaining a driving state evaluation index corresponding to the driver include: Preprocessing the multimodal information; Extracting feature data corresponding to the preprocessed multimodal information; According to the characteristic data, the driving state of the driver is analyzed and evaluated based on a pre-built evaluation model to obtain a driving state evaluation index corresponding to the driver.

4. The driving assistance method based on driver status according to claim 3, characterized in that: The preprocessing of the multimodal information includes at least one of the following: Performing data cleaning processing on the multimodal information; Alternatively, performing data standardization and alignment processing on the multimodal information; Alternatively, image enhancement processing is performed on the multimodal information.

5. The driving assistance method based on driver status according to claim 1, characterized in that: The determining, according to the interaction habit information, an information display mode corresponding to the driver, and outputting the driving suggestion to the driver based on the information display mode, comprises: Determining, according to the interaction habit information, a first frequency of the driver using the vehicle-mounted display screen and a second frequency of the driver using the voice prompt system; If the first frequency is greater than or equal to the second frequency, the driving advice is displayed through the vehicle-mounted display screen; or, if the first frequency is less than the second frequency, the driving advice is broadcast through the voice prompt system.

6. A driving assistance method based on driver status according to any one of claims 1 to 5, characterized in that: The method further comprises: If the driver is determined to be a novice driver based on the driver's license information or driving skill level, adjusting the vehicle to enter a novice driver guidance mode; Among them, the novice driver guidance mode includes functions of simulation training, real-time feedback, and guidance prompts.

7. The driving assistance method based on driver status according to claim 1, characterized in that: The method further comprises: Judging whether there are any driving safety hazards according to the driving status evaluation indicators; If it is determined that there is a driving safety hazard, the vehicle's corresponding emergency intervention mechanism will be triggered.

8. A driving assistance device based on driver status, characterized in that: The device comprises: A collection unit, used to collect multimodal information and interaction habit information of the driver; wherein the multimodal information includes the driver's voice information, facial image, eye movement information, head posture information, hand and foot movement information; A processing unit, configured to process the multimodal information, analyze the driving state of the driver through an artificial intelligence algorithm, and obtain a driving state evaluation index corresponding to the driver; wherein the driving state evaluation index includes fatigue level, concentration level, emotional fluctuation level, and driving skill level; A search unit, configured to retrieve associated driving suggestions from a pre-established driving behavior knowledge base according to the driving state evaluation index; A display unit is used to determine the information display mode corresponding to the driver according to the interaction habit information, and output the driving suggestion to the driver based on the information display mode.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a driver status-based driving assistance method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement a driver status-based driving assistance method as described in any one of claims 1-7 when executed by the processor.

Citation Information

Cited By

  • Driving behavior evaluation method, device, equipment, medium and product

    CN120828817A

  • AI voice robot for commercial vehicle and control system thereof

    CN120985641A