Man-machine interaction system, method and device based on auxiliary driving, vehicle and storage medium

Abstract

The invention discloses a man-machine interaction system, method and device based on auxiliary driving, a vehicle and a storage medium. The system comprises a data acquisition assembly, a vehicle-mounted computer and a function module, wherein the vehicle-mounted computer is respectively connected with the data acquisition assembly and the function module; the data acquisition assembly is used for acquiring driver data, environmental perception data around a vehicle and vehicle driving data, and transmitting the driver data, the environmental perception data around the vehicle and the vehicle driving data to the vehicle-mounted computer; the vehicle-mounted computer is used for analyzing the driver data, the vehicle surrounding environment sensing data and the vehicle driving data to obtain an operation instruction and sending the operation instruction to the corresponding function module; and the function module is used for executing corresponding operation based on the operation instruction so as to assist a driver in driving. According to the system, the operation instruction capable of assisting the driver is generated through the collected data, and the experience of the driver can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent transportation technology, and in particular to a human-computer interaction system, method, device, vehicle and storage medium based on assisted driving. Background Technology

[0002] With the rapid development of the logistics industry, the transportation efficiency and safety of heavy-duty trucks have become crucial. The application of Level 2 (L2) intelligent driver assistance technology in the heavy-duty truck sector is gradually increasing. However, existing L2 intelligent driver assistance human-machine interactions for heavy-duty trucks are relatively simple, lacking real-time monitoring and proactive response to driver status. Most interactions rely solely on simple voice prompts or a few indicator lights on the dashboard, resulting in a lack of interactive modalities. When faced with complex driving scenarios, existing interaction methods cannot provide drivers with efficient and intuitive information guidance or operational feedback based on a comprehensive assessment of the environment and driver status.

[0003] Therefore, how to provide drivers with a system that can adapt to complex driving scenarios and has rich human-computer interaction methods is an urgent problem to be solved in the existing technology. Summary of the Invention

[0004] This invention provides a human-computer interaction system, method, device, vehicle, and storage medium based on assisted driving, to solve the problems that the interaction methods in the prior art cannot support complex driving scenarios and that human-computer interaction methods are scarce.

[0005] According to one aspect of the present invention, a human-machine interaction system based on assisted driving is provided, the system comprising: a data acquisition component, an on-board computer, and a functional module, wherein the on-board computer is connected to the data acquisition component and the functional module respectively; The data acquisition component is used to collect driver data, environmental perception data around the vehicle, and vehicle driving data, and transmit the driver data, environmental perception data around the vehicle, and vehicle driving data to the on-board computer. The on-board computer is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions, and send the operation instructions to the corresponding functional modules; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The functional module is used to perform corresponding operations based on the operation instructions to assist the driver in driving.

[0006] According to another aspect of the present invention, a human-computer interaction method based on assisted driving is provided, applied to an in-vehicle computer in a human-computer interaction system based on assisted driving as described in any embodiment of the present invention, the method comprising: Acquire driver data, environmental perception data around the vehicle, and vehicle driving data; The driver data, the environmental perception data around the vehicle, and the vehicle driving data are analyzed to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The operation command is sent to the corresponding functional module so that the functional module performs the corresponding operation based on the operation command to assist the driver in driving.

[0007] According to another aspect of the present invention, a human-machine interaction device based on assisted driving is provided, characterized in that it can execute the human-machine interaction method based on assisted driving described in any embodiment of the present invention, the device comprising: The acquisition module is used to acquire driver data, environmental perception data around the vehicle, and vehicle driving data. The analysis module is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The sending module is used to send the operation command to the corresponding functional module, so that the functional module can perform the corresponding operation based on the operation command to assist the driver in driving.

[0008] According to another aspect of the present invention, a vehicle is provided, the vehicle comprising: at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the human-computer interaction method based on assisted driving as described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the human-computer interaction method based on assisted driving as described in any embodiment of the present invention.

[0010] This invention discloses a human-computer interaction system, method, device, vehicle, and storage medium based on assisted driving. The system includes a data acquisition component, an on-board computer, and functional modules. The on-board computer is connected to both the data acquisition component and the functional modules. The data acquisition component collects driver data, environmental perception data around the vehicle, and vehicle driving data, and transmits these data to the on-board computer. The on-board computer analyzes the driver data, environmental perception data, and vehicle driving data to obtain operation instructions, and sends these instructions to the corresponding functional modules. The operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The functional modules execute corresponding operations based on the operation instructions to assist the driver. This system generates driver-assistant operation instructions from collected data, improving the driver's experience and solving the problems of existing interaction methods being unable to support complex driving scenarios and the lack of sufficient human-computer interaction methods.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a human-computer interaction system based on assisted driving provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the installation of a depth camera according to an embodiment of the present invention; Figure 3 A schematic diagram of a touch-sensing area provided in an embodiment of the present invention; Figure 4 This invention provides a schematic diagram of an instrument displaying a normal state. Figure 5 A schematic diagram of an instrument displaying a dangerous state is provided as an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the fusion of projection information and actual road scene provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a sensor layout provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of a system request takeover early warning process provided in an embodiment of the present invention; Figure 9 This is a schematic diagram illustrating a driver's intention to take over control, provided as an embodiment of the present invention. Figure 10 This is a comparative schematic diagram of adaptive road condition information display provided by an embodiment of the present invention; Figure 11 This is a flowchart illustrating a human-computer interaction method based on assisted driving provided in Embodiment 2 of the present invention; Figure 12 This is a schematic diagram of a human-computer interaction device based on assisted driving provided in Embodiment 3 of the present invention; Figure 13 This is a schematic diagram of the vehicle structure based on the human-computer interaction method for assisted driving according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0015] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having," etc., are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] The existing L2-level intelligent driver assistance human-machine interaction for heavy-duty trucks has the following shortcomings: The human-computer interaction methods are relatively simple. For example, when the intelligent driver assistance system detects that the vehicle in front is slowing down, it only provides a voice prompt such as "Vehicle in front is slowing down, please pay attention." The driver cannot intuitively understand the specific deceleration range, distance, and other key information, nor can they obtain information through other more intuitive methods. This causes the driver to spend more time and energy to understand and judge the situation, increasing the driving burden.

[0025] There is a lack of effective perception and response to driver conditions. Existing systems generally do not monitor driver fatigue levels or concentration in real time. However, heavy truck drivers are prone to fatigue after long hours of driving. When drivers are fatigued or inattentive, existing human-machine interaction systems cannot make timely adjustments, such as increasing the intensity of reminders or adjusting the intervention level of intelligent driver assistance systems, which can easily lead to safety accidents.

[0026] There are shortcomings in the handover of control. When the intelligent driver assistance system encounters a situation it cannot handle and requires driver intervention, it simply sounds an alarm without providing advance warning or guidance. For example, in complex situations such as sudden road construction, the system directly requests driver intervention, which may not be adequately prepared, leading to prolonged intervention time and increased accident risk. Furthermore, the system cannot quickly and accurately recognize and respond to situations where the driver actively seeks to take control, potentially preventing the driver from smoothly taking over when timely control is needed.

[0027] The layout and design of the information presentation are not reasonable enough. Some heavy-duty truck intelligent driver assistance systems scatter important driving information and driver assistance status information across different screens or locations. Drivers need to frequently shift their gaze to obtain this information while driving, which not only increases the time their eyes are off the road but also easily leads to untimely or incomplete information acquisition, affecting driving safety and efficiency.

[0028] Existing human-computer interaction methods are poorly adaptable to different driving scenarios. Regardless of road conditions (highways, city roads, rural roads) or weather conditions (sunny, rainy, foggy), the methods and content of human-computer interaction remain largely unchanged. For example, in foggy conditions with low visibility, the system does not display or provide additional information about the surrounding environment, preventing drivers from promptly understanding nearby vehicles and road conditions, thus increasing driving risks.

[0029] For example, Chinese invention patent CN118107490A discloses a heavy-duty truck with L2-level assisted driving functionality. This is achieved by adding and rearranging sensing devices such as cameras, lidar, and millimeter-wave radar. The improved heavy-duty truck includes radar and camera components that are electrically connected to the intelligent driving controller. The radar components include forward-facing lidar and forward-facing center millimeter-wave radar; the camera components include a forward-facing center binocular camera and a forward-facing left camera. This approach allows for the expansion and support of multiple higher-level assisted driving functions while meeting regulatory requirements related to assisted driving, thus improving the driving comfort and safety of the heavy-duty truck. However, this patent primarily focuses on the layout of the sensing hardware to enhance assisted driving functions, with less attention paid to human-machine interaction. It lacks in-depth design regarding how the L2-level intelligent assisted driving system can better interact with the driver, such as information presentation methods and driver intent recognition feedback. For example, it does not explain how to more effectively transmit the information acquired by these sensing devices to the driver, or how the driver can interact with the system more conveniently.

[0030] Chinese invention patent CN120080874A discloses a human-machine co-driving control method, device, and vehicle. This invention predicts the user's takeover intention based on steering wheel control parameters, obtains corresponding intelligent driving control coefficients, and uses these coefficients to generate target control parameters for the final control of the vehicle's steering system. This ensures that the control level of the intelligent driving steering request parameters output by the intelligent driving system matches the user's takeover intention, solving the problem of the intelligent driving system and driver competing for the steering wheel. It also considers safety, comfort, and compliance, and, through upper and lower computer collaboration, achieves intelligent torque distribution in human-machine co-driving, improving the flexibility of intelligent driving. However, this invention primarily focuses on solving the problem of steering wheel control and is not comprehensive enough for human-machine interaction in L2-level intelligent assisted driving for heavy trucks. For example, it lacks research on other interaction methods besides steering wheel control, such as voice interaction and visual interaction. Furthermore, it does not adequately consider the special characteristics of heavy truck driving scenarios. Heavy truck drivers may need to deal with a wider variety of information and instructions, and this patent does not address these specific needs or consider how to better facilitate human-machine interaction under different road conditions and environments.

[0031] In conclusion, to build richer and more effective human-machine interaction methods, it is necessary to go beyond steering wheel control and include multimodal interactions such as voice and vision to meet the requirements of efficient information transmission between the driver and the system in L2-level intelligent driver assistance systems for heavy-duty trucks. Furthermore, it is essential to improve the human-machine interaction system's adaptability to different road conditions, environments, and driver states, ensuring stable and reliable human-machine interaction under various complex situations, thereby enhancing the safety and comfort of heavy-duty truck driving.

[0032] To address the aforementioned issues, this invention provides a human-machine interaction system based on assisted driving, primarily applied to intelligent assisted driving systems for heavy-duty trucks, to improve the interaction efficiency and safety between the driver and the vehicle's intelligent assisted driving system.

[0033] First, in addition to existing voice prompts, gesture and touch interactions have been added. For example, without taking their hands off the steering wheel, drivers can use simple gestures, such as clenching a fist or waving their hand, to quickly operate certain functions of the intelligent driver assistance system (such as changing cruise speed or turning specific assistance functions on or off). Regarding touch interaction, a touch-sensitive area can be set on the steering wheel, allowing drivers to adjust the displayed information category or zoom in and out of the map display by swiping. This enables drivers to interact with the system more conveniently without excessive distraction, improving information transmission efficiency and ensuring driving safety.

[0034] Secondly, the information presentation method has been optimized by adopting intelligent layered display technology, which dynamically adjusts the display priority of information based on the driving scenario and the importance of the information. For example, under normal driving conditions, key safety information such as vehicle speed and distance to the vehicle in front is mainly displayed; when a potential hazard is detected, such as a vehicle braking suddenly in front or an obstacle suddenly appearing, the system immediately highlights the hazard warning information in a bright color and larger font, accompanied by strong voice prompts and vibration feedback, ensuring that the driver can obtain important information immediately. At the same time, augmented reality technology is used to project environmental information around the vehicle and prompts from the driver assistance system directly onto the windshield, allowing the driver to obtain comprehensive information without taking their eyes off the road, greatly reducing the time spent looking away from the road and improving driving safety.

[0035] Secondly, the system enhances its perception and response to driver status. In-vehicle cameras and sensors monitor the driver's facial expressions, eye movements, and posture in real time to determine fatigue levels and concentration. When driver fatigue is detected, the system automatically adjusts its interaction methods, such as increasing the frequency and volume of voice prompts and displaying refreshing suggestions on the screen, such as reminding the driver to take a short break or playing upbeat music. If the driver is inattentive, the system will issue targeted reminders and appropriately increase the intervention level of the intelligent driver assistance system to ensure vehicle safety.

[0036] In addition, the control handover mechanism has been optimized. Before the intelligent driver assistance system requires the driver to take over the vehicle, it issues advance warnings to the driver through various means. For example, a gentle voice prompt informs the driver of impending complex road conditions, indicating that the system may be unable to continue providing support. Then, a countdown is displayed on the screen, gradually increasing the intensity of the warning, including flashing lights and vibration alerts, giving the driver sufficient time to prepare for takeover. When the driver actively seeks to take over the vehicle, the system can quickly and accurately recognize the driver's intention, such as through changes in steering wheel pressure or specific operational actions, and immediately and smoothly transfer vehicle control to the driver, ensuring a smooth driving experience.

[0037] Finally, the adaptability of the human-computer interaction system to different scenarios is improved. Based on road condition information, such as highways, urban roads, and mountain roads, the interaction method and information presentation content are automatically adjusted. On highways, information related to cruise control and maintaining a safe following distance is highlighted; on urban roads, information such as traffic light recognition and pedestrian warnings is emphasized. For different weather conditions, such as rain, fog, and snow, the display contrast and prompting methods are adjusted. In foggy weather, the size of the display markings for surrounding vehicles and obstacles is increased, and the frequency of audio prompts is increased, allowing drivers to clearly obtain information even in adverse weather conditions, thus improving the driving experience.

[0038] The improvements of this invention can solve the problems of existing L2-level intelligent driver assistance human-machine interaction methods for heavy trucks, such as single interaction methods, unreasonable information presentation, insufficient perception of driver status, poor handover of control, and poor scenario adaptability, ultimately achieving a more efficient, safe, and comfortable driving experience. The specific implementation is as follows: Example 1 Figure 1 This is a schematic diagram of a human-computer interaction system based on assisted driving provided in Embodiment 1 of the present invention. The method can be applied to situations where a driver is assisted in driving a vehicle.

[0039] like Figure 1 As shown, the first embodiment of the present invention provides a human-computer interaction system based on assisted driving, including: a data acquisition component 100, an on-board computer 200 and a functional module 300, wherein the on-board computer 200 is connected to the data acquisition component 100 and the functional module 300 respectively. The data acquisition component 100 is used to collect driver data, environmental perception data around the vehicle and vehicle driving data, and transmit the driver data, environmental perception data around the vehicle and vehicle driving data to the on-board computer 200. The vehicle-mounted computer 200 is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions, and send the operation instructions to the corresponding functional module 300; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. Functional module 300 is used to perform corresponding operations based on the operation instructions to assist the driver in driving.

[0040] The vehicle driving data may include parameters such as the vehicle's current motion state, location information, and assisted driving status. Function switching commands can be control commands that trigger the activation, deactivation, or mode switching of specific functions in the intelligent assisted driving system. Information display interaction commands can be control commands that adjust the layout, category, or scaling of the content displayed on the human-machine interface. Layered display commands can be control commands that adjust the priority and visual weight of instrument display information. Augmented reality display commands can be control commands that project virtual navigation guidance and assisted driving information onto a specific location on the windshield. Fatigue driving interaction commands can be control commands that warn or intervene with the driver. Control handover commands can be control commands that issue a takeover request to the driver and coordinate a smooth transfer of vehicle control. Display information adjustment commands can be control commands that adjust the information display strategy. Functional module 300 can be a module that performs specific operations; for example, functional module 300 can be an instrument controller, ambient light, warning light, steering wheel vibration motor, voice broadcast unit, augmented reality display device, etc., which are not limited in this embodiment.

[0041] In this embodiment, the data acquisition component 100 can be composed of different modules and can collect various types of data, such as driver data, environmental perception data around the vehicle, and vehicle driving data. The data acquisition component 100 can be connected to the on-board computer 200 to transmit the collected driver data, environmental perception data, and vehicle driving data to the on-board computer 200. The on-board computer 200 can analyze the driver data, environmental perception data, and vehicle driving data to obtain operation instructions, and send the operation instructions to the corresponding functional module 300. After receiving the corresponding operation instruction, the functional module 300 can execute the corresponding operation according to the operation instruction to assist the driver in driving.

[0042] It is understood that the driver-related data acquired by the data acquisition component 100 in this embodiment of the invention is collected only after the driver's authorization.

[0043] In one embodiment, the driver data includes gesture images, touch data, facial images, takeover intention signals, and driver vital sign data; the environmental perception data includes traffic scene data and natural environment data; the data acquisition component 100 includes at least a depth camera, a touch sensing area, a vehicle operating status sensor, a traffic environment sensor, a takeover intention acquisition module, a vital sign sensor, a facial recognition camera, and a natural environment sensor. The depth camera is used to capture images of the driver's hand gestures; The touch sensing area is used to collect touch data from the driver's hand; The vehicle operating status sensor is used to collect vehicle driving data; The traffic environment sensor is used to collect traffic scene data around the vehicle; The takeover intention acquisition module is used to acquire the driver's takeover intention signal; The vital signs sensor is used to collect the driver's vital signs data; The facial recognition camera is used to capture facial images of the driver; The natural environment sensor is used to collect data on the natural environment around the vehicle.

[0044] Among these, gesture images can be image data containing information on the spatial position and movement trajectory of the driver's hands. Touch data can be electrical signal data of the driver's fingers. Takeover intention signals can be physical trigger signals generated when the driver takes over the vehicle. Driver vital sign data can include data such as the driver's physiological characteristics and body posture. Traffic scene data can include data such as road structure information, traffic facility information, and traffic participant information. Natural environment data can refer to data such as weather conditions, lighting, and visibility.

[0045] In this embodiment, the data acquisition component 100 may include devices such as a depth camera, a touch-sensitive area, a vehicle operating status sensor, a traffic environment sensor, a takeover intent acquisition module, a vital sign sensor, a facial recognition camera, and a natural environment sensor. Through these different devices, information about the vehicle's interior and exterior can be collected from multiple perspectives.

[0046] For example, Figure 2 This is a schematic diagram of the installation of a depth camera provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the depth camera can be mounted above the rearview mirror inside the vehicle. It features high resolution and a high frame rate. The depth camera's field of view is precisely calibrated to fully cover the range of motion of the driver's hands during normal driving. To ensure stable operation under various lighting conditions, the depth camera can be equipped with an infrared illumination device, employing infrared structured light technology to accurately capture the three-dimensional spatial information of the hands even in low-light environments.

[0047] Figure 3 A schematic diagram of a touch-sensing area provided in an embodiment of the present invention, such as... Figure 3 As shown, the touch-sensitive area can be designed on both sides of the steering wheel, and can employ capacitive touch technology with high sensitivity and fast response. The touch-sensitive area can include multiple functional sub-areas, each corresponding to a specific interactive function. For example, the left side of the touch-sensitive area can include an up-and-down swiping area, a left-and-right swiping area, and a tapping area. The up-and-down swiping area corresponds to switching displayed information, the left-and-right swiping area corresponds to adjusting volume, and the tapping area corresponds to confirmation. The right side of the touch-sensitive area can include a map zooming area, a navigation destination selection area, etc. The touch-sensitive area can adopt a multi-layer structure design. The outermost layer is made of a wear-resistant and non-slip special material, providing a good tactile feel while preventing the driver's hand from slipping during operation. The middle layer is a capacitive sensing layer, composed of multiple capacitive sensor arrays, capable of accurately sensing the touch position and touch pressure. The bottom layer is a signal processing circuit, responsible for converting capacitance changes into electrical signals and performing preliminary signal amplification and filtering. To ensure the accuracy of touch operation, the surface of the touch-sensing area can be specially textured, such as with tiny bumps or depressions. Different functional areas correspond to different texture patterns, thereby providing tactile feedback so that the driver can more accurately locate the touch position by feel when operating.

[0048] This invention provides a human-machine interaction system based on assisted driving, comprising: a data acquisition component, an on-board computer, and functional modules. The on-board computer is connected to both the data acquisition component and the functional modules. The data acquisition component collects driver data, environmental perception data around the vehicle, and vehicle driving data, and transmits these data to the on-board computer. The on-board computer analyzes the driver data, environmental perception data, and vehicle driving data to obtain operation instructions, and sends these instructions to the corresponding functional modules. The operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The functional modules execute corresponding operations based on the operation instructions to assist the driver. This system generates operation instructions that assist the driver by collecting data, improving the driver's experience and solving the problems of existing interaction methods being unable to support complex driving scenarios and the lack of sufficient human-machine interaction methods.

[0049] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0050] In one embodiment, the takeover intent acquisition module includes a pressure sensor and a microswitch integrated on the steering wheel; The vital signs sensors include a heart rate sensor, a respiratory rate sensor, and multiple pressure sensors mounted on the driver's seat; The natural environment sensors include at least a rain sensor, a humidity sensor, a visibility sensor, and a temperature sensor.

[0051] In this embodiment, the takeover intent acquisition module may include a pressure sensor and a micro switch integrated on the steering wheel, the vital signs sensor may include a heart rate sensor, a respiratory rate sensor, and multiple pressure sensors installed on the driver's seat, and the natural environment sensor may include at least a rain sensor, a humidity sensor, a visibility sensor, and a temperature sensor.

[0052] This embodiment can collect massive amounts of data in real time using multiple sensors. For example, vehicle driving data can be collected by vehicle operation status sensors, including vehicle speed sensors, engine speed sensors, and acceleration sensors. The vehicle speed sensor acquires vehicle speed information, the engine speed sensor collects engine speed data, and the acceleration sensor monitors vehicle acceleration. Traffic scene data can include distance, angle, and speed of vehicles ahead, road sign information, etc., which can be collected jointly by traffic environment sensors such as millimeter-wave radar, lidar, and cameras. Millimeter-wave radar can monitor the distance, speed, and angle information of vehicles ahead in real time, lidar can collect three-dimensional point cloud data of the surrounding environment, and cameras can identify road signs, traffic lights, and surrounding vehicles, pedestrians, and other targets. Driver vital signs data can include heart rate, respiratory rate, etc., which can be collected by in-vehicle heart rate sensors and seat pressure sensors. All sensors can be connected to the onboard computer via a high-speed data bus to ensure that the data can be transmitted to the computer for processing in real time and accurately. The onboard computer can then use big data analytics to perform in-depth mining and analysis of this data. For example, various types of data can be cleaned and preprocessed to remove noise data and outliers, ensuring data accuracy and consistency. Then, based on machine learning algorithms, statistical analysis is performed on historical data to determine the importance weight of various types of information in different driving scenarios.

[0053] In this embodiment, the facial recognition camera can be a high-resolution camera with infrared illumination, capable of clearly capturing the driver's facial features at night or in low-light environments. The heart rate sensor can be installed on the steering wheel or seat; for example, a photoelectric heart rate sensor integrated into the steering wheel grip can collect the driver's heart rate data in real time by detecting changes in the volume of blood vessels under the driver's skin. The respiratory rate sensor can be integrated into the seat or the vehicle's air circulation system; for example, a piezoelectric thin-film sensor can be used to obtain respiratory rate information by sensing the minute vibrations of the driver's body during breathing. These physiological indicators, corroborated with the facial recognition data, can provide multi-dimensional evidence for more accurate assessment of the driver's condition. For example, when the heart rate is abnormally high, breathing is rapid, and the facial expression shows tension, the system can comprehensively determine that the driver may be in a highly stressed or dangerous state. Multiple pressure sensors can be installed on the seat to form a pressure sensor array, sensing changes in the pressure distribution of the driver's body on the seat, thereby monitoring the driver's body posture.

[0054] The takeover intent acquisition module may include high-precision pressure sensors and microswitches integrated into the steering wheel. The pressure sensors can be evenly distributed across the steering wheel grip area, enabling real-time monitoring of the magnitude, distribution, and changes in pressure applied to the steering wheel by the driver's hands. The microswitches can be installed at specific locations on the steering wheel for easy driver operation, used to detect whether the driver has pressed a specific takeover button. These hardware devices can be connected to the onboard computer via dedicated signal lines, ensuring that the acquired signals are transmitted quickly and accurately to the computer for processing.

[0055] The vehicle in this embodiment is equipped with various environmental sensors, including rain sensors, humidity sensors, visibility sensors, and temperature sensors, to monitor the weather conditions around the vehicle in real time. Rain sensors accurately measure rainfall by detecting the frequency and intensity of raindrop impacts on the sensor surface. For example, piezoelectric rain sensors can quickly and accurately convert raindrop impacts into electrical signals, which are then processed by a signal processing circuit to calculate the rainfall. Humidity sensors measure the water vapor content in the air. Capacitive humidity sensors can be used, whose capacitance changes with air humidity; humidity information is obtained by detecting these changes. Visibility sensors utilize optical principles to determine visibility range by detecting the degree of light scattering and attenuation in the air. For example, forward-scattering visibility sensors can measure visibility within a certain distance ahead in real time. Temperature sensors monitor ambient temperature in real time. Thermistor temperature sensors can be used, whose resistance changes with temperature; temperature data is obtained by measuring the resistance value. These sensors transmit the collected weather data to the onboard computer in real time via a data bus.

[0056] In one embodiment, the on-board computer 200 is specifically used for: The gesture images in the driver data are analyzed by a pre-trained gesture recognition model to obtain the driver's gesture intention, and a function switching instruction is generated based on the gesture intention. The touch data in the driver data is analyzed by a touch recognition algorithm to obtain the driver's touch actions, and information display interaction instructions are generated based on the functions corresponding to the touch actions; The current driving state of the vehicle is determined based on the environmental perception data and the vehicle driving data, and a hierarchical display instruction is generated based on the information display priority relationship corresponding to the current driving state. The vehicle driving data and route instruction information are fused to obtain fused projection information, and augmented reality display instructions are generated based on the projection information. The driver's facial images in the driver data are identified using a pre-trained deep learning model to obtain the driver's fatigue level and attention state. Based on the fatigue level and attention state, and combined with the driver's vital signs data in the driver data, fatigue driving interaction instructions are generated.

[0057] In this embodiment, the on-board computer 200 can analyze driver data, environmental perception data around the vehicle, and vehicle driving data in different ways to generate different operation commands. The on-board computer 200 can parse the gesture images in the driver data using a pre-trained gesture recognition model to obtain the driver's gesture intentions and generate function switching commands based on these intentions (e.g., a fist clench corresponds to a confirmation operation, and a hand wave corresponds to a cancellation operation). For example, after a depth camera captures a gesture image, it transmits the data to the on-board computer 200. The on-board computer 200 can analyze the image in real time using a pre-trained gesture recognition model to parse the driver's gesture intentions. Subsequently, the commands are transmitted to various functional modules of the intelligent driver assistance system via the vehicle's controller local area network bus to realize the corresponding functions. For example, when changing the cruise speed, the system can precisely adjust the parameters of the cruise control module and change the vehicle's driving speed based on the recognized gesture commands and preset speed adjustment strategies.

[0058] The pre-trained gesture recognition model utilizes deep learning algorithms, trained on a large dataset of gesture samples from real-world driving scenarios. This data covers various common gestures, such as clenching a fist, waving, and finger swiping. Each gesture is sampled from different drivers under varying lighting, angles, and speeds to enhance the model's generalization ability. The training process involves preprocessing the collected gesture samples, including data cleaning and normalization, to ensure data consistency and usability. Then, a convolutional neural network architecture is used to extract features from the pre-processed gesture images, gradually abstracting key gesture features through multiple convolutional and pooling layers. Next, a recurrent neural network (such as a long short-term memory network) is used to process the dynamic changes in gestures over time, accurately recognizing various complex gestures performed by the driver. Transfer learning techniques can also be employed, using model parameters pre-trained on large-scale public image datasets to initialize some layers of the model, accelerating the training process and improving recognition accuracy. During training, the cross-entropy loss function is used to measure the difference between the model's predictions and the true labels, and the model parameters are continuously adjusted using stochastic gradient descent to minimize the loss function. Through multiple rounds of training and validation, the accuracy of the model in gesture recognition can be improved. Taking the fist-clenching gesture as an example, a well-trained gesture recognition model can not only recognize the closed state of the hand, but also consider the continuity and speed of the movement to avoid misjudgment.

[0059] This embodiment uses a touch recognition algorithm to analyze touch data in driver data to obtain the driver's touch actions, and generates information display interaction commands based on the corresponding touch actions. Specifically, when the driver touches the touch sensing area, the electrical signal generated by the change in capacitance is quickly collected and transmitted to the onboard computer. The computer uses a built-in touch recognition algorithm to analyze the driver's operation intention based on information such as the touch position, sliding direction, and force. For example, sliding upwards on the left touch area is recognized by the system as switching to the next type of display information; pinching with two fingers on the right touch area is determined by the system as reducing the map display scale. The computer can send the analyzed commands to the corresponding display control module or navigation module to adjust the information display. Taking switching display information as an example, after receiving the command, the display control module can display the next type of information (such as switching from vehicle speed information to fuel consumption information) at a designated position on the instrument panel or central control screen according to the preset information display order, realizing the touch interaction function operation.

[0060] The addition of gesture and touch interaction allows drivers to operate the intelligent driver assistance system without taking their hands off the steering wheel or looking away from the road for too long. In emergencies, drivers can quickly turn on the hazard warning lights with gestures, avoiding distraction from the road while searching for buttons on the center console, thus greatly reducing potential dangers.

[0061] This embodiment can determine the vehicle's current driving state based on environmental perception data and vehicle driving data, and generate hierarchical display instructions based on the information display priority relationship corresponding to the current driving state. The current driving state can include normal driving state and dangerous state. The information display priority relationship is different in different scenarios. In normal highway driving scenarios, information such as vehicle speed, engine speed, and distance to the vehicle in front has higher weight; while in urban congestion scenarios, information such as traffic light status and the location of nearby pedestrians has increased weight. Detailed information display priority relationships can be preset, and the information priority can be dynamically adjusted according to different driving scenarios and real-time data changes. Figure 4 This invention provides a schematic diagram of an instrument display in normal operating condition, as shown in the embodiment of the invention. Figure 4 As shown, under normal driving conditions, basic driving information such as vehicle speed and engine speed are displayed in a large, first preset font and a prominent preset color in the center of the instrument panel. Secondary but important information, such as distance to the vehicle in front and fuel consumption, is displayed in a smaller, second preset font in the edge area of ​​the instrument panel. In dangerous situations, such as when there is a potential hazard like a vehicle braking suddenly ahead or an obstacle appearing on the road, the system can recalculate the information priority based on a hazard assessment algorithm, as well as factors such as the type of hazard, distance, and urgency. Figure 5 A schematic diagram of an instrument displaying a dangerous state is provided as an embodiment of the present invention, such as... Figure 5 As shown, if a high-risk hazard is identified, the information display priority will be to display a hazard warning message (such as "Emergency hazard ahead, take immediate action"). This hazard warning message will occupy the most prominent position in the center of the instrument panel with a flashing red, oversized, third-preset font, while the brightness of other information will be reduced or temporarily hidden to ensure that the driver's attention is instantly focused on the critical hazard information. Once the hazard has passed, the system will readjust the information priority according to the current driving scenario and restore the normal information display layout.

[0062] This embodiment fuses vehicle driving data with route instruction information to obtain fused projection information, and generates augmented reality display instructions based on the projection information. For example, an augmented reality (AR) projection device with high brightness, high contrast, and wide viewing angle can be installed on the windshield to ensure that the driver can clearly see the projection information under various lighting conditions. Simultaneously, a dedicated graphics processing unit is equipped for real-time generation and processing of AR images. This unit is connected to the onboard computer via a high-speed interface, enabling it to quickly receive environmental perception and navigation information sent by the computer. To ensure accurate matching between the AR projection and the actual road scene, a gyroscope and accelerometer can also be installed in the AR projection device to monitor vehicle attitude changes in real time, providing accurate attitude data for AR image rendering.

[0063] The onboard computer can process and fuse information about surrounding vehicles, pedestrians, and obstacles in real time, along with route guidance information provided by the navigation module. Using computer vision algorithms, it identifies and locates targets in images captured by cameras, determining the position and category of surrounding objects. Combining data from millimeter-wave radar and lidar, it accurately measures the distance and speed of targets, fusing this environmental information with navigation information. Through coordinate transformation and projection algorithms, it precisely matches and aligns virtual information with the actual road scene. Then, the fused information is transmitted to an AR projection device, which projects the information onto the windshield in augmented reality form, perfectly blending it with the road scene actually seen by the driver. Figure 6 This is a schematic diagram illustrating the fusion of projection information and actual road scene provided in an embodiment of the present invention, such as... Figure 6 As shown, during navigation, the AR projection device calculates the projection position and angle of the virtual arrow at the upcoming turn on the actual road based on the vehicle's current location and direction of travel. It accurately projects navigation information such as the turn arrow and distance to the next intersection onto the actual location on the road ahead, making the driver feel as if they are directly seeing real road signs. When a pedestrian is detected suddenly entering the road ahead, the system uses AR technology to project a prominent red border around the pedestrian's location, accompanied by a voice prompt "Caution: Pedestrian Ahead," allowing the driver to react quickly. To enhance the display effect during projection, anti-aliasing and lighting simulation technologies can also be used to make the virtual information look more realistic and natural.

[0064] Intelligent layered display dynamically adjusts based on driving scenarios and information importance, ensuring drivers always have access to critical information immediately. AR technology projects important information directly onto the windshield, blending it with the actual road scene, eliminating the need for drivers to frequently shift their gaze. In complex road conditions, AR enables rapid identification and reaction to hazards, effectively shortening reaction time and significantly reducing the probability of accidents.

[0065] This embodiment can recognize facial images in driver data based on a pre-trained deep learning model to obtain the driver's fatigue level and attention state. Based on the fatigue level and attention state, combined with the driver's vital signs data in the driver data, fatigue driving interaction instructions are generated. First, a facial recognition algorithm can be used to analyze the driver's facial expressions (such as yawning and frowning when fatigued, and wandering eyes when inattentive), eye opening and closing degree, blinking frequency, etc. Through training the deep learning model with a large number of facial images in different states, the system can accurately identify the driver's fatigue and attention state. Figure 7 This is a schematic diagram of a sensor layout provided in an embodiment of the present invention, such as... Figure 7As shown, the camera captures real-time images of the driver's face and transmits them to the onboard computer. The computer's facial recognition algorithm preprocesses the images, including grayscale conversion and normalization, and then extracts facial features using a convolutional neural network. In the CNN model, representative facial feature vectors are extracted through multiple convolutional and pooling operations. These feature vectors are compared with pre-trained feature templates for fatigue and inattention states, and the driver's condition is determined by calculating the similarity. For example, by training on a large number of facial images of fatigued individuals, a fatigue feature template is established. When fatigue features such as yawning or frowning are detected, and the eye closure time exceeds a certain threshold (e.g., continuous eye closure for more than 1 second), while the blinking frequency is below the normal range, the driver is determined to be fatigued. To improve accuracy, the facial recognition algorithm also incorporates time-series analysis to dynamically track changes in the driver's facial state, avoiding misjudgments caused by instantaneous changes in expression.

[0066] The pressure sensor array transmits the collected pressure data to the onboard computer. The computer analyzes the pressure data using specialized algorithms to identify the driver's body posture, such as whether they are leaning forward, backward, or frequently adjusting their seating position. By training on a large amount of pressure data under normal driving postures, a normal posture model can be established. When the system detects that the driver is leaning forward for an extended period and the pressure distribution deviates from the normal model by more than a certain threshold, it determines that the driver may be overly focused on the road conditions ahead or be in a state of tension. The system can then adjust its interaction strategy accordingly (such as providing soothing music). By cross-referencing body posture monitoring data with facial recognition and physiological indicator monitoring data, a multi-dimensional basis can be provided for more accurate assessment of the driver's condition.

[0067] Interaction strategies can include fatigue response strategies and inattention response strategies. Fatigue response strategy: When the system determines that the driver is fatigued through multi-dimensional monitoring, it first activates a personalized fatigue reminder mechanism. Based on the voice style (e.g., gentle female voice, steady male voice) and reminder intensity (e.g., mild, moderate, severe) previously selected by the driver in the system settings, a warm and targeted reminder is issued through the voice module. For example, for a driver who selected a mild reminder, the voice prompt is "You look somewhat tired. For your safety, we suggest you find a safe place to rest." For a driver who selected a severe reminder, the voice prompt is more urgent and strong, such as "You are in a state of fatigued driving and must rest immediately!" Simultaneously, the location information of nearby service areas or rest stops is displayed on the instrument panel with prominent icons and text, and navigation guidance is provided. If the driver does not respond within a certain period, the system takes further measures, automatically adjusting the ambient lighting in the vehicle to a bright and refreshing tone, such as blue or white, to stimulate the driver's nervous system and increase alertness by changing the lighting environment. At the same time, upbeat and stimulating music is played; the music selection can also be personalized according to the driver's preferences. If fatigue persists and worsens, the system automatically reduces the vehicle speed to a safe range and strengthens the intelligent driver assistance system's intervention, such as automatically maintaining a greater safe following distance to ensure vehicle safety until the driver's condition returns to normal after rest.

[0068] Inattention Response Strategy: When inattention is detected, the system first emits a sharp but not piercing warning sound. The frequency and volume of the warning sound can be adjusted according to the driver's previously set preferences. Simultaneously, a pop-up window appears on the instrument panel displaying the message "Please concentrate on driving," and the color and font size of the window can also be customized. If the inattention persists, the system temporarily strengthens the intelligent driver assistance system's control over the vehicle. For example, it automatically adjusts the speed to maintain a safe distance from the vehicle in front, monitoring the speed and distance of the vehicle ahead in real time using millimeter-wave radar. When the distance is detected to be too close, the system automatically reduces the speed; it also lowers the speed limit to prevent dangerous situations caused by excessive speed due to driver distraction. Furthermore, the system also enhances the alerting effect by changing the voice interaction method, such as increasing the frequency and volume of voice prompts. For example, if a voice prompt is normally given every 5 minutes, it is shortened to every 2 minutes when inattention is detected, and the voice volume is increased by 10 decibels. Additionally, combining facial recognition and body posture monitoring data, if the system detects that the driver's gaze has been off the road for an extended period, it will also alert the driver through methods such as seat vibration until the driver regains attention.

[0069] By monitoring driver status in real time across multiple dimensions, the system can promptly detect dangerous conditions such as fatigue or inattention and quickly implement personalized intervention measures. When driver fatigue is detected, timely reminders and speed adjustments can effectively prevent accidents caused by fatigued driving. Timely responses to inattention can also prevent dangerous situations caused by driver distraction, significantly improving driving safety.

[0070] In one embodiment, the on-board computer 200 is specifically used for: When the environmental perception data predicts that there are complex road conditions ahead, a control handover command is generated, which is used to remind the driver to take over the vehicle. When the driver is identified as taking over the vehicle through the takeover intent signal in the driver data, the control parameters of the driver assistance system are adjusted to transfer vehicle control to the driver.

[0071] In this embodiment, when complex road conditions are predicted ahead based on environmental perception data, the on-board computer 200 can generate a control handover command to remind the driver to take over the vehicle. If the driver is identified as taking over the vehicle through the takeover intention signal in the driver data, the control parameters of the assisted driving are adjusted to transfer vehicle control to the driver. Otherwise, intelligent control is strengthened to prevent accidents.

[0072] For example, intelligent driver assistance systems can utilize various sensors (such as cameras, radar, and lidar) to perceive the vehicle's surrounding environment in real time. Combined with map data and road condition prediction algorithms, they can predict potentially complex road conditions in advance. Map data includes information such as road types, traffic restrictions, and real-time road conditions. Cameras use image recognition technology to detect road signs, traffic conditions, and road construction ahead. Radar monitors the speed, distance, and movement of vehicles ahead in real time. LiDAR provides more accurate 3D point cloud data of the surrounding environment for detecting obstacles and complex terrain. Based on machine learning techniques, complex road condition prediction models can be built by learning from a large amount of historical road condition data and real-time sensor data. This allows analysis of characteristic patterns in sensor data, such as a sudden increase in vehicle density ahead or road signs displaying construction information. Combined with road condition information from map data, this can predict whether situations that the system cannot handle are about to occur, such as road construction causing lane narrowing or extreme weather affecting sensor performance. The road condition prediction model can be continuously updated and optimized. Through a real-time feedback mechanism, the actual road conditions are compared with the predicted results, and model parameters are adjusted to improve prediction accuracy.

[0073] Figure 8 This is a schematic diagram of a system request takeover warning process provided in an embodiment of the present invention, such as... Figure 8As shown, when complex road conditions are predicted and the system may require driver intervention, a tiered warning mechanism is activated 10-15 seconds in advance. First, the voice module provides a smooth and clear voice prompt informing the driver, "The road conditions ahead are complex; the system will soon be unable to provide support. Please prepare to take over." Simultaneously, a countdown timer is prominently displayed in the center of the instrument panel with a yellow background. After the countdown begins, the system gradually increases the intensity of the warning every 3-5 seconds, based on the remaining time and the urgency of the road conditions. For example, with 8 seconds remaining, the instrument panel lights begin to flash at a slower frequency, alerting the driver to the impending takeover request; with 5 seconds remaining, the seat begins to vibrate slightly, with the intensity adjustable according to the driver's preference, providing a tactile alert; with 3 seconds remaining, the voice prompt volume increases and the speech rate accelerates, attracting the driver's attention more strongly. When the countdown ends, if the driver has not yet taken over, the system automatically applies emergency braking. Based on the vehicle's current speed and road conditions, the braking system precisely controls the braking force to bring the vehicle to a safe stop, continuously emitting strong audible and visual warnings, such as high-frequency flashing instrument panel lights and a sharp alarm sound, until the driver takes over or the danger is over. Throughout the warning process, the system will monitor the driver's status and operating intentions in real time. If it detects that the driver is attempting to take over, such as gripping the steering wheel tightly with both hands or pressing the takeover button, it will immediately stop further warning measures and wait for the driver to smoothly take over the vehicle.

[0074] Figure 9 This is a schematic diagram of a driver's intention to take over, provided as an embodiment of the present invention. Figure 9 As shown, the onboard computer can accurately determine whether the driver intends to take over the vehicle by combining data from pressure sensors and microswitches, along with current vehicle driving status information (such as vehicle speed and steering angle), through a comprehensive analysis using a machine learning model. First, the pressure sensor data is preprocessed to remove noise and interference signals, extracting characteristic parameters of pressure changes, such as pressure peak value and pressure change rate. For microswitch signals, changes in their switching state are directly detected. Then, this data, along with vehicle driving status information, is input into a pre-trained machine learning model, such as a support vector machine model or a neural network model. The model establishes a mapping relationship between the driver's intention to take over and the input data by learning from a large amount of historical data. In actual operation, the model calculates the probability value of the driver's intention to take over based on real-time input data. When the probability value exceeds a set threshold, it determines that the driver intends to take over the vehicle. For example, if the driver suddenly increases the pressure on the steering wheel while the vehicle's driving state is relatively stable, the algorithm considers this a signal that the driver is preparing to take over; if the takeover button is pressed at this time, the driver's intention to take over is further confirmed.

[0075] Once the system detects the driver's intention to take over, the onboard computer quickly adjusts the control parameters of the intelligent driver assistance system, smoothly transferring vehicle control to the driver. First, the intelligent driver assistance system gradually reduces its intervention in vehicle steering and speed control. For example, in steering control, it gradually reduces the power steering effort, allowing the driver to gradually feel the vehicle's natural steering resistance; in speed control, it slowly releases control of the accelerator and brakes, allowing the driver to smoothly take over speed regulation. Simultaneously, the onboard computer displays handover status information on the instrument panel, such as "Control has been transferred; you may drive normally," clearly informing the driver of the successful takeover. During the control transfer process, the intelligent driver assistance system continues to monitor the vehicle's surroundings and the driver's condition, ready to intervene again if necessary. For example, if an abnormal driving condition or an emergency situation is detected shortly after the handover, the intelligent driver assistance system can quickly regain control of the vehicle to ensure driving safety.

[0076] The optimized control handover mechanism is safer and more reliable, whether the system requests takeover or the driver initiates it. Tiered warnings before a system requests takeover allow the driver sufficient time to prepare, avoiding panic caused by sudden takeover. Advance warnings and progressively stronger alerts reduce the error rate during the takeover process. Meanwhile, the rapid and accurate identification and smooth transition during voluntary driver takeover ensure the continuity of driving operations and reduce safety hazards caused by poor handover.

[0077] In one embodiment, the display information adjustment commands include instrument display adjustment commands and augmented reality projection adjustment commands, and the vehicle-mounted computer 200 is specifically used for: By combining traffic scene data from environmental perception data with the vehicle's current location and electronic map data, the type of road condition at the vehicle's current location can be determined. Generate instrument display adjustment instructions based on the road condition type; Augmented reality projection adjustment instructions are generated using natural environment data from the environmental perception data.

[0078] Among them, the instrument display adjustment command can be a control command that dynamically adjusts the content, layout, and visual weight of the instrument panel display. The augmented reality projection adjustment command can be a control command that dynamically adjusts the content, clarity, and display strategy of the augmented reality head-up display.

[0079] In this embodiment, the vehicle computer 200 can determine the road condition type of the vehicle's current location by combining traffic scene data in the environmental perception data with the vehicle's current location and electronic map data, generate instrument display adjustment instructions based on the road condition type, and generate augmented reality projection adjustment instructions by combining natural environment data in the environmental perception data.

[0080] For example, the system can obtain the vehicle's real-time location information through a positioning module and combine it with detailed electronic map data to accurately determine the type of road condition the vehicle is currently on, such as highway, urban road, mountain road, or rural road. Simultaneously, the system utilizes camera recognition of road signs and lane markings, as well as radar monitoring of surrounding vehicles' driving status, to further verify and refine the road condition judgment results. For instance, when the positioning system shows the vehicle is on a highway, and the camera identifies continuous highway signs and long, straight lane markings, while the radar detects a high vehicle speed and relatively stable distance between vehicles, it confirms the vehicle is on a highway. The electronic map data not only contains basic road information but also updates real-time with traffic congestion, construction, and other road condition information. The system interacts with a map server to obtain the latest road condition information. Furthermore, machine learning algorithms are used to analyze historical road condition data and establish road condition feature models, such as characteristic patterns of vehicle speed, acceleration, and direction of travel under different road conditions, to improve the accuracy and real-time performance of road condition judgments.

[0081] Figure 10 This is a comparative illustration of an adaptive road condition information display provided by an embodiment of the present invention, such as... Figure 10 As shown, when the system determines that the vehicle is traveling on a highway, it automatically generates instrument display adjustment commands to adjust the human-machine interaction method and information presentation content. Regarding information display, it highlights information closely related to highway driving, such as cruising speed, following distance, and current lane information, and increases the font and icon size of this information for easy identification by the driver from a distance. For example, the cruising speed is displayed in a large, pre-set font in a prominent position in the center of the instrument panel, and the following distance is displayed next to the speed information with eye-catching numbers and graphic icons. Simultaneously, the frequency of voice prompts is adjusted, reducing general prompts and only issuing voice prompts when critical events occur, such as a vehicle ahead suddenly decelerating beyond a certain threshold or deviating from its lane. The voice prompts are concise and clear, such as "The vehicle ahead is decelerating suddenly; please maintain a safe distance." When the vehicle enters urban roads, the system switches to urban driving mode, which can highlight traffic light recognition information, pedestrian warning information, nearby parking information, and real-time traffic congestion. Near intersections, the display of traffic light status is enhanced, using eye-catching colors and animations to indicate the current traffic light status and remaining time to the driver. For example, a green countdown timer displays the remaining time for a green light, while a red countdown timer displays the remaining time for a red light. The countdown numbers are displayed in a larger, pre-defined font on the dashboard or head-up display. Simultaneously, when a pedestrian is detected approaching the vehicle, the pedestrian's location is highlighted on the display screen, accompanied by a voice prompt, "Caution: Pedestrian Ahead." Real-time traffic congestion information is displayed using a combination of color-coded markers on a map and text prompts to guide drivers in choosing the optimal route and avoiding congested areas.

[0082] In rainy weather, the system automatically adjusts the contrast and brightness of the AR projection information on the windshield based on the amount of rainfall. During heavy rainfall, the contrast and brightness are significantly increased to ensure the driver can clearly see the information in the rain. For example, when rainfall exceeds a preset threshold, the brightness of the AR projection is increased by 20%, and the contrast by 30%. Simultaneously, the voice prompt volume is adjusted to adapt to the rainy environment, ensuring the driver can hear the prompts clearly. The voice prompt volume automatically adjusts based on rainfall, employing an adaptive volume adjustment algorithm that dynamically adjusts the volume by detecting the intensity of ambient noise (primarily rain noise) to ensure the prompts are always clearly audible. In foggy weather, the system uses visibility data from a visibility sensor to automatically adjust the size of surrounding vehicle and obstacle markers on the display screen. The lower the visibility, the larger the markers become to enhance their visibility. For example, when visibility is below 100 meters, the size of vehicle and obstacle markers is increased by 50%; when visibility is below 50 meters, it is further increased to 80%. Furthermore, the system increases the frequency of audio prompts, broadcasting information such as the distance, relative position, and speed of surrounding vehicles at regular intervals to help drivers better assess road conditions in low-visibility environments. For example, when visibility is less than 50 meters, nearby vehicle information is broadcast every 10 seconds. In snowy weather, in addition to adjusting the display and voice prompts, the system also fine-tunes the control parameters of the intelligent driving assistance system based on changes in snow depth and road surface friction, such as lowering the maximum cruise speed and increasing the braking distance to ensure vehicle safety. Ultrasonic sensors installed near the wheels detect snow depth, and based on a preset relationship model between snow depth and the system, the maximum cruise speed and braking distance parameters are adjusted. For example, when the snow depth reaches 1mm, the maximum cruise speed is reduced by 10%, and the braking distance is increased by 200m.

[0083] The adaptive road condition and weather function enables the system to automatically adjust the interaction method and information presentation according to different driving scenarios. On highways, the concise and clear information display and reasonable voice prompt frequency, as well as on urban roads, provide accurate traffic information display, ensuring a comfortable and convenient driving experience in various situations. For example, in inclement weather such as rain or fog, the system's targeted adjustments to the display and voice prompts improve the driver's adaptability to the environment and greatly enhance the driving experience.

[0084] This system integrates gesture, touch, and voice interaction. It uses a depth camera to accurately capture hand gestures and capacitive touch technology for steering wheel touch interaction, forming the hardware foundation for multimodal interaction. Based on deep learning algorithms trained on a large number of samples, it can accurately recognize gestures and touch intentions. The different interaction methods work together to meet the driver's needs for convenience and safety in heavy truck driving scenarios. By comprehensively processing vehicle driving data, environmental perception data, and driver status data, and dynamically adjusting information priority according to different scenarios, AR technology achieves accurate fusion of virtual information and real road scenes, improving the driving experience. Using facial recognition, physiological indicator monitoring, and body posture monitoring, it comprehensively perceives the driver's state, accurately judging conditions such as driver fatigue and inattention. Personalized and scientifically sound response strategies tailored to different states, such as reminders and speed adjustments for fatigue, and warnings and interventions for inattention, ensure driver safety. The complex road condition prediction algorithm, based on multi-sensor data and map information, accurately predicts potentially complex road conditions. A tiered warning strategy is implemented, gradually increasing the intensity of alerts based on remaining time and the urgency of the road condition, giving the driver sufficient time to take over. The takeover intent recognition algorithm, through analysis of data such as steering wheel pressure, button operations, and vehicle driving status, accurately determines the driver's intention to take over, achieving a smooth and efficient transfer of control. By acquiring real-time and accurate road and weather information through multiple sensors, different road conditions and weather conditions can be accurately identified, and scientifically reasonable adaptive adjustment strategies can be formulated for different scenarios.

[0085] Example 2 Figure 11 This is a flowchart illustrating a human-machine interaction method based on assisted driving provided in Embodiment 2 of the present invention. This method is applicable to situations where a driver is assisted in driving a vehicle. The method can be executed by a human-machine interaction device based on assisted driving, wherein the device can be implemented by software and / or hardware and is generally integrated into the vehicle. In this embodiment, the vehicle includes, but is not limited to, heavy trucks. For details not covered in this embodiment, please refer to Embodiment 1.

[0086] like Figure 11 As shown, Embodiment 2 of the present invention provides a human-computer interaction method based on assisted driving, which includes the following steps: S210: Acquire driver data, environmental perception data around the vehicle, and vehicle driving data.

[0087] In this embodiment, driver data, environmental perception data around the vehicle, and vehicle driving data can be used.

[0088] S220. Analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions.

[0089] In this embodiment, driver data, environmental perception data around the vehicle, and vehicle driving data can be analyzed to obtain operating instructions.

[0090] S230. The operation instruction is sent to the corresponding functional module so that the functional module performs the corresponding operation based on the operation instruction to assist the driver in driving.

[0091] In this embodiment, operation instructions can be sent to the corresponding functional modules so that the functional modules can perform corresponding operations based on the operation instructions to assist the driver in driving.

[0092] Embodiment 2 of the present invention provides a human-computer interaction method based on assisted driving, comprising: acquiring driver data, environmental perception data around the vehicle, and vehicle driving data; analyzing the driver data, environmental perception data around the vehicle, and vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions; and sending the operation instructions to the corresponding functional modules so that the functional modules perform corresponding operations based on the operation instructions to assist the driver in driving. This method generates operation instructions that can assist the driver by collecting data, which can improve the driver's experience and solve the problems of existing interaction methods being unable to support complex driving scenarios and the lack of human-computer interaction methods.

[0093] Example 3 Figure 12 This is a schematic diagram of a human-computer interaction device based on assisted driving provided in Embodiment 3 of the present invention. The device is applicable to situations where drivers are assisted in driving vehicles. The device can be implemented by software and / or hardware and is generally integrated into the vehicle.

[0094] like Figure 12 As shown, the device includes: The acquisition module 310 is used to acquire driver data, environmental perception data around the vehicle, and vehicle driving data. The analysis module 320 is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The sending module 330 is used to send the operation instruction to the corresponding functional module, so that the functional module can perform the corresponding operation based on the operation instruction to assist the driver in driving.

[0095] This embodiment provides a human-machine interaction device based on assisted driving, including: an acquisition module for acquiring driver data, environmental perception data around the vehicle, and vehicle driving data; an analysis module for analyzing the driver data, environmental perception data around the vehicle, and vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions; and a sending module for sending the operation instructions to corresponding functional modules, so that the functional modules perform corresponding operations based on the operation instructions to assist the driver in driving.

[0096] The aforementioned human-machine interaction device based on assisted driving can execute the human-machine interaction method based on assisted driving provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0097] Example 4 Figure 13 A schematic diagram of the structure of a vehicle 10 that can be used to implement embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0098] like Figure 13 As shown, vehicle 10 includes at least one processor 11 and a memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer program stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of vehicle 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. Input / output (I / O) interface 15 is also connected to bus 14.

[0099] Multiple components in vehicle 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows vehicle 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as human-machine interaction methods based on assisted driving.

[0101] In some embodiments, the driver assistance-based human-machine interaction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on vehicle 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the driver assistance-based human-machine interaction method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the driver assistance-based human-machine interaction method by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0108] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A human-machine interaction system based on assisted driving, characterized in that, The system includes: a data acquisition component, an on-board computer, and a functional module, wherein the on-board computer is connected to the data acquisition component and the functional module respectively. The data acquisition component is used to collect driver data, environmental perception data around the vehicle, and vehicle driving data, and transmit the driver data, environmental perception data around the vehicle, and vehicle driving data to the on-board computer. The on-board computer is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions, and send the operation instructions to the corresponding functional modules; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The functional module is used to perform corresponding operations based on the operation instructions to assist the driver in driving.

2. The system according to claim 1, characterized in that, The driver data includes gesture images, touch data, facial images, takeover intention signals, and driver vital sign data; the environmental perception data includes traffic scene data and natural environment data; the data acquisition components include at least a depth camera, a touch sensing area, a vehicle operating status sensor, a traffic environment sensor, a takeover intention acquisition module, a vital sign sensor, a facial recognition camera, and a natural environment sensor. The depth camera is used to capture images of the driver's hand gestures; The touch sensing area is used to collect touch data from the driver's hand; The vehicle operating status sensor is used to collect vehicle driving data; The traffic environment sensor is used to collect traffic scene data around the vehicle; The takeover intention acquisition module is used to acquire the driver's takeover intention signal; The vital signs sensor is used to collect the driver's vital signs data; The facial recognition camera is used to capture facial images of the driver; The natural environment sensor is used to collect data on the natural environment around the vehicle.

3. The system according to claim 2, characterized in that, The takeover intent acquisition module includes a pressure sensor and a micro switch integrated on the steering wheel; The vital signs sensors include a heart rate sensor, a respiratory rate sensor, and multiple pressure sensors mounted on the driver's seat; The natural environment sensors include at least a rain sensor, a humidity sensor, a visibility sensor, and a temperature sensor.

4. The system according to claim 1, characterized in that, The vehicle-mounted computer is specifically used for: The gesture images in the driver data are analyzed by a pre-trained gesture recognition model to obtain the driver's gesture intention, and a function switching instruction is generated based on the gesture intention. The touch data in the driver data is analyzed by a touch recognition algorithm to obtain the driver's touch actions, and information display interaction instructions are generated based on the functions corresponding to the touch actions; The current driving state of the vehicle is determined based on the environmental perception data and the vehicle driving data, and a hierarchical display instruction is generated based on the information display priority relationship corresponding to the current driving state. The vehicle driving data and route instruction information are fused to obtain fused projection information, and augmented reality display instructions are generated based on the projection information. The driver's facial images in the driver data are identified using a pre-trained deep learning model to obtain the driver's fatigue level and attention state. Based on the fatigue level and attention state, and combined with the driver's vital signs data in the driver data, fatigue driving interaction instructions are generated.

5. The system according to claim 1, characterized in that, The vehicle-mounted computer is specifically used for: When the environmental perception data predicts that there are complex road conditions ahead, a control handover command is generated, which is used to remind the driver to take over the vehicle. When the driver is identified as taking over the vehicle through the takeover intent signal in the driver data, the control parameters of the driver assistance system are adjusted to transfer vehicle control to the driver.

6. The system according to claim 1, characterized in that, The display information adjustment commands include instrument display adjustment commands and augmented reality projection adjustment commands. The on-board computer is specifically used for: By combining traffic scene data from environmental perception data with the vehicle's current location and electronic map data, the type of road condition at the vehicle's current location can be determined. Generate instrument display adjustment instructions based on the road condition type; Augmented reality projection adjustment instructions are generated using natural environment data from the environmental perception data.

7. A human-computer interaction method based on assisted driving, characterized in that, The method, applied to an in-vehicle computer in a human-machine interaction system based on assisted driving as described in claims 1-6, comprises: Acquire driver data, environmental perception data around the vehicle, and vehicle driving data; The driver data, the environmental perception data around the vehicle, and the vehicle driving data are analyzed to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The operation command is sent to the corresponding functional module so that the functional module performs the corresponding operation based on the operation command to assist the driver in driving.

8. A human-machine interaction device based on assisted driving, characterized in that, The device is capable of executing the human-machine interaction method based on assisted driving as described in claim 7, wherein the device comprises: The acquisition module is used to acquire driver data, environmental perception data around the vehicle, and vehicle driving data. The analysis module is used to analyze the driver data, the environmental perception data around the vehicle, and the vehicle driving data to obtain operation instructions; the operation instructions include at least one or more of the following: function switching instructions, information display interaction instructions, layered display instructions, augmented reality display instructions, fatigue driving interaction instructions, control handover instructions, and display information adjustment instructions. The sending module is used to send the operation command to the corresponding functional module, so that the functional module can perform the corresponding operation based on the operation command to assist the driver in driving.

9. A vehicle, characterized in that, The vehicles include: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the human-machine interaction method based on assisted driving as described in claim 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the human-computer interaction method based on assisted driving as described in claim 7.

Images