Vehicle man-machine interaction method and device and computer program product

Abstract

The invention discloses a vehicle man-machine interaction method and device and a computer program product, and the method comprises the steps: receiving and recognizing a voice instruction and a gesture instruction of a passenger in a vehicle; monitoring the state of the driver in real time; if the driver is in the normal state, executing a corresponding instruction according to a preset execution rule according to the input sequence of the voice instruction and the gesture instruction and whether a conflict exists or not; if the driver is in the fatigue or distraction state, the safety reminding instruction is executed firstly, and after confirmation of the driver is obtained, the corresponding instruction is executed according to the instruction execution logic that the driver is in the normal state. Through the innovative vehicle man-machine interaction method, the driving safety and the in-vehicle interaction experience are remarkably improved; not only is human-vehicle interaction logic optimized, but also a safer, more comfortable and more personalized driving environment is provided for a user through an intelligent control mode.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent networked vehicles, and in particular to a vehicle human-computer interaction method, device and computer program product. Background Art

[0002] With the development of intelligent vehicles, in-vehicle interactive systems are gradually evolving towards multimodal interaction, aiming to improve user experience and ease of operation. Currently, a major challenge facing the automotive industry is to achieve more high-frequency, high-perception user interaction methods and functions under the premise of cost control. Traditional single-mode interactive systems can no longer meet market demand, and multimodal interaction, such as combining eye information, voice commands and gesture recognition, has become the key to improving the flexibility and accuracy of human-computer interaction.

[0003] However, although existing technologies have achieved multimodal interaction to a certain extent, in the scenario of smart cars, how to creatively integrate voice interaction and gesture interaction, improve the practicality and experience value of interaction, expand the usage scenarios, and solve pain points such as left-behind children and left-behind items, is still a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a vehicle human-computer interaction method, device and computer program product to improve the fluency and accuracy of human-computer interaction on the basis of realizing multimodal interaction.

[0005] In order to solve the above technical problems, the present invention provides a vehicle human-computer interaction method, comprising the following steps:

[0006] Receive and recognize voice and gesture commands from passengers in the vehicle;

[0007] Real-time monitoring of driver status;

[0008] If the driver is in a normal state, the corresponding command will be executed according to the preset execution rules based on the order in which the voice command and gesture command are input and whether there is a conflict;

[0009] If the driver is in a state of fatigue or distraction, the safety reminder command will be executed first, and after obtaining the driver's confirmation, the corresponding command will be executed according to the command execution logic when the driver is in a normal state.

[0010] Preferably, if the driver is in a normal state, the corresponding command is executed according to the order in which the voice command and the gesture command are input and whether there is a conflict, according to the preset execution rules, which specifically includes:

[0011] If the voice command and gesture command are not input at the same time, they will be executed in the order in which they were input;

[0012] If the voice command and gesture command are input at the same time, it is first determined whether there is a driver command in the voice command and the gesture command. If so, the driver command is executed first; if not, it is further determined whether there is a conflict between the voice command and the gesture command, and the conflict is handled according to the preset conflict handling rules.

[0013] Preferably, the determining whether there is a conflict between the voice command and the gesture command and processing the conflict according to a preset conflict handling rule specifically includes:

[0014] If there is a conflict between a voice command and a gesture command, the driver will be reminded by voice and any conflicting command will be suspended; then other non-conflicting commands will continue to be executed according to the preset priority rules;

[0015] If there is no conflict between the voice command and the gesture command, the corresponding command is executed according to the preset priority rules.

[0016] Preferably, the preset priority rules include giving priority to driver's instructions, or giving priority to passenger's instructions, or giving priority to rear seat passenger's instructions.

[0017] Preferably, the method further comprises: if no voice command or gesture command is recognized and no power-off signal is received, maintaining real-time monitoring of the driver's status.

[0018] Preferably, the method further comprises: controlling the camera to capture the interior environment of the vehicle according to a camera angle instruction and an instruction to call the camera image remotely issued by the user, and feeding back the captured video signal to the user via a communication link.

[0019] Preferably, the method further comprises: according to a call instruction remotely issued by the user, synchronously transmitting the in-vehicle video signal and voice signal to the user, and then receiving the user's voice signal and transmitting it through a microphone to realize two-way communication.

[0020] The present invention also provides a vehicle human-machine interaction device, comprising:

[0021] A recognition module, used to receive and recognize voice commands and gesture commands from passengers in the vehicle;

[0022] Monitoring module, used to monitor the driver's status in real time;

[0023] The control module is used to execute corresponding instructions according to the preset execution rules based on the order of voice instructions and gesture instructions input and whether there is a conflict when the driver is in a normal state; or when the driver is in a fatigued or distracted state, first execute the safety reminder instruction, and after obtaining the driver's confirmation, execute the corresponding instruction according to the instruction execution logic when the driver is in a normal state.

[0024] The present invention also provides a vehicle human-machine interaction device, comprising:

[0025] one or more processors;

[0026] Memory;

[0027] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the vehicle human-computer interaction method.

[0028] The present invention also provides a computer program product, comprising computer instructions, wherein the computer instructions instruct a computer device to execute operations corresponding to the method.

[0029] The implementation of the present invention has the following beneficial effects: The present invention significantly improves driving safety and in-vehicle interactive experience through an innovative vehicle human-computer interaction method. By real-time monitoring of the driver's status, it can intelligently identify and prioritize the driver's voice and gesture commands, effectively reducing the risk of accidents caused by driver distraction or fatigue. In the event of a command conflict, the system makes decisions based on preset priority rules to ensure the rationality and smoothness of command execution. In addition, when the driver is fatigued or distracted, the system will give priority to issuing safety reminders to further ensure driving safety. Users can also remotely monitor the situation in the car through a mobile phone APP and achieve two-way voice and video communication, which enhances the practicality and convenience of the vehicle and provides users with a safer, more comfortable and personalized driving environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a flowchart of a vehicle human-computer interaction method according to an embodiment of the present invention.

[0032] Figure 2 It is a specific flow chart of a vehicle human-computer interaction method according to an embodiment of the present invention.

[0033] Figure 3 It is a schematic diagram of the implementation architecture of an embodiment of the present invention when a user enters a vehicle.

[0034] Figure 4 Schematic diagram of the steering of the human-vehicle interaction robot in an embodiment of the present invention.

[0035] Figure 5 It is a schematic diagram of the implementation process of an embodiment of the present invention in a scenario where the user is far away from the vehicle.

[0036] Figure 6 It is a schematic diagram of the implementation architecture of an embodiment of the present invention in a scenario where the user is far away from the vehicle. DETAILED DESCRIPTION

[0037] The following descriptions of the embodiments refer to the accompanying drawings to illustrate specific embodiments in which the present invention may be implemented.

[0038] Please refer to Figure 1 As shown, a vehicle human-computer interaction method is provided in a first embodiment of the present invention, comprising the following steps:

[0039] Receive and recognize voice and gesture commands from passengers in the vehicle;

[0040] Real-time monitoring of driver status;

[0041] If the driver is in a normal state, the corresponding command will be executed according to the preset execution rules based on the order in which the voice command and gesture command are input and whether there is a conflict;

[0042] If the driver is in a state of fatigue or distraction, the safety reminder command will be executed first, and after obtaining the driver's confirmation, the corresponding command will be executed according to the command execution logic when the driver is in a normal state.

[0043] Through the above steps, it can be seen that the embodiment of the present invention can effectively avoid the potential risks caused by driver misoperation when the driver is tired or distracted by real-time monitoring of the driver's state and adopting differentiated command processing strategies in different states. At the same time, for the conflict handling of voice commands and gesture commands, the preset execution rules can ensure that the system makes reasonable judgments in the shortest time and executes the most appropriate commands, thereby improving the fluency and accuracy of human-computer interaction.

[0044] Based on the traditional single in-car voice, the present invention integrates the driver status monitoring system DMS (Driver Monitoring System) to achieve multi-mode interaction in the car and add more intelligent control methods and application scenarios. Figure 2 As shown, taking the scenario of a user getting on a vehicle as an example, the specific process of the embodiment of the present invention is as follows:

[0045] The user enters the vehicle cockpit and powers on the vehicle, then logs in to the account. The AVNT (Audio Video Navigation Telematics) host, which is the main node for cockpit control, will check the current login account information:

[0046] If the account is registered or authorized, the user will be able to enjoy the full range of voice interaction and gesture interaction functions;

[0047] If the account is not registered or authorized, some of the user's voice interaction and gesture interaction permissions will be restricted and sent to the DMS.

[0048] It is understandable that the login account needs to be bound to user information in advance. At the same time, in order to enhance privacy, the user can also be required to successfully use face recognition (through DMS to identify Face ID) and voiceprint recognition (through AVNT host to identify user voiceprint) before the human-vehicle interactive robot can be used. Figure 3 , Figure 4 As shown, it should be noted that, in the embodiment of the present invention, the human-vehicle interaction robot is an intelligent vehicle-mounted system that integrates DMS and multimodal interaction technology, provides a natural and intuitive interaction method, and allows the driver and passengers to control the vehicle through voice commands or gestures, thereby achieving safer and smarter vehicle control.

[0049] Under a specific account (such as XX account), the user can make a series of settings for the human-vehicle interactive robot on the AVNT host display, including but not limited to the following aspects:

[0050] Application object setting: Users can select the application objects of the human-vehicle interaction robot. For example, the interaction function can be opened only to specific accounts, or only to the driver, or only to the rear passengers, etc., to increase the flexibility and safety of use.

[0051] Execution rule settings: including instruction execution priority settings and instruction conflict handling settings:

[0052] For the priority of command execution, users can set which command should be responded to and executed first when multiple people issue commands at the same time. For example, it can be set to give priority to the driver, or the passenger in the front passenger seat; this helps to avoid confusion when multiple people operate at the same time. Users can also set that when there are multiple commands issued in sequence, the corresponding commands will be executed in chronological order.

[0053] For command conflict handling, users can set the default conflict handling rules to handle conflicts between multiple commands. The default conflict handling rule is to issue a voice reminder and not execute any conflicting commands. This design can prevent unexpected operations or system crashes caused by command conflicts.

[0054] It should be noted that in order to ensure the core functions of DMS and security monitoring needs, the above settings cannot conflict with the core functions of DMS.

[0055] After the setting is completed, the human-vehicle interaction robot is activated and turns to the driver to monitor the driver's status. It should be noted that after the vehicle was powered off last time, the human-vehicle interaction robot defaults to the center of the central control; after the vehicle is powered on, the human-vehicle interaction robot defaults to turning left and facing the driver. Of course, it can also be set that after the vehicle is powered on, the human-vehicle interaction robot turns right and faces the passenger in the front passenger seat.

[0056] The human-vehicle interaction robot monitors in real time whether there are voice commands or gesture commands. When a passenger, including the driver, wakes up the voice through the wake-up word, the human-vehicle interaction robot adjusts the steering direction to the current activated voice zone position (which needs to be calibrated in advance, such as facing left for the driver's call, facing right for the co-driver's call, and center for the rear seat call) according to the voice command, and then executes it, while providing voice feedback on the process; when a passenger, including the driver, makes a gesture command in front of the camera, the human-vehicle interaction robot can adjust the direction according to the gesture, and further recognize and execute the relevant gesture command.

[0057] When the human-vehicle interaction robot recognizes a voice command or a gesture command, it determines whether the current driver's state is normal, fatigued or distracted. It is understandable that DMS can determine the driver's state in a variety of ways. For example, DMS captures the driver's facial image through a camera and analyzes the driver's eye movements, blinking frequency, pupil changes, and yawning. If the driver frequently closes his eyes, blinks for too long, or yawns, it is determined that the driver may be in a state of fatigue. DMS monitors the driver's line of sight through a camera to determine whether he or she is not looking at the road ahead for a long time. If the driver frequently checks his or her mobile phone, adjusts the on-board equipment, or is distracted by observing other things in the car, distraction can be identified. When DMS does not detect the above fatigue or distraction, it can be determined that the driver is in a normal driving state.

[0058] When the driver is in a normal state (i.e., not fatigued or distracted), the human-vehicle interaction robot will process the recognized voice and gesture commands according to the following steps:

[0059] The human-vehicle interaction robot makes a preliminary judgment based on the order in which the voice commands and gesture commands are input.

[0060] If the voice command and gesture command are not input at the same time, the human-vehicle interaction robot will execute the commands in sequence according to the principle of "first input, first executed".

[0061] If the voice command and gesture command are input at the same time, the human-vehicle interaction robot first determines whether there is a clear driver command. The driver's command usually has a higher priority and is issued through specific gestures, voice keywords or preset shortcut keys. The human-vehicle interaction robot will immediately execute the driver's corresponding command. After executing the driver's command, the system will continue to process other commands.

[0062] If the driver's command is not detected in the simultaneously input voice command and gesture command, the human-vehicle interaction robot will further determine whether there is a conflict between the voice command and the gesture command. This conflict may occur when the two commands require the vehicle to perform completely different operations.

[0063] If there is a conflict, the human-vehicle interaction robot will handle it according to the preset conflict handling rules (such as the default voice reminder and not executing all conflicting commands). Specifically, the human-vehicle interaction robot will issue a voice reminder to inform the driver of the command conflict and suspend the execution of any conflicting commands to ensure driving safety.

[0064] After resolving the conflict, the human-vehicle interaction robot continues to execute other non-conflicting instructions, which will be executed according to the preset priority rules (i.e., high-priority instructions are executed first, and low-priority instructions are executed later).

[0065] When the driver is in a state of fatigue or distraction, the execution logic of the human-vehicle interaction robot will focus on ensuring driving safety, as follows:

[0066] First, the human-vehicle interaction robot reminds the driver through voice, clearly pointing out that the driver is currently in an unsafe state, and suggests that he suspend non-essential operations and concentrate on driving.

[0067] After issuing a reminder, the human-vehicle interaction robot will wait for the driver's confirmation response. This confirmation can be achieved through specific voice keywords, gestures, or confirmation buttons on the touch screen. This step is to ensure that the driver is aware of his or her status and is ready to take over or authorize the system to execute subsequent instructions.

[0068] After obtaining the driver's explicit confirmation, the human-vehicle interaction robot will process the received voice or gesture command according to the command execution logic when the driver is in a normal state. As mentioned above, the command execution logic includes executing the corresponding command according to the order in which the voice command and gesture command are input, and whether there is a conflict between the voice command and gesture command, according to the preset execution rules.

[0069] After the command is executed, the human-vehicle interaction robot returns to the last command issuing area. The last command issuing area refers to the area where the last recognized and executed command (voice command or gesture command) came from. It may be the driver's seat, the co-pilot seat, or any other area in the car that can issue valid commands.

[0070] The above process ensures that even when the driver is in an unsafe state, the human-vehicle interaction robot can accurately respond to and execute the driver's instructions while ensuring driving safety.

[0071] When the human-vehicle interaction robot does not recognize voice commands or gesture commands and does not receive a power-off signal, the driver monitoring function is maintained. If a power-off signal is received, the human-vehicle interaction robot returns to the power-off default position (center).

[0072] It should be noted that, in the embodiment of the present invention, the human-vehicle interactive robot can adjust the direction according to the source of the voice command, and the direction control can be completed by the DMS controller. The DMS controller can also be integrated in the AVNT host.

[0073] The embodiment of the present invention can also be used in a scenario where the user is far away from the vehicle, which is described in detail below.

[0074] Please refer to Figure 5 and Figure 6 As shown, the user first needs to log in to the account or obtain corresponding authorization in the mobile APP to ensure that only legitimate users can access and control the human-vehicle interactive robot, increasing security.

[0075] After logging in, the user enters the interface of the human-vehicle interaction robot. The user can choose to view the situation inside the car by remotely controlling the camera on the human-vehicle interaction robot. The user can also adjust the angle of the camera to see if there are any items or living things left in the car. During this process, the AVNT host sends the camera angle command issued by the user and the command to call the camera screen to the DMS. After receiving the command, the DMS controls the camera and feeds back the video signal to the user through the link of DMS-AVNT-CCU (Central Control Unit)-TBox (Telematics Box)-Car Cloud-Mobile APP.

[0076] In addition to checking the situation inside the car, users can also remotely control the camera and microphone to communicate with the inside of the car. Specifically, the AVNT host transmits the video signal fed back by the DMS and the received voice signal to the mobile phone APP synchronously. The mobile phone APP then feeds back the user's voice signal to the AVNT host and transmits it through the microphone, thus realizing two-way communication.

[0077] When the user exits the APP, the human-vehicle interaction robot will return to the center position and then automatically power off within the specified time.

[0078] If the camera of the human-vehicle interaction robot in the embodiment of the present invention is expanded, for example, a forward wide-angle camera is installed at the back, it can also be used for forward exterior monitoring of the vehicle after parking.

[0079] Corresponding to the vehicle human-computer interaction method described in the first embodiment of the present invention, the second embodiment of the present invention further provides a vehicle human-computer interaction device, including:

[0080] A recognition module, used to receive and recognize voice commands and gesture commands from passengers in the vehicle;

[0081] Monitoring module, used to monitor the driver's status in real time;

[0082] The control module is used to execute corresponding instructions according to the preset execution rules based on the order of voice instructions and gesture instructions input and whether there is a conflict when the driver is in a normal state; or when the driver is in a fatigued or distracted state, first execute the safety reminder instruction, and after obtaining the driver's confirmation, execute the corresponding instruction according to the instruction execution logic when the driver is in a normal state.

[0083] Corresponding to the vehicle human-computer interaction method described in the first embodiment of the present invention, the third embodiment of the present invention further provides a vehicle human-computer interaction device, including:

[0084] one or more processors;

[0085] Memory;

[0086] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the vehicle human-computer interaction method described in the aforementioned embodiment 1 of the present invention.

[0087] Corresponding to the vehicle human-computer interaction method described in the aforementioned embodiment of the present invention, embodiment 4 of the present invention also provides a computer program product, including computer instructions, and the computer instructions instruct a computer device to perform operations corresponding to the vehicle human-computer interaction method described in the aforementioned embodiment of the present invention.

[0088] Preferably, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the device, and various parts of the device are connected using various interfaces and lines.

[0089] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0090] It should be noted that the above-mentioned device may include but is not limited to a processor and a memory, which can be understood by those skilled in the art.

[0091] For the working principle and process of the above embodiment, please refer to the description of the above embodiment of the present invention, which will not be repeated here.

[0092] It can be seen from the above description that compared with the prior art, the beneficial effect of the present invention is that the present invention significantly improves driving safety and in-vehicle interactive experience through an innovative vehicle human-computer interaction method. By real-time monitoring of the driver's status, it can intelligently identify and prioritize the driver's voice and gesture commands, effectively reducing the risk of accidents caused by driver distraction or fatigue. In the event of a conflict in commands, the system makes decisions based on preset priority rules to ensure the rationality and smoothness of command execution. In addition, when the driver is fatigued or distracted, the system will give priority to issuing safety reminders to further ensure driving safety. Users can also remotely monitor the situation in the car through a mobile phone APP and realize two-way voice and video communication, which enhances the practicality and convenience of the vehicle, and also provides users with a safer, more comfortable and personalized driving environment.

[0093] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A vehicle human-computer interaction method, characterized in that: The following steps are involved: Receive and recognize voice and gesture commands from passengers in the vehicle; Real-time monitoring of driver status; If the driver is in a normal state, the corresponding command will be executed according to the preset execution rules based on the order in which the voice command and gesture command are input and whether there is a conflict; If the driver is in a state of fatigue or distraction, the safety reminder command will be executed first, and after obtaining the driver's confirmation, the corresponding command will be executed according to the command execution logic when the driver is in a normal state.

2. The method according to claim 1, characterized in that If the driver is in a normal state, the corresponding command is executed according to the sequence of input of the voice command and the gesture command and whether there is a conflict, according to the preset execution rules, specifically including: If the voice command and gesture command are not input at the same time, they will be executed in the order in which they were input; If the voice command and gesture command are input at the same time, it is first determined whether there is a driver command in the voice command and the gesture command. If so, the driver command is executed first; if not, it is further determined whether there is a conflict between the voice command and the gesture command, and the conflict is handled according to the preset conflict handling rules.

3. The method according to claim 2, characterized in that The determining whether there is a conflict between the voice command and the gesture command and processing the conflict according to a preset conflict handling rule specifically includes: If there is a conflict between a voice command and a gesture command, the driver will be reminded by voice and any conflicting command will be suspended; then other non-conflicting commands will continue to be executed according to the preset priority rules; If there is no conflict between the voice command and the gesture command, the corresponding command is executed according to the preset priority rules.

4. The method according to claim 3, characterized in that The preset priority rules include driver's instructions first, front passenger seat passenger's instructions first, or rear passenger's instructions first.

5. The method according to claim 1, characterized in that Also includes: If no voice command or gesture command is recognized and no power-off signal is received, the driver's status is maintained in real time.

6. The method according to claim 1, characterized in that Also includes: According to the camera angle instructions and camera image calling instructions issued remotely by the user, the camera is controlled to capture the interior environment of the vehicle, and the captured video signal is fed back to the user through a communication link.

7. The method according to claim 1, characterized in that Also includes: According to the call commands issued remotely by the user, the video signal and voice signal in the car are transmitted to the user synchronously, and then the user's voice signal is received and transmitted through the microphone to realize two-way communication.

8. A vehicle human-machine interaction device, characterized in that: include: A recognition module, used to receive and recognize voice commands and gesture commands from passengers in the vehicle; Monitoring module, used to monitor the driver's status in real time; The control module is used to execute corresponding instructions according to the preset execution rules based on the order of voice instructions and gesture instructions input and whether there is a conflict when the driver is in a normal state; or when the driver is in a fatigued or distracted state, first execute the safety reminder instruction, and after obtaining the driver's confirmation, execute the corresponding instruction according to the instruction execution logic when the driver is in a normal state.

9. A vehicle human-machine interaction device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the vehicle human-computer interaction method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions instruct a computer device to execute operations corresponding to the method according to any one of claims 1 to 7.

Images