Multi-mode man-machine interaction system for automatic driving intelligent cabin

Abstract

The invention provides a multi-mode man-machine interaction system for an automatic driving intelligent cabin, and relates to the technical field of intelligent cabins. The multi-mode man-machine interaction system for the automatic driving intelligent cabin comprises a voice module, a human face module, a key touch control module and an intelligent control module, the voice module comprises a voice recognition module, a voice judgment module and a voice operation module, the voice module is used for a natural language processing technology, and efficient and safe communication and manual-free operation between a driver and a vehicle are achieved. The method comprises the following steps: recording and recognizing lip language of a lip through a camera, sending recognized signals to a voice recognition module in a same interaction manner, comparing with an expanded low-resource dialect data acquisition module through a lip language and dialect comparison module, and sending an instruction to a voice operation module when the same signals are reached; and when the identification signal is not reached, the operation is terminated, thereby preventing false instruction operation from influencing driving safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit technology, specifically to a multimodal human-machine interaction system for autonomous driving intelligent cockpits. Background Technology

[0002] The intelligent cockpit aims to integrate various IT and artificial intelligence technologies to create a brand-new integrated digital platform within the vehicle, providing drivers with an intelligent experience and promoting driving safety. Currently, there is considerable research progress both domestically and internationally, such as installing cameras on the A and B pillars and rearview mirrors of vehicles to provide functions like emotion recognition, age detection, abandoned object detection, and seatbelt detection.

[0003] Human-computer interaction (HCI) is a discipline that studies the interactive relationship between a system and its users. A system can be various types of machines, as well as computerized systems and software. The HCI typically refers to the part visible to the user. Users interact with and operate the system through the HCI. Examples range from small things like a radio's play button to large things like an airplane's instrument panel or a power plant's control room.

[0004] Existing technologies have seamlessly integrated smart cockpits and human-machine interaction with vehicle intelligence, achieving greater intelligence. However, existing smart cockpit human-machine interaction still has limitations. Firstly, middle-aged and older users may have heavy accents when interacting with the system, making it difficult to communicate smoothly in standard Mandarin. Secondly, the commands executed by the system are relatively simple, and the adapted instructions often do not match the user's needs, which can easily distract the user and lead to incorrect commands, affecting driving safety. Therefore, further improvements are needed to address these issues.

[0005] When applying for this invention, the applicant, through a search, discovered a Chinese patent disclosing a "Multimodal Human-Machine Interaction System for Autonomous Driving Intelligent Cockpits," application number "202111528535.2." This patent primarily uses a configured distance reminder strategy to determine whether the driver's viewing distance is too close to the display screen. If the distance is too close, a reminder voice is emitted through a speaker to maintain a safe distance, preventing nearsightedness in drivers and passengers while watching movies, thereby improving the driver's user experience in the cockpit. However, incorrect operations during vehicle-machine interaction affect the interaction with the vehicle-machine system, making it impossible to successfully execute designated functions. Therefore, based on the applicant's invention, a multimodal human-machine interaction system for autonomous driving intelligent cockpits has been invented. This system solves the problems of relatively simple execution commands in vehicle-machine systems, frequent inconsistencies between the adapted instructions and the user's required instructions, which can easily cause user distraction during driving and affect smooth and safe driving due to incorrect instructions. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a multimodal human-machine interaction system for intelligent cockpits in autonomous driving. This system solves the problems of relatively simple execution commands in vehicle systems, frequent inconsistencies between the adapted instructions and the user's required instructions, which can easily distract the user and cause erroneous instructions to affect smooth and safe driving.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution: a multimodal human-computer interaction system for autonomous driving intelligent cockpits, comprising: a voice module, a face module, a button touch module, and an intelligent control module;

[0010] The voice module includes a voice recognition module, a voice judgment module, and a voice operation module. The voice module uses natural language processing technology to achieve efficient and safe communication between the driver and the vehicle and to enable manual operation without manual intervention.

[0011] The face module includes a face analysis module, a face judgment module, and a face operation module. When the driver sits on the seat assembly, the face module is used for biometric authentication and real-time status monitoring, which significantly improves driving safety and personalized experience, while also enabling keyless start and access management.

[0012] The intelligent control module includes a camera-lidar recognition module, a harsh environment recording module, a recognition and comparison module, and an intelligent operation module. The intelligent control module is used for multimodal interaction and AI decision-making, realizing a leapfrog upgrade from passive response to proactive service.

[0013] As a preferred technical solution of the present invention, the speech recognition module further includes a lip visual analysis module and an expanded low-resource dialect data acquisition module. The lip visual analysis module captures the driver's lip movements and micro-expressions in real time, realizing an interactive upgrade from passive response to active prediction, and can prevent the influence of accents from causing misjudgments in voice operation during human-computer interaction.

[0014] As a preferred technical solution of the present invention, the expanded low-resource dialect data acquisition module systematically records the speech and text information of rare languages ​​or dialects to provide underlying support for human-computer interaction technology. It first uses a small amount of labeled data to train a basic model, and then uses the model prediction results to select high-value samples for manual labeling, and iteratively optimizes the dataset.

[0015] As a preferred technical solution of the present invention, the button touch module includes a signal receiving module. The voice operation module, face operation module, signal receiving module and intelligent operation module send signals to the controller module through the signal transmission module. The controller module can then send control signals to the corresponding power controllers based on the received signals.

[0016] As a preferred technical solution of the present invention, the input terminals of the voice module, face module, button touch module and intelligent control module are all input by the user to send operation commands, thereby completing the corresponding operation of human-computer interaction.

[0017] As a preferred technical solution of the present invention, the speech recognition module transmits the received speech signals to the expanded low-resource dialect data acquisition module for comparison, and then quickly recognizes the corresponding dialect signal with the received speech signal before returning it to the speech recognition module.

[0018] As a preferred technical solution of the present invention, the lip visual analysis module records and recognizes lip movements using a camera, and sends the recognized signal to the speech recognition module. The lip movement and dialect comparison module compares the signal with the extended low-resource dialect data acquisition module. If the signal matches, the speech operation module can send a command. If the signal does not match, the operation will be terminated to prevent erroneous commands from affecting driving safety.

[0019] (III) Beneficial Effects

[0020] This invention provides a multimodal human-machine interaction system for autonomous driving intelligent cockpits. It has the following beneficial effects:

[0021] 1. This invention incorporates a lip visual analysis module, an expanded low-resource dialect data acquisition module, and a lip-reading and dialect comparison module. A camera records and recognizes lip movements, and the recognized signal is sent to a speech recognition module. This module compares the lip-reading and dialect comparison signal with the expanded low-resource dialect data acquisition module. If a match is found, a command is sent to the speech control module; otherwise, the operation terminates, preventing erroneous commands from affecting driving safety. This structure enables multimodal human-machine interaction, improving accuracy and enhancing cockpit intelligence.

[0022] 2. This invention features a lip visual analysis module that captures the driver's lip movements and micro-expressions in real time, achieving an interactive upgrade from passive response to active prediction. This prevents misjudgments of voice operations caused by accents during human-computer interaction. The structure can collect and analyze lip information, reducing analysis time when the same operation occurs again.

[0023] 3. This invention includes an expanded low-resource dialect data acquisition module, which systematically records the speech and text information of rare languages ​​or dialects to provide underlying support for human-computer interaction technology. It first trains a basic model using a small amount of labeled data, then selects high-value samples for manual labeling based on the model's prediction results, and iteratively optimizes the dataset. This structure completes the storage of dialect data information and reduces the limitations of existing intelligence. Attached Figure Description

[0024] Figure 1 This is an interaction flowchart of a multimodal human-computer interaction system for an autonomous driving intelligent cockpit proposed in this invention.

[0025] Figure 2 This is a flowchart illustrating the analysis and processing of a multimodal human-computer interaction system for an autonomous driving intelligent cockpit proposed in this invention.

[0026] The system comprises: 1. User; 2. Voice module; 201. Voice recognition module; 2011. Lip visual analysis module; 2012. Expanded low-resource dialect data acquisition module; 202. Voice judgment module; 203. Voice operation module; 3. Face module; 301. Face analysis module; 302. Face judgment module; 303. Face operation module; 4. Button touch module; 401. Signal receiving module; 5. Intelligent operation module; 501. Camera LiDAR recognition module; 502. Recognition and comparison module; 503. Recording harsh environments; 504. Intelligent operation module; 6. Signal transmission module; 7. Controller module. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1:

[0029] like Figure 1-2 As shown, this embodiment of the invention provides a multimodal human-machine interaction system for autonomous driving intelligent cockpits, including: a voice module 2;

[0030] Voice module 2 includes a voice recognition module 201, a voice judgment module 202, and a voice operation module 203. Voice module 2 utilizes natural language processing technology to achieve efficient and safe communication between the driver and the vehicle, enabling manual operation without manual intervention. Voice recognition module 201 also includes a lip visual analysis module 2011 and an expanded low-resource dialect data acquisition module 2012. The lip visual analysis module 2011 captures the driver's lip movements and micro-expressions in real time, achieving an interactive upgrade from passive response to active prediction. This prevents misjudgments of voice operations due to accent interference during human-computer interaction. The expanded low-resource dialect data acquisition module 2012 systematically records the speech and text information of rare languages ​​or dialects, providing underlying support for human-computer interaction technology. Initially, a small amount of labeled data is used for training. The system uses a basic model, then selects high-value samples based on the model prediction results for manual annotation, and iteratively optimizes the dataset. The speech recognition module 201 transmits the received speech signals to the expanded low-resource dialect data acquisition module 2012 for comparison. The corresponding dialect signal is quickly recognized by the received speech signal and then transmitted back to the speech recognition module 201. The lip visual analysis module 2011 records and recognizes lip movements using a camera, and sends the recognized signal back to the speech recognition module 201. The lip movement and dialect comparison module 2012 compare the signals with the expanded low-resource dialect data acquisition module 2012. If the signals match, the system sends a command to the speech operation module 203. If the recognition signal is not found, the operation is terminated to prevent erroneous commands from affecting driving safety.

[0031] Example 2:

[0032] The difference between this embodiment and Embodiment 1 is that: a multimodal human-computer interaction system for autonomous driving intelligent cockpits includes a voice module 2, a face module 3, a button touch module 4, and an intelligent control module 5;

[0033] Voice module 2 includes a voice recognition module 201, a voice judgment module 202, and a voice operation module 203. Voice module 2 utilizes natural language processing technology to achieve efficient and safe communication between the driver and the vehicle, enabling manual operation without manual intervention. Voice recognition module 201 also includes a lip visual analysis module 2011 and an expanded low-resource dialect data acquisition module 2012. The lip visual analysis module 2011 captures the driver's lip movements and micro-expressions in real time, achieving an interactive upgrade from passive response to active prediction. This prevents misjudgments of voice operations due to accent interference during human-computer interaction. The expanded low-resource dialect data acquisition module 2012 systematically records the speech and text information of rare languages ​​or dialects, providing underlying support for human-computer interaction technology. Initially, a small amount of labeled data is used for training. The system uses a basic model, then selects high-value samples based on the model prediction results for manual annotation, iteratively optimizes the dataset, and the speech recognition module 201 transmits the received speech signals to the expanded low-resource dialect data acquisition module 2012 for comparison. The corresponding dialect signal is quickly recognized by the received speech signal and then transmitted back to the speech recognition module 201. The lip visual analysis module 2011 records and recognizes the lip movements of the lips through a camera, and sends the recognized signal back to the speech recognition module 201. The lip movement and dialect comparison module 2012 compare the signals with the expanded low-resource dialect data acquisition module 2012. If the signals are the same, the system can send a command to the voice operation module 203. If the recognition signal is not reached, the operation will be terminated to prevent erroneous commands from affecting driving safety.

[0034] The face module 3 includes a face analysis module 301, a face judgment module 302, and a face operation module 303. When the driver sits on the seat assembly, the face module 3 is used to perform biometric authentication and real-time status monitoring, which significantly improves driving safety and personalized experience, while also achieving keyless start and access management.

[0035] The intelligent control module 5 includes a camera-LiDAR recognition module 501, a harsh environment recording module 503, a recognition and comparison module 502, and an intelligent operation module 504. The intelligent control module 5 is used for multimodal interaction and AI decision-making, realizing a leapfrog upgrade from passive response to proactive service. By recording harsh environments 503, the system can record the vehicle's simulated harsh environment scenarios. When the vehicle drives into a harsh environment, the camera-LiDAR recognition module 501 can complete the comparison and recognition of the harsh environment, which can prevent the harsh environment from affecting driving safety.

[0036] The button touch module 4 includes a signal receiving module 401, a voice operation module 203, a face operation module 303, and an intelligent operation module 504. The signal receiving module and the intelligent operation module 504 send signals to the controller module 7 through the signal transmission module. The controller module 7 can send control signals to the corresponding power controllers through the received signals. The input terminals of the voice module 2, face module 3, button touch module 4 and intelligent control module 5 are all input by the user 1 to send operation commands, thereby completing the corresponding operation of human-computer interaction.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-modal human-machine interaction system for an autonomous driving intelligent cockpit, characterized in that: Including voice module (2), face module (3), key touch module (4) and intelligent control module (5); The voice module (2) includes a voice recognition module (201), a voice judgment module (202) and a voice operation module (203), and the voice module (2) uses natural language processing technology to realize efficient and safe communication between the driver and the vehicle and hands-free operation. The face module (3) includes a face analysis module (301), a face judgment module (302) and a face operation module (303), and when the driver sits on the seat assembly, the face module (3) uses biometric authentication and real-time state monitoring to significantly improve driving safety and personalized experience while achieving keyless start and permission management. The intelligent control module (5) includes a camera laser radar recognition module (501), an entry into a harsh environment (503), an identification comparison module (502) and an intelligent operation module (504), and the intelligent control module (5) uses multi-modal interaction and AI decision to realize a leap from passive response to active service.

2. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 1, characterized in that: The voice recognition module (201) further includes a lip visual analysis module (2011) and an expanded low-resource dialect data collection module (2012), the lip visual analysis module (2011) captures the driver's lip movement and micro-expression in real time, realizes the interactive upgrade from passive response to active prediction, and can prevent the influence of accent and misjudgment of voice operation during human-computer interaction.

3. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 2, characterized in that: The expanded low-resource dialect data collection module (2012) systematically records the voice and text information of rare languages or dialects to provide bottom support for human-computer interaction technology, first trains a basic model using a small amount of labeled data, then selects high-value samples for manual annotation through model prediction results, and iteratively optimizes the data set.

4. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 1, characterized in that: The key touch module (4) includes a signal receiving module (401), the voice operation module (203), the face operation module (303), the signal receiving module and the intelligent operation module (504) transmit signals to the controller module (7) through a signal transmission module, and the controller module (7) can control the corresponding power controller based on the received signals.

5. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 1, characterized in that: The input ends of the voice module (2), the face module (3), the key touch module (4) and the intelligent control module (5) are input by the user (1) to send operation instructions, thereby completing corresponding operations of human-computer interaction.

6. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 2, characterized in that: The voice recognition module (201) transmits the received voice signal to the expanded low-resource dialect data collection module (2012) for comparison, and returns the corresponding dialect signal and the received voice signal to the voice recognition module (201) after rapid recognition.

7. The multi-modal human-machine interaction system for an autonomous driving intelligent cockpit according to claim 6, characterized in that: The lip visual analysis module (2011) records and recognizes lip language of the lip through a camera, and sends the recognized signal to the speech recognition module (201). Compared with the lip language and dialect comparison module, the extended low-resource dialect data collection module (2012) reaches the same signal and sends an instruction to the speech operation module (203). When the recognized signal is not reached, the operation is terminated to prevent misinstruction operation from affecting driving safety.

Images