Safety control method and device of intelligent cockpit voice interaction system

Abstract

The invention discloses a safety control method and device for an intelligent cockpit voice interaction system. The method comprises the steps that vehicle dynamic information and driver dynamic information are acquired; judging whether the driver is in a fatigue or sleep state or not according to the dynamic information of the driver so as to evaluate the dangerous state of the driver; according to the vehicle dynamic information, the state of the vehicle and whether dangerous conditions exist around are detected, and therefore vehicle dangerous state evaluation is achieved; according to the driver and vehicle dangerous state evaluation result, the intelligent cockpit voice interaction system is switched to a corresponding working mode, the working mode comprises a common working mode, a safe working mode and a dangerous environment mode, in the common working mode, the vehicle receives all voice commands and executes corresponding operation, and in the dangerous environment mode, the vehicle receives all the voice commands; in the safe working mode, the vehicle receives a voice command of a registered person and executes corresponding operation, and in the dangerous environment mode, the vehicle receives physical key triggering and executes corresponding operation in cooperation with a voice command of a driver.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent cockpit voice safety and intelligent driving assistance, and in particular relates to a safety control method and device for an intelligent cockpit voice interaction system. Background Art

[0002] With the development of vehicle intelligence and the introduction of the concept of smart cockpit, voice interaction system has become an important way for drivers to interact with vehicles. However, the current traditional voice control system usually relies only on simple voice recognition and fixed command sets. This system cannot dynamically adjust according to the real-time changing driving environment and driver status. For example, when the driver is resting in the car, the traditional system cannot adjust the voice interaction mode in time. And when responding to emergencies, complex voice commands may not be as direct and reliable as physical buttons, further reducing the safety and reliability of the system. Summary of the invention

[0003] In view of the problems existing in the prior art, the purpose of the embodiments of the present application is to provide a safety control method and device for an intelligent cockpit voice interaction system, which aims to automatically switch the working modes of the intelligent cockpit and voice system according to the vehicle and driver status, and enhance the safety of the intelligent cockpit voice interaction system.

[0004] According to a first aspect of an embodiment of the present application, a safety control method for an intelligent cockpit voice interaction system is provided, comprising:

[0005] Acquiring vehicle dynamic information and driver dynamic information, wherein the vehicle dynamic information includes vehicle speed, GPS data, and IMU data, and the driver dynamic information is a driver's facial video;

[0006] According to the driver's dynamic information, determine whether the driver is in a fatigue or sleep state, thereby achieving a driver's dangerous state assessment;

[0007] According to the vehicle dynamic information, the vehicle status and whether there are any dangerous situations around it are detected, so as to realize the dangerous status assessment of the vehicle;

[0008] According to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, the intelligent cockpit voice interaction system is switched to the corresponding working mode, wherein the working modes include normal working mode, safe working mode and dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations. In the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations. In the dangerous environment mode, the vehicle accepts physical button triggers and cooperates with the driver's voice commands to perform corresponding operations.

[0009] Further, judging whether the driver is in a fatigue or sleep state based on the driver dynamic information, thereby evaluating whether the driver is in a dangerous state, includes:

[0010] Preprocessing the images of each frame in the driver's facial video;

[0011] Based on the preprocessed images, the rPPG algorithm is used to extract the driver's physiological information;

[0012] The preprocessed image is input into the fatigue state prediction model, and combined with the driver's physiological information, it is determined whether the driver is in a fatigue or sleep state. If the driver is in a fatigue or sleep state, it is determined that the driver is in a dangerous state.

[0013] Furthermore, based on the vehicle dynamic information, detecting the state of the vehicle and whether there is a dangerous situation around it includes:

[0014] Based on the vehicle speed, GPS data and IMU data, determining whether the vehicle is in a driving state or a parking state;

[0015] Detects whether there is an advanced voice command that a non-administrator user is using to control the vehicle. If so, determines that there is a dangerous situation around the vehicle.

[0016] Furthermore, the voice commands are classified into high-level voice commands and low-level voice commands according to the degree of safety sensitivity. The high-level voice commands include basic driving control, body control, and driving assistance function control; the low-level voice commands include comfort control, entertainment system control, and in-vehicle interaction control.

[0017] Furthermore, the process of determining whether the user who issues the voice command is an administrator-authorized user includes:

[0018] Get the multi-channel audio stream collected by the microphone array inside the vehicle;

[0019] Preprocessing the multi-channel audio stream, including noise filtering and signal enhancement;

[0020] The preprocessed multi-channel audio stream is input into the pre-trained ECAPA-TDNN model to extract voiceprint features;

[0021] The extracted voiceprint features are matched with the registered voiceprint features stored in the voiceprint database. If the similarity exceeds a predetermined similarity threshold, the match is successful, that is, the user who issued the voice command is registered, otherwise, the user who issued the voice command is not registered;

[0022] If the user who issues the voice command has been registered, it is determined whether he is an administrator user based on the corresponding information stored in the voiceprint database.

[0023] Furthermore, according to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, the intelligent cockpit voice system is switched to a corresponding working mode, specifically:

[0024] When the vehicle is not in motion, the driver is not in danger, and there is no danger around the vehicle, the intelligent cockpit voice interaction system is in normal working mode;

[0025] If the driver is detected to be in a dangerous state or not in the driving seat when the vehicle is in driving or parking state, the intelligent cockpit voice interaction system switches to a safe working mode;

[0026] If there is a dangerous situation around the vehicle, the intelligent cockpit voice interaction system will switch to the dangerous environment mode.

[0027] According to a second aspect of an embodiment of the present application, there is provided a safety control device for an intelligent cockpit voice interaction system, comprising:

[0028] An acquisition module, used to acquire vehicle dynamic information and driver dynamic information, wherein the vehicle dynamic information includes vehicle speed, GPS data, and IMU data, and the driver dynamic information is a driver's facial video;

[0029] A driver danger state assessment module, used to determine whether the driver is in a fatigue or sleep state based on the driver dynamic information, thereby implementing driver danger state assessment;

[0030] A vehicle dangerous state assessment module is used to detect the state of the vehicle and whether there are dangerous situations around it according to the vehicle dynamic information, so as to achieve vehicle dangerous state assessment;

[0031] The working mode switching module is used to switch the intelligent cockpit voice interaction system to the corresponding working mode according to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, wherein the working modes include normal working mode, safe working mode and dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations; in the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations; in the dangerous environment mode, the vehicle accepts physical button triggers and performs corresponding operations in conjunction with the driver's voice commands.

[0032] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0033] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0034] one or more processors;

[0035] A memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0037] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0038] The present invention includes the following key links:

[0039] 1. Driver and vehicle status assessment: Based on sensor and camera data, the current status of the driver and vehicle is analyzed and assessed, including detecting whether the driver is fatigued or asleep, and whether there are possible dangerous factors around the vehicle.

[0040] 2. Deep learning-based perception system: This system uses the evaluation results of the driver and vehicle status to switch the smart cockpit working mode in real time according to the driver and vehicle status. Through this perception technology, while ensuring the convenience of voice control, it can respond to various emergencies in a timely manner to protect the driver's safety.

[0041] 3. Voice control system: In normal working mode, people in the car can directly control the car through voice commands; in safe working mode, voice recognition will be performed in conjunction with voiceprint recognition and voice command classification; when in dangerous environment mode, it supports the use of physical buttons in conjunction with voice commands to control the vehicle to ensure the safety of the driver.

[0042] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0043] It can be seen from the above embodiments that the present application utilizes on-board sensors, cameras, microphone arrays and other equipment to obtain dynamic information of the vehicle and the driver, and combines deep learning-based perception technology to evaluate the driver's risk status in real time. It can promptly detect when the driver is in a dangerous state or the vehicle is in a dangerous environment, thereby automatically switching the smart cockpit system to the most suitable working mode and adjusting the voice control method, which can effectively avoid interference from others and potential dangerous factors, ensure the safety and efficiency of operation, and thus improve the overall driving experience and driving safety.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 The present invention is a flowchart of a safety control method of an intelligent cockpit voice interaction system according to an exemplary embodiment.

[0047] Figure 2 The figure is a schematic diagram of voice command classification according to an exemplary embodiment.

[0048] Figure 3 The figure is a schematic diagram of a voiceprint recognition process according to an exemplary embodiment.

[0049] Figure 4 It is a block diagram of a safety control device of an intelligent cockpit voice interaction system according to an exemplary embodiment.

[0050] Figure 5 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0052] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0053] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0054] like Figure 1 As shown, the present application provides a safety control method for an intelligent cockpit voice interaction system, comprising:

[0055] (1) Obtain vehicle dynamic information and driver dynamic information;

[0056] Specifically, the vehicle dynamic information obtained through vehicle sensors includes vehicle speed, GPS data, IMU data, etc., and the driver dynamic information is the driver's facial video obtained through the in-vehicle camera, etc.

[0057] (2) judging whether the driver is in a fatigue or sleep state based on the driver's dynamic information, thereby realizing the driver's dangerous state assessment;

[0058] The driver's behavior is detected based on vision, including whether there is fatigue driving, whether the driver is in a sleep state, etc. Real-time and effectiveness are important indicators of driver behavior detection. Real-time requires the ability to monitor these dynamic information in real time, so that warnings can be issued immediately or corresponding safety measures can be taken when the driver is in a dangerous state to avoid possible traffic accidents. At the same time, effectiveness requires the ability to accurately identify the driver's true state, effectively avoid false alarms and missed alarms, and accurately judge the driver's dangerous state, so as to trigger corresponding warnings or intervention measures when necessary to ensure driving safety, specifically including: preprocessing the image of each frame in the driver's facial video; extracting the driver's physiological information based on the preprocessed image using the rPPG algorithm; inputting the preprocessed image into the fatigue state prediction model, combining the driver's physiological information to determine whether the driver is in a fatigue or sleep state, and if the driver is in a fatigue or sleep state, the driver is determined to be in a dangerous state.

[0059] In a specific implementation, a CNN-based target detection method such as YOLOv4, YOLOv3, EfficientDet, Faster-RCNN, RetinaNet, R-FCN and other algorithms can be used to detect driver fatigue or sleep. In this embodiment, the YOLOv5 algorithm, which is more efficient and suitable for single GPU training, is selected. Before detecting the driver's facial image using the YOLOv5 algorithm, the driver's facial image needs to be normalized. The normalization adopts the z-score normalization method based on the mean and standard deviation of the original data. The processed data conforms to the standard normal distribution, and the conversion function is:

[0060]

[0061] In the formula, is the normalized driver facial image data value, x is the original driver facial image data value, μ is the mean value of all driver facial image data, is the standard deviation of all driver facial image data.

[0062] rPPG is a non-contact physiological signal detection technology that uses ordinary cameras to capture skin color changes to extract physiological signals such as heart rate, respiratory rate and heart rate variability. rPPG technology detects tiny color changes related to the heart cycle by analyzing changes in the intensity of light reflected by the skin. These changes can be extracted through image processing and signal processing techniques for monitoring cardiovascular activity. In this application, the facial video of the driver to be detected is simultaneously fed into the Yolov5 model and the rPPG model. The video length of the input model should be at least 2 seconds. In the specific implementation, a 10-15 second clip can be taken. The physiological information (such as heart rate, respiratory rate, heart rate variability, etc.) extracted from the video by the rPPG model is used to assist in determining whether the driver is in a state of sleep or fatigue.

[0063] In one embodiment, the algorithm fusion process is as follows: 1) perform image preprocessing operations such as cropping, scaling, median filtering, etc. on the collected driver behavior image frame sequence and use the above-mentioned z-score algorithm to normalize the image pixels to obtain a 128 pixel × 128 pixel image; 2) input the preprocessed video into the rPPG algorithm to obtain the driver's heart rate, respiratory rate and heart rate variability; 3) input the preprocessed image into the Yolov5 model to obtain whether the driver is in a state of yawning, closing eyes, etc., and combine the results of the rPPG algorithm to determine whether the driver is in a state of fatigue or sleep; specifically, if it is detected that the driver is in a yawning state for a period of time exceeding the first predetermined threshold and the heart rate variability is detected to decrease, the driver is judged to be in a fatigue state; if it is monitored that the driver is in a closed eye state for a period of time exceeding the second predetermined threshold and the heart rate and respiratory rate decrease, the driver is judged to be in a sleep stage; if in a fatigue or sleep stage, the driver is judged to be in a dangerous state at this time. It should be noted that the first predetermined threshold and the second predetermined threshold are set according to actual conditions, and this application does not limit them.

[0064] In this embodiment, the Yolov5 model needs to be pre-trained in advance, using the CSPDarknet53 feature extractor pre-trained on the COCO dataset.

[0065] (3) Detecting the state of the vehicle and whether there are any dangerous situations around it based on the vehicle dynamic information, thereby realizing the dangerous state assessment of the vehicle;

[0066] Specifically, the vehicle status and surrounding dangerous behaviors include: whether the vehicle is in a driving state, dangerous behaviors such as attempts by non-administrator privilege users to use advanced voice commands, etc.;

[0067] (3.1) Based on the vehicle speed, GPS data and IMU data, the movement trajectory and acceleration change of the vehicle are monitored, and when the acceleration and position change or the vehicle speed is greater than 0, indicating that the vehicle is moving, it is determined that the vehicle is in a driving state.

[0068] (3.2) Detect whether there is a non-administrator user using voice commands to control the vehicle's advanced voice commands, and if so, determine that there is a dangerous situation around the vehicle;

[0069] like Figure 2 As shown in the figure, voice commands are classified into high-level voice commands and low-level voice commands according to their security sensitivity; high-level voice commands include basic driving control (such as starting / stopping the engine, setting cruise control, etc.), body control (such as controlling door and window switches, etc.) and driving assistance function control (such as parking assistance, lane keeping, etc.); low-level voice commands include comfort control (such as air conditioning adjustment, non-driving seat adjustment, etc.), entertainment system control (such as multimedia device control, etc.) and in-car interaction control (such as controlling the interior ambient light, using voice assistant, etc.). Only those who have registered through voiceprint in advance and are set as administrators can use high-level voice commands, while other people can only use low-level voice commands; voiceprint recognition uses a method based on time delay neural network to recognize voiceprints and determine whether the user who issues the voice command is an administrator user, such as Figure 3 As shown, the details are as follows:

[0070] (3.2.1) Obtain multi-channel audio stream collected by the microphone array inside the vehicle;

[0071] Microphone arrays are installed in appropriate locations in the vehicle (such as the roof, near the seat headrests, etc.) to ensure that the voices of the driver and passengers can be clearly captured. The microphone array collects sound data inside the vehicle in real time and generates a multi-channel audio stream.

[0072] (3.2.2) preprocessing the multi-channel audio stream, including noise filtering and signal enhancement;

[0073] The collected multi-channel audio data is preprocessed, including noise filtering and signal enhancement. Specifically, a bandpass filter is used to remove background noise in non-target frequency bands and retain the sound signal in the target frequency band. The parameters of the bandpass filter should be adjusted according to the internal noise characteristics of the vehicle and the target sound frequency range; further noise suppression algorithms, such as spectral subtraction or adaptive noise suppression, are applied to reduce background noise interference and improve signal clarity; the AGC algorithm is used to dynamically adjust the gain according to the strength of the real-time audio signal to enhance the signal strength of the target sound source and ensure that the signal can be clearly identified at different volume levels; further signal enhancement algorithms, such as beamforming technology, are applied to improve the signal-to-noise ratio of the target sound source by focusing the signals collected by multiple microphones.

[0074] (3.2.3) Input the preprocessed multi-channel audio stream into the pre-trained ECAPA-TDNN model to extract voiceprint features;

[0075] The preprocessed audio data is input into the pre-trained ECAPA-TDNN model. The ECAPA-TDNN model extracts voiceprint features from audio signals through deep convolutional neural networks (CNN) and time delay neural networks (TDNN). Feature extraction involves capturing the time-frequency characteristics of audio signals and generating high-dimensional feature vectors.

[0076] (3.2.4) Matching the extracted voiceprint features with the registered voiceprint features stored in the voiceprint database. If the similarity exceeds a predetermined similarity threshold, the match is successful, i.e., the user who issued the voice command is registered; otherwise, the user who issued the voice command is not registered;

[0077] The extracted voiceprint features are matched with the pre-registered voiceprint database, where the voiceprint database can set the permission level of the voiceprint. The driver can be set as the administrator in advance to ensure that advanced voice commands can be executed. The similarity between voiceprint features is calculated using methods such as cosine similarity or Euclidean distance. According to the similarity score, the matching degree of the voiceprint features is determined. If the similarity exceeds the predetermined similarity threshold, it is determined that the match is successful, that is, the user who issued the voice command is registered, otherwise the user who issued the voice command is not registered;

[0078] (3.2.5) If the user who issues the voice command has been registered, determine whether he or she is an administrator user based on the corresponding information stored in the voiceprint database.

[0079] (4) Switching the intelligent cockpit voice interaction system to a corresponding working mode according to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, wherein the working modes include a normal working mode, a safe working mode and a dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations. In the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations. In the dangerous environment mode, the vehicle accepts physical button triggers and performs corresponding operations in conjunction with the driver's voice commands.

[0080] Specifically, when the vehicle is not in driving, the driver is not in sleeping state and the surroundings of the vehicle are safe, the vehicle should be in normal working mode and able to receive voice commands normally; once the vehicle is in driving state or detects that the driver is in a dangerous state when parked or does not detect that the driver is in the driving seat when parked, the smart cockpit will automatically switch to the safe working mode, where whether the driver is in the driving seat can be determined by the Yolov5 algorithm or the pressure sensor on the driving seat; if it is detected that a person without registered administrator privileges attempts to use voice commands to control advanced security level instructions, it will be forced to enter the dangerous environment mode and trigger the external alarm of the entire vehicle.

[0081] In different smart cockpit working modes, the smart cockpit voice interaction system switches the corresponding voice control mode according to the current working mode:

[0082] Normal working mode, the vehicle can normally accept voice commands and perform corresponding operations.

[0083] In the secure working mode, steps (3.2.1) to (3.2.5) are used to determine whether the voiceprint of the voice command has been registered. If it has been registered, its permission level (administrator permission or general permission) is also determined. After successful identity authentication, the system will grant the corresponding operation permission and execute the voice command issued by the user; after failed identity authentication, the system will refuse to execute the voice command and may issue a warning prompt.

[0084] In dangerous environment mode, the vehicle will block all voice commands to ensure that the vehicle cannot be controlled by other people, improving the safety of the driver in dangerous conditions; only when the driver presses and holds the physical button and passes the voiceprint recognition with administrator privileges, the vehicle will perform the corresponding operation according to the voice command.

[0085] The voiceprint recognition in the above steps can monitor the audio information collected by the microphone array in real time and perform recognition and analysis. Only the voice characteristics of drivers registered in advance can be recognized, and other unregistered voices cannot pass the voiceprint recognition system, and different personnel are given control permissions by judging the permission level. The benefits of this design include: improving security, preventing external interference, using voiceprint recognition technology, and only allowing the smart cockpit voice interaction system to execute advanced voice commands issued by registered personnel with administrator privileges, thereby avoiding the impact of interference from other personnel on vehicle safety. To improve user experience, the system can dynamically adjust the parameters of the voiceprint recognition algorithm according to changes in environmental noise and driver position to ensure accurate recognition and response to voice commands in various driving environments.

[0086] Corresponding to the aforementioned embodiment of a safety control method for an intelligent cockpit voice interaction system, the present application also provides an embodiment of a safety control device for an intelligent cockpit voice interaction system.

[0087] Figure 4 is a block diagram of a safety control device of an intelligent cockpit voice interaction system according to an exemplary embodiment. Figure 4 , the device may include:

[0088] An acquisition module 21 is used to acquire vehicle dynamic information and driver dynamic information, wherein the vehicle dynamic information includes vehicle speed, GPS data, and IMU data, and the driver dynamic information is a driver's facial video;

[0089] A driver danger state assessment module 22, used to determine whether the driver is in a fatigue or sleep state according to the driver dynamic information, thereby implementing driver danger state assessment;

[0090] The vehicle dangerous state assessment module 23 is used to detect the state of the vehicle and whether there are dangerous situations around it according to the vehicle dynamic information, so as to achieve the vehicle dangerous state assessment;

[0091] The working mode switching module 24 is used to switch the intelligent cockpit voice interaction system to the corresponding working mode according to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, wherein the working modes include normal working mode, safe working mode and dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations. In the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations. In the dangerous environment mode, the vehicle accepts physical button triggers and cooperates with the driver's voice commands to perform corresponding operations.

[0092] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0093] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0094] Accordingly, the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned safety control method of the intelligent cockpit voice interaction system. Figure 5 As shown in FIG. 1 , a hardware structure diagram of a safety control device of an intelligent cockpit voice interaction system provided by an embodiment of the present invention is provided in any device with data processing capability, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0095] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, a safety control method for an intelligent cockpit voice interaction system as described above is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0096] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. A safety control method for an intelligent cockpit voice interaction system, characterized in that: include: Acquiring vehicle dynamic information and driver dynamic information, wherein the vehicle dynamic information includes vehicle speed, GPS data, and IMU data, and the driver dynamic information is a driver's facial video; According to the driver's dynamic information, determine whether the driver is in a fatigue or sleep state, thereby achieving a driver's dangerous state assessment; According to the vehicle dynamic information, the vehicle status and whether there are any dangerous situations around it are detected, so as to realize the dangerous status assessment of the vehicle; According to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, the intelligent cockpit voice interaction system is switched to the corresponding working mode, wherein the working modes include normal working mode, safe working mode and dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations. In the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations. In the dangerous environment mode, the vehicle accepts physical button triggers and cooperates with the driver's voice commands to perform corresponding operations.

2. The method according to claim 1, characterized in that Judging whether the driver is in a fatigue or sleep state based on the driver dynamic information, thereby evaluating whether the driver is in a dangerous state, including: Preprocessing the images of each frame in the driver's facial video; Based on the preprocessed images, the rPPG algorithm is used to extract the driver's physiological information; The preprocessed image is input into the fatigue state prediction model, and combined with the driver's physiological information, it is determined whether the driver is in a fatigue or sleep state. If the driver is in a fatigue or sleep state, it is determined that the driver is in a dangerous state.

3. The method according to claim 1, characterized in that According to the vehicle dynamic information, the vehicle status and whether there is a dangerous situation around the vehicle are detected, including: Based on the vehicle speed, GPS data and IMU data, determining whether the vehicle is in a driving state or a parking state; Detects whether there is an advanced voice command that a non-administrator user is using to control the vehicle. If so, determines that there is a dangerous situation around the vehicle.

4. The method according to claim 3, characterized in that: The voice commands are classified into high-level voice commands and low-level voice commands according to the degree of security sensitivity. The high-level voice commands include basic driving control, body control, and driving assistance function control; the low-level voice commands include comfort control, entertainment system control, and in-vehicle interaction control.

5. The method according to claim 3, characterized in that: The process of determining whether the user issuing the voice command is an administrator user includes: Get the multi-channel audio stream collected by the microphone array inside the vehicle; Preprocessing the multi-channel audio stream, including noise filtering and signal enhancement; The preprocessed multi-channel audio stream is input into the pre-trained ECAPA-TDNN model to extract voiceprint features; The extracted voiceprint features are matched with the registered voiceprint features stored in the voiceprint database. If the similarity exceeds a predetermined similarity threshold, the match is successful, that is, the user who issued the voice command is registered, otherwise, the user who issued the voice command is not registered; If the user who issues the voice command has been registered, it is determined whether he is an administrator user based on the corresponding information stored in the voiceprint database.

6. The method according to claim 1, characterized in that According to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, the intelligent cockpit voice system is switched to the corresponding working mode, specifically: When the vehicle is not in motion, the driver is not in danger, and there is no danger around the vehicle, the intelligent cockpit voice interaction system is in normal working mode; If the driver is detected to be in a dangerous state or not in the driving seat when the vehicle is in driving or parking state, the intelligent cockpit voice interaction system switches to a safe working mode; If there is a dangerous situation around the vehicle, the intelligent cockpit voice interaction system will switch to the dangerous environment mode.

7. A safety control device for an intelligent cockpit voice interaction system, characterized in that: include: An acquisition module, used to acquire vehicle dynamic information and driver dynamic information, wherein the vehicle dynamic information includes vehicle speed, GPS data, and IMU data, and the driver dynamic information is a driver's facial video; A driver danger state assessment module, used to determine whether the driver is in a fatigue or sleep state based on the driver dynamic information, thereby implementing driver danger state assessment; A vehicle dangerous state assessment module is used to detect the state of the vehicle and whether there are dangerous situations around it according to the vehicle dynamic information, so as to achieve vehicle dangerous state assessment; The working mode switching module is used to switch the intelligent cockpit voice interaction system to the corresponding working mode according to the results of the driver's dangerous state assessment and the vehicle's dangerous state assessment, wherein the working modes include normal working mode, safe working mode and dangerous environment mode. In the normal working mode, the vehicle accepts all voice commands and performs corresponding operations; in the safe working mode, the vehicle accepts voice commands from registered personnel and performs corresponding operations; in the dangerous environment mode, the vehicle accepts physical button triggers and performs corresponding operations in conjunction with the driver's voice commands.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images