Driver fatigue awakening device

Abstract

The invention discloses a driver fatigue awakening device, and relates to the technical field of voice interaction. The vehicle-mounted terminal comprises an instruction sending module, an intelligent voice conversation module, a voice synthesis module and an automatic voice recognition module; the instruction sending module is used for sending a voice interaction instruction when the driver is in a fatigue state; the intelligent voice dialogue module is used for outputting a dialogue text by using a large language model after receiving the voice interaction instruction; the dialogue text is a text conforming to the style preference of the driver; the voice synthesis module is used for converting the dialogue text into output voice; the automatic voice recognition module is used for recognizing the answer voice of the driver when the driver answers the output voice to obtain an answer text; the answer text is fed back to the intelligent voice dialogue module, the intelligent voice dialogue module uses a large language model to output a dialogue text according to the answer text, the fatigue state of the driver is relieved, and the driving safety is improved.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a driver fatigue awakening device. Background Art

[0002] Driver fatigue driving is one of the important causes of traffic accidents. At present, the existing technology only alarms and reports to the backend for fatigue driving and bad driving habits, but there is no effective means of intervention when it is inconvenient to stop and rest due to fatigue. There are no targeted measures to improve the driver's fatigue state. For example, it takes a certain distance and time for the driver to reach the rest point, and continuing to drive in a fatigued state poses a great safety hazard.

[0003] Therefore, a driver fatigue awakening device is urgently needed. Summary of the invention

[0004] The purpose of this application is to provide a driver fatigue awakening device, which can effectively intervene in the situation where the driver is in a fatigued state and it is inconvenient to stop and rest, relieve the driver's fatigue state, and improve driving safety.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] The present application provides a driver fatigue awakening device, including a vehicle-mounted terminal; the vehicle-mounted terminal includes a command issuing module, an intelligent voice dialogue module, a voice synthesis module and an automatic voice recognition module;

[0007] The command issuing module is used to issue voice interaction commands when the driver is in a fatigue state;

[0008] An intelligent voice dialogue module is used to: after receiving a voice interaction instruction, output a dialogue text using a large language model; the dialogue text is a text that conforms to the driver's style preference;

[0009] The speech synthesis module is used to convert the conversation text into output speech;

[0010] The automatic speech recognition module is used to: when the driver responds to the output speech, recognize the driver's response speech and obtain the response text;

[0011] The answer text is fed back to the intelligent voice dialogue module, and the intelligent voice dialogue module uses a large language model to output a dialogue text based on the answer text.

[0012] Optionally, the driver fatigue awakening device also includes a Web voice chat room module; the Web voice chat room module is used to: make voice calls with listed friends through the voice chat room.

[0013] Optionally, the driver fatigue awakening device also includes a user terminal; the user terminal is used to enter the voice chat room and set the parameters of the voice chat room.

[0014] Optionally, the speech synthesis module includes a text analysis unit, a language synthesis unit and a prosody processing unit;

[0015] The text analysis unit is used to: analyze the dialogue text to obtain the analysis result of the dialogue text; the analysis result includes the structure of the dialogue text and the phoneme composition of each word in the dialogue text;

[0016] A speech synthesis unit, for: converting the dialogue text into speech waveform based on the analysis result;

[0017] The prosody processing unit is used to perform prosody processing on the speech waveform to obtain output speech.

[0018] Optionally, the speech synthesis module is used to convert the dialogue text into output speech using a trained deep learning model.

[0019] Optionally, the speech synthesis module includes a speaker; the speaker is used to play the output speech.

[0020] Optionally, the automatic speech recognition module includes a microphone; the microphone is used to collect the driver's answer voice and convert the answer voice into a digital signal.

[0021] Optionally, the automatic speech recognition module includes a feature extraction unit and a text output unit;

[0022] The feature extraction unit is used to: extract features from the digital signal to obtain pronunciation feature data;

[0023] The text output unit is used to obtain the answer text according to the pronunciation feature data.

[0024] Optionally, in extracting features from digital signals to obtain pronunciation feature data, the feature extraction unit is used to: utilize a Mel-frequency cepstral coefficient algorithm to extract features from digital signals to obtain pronunciation feature data.

[0025] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0026] The present application provides a device for awakening a driver from fatigue. Through a large language model, questions asked to awaken the driver from fatigue are more intelligent, can better arouse the driver's interest, and relieve the driver's fatigue state. For situations where it is inconvenient to stop and rest in a fatigued state, effective intervention can be carried out to relieve the driver's fatigue state and improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of the structure of a driver fatigue awakening device provided in one embodiment of the present application;

[0029] Figure 2 A schematic diagram of the functional modules of a driver fatigue awakening device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] In an exemplary embodiment of the present application, Figure 1 As shown, a driver fatigue awakening device is provided, including a vehicle-mounted terminal; the vehicle-mounted terminal includes a command issuing module, an intelligent voice dialogue module, a voice synthesis module and an automatic voice recognition module.

[0033] The command issuing module is used to issue voice interaction commands when the driver is in a fatigue state. The fatigue state can be identified by the driver's facial and hand features.

[0034] The intelligent voice dialogue module is used to: after receiving the voice interaction instruction, output the dialogue text using the large language model; the dialogue text is the text that conforms to the driver's style preference.

[0035] The speech synthesis module is used to convert the conversation text into output speech.

[0036] The automatic speech recognition module is used to: when the driver responds to the output speech, recognize the driver's response speech and obtain the response text.

[0037] The answer text is fed back to the intelligent voice dialogue module, and the intelligent voice dialogue module uses a large language model to output a dialogue text based on the answer text.

[0038] 1) Intelligent voice dialogue module:

[0039] When the driver is in a state of deep fatigue, the large language model connected to the system needs to ask the driver targeted questions based on the actual situation, helping the driver to improve brain activity and get rid of fatigue as soon as possible; at the same time, the large language model will record the driver's style preferences, ask questions that the driver is more interested in, and help the driver think about the problem faster.

[0040] The driver fatigue awakening device provided in the present application is suitable for various vehicles or embedded in other fatigue monitoring systems, and is particularly suitable for long-distance transportation and professional driving scenarios.

[0041] When the fatigue monitoring system detects fatigue, it sends a signal, that is, a voice interaction command is issued through the command issuing module. The intelligent voice dialogue module uses a large language model to obtain a dialogue text based on the driver's style preference. The dialogue text is some common sense questions, and the driver answers the above questions to get rid of the monotonous driving situation. The intelligent voice dialogue module will record the driver's historical dialogues, and select questions or topics that are more targeted at the driver's cognitive level based on the historical dialogues as dialogue texts, which can help the driver maintain driving alertness and brain activity for a long time, effectively prevent entering a driving fatigue state, and improve the driver's transportation efficiency.

[0042] It should be noted that the large language model uses the open API of the more mature large language model on the market to access the large language model to ensure that the driver can communicate with artificial intelligence.

[0043] Figure 2 In the dialogue management, it maintains and manages the dialogue process with the user, and determines the next decision and judgment of the artificial intelligence.

[0044] 2) The speech synthesis module is used to convert the dialogue text into output speech using a trained deep learning model. The trained deep learning model is a model trained with sample dialogue text as input and sample output speech corresponding to the sample dialogue text as labels.

[0045] The speech synthesis module includes a text analysis unit, a language synthesis unit and a prosody processing unit.

[0046] The text analysis unit is used to analyze the dialogue text to obtain the analysis result of the dialogue text; the analysis result includes the structure of the dialogue text and the phoneme composition of each word in the dialogue text.

[0047] The speech synthesis unit is used to convert the dialogue text into speech waveform based on the analysis result.

[0048] The prosody processing unit is used to perform prosody processing on the speech waveform to obtain output speech.

[0049] The speech synthesis module uses a deep learning-based speech synthesis algorithm to play the conversation content initiated by the language model to the driver using the driver's customized prompt sound. The speech synthesis module mainly includes the following three processing contents:

[0050] Text analysis: Linguistic analysis of the input text to determine the structure of sentences and the phonemic composition of each word.

[0051] Language synthesis: Extract the processed text from the speech synthesis library trained by deep learning and convert the linguistic description into speech waveform.

[0052] Rhythm processing: By processing the synthesized speech waveform, the output speech is made more coherent and natural, reducing the driver's discomfort.

[0053] The speech synthesis algorithm based on deep learning can adopt the WaveNet model, MelGAN model, Tacotron model, FastSpeech model and Deep Voice model.

[0054] The speech synthesis module includes a speaker; the speaker is used to play the output speech. The speech synthesis module uses a deep learning method to extract the dialogue text content from the trained speech synthesis library, and performs speech synthesis based on the sound timbre defined in advance by the user, thereby realizing the broadcast of the output content of the AI ​​language model.

[0055] 3) Automatic speech recognition module: When the system detects that the driver is in a fatigued state and artificial intelligence intervenes (i.e., the intelligent voice dialogue module outputs the output speech corresponding to the dialogue text to the driver), the sound signal is converted into a digital signal after collecting the driver's voice information, and after feature recognition, it is compared with the data in the system voice library to realize automatic recognition of the driver's answer voice. It is converted into text and fed back to the artificial intelligence, and the next step is carried out after the artificial intelligence makes a judgment.

[0056] The automatic speech recognition module includes a microphone; the microphone is used for collecting and processing speech signals, for collecting the driver's answering speech, and converting the answering speech into a digital signal to facilitate subsequent computer processing.

[0057] The automatic speech recognition module includes a feature extraction unit and a text output unit.

[0058] The feature extraction unit is used to extract features from the digital signal to obtain pronunciation feature data. The text output unit is used to obtain the answer text according to the pronunciation feature data.

[0059] In terms of extracting features from digital signals to obtain pronunciation feature data, the feature extraction unit is used to extract features from digital signals using the Mel Frequency Cepstrum Coefficient (MFCC) algorithm to obtain pronunciation feature data, thereby facilitating accurate recognition of speech content. The pronunciation feature data includes features such as initials and finals.

[0060] The calculation process of the MFCC algorithm is as follows:

[0061] 1. Audio acquisition: Record the sound signal through a microphone and convert it into a digital signal. This usually involves sampling and quantization, converting the continuous analog sound signal into a discrete digital form.

[0062] 2. Preprocessing: Preprocess the digital signal to obtain a preprocessed digital signal. Preprocessing includes removing DC offset and applying a pre-emphasis filter to increase the energy of high-frequency components.

[0063] 3. Framing and windowing: The preprocessed digital signal is divided into multiple frames, each of which is usually 20-40 milliseconds long. A window function (such as a Hamming window) is then applied to each frame to reduce boundary effects and obtain a windowed frame.

[0064] 4. Fourier transform: Perform fast Fourier transform (FFT) on each windowed frame to convert the time domain signal of the windowed frame into a frequency domain signal.

[0065] 5. Mel filter bank: The frequency domain signal is filtered through a set of Mel filters to simulate the auditory perception of the human ear and obtain the frequency band after the Mel filter bank. The center frequencies of these filters are evenly distributed on the Mel scale.

[0066] 6. Logarithmic transformation: Perform logarithmic transformation on the energy of each frequency band after passing through the Mel filter bank to obtain logarithmic energy.

[0067] 7. Discrete Cosine Transform (DCT): Perform discrete cosine transform on the logarithmic energy to obtain MFCC coefficients, i.e. pronunciation feature data. Since MFCC coefficients contain most of the speech information, usually only the first few coefficients are retained.

[0068] The input of the MFCC algorithm is the digital signal of the sound signal after it is collected and converted by the microphone. During the whole process, the original analog sound signal has been converted into a discrete time signal that can be processed by the digital signal processing algorithm.

[0069] The text output unit is used for voice model comparison. Specifically, the extracted pronunciation feature data is compared and matched with the huge sound model in the system, and the corresponding text content of the input is determined by comparing the similarity between the pronunciation feature data and the data in the sound model to obtain the answer text.

[0070] The sound model is a model or data structure used for speech recognition, which can be used to store and analyze speech features so as to match them with input speech features. The sound model can adopt acoustic models, language models, pronunciation models, etc.

[0071] The acoustic model is used to map sound signals (such as MFCC features) to possible phonemes or speech units. The acoustic model can be obtained by training a deep neural network (such as a convolutional neural network (CNN), a recurrent neural network (RNN) or a Transformer).

[0072] Language models are used to predict text content based on the probability of word occurrence. Language models are trained based on large amounts of text data to improve the accuracy of speech recognition.

[0073] The pronunciation model is used to process the mapping relationship between phonemes and words. The pronunciation model can adopt a finite state machine or a rule-based system.

[0074] The process of comparing the similarity between the pronunciation feature data and the data in the sound model is as follows:

[0075] When comparing similarity, pronunciation feature data (such as MFCC coefficients) are used. These feature data include tones and finals:

[0076] Tone: During feature extraction, the energy characteristics of high-frequency components may contain tone information. Tone features are identified by analyzing the frequency distribution and modulation patterns in feature data.

[0077] Finals: The characteristics of finals are determined by analyzing the frequency distribution pattern and formants in the MFCC coefficients.

[0078] When comparing the pronunciation feature data with the data in the sound model, the pronunciation feature data extracted by the MFCC algorithm is matched with the data feature templates in the acoustic model. These templates are trained based on a large amount of speech data and are used to describe the MFCC feature distribution corresponding to various phonemes (including tones and finals).

[0079] The comparison process is mainly to match the MFCC features with the feature templates in the sound model, rather than directly comparing with some clear "tone" or "final" data. Instead, it is done indirectly through deep learning models and statistical models. Specifically:

[0080] When comparing similarities, the distance or similarity between the pronunciation feature data and the data feature template stored in the acoustic model is calculated, and the phoneme or word that the input speech is most likely to correspond to is determined based on the calculated distance. In an example of the present application, the Euclidean distance algorithm can be used to calculate the distance between the pronunciation feature data and the data feature template stored in the acoustic model, and the cosine similarity algorithm can also be used to calculate the similarity between the pronunciation feature data and the data feature template stored in the acoustic model.

[0081] 4) Web Voice Chat Room Module:

[0082] like Figure 2 As shown, in addition to awakening the driver who has fallen into driving fatigue, the present application also establishes a Web voice chat room module to help prevent the driver from falling into a state of driving fatigue by allowing the driver to be in an environment where he can communicate with others in real time.

[0083] The driver fatigue awakening device also includes a Web voice chat room module; the Web voice chat room module is used to: make voice calls with listed friends through the voice chat room.

[0084] Among them, the voice chat room is built using WebSocket.

[0085] When the system detects that the driver is in a state of driving fatigue, it can proactively intervene and prevent the existing or potential driving fatigue through intelligent human-computer dialogue or by providing the driver with a customized instant voice chat room, thereby increasing the driver's brain activity and preventing the driver from falling into a state of driving fatigue and thus causing traffic accidents.

[0086] The driver fatigue awakening device also includes a user terminal; the user terminal is used to enter the voice chat room and set the parameters of the voice chat room. The user (driver, relative, friend or fleet manager) can set the voice chat room at any time through the user terminal, and provide a convenient solution for the user to join the voice chat room. The user terminal can be a mobile phone, tablet computer and other devices.

[0087] The Web voice chat room module mainly consists of five parts:

[0088] (1) Front-end user interface: Users can join a voice chat room through a web interface or application. This interface includes voice call controls, user list, chat message window, etc. Users can choose to turn on microphones, speakers, etc. for voice calls.

[0089] (2) Audio capture and transmission: The user's voice is captured through a microphone and converted into digital audio data. The above audio data is sent to a server or directly transmitted to other users (peer-to-peer connection) via the Internet using a real-time transport protocol (such as WebRTC).

[0090] (3) Server processing: In a centralized system, the server can be responsible for coordinating voice data flows, managing user connections and permissions, and handling any data synchronization needs, such as voice activity indication, online status updates, etc.

[0091] For the driver fatigue awakening device using peer-to-peer technology, the role of the server is mainly to help users establish a connection (i.e., exchange signals) during the initial connection, and the subsequent audio data is directly transmitted between users to reduce latency and bandwidth usage. This application uses WebRTC technology to build a voice chat room.

[0092] The driver fatigue awakening device also includes a monitoring center; the monitoring center is a monitoring platform set up by the fleet or relevant industry management departments, which is used to: monitor and record driver and vehicle information in real time, and the management personnel can command and manage the driver through the monitoring center.

[0093] The server provides data connection and data processing services for vehicle terminals, mobile phones and monitoring centers. It has API access (application programming interface) functions for network information services, information exchange services for each terminal, and server functions for dialogue (chat) functions.

[0094] Application Programming Interface (API access): It can provide information sources through API access of traffic information platforms, meteorological information platforms, AI language model providers or other network consulting platforms, and thus provide network information services for vehicle-mounted terminals.

[0095] (4) Audio reception and playback: The received audio data is decoded and played at the user end. Echo cancellation and noise suppression methods are used to improve call quality.

[0096] (5) Chat room management function: Users can create chat rooms, invite other users to join, or manage chat room settings such as access control, voice activity indication, etc.

[0097] The driver fatigue awakening device also includes a communication module, and the communication module can communicate with a server, a monitoring center, and a mobile phone terminal through the operator's network.

[0098] The main function of the Web voice chat room module is to enable the driver to stay active at all times by opening the voice chat function when the driver needs to drive long distances and prevent him from falling into a state of deep fatigue driving. When the AI ​​wake-up measures (through the intelligent voice dialogue module) fail to affect the driver, other people can effectively supervise the driver or even wake him up urgently, effectively reducing the probability of traffic accidents caused by deep fatigue driving.

[0099] This application is connected to a large language model, which makes the selection of questions to awaken the driver's fatigue more intelligent, more able to arouse the driver's interest and relieve the driver's fatigue state; at the same time, this application provides a reminder sound recording function, and through the use of a deep learning voice fusion algorithm, the driver can customize the reminder sound, which is more conducive to the driver getting rid of fatigue; in addition, this application uses WebSocket to build a web chat room, so that the driver can communicate with relatives, friends or fleet managers in real time, prevent the driver from falling into a fatigue state, and ensure the driver's driving safety. This application is a system that uses AI language models and multimodal active intervention to prevent driver fatigue. Through the comprehensive use of computer technology, human-computer interaction, human-human interaction and physical stimulation measures, it can effectively prevent the driver's fatigue driving behavior, intervene in deep fatigue states in a timely manner, and significantly improve the safety and efficiency of transportation.

[0100] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A driver fatigue awakening device, characterized in that: The driver fatigue awakening device comprises a vehicle-mounted terminal; the vehicle-mounted terminal comprises a command issuing module, an intelligent voice dialogue module, a voice synthesis module and an automatic voice recognition module; The command issuing module is used to issue voice interaction commands when the driver is in a fatigue state; An intelligent voice dialogue module is used to: after receiving a voice interaction instruction, output a dialogue text using a large language model; the dialogue text is a text that conforms to the driver's style preference; The speech synthesis module is used to convert the conversation text into output speech; The automatic speech recognition module is used to: when the driver responds to the output speech, recognize the driver's response speech and obtain the response text; The answer text is fed back to the intelligent voice dialogue module, and the intelligent voice dialogue module uses a large language model to output a dialogue text based on the answer text.

2. The driver fatigue awakening device according to claim 1, characterized in that: The driver fatigue awakening device also includes a Web voice chat room module; the Web voice chat room module is used to: make voice calls with listed friends through the voice chat room.

3. The driver fatigue awakening device according to claim 2, characterized in that: The driver fatigue awakening device also includes a user terminal; the user terminal is used to enter the voice chat room and set the parameters of the voice chat room.

4. The driver fatigue awakening device according to claim 1, characterized in that: The speech synthesis module includes a text analysis unit, a language synthesis unit and a prosody processing unit; The text analysis unit is used to: analyze the dialogue text to obtain the analysis result of the dialogue text; the analysis result includes the structure of the dialogue text and the phoneme composition of each word in the dialogue text; A speech synthesis unit, for: converting the dialogue text into speech waveform based on the analysis result; The prosody processing unit is used to perform prosody processing on the speech waveform to obtain output speech.

5. The driver fatigue awakening device according to claim 1, characterized in that: The speech synthesis module is used to convert the dialogue text into output speech using a trained deep learning model.

6. The driver fatigue awakening device according to claim 1, characterized in that: The speech synthesis module includes a speaker; the speaker is used to play the output speech.

7. The driver fatigue awakening device according to claim 1, characterized in that: The automatic speech recognition module includes a microphone; the microphone is used to collect the driver's answer voice and convert the answer voice into a digital signal.

8. The driver fatigue awakening device according to claim 7, characterized in that: The automatic speech recognition module includes a feature extraction unit and a text output unit; The feature extraction unit is used to: extract features from the digital signal to obtain pronunciation feature data; The text output unit is used to obtain the answer text according to the pronunciation feature data.

9. The driver fatigue awakening device according to claim 8, characterized in that: In terms of extracting features from digital signals to obtain pronunciation feature data, the feature extraction unit is used to extract features from digital signals using a Mel-frequency cepstral coefficient algorithm to obtain pronunciation feature data.

Images