Intelligent cockpit system and biological signal anomaly detection method, device and equipment thereof, medium and product

Abstract

The invention discloses an intelligent cabin system and a biological signal anomaly detection method, device and equipment thereof, a medium and a product, and relates to the field of safe driving monitoring, and the method comprises the steps: obtaining the vital sign data of a user and the pressure of the user on a steering wheel based on the intelligent cabin system; the vital sign data comprises respiratory rate, heart rate and blood oxygen; performing anomaly detection on the vital sign data and the pressure by using a multi-source fusion algorithm to generate an anomaly detection result; controlling the vehicle to execute corresponding operation according to the abnormal detection result; the corresponding operation comprises the steps of cutting off a vehicle execution power system, turning on a warning lamp, transmitting a message to give an alarm, sharing position information and turning on an auxiliary automatic driving function. According to the application, the vital sign data of a driver can be monitored in real time, the potential health risk of the driver can be predicted, and sudden diseases can be monitored in real time; and the traffic accident rate is reduced.

Description

Technical Field

[0001] The present application relates to the field of safe driving monitoring, combining technologies such as smart cockpit technology, biomedical signal processing, anomaly detection algorithms and artificial intelligence, and in particular to a smart cockpit system and its bio-signal anomaly detection method, device, equipment, medium and product to achieve real-time monitoring and intelligent response to the driver's status, thereby improving driving safety. Background Art

[0002] Intelligence, networking and sharing have become the development trend of today's automobile industry. Cars are gradually transforming from simple means of transportation to smart terminals. As one of the key factors for consumers to consider when buying and using cars, the car cockpit has gradually undergone tremendous changes in form, function and interaction mode, and the concept of smart cockpit perception system has also emerged. The smart cockpit perception system refers to the intelligent perception and control of the vehicle's internal environment through various sensors and artificial intelligence technologies, providing a safer, more comfortable and more personalized driving and riding experience.

[0003] Driving safety is the focus of the design of smart cockpit perception systems. The Driver Monitoring System (DMS) monitors the driver in real time to prevent him from losing control of the vehicle due to distraction or fatigue. In 2018, China has already mandated the installation of DMS systems on commercial vehicles such as "two dangerous goods and one passenger vehicle". It is expected that ordinary passenger cars will also be popularized in accordance with regulations in 2027. Other regions such as the European Union also require new models to be equipped with similar systems such as fatigue driving and distraction monitoring. The main functions of the existing DMS system include: fatigue driving detection, distraction detection, and dangerous driving behavior (calling, smoking) detection. In addition, some traffic accidents are still caused by diseases and complications during driving, leading to hypoglycemia, arrhythmia, respiratory decompensation and other problems. Most acute complications can be screened out through routine physical examinations in the early stages of abnormal indicators, and abnormal indicators can be used as a reference for driving risk assessment factors, thereby further improving driving safety.

[0004] At present, the rapid physical examination before driving has been tested in the public transportation system in some parts of China. After the physical examination at the station, the driver cooperates with the DMS system to provide multiple guarantees for driving safety. However, the current system's detection method for the driver's abnormal state mostly uses visual detection methods, which cannot directly detect abnormal data from the human body. Instead, it requires the human body to show symptoms of discomfort before the problem can be detected. However, when the human body shows symptoms of discomfort, it is still very dangerous for the vehicle to start changing the operating state. It is impossible to predict the driver's potential health risks and monitor sudden diseases in real time. When such diseases occur, it is often difficult for the driver to stop in time. In addition, abnormal conditions such as cardiovascular and cerebrovascular diseases, high tension, and high blood pressure may also lead to operational errors, thereby causing traffic accidents. Summary of the invention

[0005] The purpose of this application is to provide a smart cockpit system and its bio-signal abnormality detection method, device, equipment, medium and product to solve the problem of being unable to predict the driver's potential health risks and real-time monitoring of sudden illnesses, and prone to operating errors when the driver's body is in an abnormal state, resulting in a high incidence of traffic accidents.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a smart cockpit system, including: a master device, a wearable slave device, and a cloud platform;

[0008] The main device includes a vehicle-mounted platform, a millimeter-wave radar and a piezoelectric sensor; the millimeter-wave radar is connected to the vehicle-mounted platform and is used to detect the user's breathing frequency; the piezoelectric sensor is arranged on the steering wheel and is connected to the vehicle-mounted platform and is used to collect the user's pressure on the steering wheel;

[0009] The wearable slave device includes a heart rate sensor and a blood oxygen sensor; the heart rate sensor is used to detect the user's heart rate; the blood oxygen sensor is used to detect the user's blood oxygen;

[0010] The vehicle-mounted platform is connected to the millimeter-wave radar, the heart rate sensor and the blood oxygen sensor, and is used to use a multi-source fusion algorithm to perform abnormal detection on the user's vital sign data and the pressure, and upload the vital sign data, the pressure and the abnormal detection results to the cloud platform, and control the vehicle to perform corresponding operations according to the abnormal detection results; the corresponding operations include cutting off the vehicle execution power system, turning on the warning light, messaging alarm, sharing location information and turning on the auxiliary automatic driving function; the vital sign data include respiratory rate, heart rate and blood oxygen;

[0011] The cloud platform is used to send the vital sign data, the pressure and the abnormality detection result to the client.

[0012] In a second aspect, the present application provides a method for detecting abnormal biosignals based on a smart cabin system, comprising:

[0013] Based on the above-mentioned smart cabin system, the user's vital signs data and the pressure of the user on the steering wheel are obtained; the vital signs data include respiratory rate, heart rate and blood oxygen;

[0014] Performing anomaly detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate anomaly detection results;

[0015] The vehicle is controlled to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

[0016] In a third aspect, the present application provides a biological signal anomaly detection device based on a smart cabin system, comprising:

[0017] The smart cabin seat system is used to obtain the user's vital signs data and the pressure the user applies to the steering wheel; the vital signs data include breathing rate, heart rate and blood oxygen;

[0018] A central processing unit, configured to perform abnormality detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate an abnormality detection result;

[0019] A control module is used to control the vehicle to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

[0020] In a fourth aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned methods for detecting abnormal biological signals based on a smart cabin seat system.

[0021] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for detecting abnormal biological signals based on a smart cabin system.

[0022] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned biological signal anomaly detection methods based on a smart cabin system.

[0023] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0024] This application sets up multiple sensors of different categories to collect the user's vital signs data and the pressure on the steering wheel in real time, and uses a multi-source fusion algorithm to detect anomalies in the user's vital signs data and pressure, so as to control the vehicle to perform corresponding operations according to the abnormal detection results, that is, once it is identified that the driver's ability is limited, the vehicle will be subjected to abnormal warnings, turn on warning lights, speed limits, message transmission alarms, share location information and other corresponding operations according to the model output results. At the same time, the auxiliary automatic driving function is turned on to take over the vehicle when necessary and park safely, thereby effectively reducing the high incidence of traffic accidents. This application can improve the driver's safety and driving experience, and provide strong support for vehicle management and rescue, which is of great significance to protecting drivers, passengers and road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a design block diagram of a smart cockpit system in one embodiment of the present application;

[0027] Figure 2 A flowchart of a method for detecting abnormal biological signals based on a smart cabin system according to an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a pulse wave signal preprocessing process provided in an embodiment of the present application; wherein: Figure 3 (a) is a schematic diagram of the original PPG signal; Figure 3 (b) is a schematic diagram of PPG signal removing DC component; Figure 3 (c) is a schematic diagram of PPG detrending signal;

[0029] Figure 4 A schematic diagram of the filtering effect of a Savitzky-Golay filter under different window lengths FL provided in an embodiment of the present application; wherein, Figure 4 (a) is a schematic diagram of the filtering effect when the filter order (Order, ORD) = 3 and FL = 11; Figure 4 (b) is a schematic diagram of the filtering effect when ORD=3, FL=21; Figure 4 (c) is a schematic diagram of the filtering effect when ORD=3, FL=51; Figure 4 (d) is a schematic diagram of the filtering effect when ORD=3, FL=101;

[0030] Figure 5 A logic diagram for abnormality detection of biological signals based on DS evidence theory provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a speech recognition system provided in one embodiment of the present application;

[0032] Figure 7 A schematic diagram of the deployment connection between a master device and a slave device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] The present application provides a smart cockpit system, including: a master device, a wearable slave device and a cloud platform.

[0036] The main device includes a vehicle-mounted platform, a millimeter-wave radar and a piezoelectric sensor; the millimeter-wave radar is connected to the vehicle-mounted platform and is used to detect the user's breathing frequency; the piezoelectric sensor is arranged on the steering wheel and is connected to the vehicle-mounted platform and is used to collect the user's pressure on the steering wheel.

[0037] The wearable slave device includes a heart rate sensor and a blood oxygen sensor; the heart rate sensor is used to detect the user's heart rate; the blood oxygen sensor is used to detect the user's blood oxygen.

[0038] The vehicle-mounted platform is connected to the millimeter-wave radar, the heart rate sensor and the blood oxygen sensor, and is used to use a multi-source fusion algorithm to perform abnormality detection on the user's vital signs data and the pressure, and upload the vital signs data, the pressure and the abnormality detection results to the cloud platform, and control the vehicle to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's execution power system, turning on warning lights, messaging alarms, sharing location information and turning on assisted automatic driving functions; the vital signs data include respiratory rate, heart rate and blood oxygen.

[0039] The cloud platform is used to send the vital sign data, the pressure and the abnormality detection result to the client.

[0040] In practical applications, the master device may be a cockpit and the slave device may be a wristband, such as Figure 1 As shown, the master device and the slave device can usually be developed in C / C++ and Python. From the functional architecture design, it is divided into five layers, including physical hardware layer, device driver layer, operating system layer, functional module layer and business application layer.

[0041] The physical hardware layer is mainly responsible for providing a hardware support environment for the smart cockpit system. The hardware modules of the master device can adopt the GD32 board, HorizonX3M board, base power supply motherboard, WI-FIBLE module, multi-protocol interface expansion module, microphone and touch display provided by the Xiaomi AIOT platform. In addition, the master device also has hardware modules such as millimeter wave radar, piezoelectric sensor, GPS positioning module, etc. The hardware modules of the slave device include Arduino Nano board, HC-05 Bluetooth module, heart rate and blood oxygen sensor.

[0042] The device driver layer is the interface between the operating system and the hardware device. It provides a set of APIs that allow applications to use the functions of the hardware device. The driver and firmware of the main device include the hardware abstraction layer (HAL), I2C driver, UART driver, piezoelectric sensor driver, base motherboard driver, LD millimeter wave radar driver, GPS driver, WI-FI+BLE driver. The above driver components provide the underlying hardware control support for this device to ensure that the hardware can be effectively controlled.

[0043] The operating system layer is the core of the device. It manages and controls hardware resources, provides hardware abstraction, and encapsulates system services for application calls. The GD32 board of the master device uses the Nuttx embedded real-time operating system. The Horizon X3M board uses the classic Ubuntu operating system. The slave device is programmed based on bare metal and does not have an operating system.

[0044] The functional module layer is a key part of the smart cockpit system, which realizes the core functions of the smart cockpit. The core functional modules of the master device include pressure acquisition perception, respiratory spectrum acquisition perception, GPS data acquisition perception, Kaldi speech recognition algorithm, multi-mode data fusion perception algorithm, serial port communication, I2C communication and WI-FI+BLE communication. The core functional modules of the slave device include heart rate acquisition perception, blood oxygen acquisition perception and BLE serial port communication.

[0045] The business application layer is a layer that directly interacts with users in the smart cockpit system. It implements specific business logic and application functions based on the API provided by the functional module layer. The master device provides multi-mode data fusion life status monitoring, in-cabin emergency voice recognition, cockpit data cloud platform sharing, and anti-mistouch emergency mechanism. The slave device provides heart rate and blood oxygen data monitoring, master device Bluetooth communication and other user services, and provides the above business functions to users for scheduling and use, bringing users an excellent new generation of smart cockpit experience.

[0046] In order to provide users with a better experience, the main device also includes: a voice recognition system; the voice recognition system includes a microphone and a voice recognition module; the microphone is connected to the voice recognition module; the microphone is used to collect voice commands; the voice recognition module is used to convert the voice commands into phonemes or words using an acoustic model, and use a language model to predict the probability of a word sequence based on the phonemes or words; the vehicle-mounted platform is also used to control the vehicle to perform corresponding operations based on the probability of the word sequence.

[0047] Furthermore, the present application is equipped with an emergency voice recognition control assistance function based on voice recognition technology and a cockpit data visualization sharing function, which facilitates the driver to voice control the in-vehicle equipment in an emergency. At the same time, it can display the driver's physical health status in real time and share the human and vehicle information on the cloud platform, making it convenient for the current user and his family to obtain real-time human and vehicle information.

[0048] In an exemplary embodiment, after various sensor data are uploaded to the vehicle-mounted platform in real time, the smart cockpit main device is set to display the multi-source data perceived, and at the same time, the multi-source bio-signal data is integrated with the abnormal detection results to update the display in real time, and at the same time, it is further uploaded to the cloud platform or the corresponding App server as needed with the help of the vehicle-mounted communication module or the public network.

[0049] The main device of the smart cockpit is equipped with a comprehensive display interface, which is convenient for the driver to observe his own physical condition. At the same time, passengers in the car, their families, and rescue personnel can access the corresponding App to observe vehicle and personnel information in real time, including vital signs such as heart rate data, blood oxygen data, respiratory rate data, and vehicle location data, so as to achieve the effect of full ecological information sharing between people and vehicles. Specifically:

[0050] It is mainly divided into two parts: one is cloud platform access and data sharing, and the other is the comprehensive display of the smart cockpit main equipment.

[0051] Cloud platform access and data sharing are mainly achieved through the creation of products and WI-FI+BLE modules on the Xiaomi IOT platform, and data such as heart rate, blood oxygen, pulse, respiratory rate, and vehicle location information are uploaded to the Xiaomi IOT platform in real time based on the public network. Cloud platform access and data sharing are achieved through the WI-FI+BLE module, in which the Xiaomi module integrates the standard firmware program of the Xiaomi IoT platform (including basic capabilities such as network configuration, communication, OTA, and security). Developers only need to develop the MCU program to realize the communication between the MCU and the module serial port, and create products on the Xiaomi IOT platform to enable the current device to access the cloud platform.

[0052] The main device integrated display function of the smart cockpit system is developed based on QT and deployed on Horizon X3M. Qt Creator is mainly used to create QtWidgets application projects. The smart cockpit perception system interface is designed in Qt Designer, including various controls, layouts and interactive elements, and the style sheet provided by Qt is used to beautify and customize the interface. For interface logic writing, the logic code of the interface is written in Qt Creator, and the user interaction is processed using the C++ language or the signal slot mechanism provided by Qt. The communication with other sensors in the smart cockpit perception system is realized through the serial port. For data visualization, the chart library provided by Qt is used to realize data visualization, draw real-time curves of sensor data and display vehicle status information. Multi-threaded coding is used to ensure interface fluency and system responsiveness. After the interface development is completed, it is deployed on the Horizon X3M board through the aarch64 cross compiler, and connected to the touch screen for local visualization. The main display information includes human health status (heart rate, blood oxygen concentration, pulse rate, respiratory rate, vehicle location information), and can prompt the user whether the current physical state is suitable for driving the vehicle.

[0053] The method for detecting abnormal biological signals based on the intelligent cabin seat system provided in the embodiment of the present application is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method includes the following steps: Figure 2 shown.

[0054] S1: Based on the smart cabin system, the user's vital signs data and the pressure the user applies to the steering wheel are obtained; the vital signs data include respiratory rate, heart rate and blood oxygen.

[0055] S2: Perform anomaly detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate anomaly detection results.

[0056] S3: Controlling the vehicle to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

[0057] In an exemplary embodiment, based on short-range communication technologies such as WI-FI, Bluetooth and Star Flash, the user's vital signs data and pressure collected by the smart cabin system are uploaded to the vehicle-mounted platform in real time. The vehicle-mounted platform uses artificial intelligence algorithms or other multi-source fusion algorithms, such as the ubiquitous biological signal abnormal state detection method based on DS evidence theory, to detect and fuse multi-source data. Since multiple sensors can more comprehensively consider multiple indicators when making expert system judgments, the system's recognition resolution of abnormal states can be improved.

[0058] Among them, the anomaly detection results generated by the multi-source fusion algorithm have strong noise resistance and can reduce the chance of misjudgment; at the same time, multiple types of sensors comprehensively consider multiple indicators when making expert system judgments, thereby improving the system's recognition resolution of abnormal conditions.

[0059] In an exemplary embodiment, step S2 can be replaced by the following steps: preprocessing the vital signs data to determine the preprocessed vital signs data; extracting biomedical indicators based on the time domain characteristics and frequency domain characteristics of the preprocessed vital signs data; the biomedical indicators include heart rate indicators, respiratory rate indicators, blood oxygen indicators, SDNN indicators, NN50 indicators and LF / HF ratio indicators; based on the medical database and expert system, evaluating the biomedical indicators to determine the importance of the biomedical indicators to the abnormal detection results and the credibility of the biomedical indicators themselves; based on the importance and the credibility, generating abnormal detection results using DS evidence theory; the abnormal detection results include healthy status, abnormal status and dangerous status.

[0060] In order to clearly explain how the vehicle-mounted platform uses artificial intelligence algorithms or other multi-source fusion algorithms to make multi-source data fusion decisions, this application performs anomaly detection on multi-source biological signals based on DS evidence theory. The specific steps are as follows.

[0061] S21: Perform pre-processing such as detrending and filtering on the signal to remove fixed-frequency interference such as DC and power spurs. The physiological activity information of the human body is retained in the pre-processed biological signal.

[0062] Since the pulse wave signal contains a DC component and has a large baseline offset error, it is necessary to first remove the DC component. The DC component can be removed by subtracting the mean value in the waveform. For slow baseline drift errors, the baseline drift of the waveform can be removed using a polynomial fitting method, such as Figure 3 The pulse wave signal preprocessing process based on photoplethysmography (PPG) is shown. After the pulse wave is preprocessed, photoplethysmography can be used for pulse wave analysis to obtain indicators such as heart rate and blood oxygen saturation rate.

[0063] Before peak detection, smoothing filtering is required to reduce the impact of burrs on peak detection. Figure 4 The Savitzky-Golay filter shown in the figure has filtering effects under different window lengths FL. After smoothing the waveform, peak detection can be used to obtain the number of blood flow peaks within a period of time. Since the change in blood flow is directly affected by cardiac activity, the number of peaks within one minute is the heart rate, and blood oxygen can be calculated from the reflectivity of red light (600-800nm) and infrared light (800-1000nm) within the same period of time. The reason is that there are two forms of hemoglobin in the blood: oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb). They have different absorption characteristics for light of different wavelengths. The absorption coefficient of Hb is higher between 600-800nm, and the absorption coefficient of HbO2 is higher between 800-1000nm. The blood oxygen content can be obtained by calculating the reflectivity of the two lights.

[0064] S22: Extract biomedical indicators based on time-frequency domain feature analysis.

[0065] For example, heart rate is one of the objective indicators that can reflect the cardiovascular function of the human body. In the time domain, the heart rate index is obtained by calculating the pulse wave and the RR interval of the heartbeat phase. Taking the pulse wave as an example, the frequency domain feature of heart rate variability (HRV) frequency domain analysis quantitatively evaluates the regulatory role of the sympathetic nerve and the vagus nerve.

[0066] It should be noted that HRV divides the ECG pulse signal into ultra low frequency (ULF) (0-0.0033Hz), very low frequency (VLF) (0.0033-0.04Hz), low frequency (LF) (0.04-0.15Hz) and high frequency (HF) (0.15-0.4Hz). The heart rate pulsation is generally considered to be periodic. By calculating the power spectral density, it can effectively evaluate the cardiovascular nerve function state and can be used as a relatively independent indicator for judging the prognosis of various cardiovascular diseases and sudden cardiac death.

[0067] S23: Based on clinical symptom observations, the abnormality of the same physiological indicator may be caused by different factors, and a certain disease may cause abnormalities in multiple physiological indicators.

[0068] When a physiological indicator is abnormal, we can only determine whether it is a dangerous abnormal state with a certain probability. The same disease can also manifest as abnormalities in multiple indicators. Multi-sensors can effectively handle the uncertainty of state detection, fuse information from multiple dimensions, and improve the accuracy and robustness of the fusion results.

[0069] SDNN is the standard deviation of the normal heart beat interval (NN interval), a key indicator for evaluating HRV, reflecting the function of the autonomic nervous system and heart health. A higher SDNN value indicates greater heart rate variability and stronger heart adaptability; a lower value may indicate impaired autonomic nervous function and poor heart response to environmental changes, which may be associated with heart disease.

[0070] NN50 refers to the number of adjacent interval pairs in which the difference between consecutive NN intervals exceeds 50 milliseconds. It is another indicator for evaluating HRV, especially for detecting rapid changes in the heart and is often used to evaluate parasympathetic nerve activity.

[0071] The LF / HF ratio is an important indicator of HRV and assesses the balance of the autonomic nervous system. A higher ratio indicates strong sympathetic activity, which may be associated with stress and anxiety; a lower ratio indicates strong parasympathetic activity, which is usually associated with a state of relaxation.

[0072] With the application of sensors, DS evidence theory has played a prominent role in expressing "certainty" and "uncertainty" in the field of multi-sensors. Compared with traditional probability theory methods, DS evidence theory can more effectively deal with the unknown and uncertainty of multi-source information fusion problems. The improved DS fusion algorithm can also handle contradictory sensor data.

[0073] The core of DS evidence theory lies in the basic probability distribution of the evidence body, which directly affects the accuracy of the fusion results. Common basic probability distribution methods include: preliminarily estimating the probability distribution based on prior knowledge and information provided by domain experts; performing probability estimation through statistical analysis of historical data, and using Bayesian inference to combine prior knowledge and new observation data to update the probability distribution and improve accuracy.

[0074] S24: Deploy a multi-sensor fusion algorithm based on DS evidence theory on the vehicle platform to perform decision-level fusion of various sensor data and perform abnormal state detection. The specific detection process logic Figure 5 After preprocessing, multi-source biological signals can obtain multiple physiological indicators in the time-frequency domain. For a certain indicator, its importance to the final judgment result and its feasibility need to be determined.

[0075] This application divides the final abnormality detection results into three states: healthy, abnormal, and dangerous. The importance of the indicator is reflected in the probability of the three states corresponding to a certain range, which is called the basic probability distribution. For example, when the heart rate is higher than 180 beats / minute, the corresponding basic probability distribution is [0.05, 0.15, 0.8], and when the blood oxygen is lower than 95%, the basic probability distribution is [0.2, 0.6, 0.2]. The above probability distribution shows that the basic probabilities of the corresponding "dangerous" states are different when the heart rate is abnormal and the blood oxygen is abnormal, which indicates the degree of influence of different physiological indicators on the diagnosis of dangerous states. The basic probability distribution mainly comes from the statistical data and expert experience in the medical database. If there is an error between the fusion result and the actual situation, it is necessary to reversely adjust the basic probability distribution based on the data.

[0076] The credibility of an indicator refers to the conflict coefficient calculated based on the degree of difference between data sources when different data sources of the same indicator conflict (such as heart rate detection data from a bracelet and a cockpit). The conflict coefficient affects the effectiveness of such indicators during data fusion. When the conflict coefficient is large, it can be considered that there is a large error in the sampling of such indicators and it will not be used as a reference.

[0077] In an exemplary embodiment, step S3 can be replaced by the following steps: based on the abnormality detection result, using a microphone to collect voice instructions; using the acoustic model in the speech recognition module to convert the voice instructions into phonemes or words; using the language model in the speech recognition module to predict the probability of a word sequence based on the phonemes or words; and controlling the vehicle to perform corresponding operations according to the probability of the word sequence.

[0078] In an exemplary embodiment, the voice command is converted into phonemes or words using an acoustic model in a speech recognition module, specifically including: extracting the acoustic features of the voice command using different analysis methods; the different analysis methods include time domain analysis methods, frequency domain analysis methods and inverse frequency domain analysis methods; performing inverse spectrum, mean, variance and normalization processing on the acoustic features in turn to determine the processed acoustic features; based on the acoustic model, converting the processed acoustic features into phonemes or words; the acoustic model is a Gaussian mixture model.

[0079] The method of converting the voice command into phonemes or words by using the acoustic model in the voice recognition module further includes: formulating priority sorting rules based on pronunciation analysis adjusted by proprietary words in the emergency voice recognition in the cabin; adding the priority sorting rules to the pronunciation dictionary to identify the phonemes or words.

[0080] In an exemplary embodiment, based on the abnormal detection results, the driver and the cockpit are interacted with by voice recognition technology, etc., to perform multi-level vehicle mode control. Specifically:

[0081] S31: By collecting a large amount of in-vehicle scene voice data, which includes voice commands in various emergency situations, such as keywords such as "emergency stop", "turn on warning lights", and "help", the collected voice data is pre-processed, including silence removal and noise suppression, to improve the accuracy of voice recognition.

[0082] Then use speech recognition tools, such as sherpa-ncnn, to perform model training, including acoustic model training and language model training.

[0083] Among them, the acoustic model is used to convert voice signals into phonemes or words, and the language model is used to predict the probability of word sequences. After training is completed, the voice recognition system is tested and optimized, and the model parameters are adjusted according to the test results. Finally, the trained voice recognition system is deployed on the terminal side to provide voice recognition services for the vehicle.

[0084] Speech recognition technology is mainly used to recognize the user's voice commands to interact with the cockpit, including voice recognition to control the vehicle to switch to a specific mode, voice recognition to turn on the double flash lights, voice recognition to intelligently slow down, and voice recognition to send distress messages. At the same time, the cockpit supports customized voice commands to meet the needs of specific users or scenarios, providing users with a smarter and more convenient way of interaction, allowing drivers to focus more on the driving process while enjoying personalized smart cockpit services.

[0085] The structure of the specific speech recognition system is as follows: Figure 6 As shown, it mainly includes two processes: model training and speech recognition.

[0086] Among them, model training is the core work of the speech recognition system. The process includes five parts: feature extraction, acoustic model, pronunciation dictionary, language model, and speech decoding.

[0087] After the model training is completed, the trained model is used to decode the test speech in the speech library, and then the score is given, and the bit error rate indicator is used to evaluate the recognition rate of the model.

[0088] The process of speech recognition is similar to the process of scoring the model. It can basically be considered that the test voice file is replaced with the actual input audio.

[0089] The specific five-part functional design process corresponding to the speech recognition system is as follows:

[0090] Feature extraction: First, analyze and extract the characteristic parameters that can represent the essence of the speech, and then use the characteristic parameters for subsequent processing. It should be noted that according to different methods of extracting parameters, speech signal analysis can be divided into time domain, frequency domain, inverse frequency domain, and other domain analysis methods. According to different analysis methods, speech signal analysis can be divided into model analysis methods and non-model analysis methods. After feature extraction, it must also be processed by inverse spectrum mean variance normalization to ensure that the form of each characteristic parameter is consistent.

[0091] Acoustic model: After obtaining the acoustic features, these features need to be modeled. This application uses a three-factor Gaussian mixture model and neural network training, and the training generally requires multiple iterations. It should be noted that although the model after each iteration can be used, the more iterations the better the model.

[0092] Pronunciation dictionary: Generally, it is a table file of characters or words and their spoken phonemes. There are some homophones in the original pronunciation dictionary, so this application has made special treatment for the proprietary words in the emergency speech recognition in the cabin. Specifically, it includes: adding priority sorting rules to the pronunciation dictionary, so that the speech recognition system can adjust the pronunciation analysis of words according to the context, and improve the recognition accuracy.

[0093] Language model: It should be emphasized that the language model here is not modeled by explicitly specifying grammatical rules but by statistical models. For example, the N-gram language model can be used. The N-gram model has been widely used in speech recognition because of its simplicity and effectiveness. Its goal is to calculate the probability of a string appearing as a sentence. When decoding, it searches for the word order combination with the highest probability of matching. The language model is generally trained using the N-gram tool on text information from various books, web pages, newspapers, news and other resources. Generally, the more resources, the better the trained model.

[0094] Speech decoding: The acoustic model, pronunciation dictionary model, and language model trained earlier can basically be considered as graph data structures. The essence of decoding is to find the optimal path in the graph network. In order to accelerate the decoding recognition efficiency, a weighted finite state transducer (WFST) decoding mechanism is introduced, so that the dynamic knowledge source can be extracted and compiled to form a static network, which can be directly called during decoding. From the input HMM state sequence, the word sequence and its related scores are directly obtained.

[0095] S32: A multi-source fusion algorithm based on a neural network is deployed in the cockpit to perform real-time calculation and analysis on various sensor data. Once it is identified that the driver's ability is limited, the vehicle will be operated according to the model output results, such as cutting off the power system, turning on the warning lights, sending messages and alarms, sharing location information, etc. At the same time, the auxiliary automatic driving function will take over the vehicle and park it safely when necessary.

[0096] The driving takeover mode includes three modes, namely, normal mode, novice mode and guarantee mode.

[0097] When the normal mode is enabled and the vehicle is driving normally (for example, at a speed of 70 miles per hour), the multimodal data perception model regularly detects the overall health status of the human body and reports it on the host computer interface. If the heart rate is too high, the blood oxygen concentration is too low, the breathing rate is too fast, the heart rate variability is too large, etc., the user will be reminded of the possible risks in real time on the host computer interface.

[0098] When the protection mode is enabled, the multimodal data perception model regularly detects the overall health status of the human body and reports it in the host computer interface. If an abnormal condition such as a high heart rate occurs, the user will be reminded of possible risks in real time in the host computer interface. In addition, when the multimodal data perception model detects an abnormality in the health status of the human body, the automatic driving mode is turned on to take over the vehicle, and it is parked at a nearby location, while the double flashes are turned on and a distress message is sent.

[0099] When the novice mode is enabled, the vehicle speed is limited, for example, the speed cannot exceed 60 miles per hour (assuming the current speed is 50 miles per hour) and the vehicle will not explode if the accelerator is stepped on suddenly. The multimodal data perception model regularly detects the overall health status of the human body and reports it in the host computer interface. If there are abnormal conditions such as high heart rate, the user will be reminded of the possible risks in real time in the host computer interface. In addition, when the multimodal data perception model detects abnormalities in the health status of the human body, the autonomous driving mode is turned on to take over the vehicle, and the vehicle is parked at a nearby location, while the double flash is turned on and a distress message is sent.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a biosignal anomaly detection device based on a smart cabin system for implementing the above-mentioned biosignal anomaly detection method based on a smart cabin system. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the biosignal anomaly detection device based on a smart cabin system provided below can refer to the above-mentioned limitations on the biosignal anomaly detection method based on a smart cabin system, and will not be repeated here.

[0101] In an exemplary embodiment, the present application also provides a biological signal abnormality detection device based on a smart cabin system, comprising:

[0102] The smart cabin seat system is used to obtain the user's vital signs data and the pressure the user applies to the steering wheel; the vital signs data include breathing rate, heart rate and blood oxygen;

[0103] A central processing unit, configured to perform abnormality detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate an abnormality detection result;

[0104] A control module is used to control the vehicle to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

[0105] In an exemplary embodiment, Figure 7 As shown in the figure, it is developed based on Nuttx, the underlying system of Xiaomi Pengpai OS, and uses the Xiaomi AIOT development platform as the in-vehicle control terminal.

[0106] The Xiaomi base motherboard is used for centralized power supply, GD32 is used as an actuator for data collection, the development board is connected to the smart bracelet via Bluetooth through the WI-FI-BLE module, the smart bracelet is connected to the heart rate and blood oxygen sensor to detect the user's heart rate, blood oxygen concentration and other data information, and then the data is sent to the GD32 development board in real time;

[0107] A millimeter-wave radar is connected to the GD32 development board to detect the user's heart rate and breathing phase, and a pressure-sensitive film is used to detect information such as the degree of steering wheel grip. A multi-sensor fusion algorithm based on DS evidence theory is deployed on the board to perform decision-level fusion of sensor data and perform abnormal state detection.

[0108] The Horizon X3M board is used as the central processor, which is responsible for the deployment of the voice recognition model on the terminal side, and is connected to the microphone to provide voice recognition for the driver or passengers to control the cabin equipment, emergency calls and other functions. The GPS positioning module is connected to provide the driver with vehicle information positioning function. At the same time, the U2 board is used as the central control of the smart cockpit host computer, which receives the GD32 human body status detection results, interacts with the user through the on-board screen, and provides real-time status overview, abnormal status warning, network communication, and driving mode switching functions.

[0109] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store biological signal anomaly detection data based on a smart cabin system. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a biological signal anomaly detection method based on a smart cabin system is implemented.

[0110] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0111] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above method when executed by a processor.

[0112] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ReadOnlyMemory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (Magnetoresistive RandomAccess Memory, MRAM), ferroelectric random access memory (Ferroelectric RandomAccess Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (RandomAccess Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0114] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0115] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A smart cockpit system, characterized in that: include: Master device, wearable slave device and cloud platform; The main equipment includes a vehicle-mounted platform, a millimeter-wave radar, and a piezoelectric sensor; The millimeter wave radar is connected to the vehicle-mounted platform and is used to detect the user's breathing frequency; the piezoelectric sensor is provided on the steering wheel and is connected to the vehicle-mounted platform and is used to collect the pressure of the user on the steering wheel; The wearable slave device includes a heart rate sensor and a blood oxygen sensor; the heart rate sensor is used to detect the user's heart rate; the blood oxygen sensor is used to detect the user's blood oxygen; The vehicle-mounted platform is connected to the millimeter-wave radar, the heart rate sensor and the blood oxygen sensor, and is used to use a multi-source fusion algorithm to perform abnormal detection on the user's vital sign data and the pressure, and upload the vital sign data, the pressure and the abnormal detection results to the cloud platform, and control the vehicle to perform corresponding operations according to the abnormal detection results; the corresponding operations include cutting off the vehicle execution power system, turning on the warning light, messaging alarm, sharing location information and turning on the auxiliary automatic driving function; the vital sign data include respiratory rate, heart rate and blood oxygen; The cloud platform is used to send the vital sign data, the pressure and the abnormality detection result to the client.

2. The smart cockpit system according to claim 1, characterized in that: The main device also includes: a speech recognition system; The speech recognition system includes a microphone and a speech recognition module; The microphone is connected to the speech recognition module; the microphone is used to collect voice instructions; the speech recognition module is used to convert the voice instructions into phonemes or words using an acoustic model, and use a language model to predict the probability of a word sequence based on the phonemes or words; The vehicle-mounted platform is also used to control the vehicle to perform corresponding operations according to the probability of the word sequence.

3. A biological signal anomaly detection method based on a smart cabin system, characterized in that: include: Based on the smart cabin seat system described in claims 1-2, the user's vital signs data and the pressure the user applies to the steering wheel are obtained; the vital signs data include respiratory rate, heart rate and blood oxygen; Performing anomaly detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate anomaly detection results; The vehicle is controlled to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

4. The method for detecting abnormal biological signals based on the intelligent cabin seat system according to claim 3 is characterized in that: The multi-source fusion algorithm is used to perform abnormality detection on the vital sign data and the pressure to generate abnormality detection results, specifically including: Preprocessing the vital sign data to determine preprocessed vital sign data; Extracting biomedical indicators according to the time domain characteristics and frequency domain characteristics of the preprocessed vital sign data; the biomedical indicators include a heart rate indicator, a respiratory rate indicator, a blood oxygen indicator, a SDNN indicator, a NN50 indicator, and a LF / HF ratio indicator; Based on the medical database and the expert system, the biomedical indicator is evaluated to determine the importance of the biomedical indicator to the abnormal detection result and the credibility of the biomedical indicator itself; Based on the importance and the credibility, the DS evidence theory is used to generate an abnormality detection result; the abnormality detection result includes a healthy state, an abnormal state and a dangerous state.

5. The method for detecting abnormal biological signals based on the intelligent cabin seat system according to claim 3 is characterized in that: Controlling the vehicle to perform corresponding operations according to the abnormality detection result specifically includes: Based on the abnormal detection result, using a microphone to collect voice instructions; Converting the voice command into phonemes or words using an acoustic model in a voice recognition module; Using the language model in the speech recognition module, predicting the probability of a word sequence based on the phonemes or words; According to the probability of the word sequence, the vehicle is controlled to perform corresponding operations.

6. The method for detecting abnormal biological signals based on the intelligent cabin seat system according to claim 5, characterized in that: The voice command is converted into phonemes or words using an acoustic model in a voice recognition module, specifically including: Extracting acoustic features of the voice command using different analysis methods; the different analysis methods include time domain analysis method, frequency domain analysis method and inverse frequency domain analysis method; The acoustic features are sequentially subjected to cepstrum, mean, variance and normalization processing to determine the processed acoustic features; Based on the acoustic model, converting the processed acoustic features into phonemes or words; the acoustic model is a Gaussian mixture model; The method further comprises: converting the voice command into phonemes or words by using an acoustic model in a voice recognition module; and then: Formulate priority sorting rules based on pronunciation analysis of proprietary words adjusted in emergency voice recognition in the cabin; The priority sorting rules are added to the pronunciation dictionary to identify the phonemes or words.

7. A biological signal anomaly detection device based on a smart cabin system, characterized in that: The biological signal anomaly detection device based on the intelligent cabin system includes: The smart cabin seat system is used to obtain the user's vital signs data and the pressure the user applies to the steering wheel; the vital signs data include breathing rate, heart rate and blood oxygen; A central processing unit, configured to perform abnormality detection on the vital sign data and the pressure using a multi-source fusion algorithm to generate an abnormality detection result; A control module is used to control the vehicle to perform corresponding operations according to the abnormality detection results; the corresponding operations include cutting off the vehicle's power system, turning on warning lights, messaging alarms, sharing location information, and turning on assisted autonomous driving functions.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the biological signal anomaly detection method based on the smart cabin seat system according to any one of claims 3 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the biological signal anomaly detection method based on the smart cabin system described in any one of claims 3 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the biological signal anomaly detection method based on the smart cabin system described in any one of claims 3 to 6 is implemented.

Images