A driving state detection and warning system and method based on multimodal technology

Abstract

The present invention relates to a driving state detection and early warning system and method based on multimodal technology, so as to solve the technical problem that the driver is prone to red vision or black vision during high-speed driving, which leads to serious driving accidents. The system includes a helmet unit and a remote control module; the helmet unit includes a shell, an eye tracking detection module, a cerebral blood oxygen detection module, a signal processing module, a voice interaction module, a power supply module and a reference coordinate construction module. The method includes: 1. Wearing a helmet unit, constructing a reference coordinate system of the driving area, and dividing at least one high-heat area; 2. Starting each module of the helmet unit, real-time detection of the driver's eye capillary texture characteristics, attention mechanism distribution characteristics and cerebral blood oxygen real-time data characteristics, and issuing a warning signal according to the detection results. The present invention conducts a real-time safety assessment of the driver's state and issues a warning through multi-channel, multi-dimensional and multi-modal analysis.

Description

Technical Field

[0001] The present invention relates to a driving state detection system and method, and in particular to a driving state detection and early warning system and method based on multimodal technology. Background Art

[0002] When flying an airplane or driving a vehicle, when the speed is too fast, the driver will frequently experience "red vision" or "black vision", that is, when the overload on the driver is too great, resulting in blood blockage in the human body, ischemia of the eyes on the head, blurred vision, and inability to see things. At this time, the attention mechanism will become abnormal, which is black vision; if blood rushes to the head, the driver's brain will become congested and high pressure will be formed. At this time, the head will be in severe pain, the face will be swollen, and the capillaries of the eyeballs will rupture, which is red vision. The driver in this state looks at the outside world as if he is wearing a pair of red glasses, and everything around him is red, which is red vision.

[0003] When red vision occurs, the amount of blood in the human eyeball will change significantly; whether it is "red vision" or "black vision", the brain blood oxygen content will change dramatically, which can easily cause serious driving errors. Therefore, there is an urgent need for a device and method to detect the driver's driving status in real time and issue a warning based on the detection results. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problem that drivers are prone to red vision or black vision phenomenon that seriously affects driving operations during high-speed driving, which in turn leads to serious driving accidents. A driving status detection and early warning system and method based on multimodal technology are proposed. Through eye tracking technology, eye capillary detection technology and brain blood oxygen detection technology, the distribution characteristics of the driver's attention mechanism, the texture characteristics of the eye capillaries and the real-time data characteristics of brain blood oxygen are deeply excavated, and multi-channel, multi-dimensional, and multi-modal fusion analysis are performed to conduct a real-time safety assessment of the driver's status and issue a corresponding early warning.

[0005] The technical solution of the present invention is:

[0006] A driving state detection and warning system based on multimodal technology, which is special in that it includes a helmet unit and a remote control module;

[0007] The helmet unit includes a shell and an eye tracking detection module, a cerebral blood oxygen detection module, a signal processing module, a voice interaction module, a power supply module and a reference coordinate construction module arranged on the shell;

[0008] The eye tracking detection module includes a first adjustable bracket arranged on the housing and a camera module integrated on the first adjustable bracket;

[0009] The brain blood oxygen detection module includes at least one brain blood oxygen sensor, and the brain blood oxygen sensor is arranged in close contact with the brain;

[0010] The signal processing module is used to receive the output information of the camera module and the brain blood oxygen detection module, and process and output the information;

[0011] The voice interaction module is used to send the warning information output by the signal processing module to the driver in the form of voice;

[0012] The power supply module serves as a power source and is electrically connected to the camera module, the cerebral blood oxygen detection module, the signal processing module, the voice interaction module and the reference coordinate construction module;

[0013] The reference coordinate construction module is used to construct a reference coordinate system of the driving area, divide at least one high-heat area with a regular shape in the reference coordinate system, and transmit the reference coordinate system and the high-heat area to the signal processing module; the remote control module receives the output of the signal processing module and performs remote control according to the output information.

[0014] Furthermore, the helmet unit also includes front goggles;

[0015] The front goggles are arranged on the housing and are located between the camera module and the driver's eyes;

[0016] The voice interaction module includes a second adjustable bracket arranged on the shell and a voice interaction component arranged on the second adjustable bracket. The voice interaction component controls the switch of the front goggles through the signal processing module. The front goggles are electric goggles.

[0017] Furthermore, the camera module includes an infrared camera and a visible light camera; the infrared camera is used to capture the driver's real-time line of sight; the visible light camera is used to collect eye images, monitor the congestion of the driver's sclera and iris in real time, and obtain the texture characteristics of the eye capillaries.

[0018] Further, the processing module includes a data receiving submodule, a feature operation submodule and an instruction output submodule;

[0019] The data receiving submodule is used to receive output information from the camera module, the cerebral blood oxygen detection module and the reference coordinate construction module;

[0020] The feature operation submodule includes a variety of calculation models for calculating and analyzing the information received by the data receiving submodule;

[0021] The command output submodule is used to feed back the calculation and analysis results of the feature operation submodule to the driver or the remote control module.

[0022] Furthermore, the cerebral blood oxygen detection module is arranged at a position on the housing corresponding to the forehead of the brain, and the cerebral blood oxygen detection module is attached to the forehead in a patch manner;

[0023] The cerebral blood oxygen sensor is arranged inside the patch.

[0024] At the same time, the present invention also provides a driving state detection and warning method based on multimodal technology, which is special in that the driving state detection and warning system based on multimodal technology is adopted, including the following steps:

[0025] S1. Wearing a helmet unit, constructing a reference coordinate system of the driving area according to a reference coordinate construction module, and dividing at least one high-heat area with a regular shape in the reference coordinate system according to the weight distribution of the gaze area; and inputting the reference coordinate system and the high-heat area into a signal processing module of the helmet unit;

[0026] S2, starting each module of the helmet unit, performing real-time detection of multiple status characteristics of the driver and issuing a warning signal according to the detection results;

[0027] 2.1 Eye capillary texture features: Use the camera module to obtain the driver's facial image in real time, extract the eye area RGB image from the driver's facial image according to the visual matching algorithm, and input the eye area RGB image into the signal processing module; the signal processing module processes the eye area RGB image to obtain a binary eye capillary image; according to the eye capillary texture features in the binary eye capillary image, set the normal value of the capillary area feature value to R, set the risk threshold to 1.5R, and when 0.5R<capillary area feature value≤1.5R is normal; when 1.5R<capillary area feature value≤2R, a safety warning is given to the driver through the language interaction module; when the capillary area feature value is greater than 2R, a safety warning is issued to the remote control module;

[0028] 2.2 Attention mechanism distribution characteristics: The real-time sight line position coordinates of the driver are obtained by using the camera module and input into the signal processing module. According to the reference coordinate system and the high-heat area in step S1, the real-time sight line position coordinates are matched with the high-heat area; if the real-time sight line position coordinates are available and within the high-heat area, the driver is driving normally; if the real-time sight line position coordinates are available but continuously outside the high-heat area, the driver's attention is distracted, and a safety warning is issued to the driver through the language interaction module; if the real-time sight line position coordinates cannot be obtained continuously, it is determined that the driver has fainted, and a safety warning is issued to the remote control module;

[0029] 2.3 Real-time data characteristics of brain blood oxygen: The real-time data of brain blood oxygen is obtained by using the brain blood oxygen detection module, and the real-time data of brain blood oxygen is output to the signal processing module. The driver's brain blood oxygen data is predicted in real time through the long short-term memory model in the signal processing module. If the predicted brain blood oxygen saturation is greater than 90%, the driver is normal; if 80% < predicted brain blood oxygen data ≤ 90%, a safety warning is issued to the driver through the language interaction module; if the predicted brain blood oxygen data ≤ 80%, it meets the characteristics of syncope, and a safety warning is issued to the remote control module;

[0030] Step 2.1, step 2.2 and step 2.3 are performed simultaneously.

[0031] Further, in step S1, at least one high-heat area with a regular shape is divided in the reference coordinate system. Specifically, n non-overlapping rectangular high-heat areas are divided in the reference coordinate system, where n is an integer and n≥1, and the coordinate range of the first high-heat area is from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (z1, k1), and the coordinate range of the second high-heat area is from the upper left corner coordinate (x2, y2) to the lower right corner coordinate (z2, k2), until the coordinate range of the nth high-heat area is from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (z1, k1). n ,y n ) to the lower right corner coordinate (z n ,k n );

[0032] In step 2.2, the real-time sight position coordinates are matched with the high-heat area as follows:

[0033] If the real-time line of sight position coordinates (x, y) can be obtained and satisfy x i <x<z i And y i <y<k i (i=1,2,…n), the driver’s attention mechanism is distributed in the high-heat area and works normally;

[0034] If the real-time sight line position coordinates (x, y) can be obtained and does not continuously satisfy x i <x<z i And y i <y<k i (i=1,2,…n), the driver’s attention mechanism distribution deviates from the high-heat area;

[0035] If the real-time line of sight position coordinates (x, y) cannot be obtained continuously, it is determined that the driver has fainted.

[0036] Furthermore, in step 2.1, the camera module is used to obtain the driver's facial image in real time, using a visible light camera in the camera module;

[0037] The visual matching algorithm is the DLIB-68 visual matching algorithm;

[0038] The signal processing module processes the RGB image of the eyeball area specifically by using the CNN neural network model in the signal processing module to process the RGB image of the eyeball area to generate a binary eyeball capillary image.

[0039] Furthermore, the CNN neural network model includes 14 layers, the 1st to 12th layers are single-channel convolutional layers connected in sequence, the 13th layer is a three-channel convolutional layer, and the 14th layer is a fully connected layer;

[0040] The 13th layer includes a 13-a subchannel, a 13-b subchannel, and a 13-c subchannel; the 13-a subchannel includes a convolution layer, the 13-b subchannel includes two sequentially connected convolution layers 13-b-1 and 13-b-2, and the 13-c subchannel includes three sequentially connected convolution layers 13-c-1, 13-c-2, and 13-c-3;

[0041] The 12th layer copies the processed data information three times and outputs them to the 13-a subchannel, 13-b subchannel and 13-c subchannel for processing respectively;

[0042] The 13-a subchannel performs a RELU activation function operation on the data information output by the 12th layer, and outputs the result to the 14th layer;

[0043] The 13-b subchannel performs two RELU activation function operations on the data information output by the 12th layer according to the processing order of the 13-b-1 convolution layer and the 13-b-2 convolution layer, and outputs the result to the 14th layer;

[0044] The 13-c subchannel performs three RELU activation function operations on the data information output by the 12th layer in the processing order of the 13-c-1 convolution layer, the 13-c-2 convolution layer and the 13-c-3 convolution layer, and outputs the result to the 14th layer.

[0045] Furthermore, in step 2.2, the camera module is used to obtain the real-time line of sight position coordinates of the driver, using the infrared camera in the camera module.

[0046] Beneficial effects of the present invention:

[0047] 1. The appearance of black vision and red vision represents the physiological limit of human beings. If the physiological limit is not quickly escaped, an accident will occur within a few seconds. The driving status detection and warning system based on multimodal technology provided by the present invention can detect the driver's status in real time, perform real-time safety assessment and warning mechanism, detect black vision and red vision phenomena in time and make safety assessment, provide warning mechanism, and provide strong protection for the life safety of car drivers or airplane pilots.

[0048] 2. The method provided by the present invention combines eye tracking technology, eye capillary detection technology and brain blood oxygen detection technology to deeply explore the distribution characteristics of the driver's attention mechanism, capillary texture characteristics and brain blood oxygen real-time data characteristics, and conducts multi-channel, multi-dimensional and multi-modal fusion analysis, which can avoid the misjudgment of single-modal detection technology to the greatest extent, enhance system stability and improve system fault tolerance.

[0049] 3. The method provided by the present invention has a fast detection reaction speed and can quickly provide early warning feedback. The eyeball capillary detection technology and brain blood oxygen detection technology used in the present invention are both combined with the neural network model for processing, giving full play to the deep learning characteristics of the neural network model, and the neural network model has strong computing and processing performance and fast speed, which can realize end-to-end processing and eliminate the cumbersome preprocessing steps in traditional signal processing methods.

[0050] 4. In the CNN neural network model of the present invention, a single-channel convolution layer is divided into a three-channel convolution layer. Compared with the traditional single-channel convolution layer, this method reduces redundant parameters and enhances feature recognition ability and efficiency. At the same time, compared with the single-channel convolution layer of the existing CNN model that only uses the ReLU activation function once, the neural network model uses the ReLU activation function six times through the three-channel convolution layer, further enhancing the feature extraction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of a helmet unit in an embodiment of a driving state detection and warning system based on multimodal technology of the present invention;

[0052] Figure 2 It is a flow chart of the driving state detection and early warning method based on multimodal technology of the present invention;

[0053] Figure 3 A schematic diagram of the structure of a CNN neural network model used for eyeball capillary texture feature detection in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the LSTM model structure used in real-time cerebral blood oxygen data feature detection in an embodiment of the present invention.

[0055] The reference numerals are as follows:

[0056] 1-housing, 2-first adjustable bracket, 3-camera module, 4-voice interaction component, 5-front goggles, 6-brain blood oxygen detection module, 7-signal processing module, 8-power supply module, 9-goggles knob, 10-second adjustable bracket. DETAILED DESCRIPTION

[0057] In this embodiment, "left, right, up, and down" are the directions of the driver in the driving state.

[0058] See also Figure 2 This embodiment provides a driving state detection and warning method based on multimodal technology, and the method includes the following steps:

[0059] S1, Wear helmet unit; see Figure 1 The helmet unit includes a shell 1 and an eye tracking detection module, a cerebral blood oxygen detection module 6, a signal processing module 7, a front goggles 5, a voice interaction module, a power supply module 8 and a reference coordinate construction module arranged on the shell 1.

[0060] An air cushion sponge layer is arranged on the inner side of the outer shell 1 of the helmet unit to increase the wearing comfort and play a shock absorbing and buffering role.

[0061] The eye tracking detection module is located at the upper right outside of the front visor of the helmet, and includes a first adjustable bracket 2 connected to the shell 1 and a camera module 3 integrated on the first adjustable bracket 2. The height, angle and distance can be freely adjusted through the first adjustable bracket 2, so that the eye tracking detection module does not affect the driver's normal line of sight; the camera module 3 integrates an infrared camera and a visible light camera. The infrared camera is used to divide the pupil and the iris, and a coordinate mapping relationship is established between the infrared light spot on the eyeball and the center point of the pupil to obtain the driver's real-time line of sight and determine whether the driver's line of sight deviates from the high-heat area or is completely out of the high-heat area coordinate system; the visible light camera is used to collect eye images, detect the congestion of the driver's sclera and iris area in real time, and extract capillary texture features to determine whether the driver has "red vision" and its severity.

[0062] The cerebral blood oxygen detection module 6 includes at least one cerebral blood oxygen sensor, which is arranged in close contact with the brain. Specifically, since there is no hair interference in the forehead of the human body, the cerebral blood oxygen detection module 6 is arranged at a position corresponding to the forehead of the human body on the shell 1, and the cerebral blood oxygen detection module 6 is attached to the forehead in the form of a patch; the cerebral blood oxygen sensor is arranged inside the patch.

[0063] The signal processing module 7 is used to receive the output information of the camera module 3, the cerebral blood oxygen detection module 6 and the reference coordinate construction module, and to process and output the information. Specifically, the signal processing module 7 includes a data receiving submodule, a feature operation submodule and an instruction output submodule; the data receiving submodule is used to receive in real time the driver's line of sight coordinates obtained by the infrared camera in the camera module 3 and the eye area image taken by the visible light camera, the cerebral blood oxygen data detected in real time by the cerebral blood oxygen sensor patch of the cerebral blood oxygen detection module 6, and the reference coordinate system constructed by the reference coordinate construction module and the high-temperature area divided based on the reference coordinate system; the feature operation submodule includes a variety of calculation models for calculating and analyzing the information received by the data receiving submodule; the instruction output submodule is used to feed back the calculation and analysis results of the feature operation submodule to the driver or the remote control module, and the remote control module receives the output of the signal processing module 7 and performs remote control according to the output information.

[0064] The front goggles 5 are electric goggles, which are arranged on the housing 1 and are located between the camera module 3 and the driver's eyes. The electric goggles are hinged to the housing 1 through the goggles knob 9, and can be electrically adjusted up and down through the goggles knob 9 or the voice interaction module command "open the goggles" or "close the goggles" to protect the driver's safety.

[0065] The voice interaction module is located at the lower left side of the outside of the helmet, and includes a second adjustable bracket 10 and a voice interaction component 4 arranged on the second adjustable bracket 10. The voice interaction component 4 is integrated with an acoustic-to-electric conversion unit that converts acoustic signals into electrical signals and an electro-acoustic conversion unit that converts electrical signals into acoustic signals. The voice interaction component 4 can freely adjust the angle and distance through the second adjustable bracket 10. The voice interaction component 4 controls the adjustment of the electric goggles through the signal processing module 7, and is used to send warning information to the driver in voice form according to the output result of the signal processing module 7.

[0066] The power supply module is located at the top of the helmet and provides real-time power to the eye tracking detection module, the cerebral blood oxygen detection module 6, the voice interaction module, the signal processing module 7, the reference coordinate construction module and the electric goggles.

[0067] The reference coordinate construction module is used to construct a reference coordinate system of the driving area, which is a two-dimensional plane coordinate system XOY of the gaze area; the weight is allocated according to the gaze area, specifically, within a certain driving time limit, the area with a weight of ≥20% of the eye movement gaze time is a high-heat area, and the area with a weight of <20% of the eye movement gaze time is a low-heat area. In the reference coordinate system, at least one high-heat area with a regular shape is divided, and the remaining areas where the driver has little tendency to gaze are set as low-heat areas; the reference coordinate system and the high-heat area are input into the signal processing module 7 of the helmet unit; in this embodiment, according to The driver's highly inclined gaze area is divided into three rectangular high-heat areas, and each high-heat area does not contain repeated areas, that is, the three rectangular areas do not overlap with each other; the coordinate range of the first high-heat area is from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (z1, k1), the coordinate range of the second high-heat area is from the upper left corner coordinate (x2, y2) to the lower right corner coordinate (z2, k2), and the coordinate range of the third high-heat area is from the upper left corner coordinate (x3, y3) to the lower right corner coordinate (z3, k3). The rectangular range of the high-heat area can be determined according to the upper left corner and the lower right corner.

[0068] S2, starting each module of the helmet unit, performing real-time detection of multiple status characteristics of the driver and issuing a warning signal according to the detection results;

[0069] 2.1 Eye capillary texture features: The visible light camera in the camera module 3 is used to obtain the driver's facial image in real time, and the DLIB-68 visual matching algorithm is used to extract 68 feature points from the face, and the 36th to 39th feature points are screened out, that is, the range between the leftmost feature point and the rightmost feature point of the eyeball area, to obtain the RGB image of the eyeball area. It can be understood that the collection of feature points is not limited to the 36th to 39th feature points, but considering that the more feature points are collected, the greater the amount of calculation in subsequent calculations, the 36th to 39th feature points are selected on the premise of being as accurate as possible.

[0070] The RGB image of the eyeball area is input into the signal processing module 7; the CNN neural network model pre-trained in the signal processing module 7 is used to process the RGB image of the eyeball area to obtain a binary eyeball capillary image; according to the eyeball capillary texture characteristics in the binary eyeball capillary image, the normal value of the capillary area characteristic value is set to R, where the capillary area refers to the capillary texture characteristic area in the binary eyeball capillary image, reflecting the capillary texture characteristics under different conditions of capillary congestion in the sclera and iris; when driving normally, the capillary area changes in a small range, the characteristics are not significant, and the sclera is mainly white as a whole; when performing intense driving operations, the capillary area increases, the characteristics will gradually become significant, the capillary range expands, and red vision is caused. The risk threshold is set to 1.5R. When 0.5R<capillary area characteristic value≤1.5R, it is normal; when 1.5R<capillary area characteristic value≤2R, a safety warning is given to the driver through the language interaction module; when the capillary area characteristic value is greater than 2R, a safety warning is issued to the remote control module.

[0071] Extracting the eyeball capillary texture features in the eyeball area RGB image may include a traditional threshold segmentation method, a machine learning model and a neural network model. This embodiment uses a CNN neural network model; see Figure 3 The CNN neural network model includes 14 layers, the 1st to 12th layers are single-channel convolutional layers connected in sequence, the 13th layer is a three-channel convolutional layer, and the 14th layer is a fully connected layer, among which downsampling pooling is performed after the 2nd convolutional layer, the 4th convolutional layer, the 7th convolutional layer and the 10th convolutional layer respectively; the 13th layer includes 13-a sub-channel, 13-b sub-channel and 13-c sub-channel; the 13-a sub-channel contains one convolutional layer, the 13-b sub-channel contains two 13-b-1 convolutional layers and 13-b-2 convolutional layers connected in sequence, and the 13-c sub-channel contains three 13-c-1 convolutional layers, 13-c-2 convolutional layers and 13-c-3 convolutional layers connected in sequence.

[0072] The 12th layer copies the processed data information three times and outputs it to the 13-a subchannel, 13-b subchannel and 13-c subchannel for processing respectively; the 13-a subchannel performs a RELU activation function operation on the data information output by the 12th layer, and outputs the result to the 14th layer; the 13-b subchannel performs two RELU activation function operations on the data information output by the 12th layer according to the processing order of the 13-b-1 convolution layer and the 13-b-2 convolution layer, and outputs the result to the 14th layer; the 13-c subchannel performs three RELU activation function operations on the data information output by the 12th layer according to the processing order of the 13-c-1 convolution layer, the 13-c-2 convolution layer and the 13-c-3 convolution layer, and outputs the result to the 14th layer; the entire operation of the 13th layer performs a total of 6 activation function operations.

[0073] Compared with the traditional single-channel convolution layer, this method reduces the redundant parameters of the 13th layer, enhances the feature recognition ability and efficiency, and compared with the single-channel convolution layer that only uses the ReLU activation function once, the network uses the ReLU activation function six times in three channels. The formula is as follows:

[0074]

[0075] Among them, x is data information, and i ranges from (1 to +∞); multiple ReLU activation function operations overcome the problem of gradient vanishing, speed up training, and increase the nonlinearity of the neural network model. The neural network can arbitrarily approximate any nonlinear function. Therefore, multiple uses of the ReLU activation function can enhance the ability to extract features.

[0076] 2.2 Attention mechanism distribution characteristics: The infrared camera in the camera module 3 is used to obtain the driver's real-time line of sight position coordinates, and input them into the signal processing module 7. According to the reference coordinate system and the high-heat area in step S1, the real-time line of sight position coordinates are matched with the high-heat area in real time; if the real-time line of sight position coordinates can be obtained and are within the high-heat area, the driver is driving normally; if the real-time line of sight position coordinates can be obtained but are continuously outside the high-heat area, the driver's attention is distracted, and a safety warning is issued to the driver through the language interaction module; if the real-time line of sight position coordinates cannot be obtained continuously, it is determined that the driver has fainted, and a safety warning is issued to the remote control module.

[0077] Specifically, for the high-heat areas in the three rectangular areas, if the real-time line of sight position coordinates are (x, y), they can be obtained and satisfy x i <x<z i And y i <y<k i (i=1,2,3), the driver's attention mechanism is located in the high heat area and works normally; if the real-time sight position coordinates are (x, y) can be obtained, the x is not satisfied continuously. i <x<z i And y i <y<k i (i=1,2,3), the driver's attention mechanism distribution deviates from the high-heat area, and a safety warning is given to the driver through the language interaction module; if the real-time line of sight position coordinates (x, y) cannot be obtained continuously, it is determined that the driver has fainted, and a safety warning is issued to the remote control module.

[0078] 2.3 Real-time data characteristics of cerebral blood oxygen: The real-time data of cerebral blood oxygen is obtained by using the cerebral blood oxygen detection module 6, and the real-time data of cerebral blood oxygen is output to the signal processing module 7. The driver's cerebral blood oxygen data is predicted in real time through the long short-term memory model (LSTM model) in the signal processing module 7. If the predicted cerebral blood oxygen saturation is greater than 90%, the driver is normal; if 80% < the predicted cerebral blood oxygen data is greater than ≤90%, a safety warning is issued to the driver through the language interaction module; if the predicted cerebral blood oxygen data is ≤80%, it meets the characteristics of syncope, and a safety warning is issued to the remote control module.

[0079] It is understandable that the LSTM long-short time model has a memory function, so it can be combined with the brain blood oxygen value of the driver who has previously experienced dizziness or syncope and the characteristics of the up and down changes in the data for learning and memory. The LSTM long-short time memory model can avoid the problem of gradient vanishing in traditional neural networks; the actual limitation of gradient vanishing is that the model cannot learn long-term dependencies. In the training of traditional neural networks, as the training time increases and the number of network layers increases, it is easy to have gradient explosion or gradient vanishing problems, resulting in the inability to process longer sequence data and thus the inability to obtain information from long-distance data. The LSTM long-short time memory model can run in real time for a long time and has individual differentiation processing.

[0080] The Sigmoid function is defined by the following formula S(x), which is used for the output of hidden layer neurons and has a value range of (0,1). It can map a real number to the interval (0,1). The tanh activation function is a symmetric function centered at zero and has a fast convergence speed; the function expression is as follows:

[0081]

[0082] Among them, x is the data input from outside the LSTM network;

[0083] like Figure 4 As shown in the LSTM long short-term memory model, the degree of information forgetting (retention) is first calculated to obtain f t , where W f and b f Parameters generated for full connection:

[0084] f t =S(W f ·[h t-1 ,x t ]+b f )

[0085] Then calculate the input part, one part is how much information needs to be saved to the unit state at the current time t, and get i t :

[0086] it =S(W i ·[h t-1 ,x t ]+b i )

[0087] The other part is to add the new information generated by the current input at time t to the unit state to obtain c t ～ :

[0088] c t ～ =tanh(W c ·[h t-1 ,x t ]+b c )

[0089] These two parts generate a new memory state. Therefore, the unit state at the current time t is the product of the input of the forget gate retention degree and the state at time t-1 plus the product of the two parts of the input gate:

[0090] c t =ft·c t-1 +i t ·c t ～

[0091] Finally, calculate the degree to which the information is output at the current time t:

[0092] o t =S(W o ·[h t-1 ,x t ]+b o )

[0093] h t =o t ·tanh(c t )

[0094] Among them, f t is the forget gate, W f and b f is the fully connected parameter of the forget gate, h t is the short-term memory unit of the LSTM unit at time t, h t-1 is the short-term memory unit of the LSTM unit at time t-1, x t is the data input from outside the LSTM network at time t, c t is the input gate, i t Preserve information for the input gate, W i and b i The fully connected parameters that preserve information for the input gate, c t ～Add information to the input gate, W c and b c The fully connected parameters that add information to the input gate, C t is the long-term memory unit of the LSTM unit at time t, C t-1 is the long-term memory unit of the LSTM unit at time t-1, o t is the output gate, W o and b o is the fully connected parameter of the output gate.

[0095] The above detection of eye capillary texture characteristics, attention mechanism distribution characteristics and brain blood oxygen real-time data characteristics is carried out simultaneously.

[0096] In the method provided in this embodiment, the signal processing module 7 performs multi-channel, multi-dimensional, and multi-modal fusion analysis on the driver's eye capillary texture features, attention mechanism distribution features, and brain blood oxygen real-time data features, and feeds back the analysis structure to the driver in the form of voice interaction or transmits it to the remote control module in an emergency situation; it can be understood that in order to ensure safe driving, when at least one of the eye capillary texture features, attention mechanism distribution features, and brain blood oxygen real-time data features meets the requirement of providing a safety warning to the driver, a safety warning is provided to the driver through the language interaction module, and the driver can perform emergency operations to avoid causing greater losses; when at least one of the eye capillary texture features, attention mechanism distribution features, and brain blood oxygen real-time data features meets the requirement of syncope, a safety warning is issued to the remote control module. According to the information obtained by the remote control module, if it is detected that the driver's state has exceeded the load limit and syncope has occurred, further measures can be taken, such as remotely controlling the aircraft to start the autopilot mode to effectively protect the lives of relevant personnel.

Claims

1. A driving status detection and early warning method based on multimodal technology, characterized in that: A driving state detection and warning system based on multimodal technology is adopted, and the driving state detection and warning system based on multimodal technology comprises a helmet unit and a remote control module; the helmet unit comprises a shell (1) and an eye tracking detection module, a brain blood oxygen detection module (6), a signal processing module (7), a voice interaction module, a power supply module (8) and a reference coordinate construction module arranged on the shell (1); the eye tracking detection module comprises a first adjustable bracket (2) arranged on the shell (1) and a camera module (3) integrated on the first adjustable bracket (2); the brain blood oxygen detection module (6) comprises at least one brain blood oxygen sensor, and the brain blood oxygen sensor is arranged in contact with the brain; the signal processing module (7) is used to receive the camera module (3) and the brain blood oxygen detection module (6). The output information of the signal processing module (6) is processed and output; the voice interaction module is used to send the warning information output by the signal processing module (7) to the driver in the form of voice; the power supply module (8) is used as a power supply and is electrically connected to the camera module (3), the cerebral blood oxygen detection module (6), the signal processing module (7), the voice interaction module and the reference coordinate construction module; the reference coordinate construction module is used to construct a reference coordinate system of the driving area, divide at least one high-heat area with a regular shape in the reference coordinate system, and transmit the reference coordinate system and the high-heat area to the signal processing module (7); the remote control module receives the output of the signal processing module (7) and performs remote control according to the output information; the camera module (3) includes an infrared camera and a visible light camera; The infrared camera is used to capture the driver's real-time line of sight; the visible light camera is used to collect eye images, monitor the congestion of the driver's sclera and iris in real time, and obtain the texture characteristics of the eyeball capillaries; The signal processing module (7) comprises a data receiving submodule, a feature operation submodule and an instruction output submodule; the data receiving submodule is used to receive output information of the camera module (3), the cerebral blood oxygen detection module (6) and the reference coordinate construction module; The feature operation submodule includes a variety of calculation models for calculating and analyzing the information received by the data receiving submodule; the instruction output submodule is used to feed back the calculation and analysis results of the feature operation submodule to the driver or the remote control module; The helmet unit further comprises a front goggles (5); the front goggles (5) are arranged on the housing (1) and are located between the camera module (3) and the driver's eyes; the voice interaction module comprises a second adjustable bracket (10) arranged on the housing (1) and a voice interaction component (4) arranged on the second adjustable bracket (10); the voice interaction component (4) controls the switch of the front goggles (5) through the signal processing module (7); the front goggles (5) are electric goggles; the brain blood oxygen detection module (6) is arranged on the housing (1) at a position corresponding to the forehead of the brain, and the brain blood oxygen detection module (6) is attached to the forehead in the form of a patch; the brain blood oxygen sensor is arranged inside the patch; the steps are as follows: S1. Wearing a helmet unit, constructing a reference coordinate system of the driving area according to a reference coordinate construction module, and dividing at least one high-heat area with a regular shape in the reference coordinate system according to the weight distribution of the gaze area; and inputting the reference coordinate system and the high-heat area into a signal processing module (7) of the helmet unit; dividing at least one high-heat area with a regular shape in the reference coordinate system is specifically as follows: dividing n non-overlapping rectangular high-heat areas in the reference coordinate system, n is an integer and n≥1, the coordinate range of the first high-heat area is from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (z1, k1), the coordinate range of the second high-heat area is from the upper left corner coordinate (x2, y2) to the lower right corner coordinate (z2, k2), until the coordinate range of the nth high-heat area is from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (z1, k1), and the coordinate range of the nth high-heat area is from the upper left corner coordinate (x2, y2) to the lower right corner coordinate (z2, k2). n ,y n ) to the lower right corner coordinate (z n ,k n ); S2, starting each module of the helmet unit, performing real-time detection of multiple status characteristics of the driver and issuing a warning signal according to the detection results; 2.1 Eyeball capillary texture features: The camera module (3) is used to obtain the driver's facial image in real time, and the eyeball area RGB image is extracted from the driver's facial image according to the visual matching algorithm, and the eyeball area RGB image is input into the signal processing module (7); the signal processing module (7) processes the eyeball area RGB image to obtain a binary eyeball capillary image; the signal processing module (7) processes the eyeball area RGB image specifically as follows: the CNN neural network model in the signal processing module (7) is used to process the eyeball area RGB image to generate a binary eyeball capillary image The CNN neural network model includes 14 layers, the 1st to 12th layers are single-channel convolutional layers connected in sequence, the 13th layer is a three-channel convolutional layer, and the 14th layer is a fully connected layer; the 13th layer includes 13-a subchannel, 13-b subchannel and 13-c subchannel; 13-a subchannel includes one convolutional layer, 13-b subchannel includes two 13-b-1 convolutional layers and 13-b-2 convolutional layers connected in sequence, and 13-c subchannel includes three 13-c-1 convolutional layers, 13-c-2 convolutional layers and 13-c-3 convolutional layers connected in sequence; the 12th layer processes the processed The data information is copied three times and output to the 13-a sub-channel, 13-b sub-channel and 13-c sub-channel for processing respectively; the 13-a sub-channel performs a RELU activation function operation on the data information output by the 12th layer, and outputs the result to the 14th layer; the 13-b sub-channel performs two RELU activation function operations on the data information output by the 12th layer according to the processing order of the 13-b-1 convolution layer and the 13-b-2 convolution layer, and outputs the result to the 14th layer; the 13-c sub-channel performs two RELU activation function operations on the data information output by the 12th layer according to the processing order of the 13-c-1 convolution layer, The processing order of the 13-c-2 convolution layer and the 13-c-3 convolution layer is to perform three RELU activation function operations, and the results are output to the 14th layer; according to the eye capillary texture features in the binary eye capillary image, the normal value of the capillary area feature value is set to R, and the risk threshold is set to 1.5R. When 0.5R<capillary area feature value≤1.5R, it is normal; when 1.5R<capillary area feature value≤2R, a safety warning is given to the driver through the language interaction module; when the capillary area feature value is greater than 2R, a safety warning is issued to the remote control module; 2.2 Attention mechanism distribution characteristics: The real-time sight line position coordinates of the driver are obtained by using the camera module (3) and input into the signal processing module (7). According to the reference coordinate system and the high-heat area in step S1, the real-time sight line position coordinates are matched with the high-heat area; if the real-time sight line position coordinates are available and are within the high-heat area, the driver is driving normally; if the real-time sight line position coordinates are available but are continuously outside the high-heat area, the driver's attention is distracted, and a safety warning is issued to the driver through the language interaction module; if the real-time sight line position coordinates cannot be obtained continuously, it is determined that the driver has fainted, and a safety warning is issued to the remote control module; matching the real-time sight line position coordinates with the high-heat area is specifically as follows: If the real-time line of sight position coordinates (x, y) can be obtained and satisfy x i <x<z i And y i <y<k i (i=1,2,…n), the driver’s attention mechanism is located in the high-heat area and works normally; if the real-time sight position coordinates (x, y) can be obtained and x is not satisfied continuously i <x<z i And y i <y<k i (i=1,2,…n), the driver’s attention mechanism distribution deviates from the high-heat area; if the real-time sight line position coordinates (x, y) cannot be obtained continuously, it is determined that the driver has fainted; 2.3 Cerebral blood oxygen real-time data characteristics: The cerebral blood oxygen detection module (6) is used to obtain the real-time data of cerebral blood oxygen, and the real-time data of cerebral blood oxygen is output to the signal processing module (7). The driver's cerebral blood oxygen data is predicted in real time through the long short-term memory model in the signal processing module (7). If the predicted cerebral blood oxygen saturation is greater than 90%, the driver is normal; if 80% < predicted cerebral blood oxygen data ≤ 90%, a safety warning is issued to the driver through the language interaction module; if the predicted cerebral blood oxygen data ≤ 80%, it meets the characteristics of syncope, and a safety warning is issued to the remote control module; Step 2.1, step 2.2 and step 2.3 are performed simultaneously.

2. The driving state detection and early warning method based on multimodal technology according to claim 1 is characterized by: In step 2.1, the camera module (3) is used to obtain the driver's facial image in real time, using the visible light camera in the camera module (3); The visual matching algorithm is the DLIB-68 visual matching algorithm.

3. The driving state detection and early warning method based on multimodal technology according to claim 2 is characterized by: In step 2.2, the camera module (3) is used to obtain the real-time line of sight position coordinates of the driver, using the infrared camera in the camera module (3).

Images