Driver state monitoring method, system, medium and product
Through multimodal biometric technology and intelligent learning module, combined with facial, voice and heart rate data, accurate judgment and personalized monitoring of driver status are achieved, solving the problems of low recognition accuracy and slow response speed in existing systems.
Patent Information
- Application Number
- CN202411385656.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-14
AI Technical Summary
The existing driver status monitoring system has low recognition accuracy, slow response speed and lacks personalization, making it difficult to adapt to the physiological and behavioral differences of different drivers, resulting in more false alarms or missed reports.
Multimodal biometric technology is adopted to obtain facial, voice and heart rate data, feature extraction and attention weight calculation, integrate global feature vectors to judge the driver's status, and combine intelligent learning and dynamic adjustment modules to adjust and alert in real time.
It improves the identification accuracy and response speed of driver status monitoring, adapts to the personalized needs of different drivers, and reduces false alarms and missed reports.
Smart Images
Figure CN119305569A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of biometric identification technology, and in particular to a driver status monitoring method, system, medium and product. Background Art
[0002] Existing driver status monitoring systems mainly rely on a single biometric technology (such as facial recognition or heart rate monitoring) to determine the driver's status. These systems usually use traditional image processing algorithms (such as Haar features) or basic heart rate variability analysis to determine the driver's fatigue or attention level. However, in complex lighting conditions or noisy environments, a single biometric method is prone to errors, resulting in system misjudgment; and due to the complexity of the algorithm and the processing capacity of sensor data, the system's response speed is slow. The system cannot adapt to the physiological and behavioral differences of different drivers, resulting in a large number of false alarms or missed alarms.
[0003] Some advanced systems have begun to introduce multimodal biometric technology to improve monitoring accuracy by combining facial recognition, voice analysis and heart rate monitoring. These systems usually use machine learning algorithms to conduct comprehensive analysis of multiple sensor data to improve the overall judgment ability of the system. However, most existing multimodal fusion algorithms are based on simple weighted averages of rules, which makes it difficult to fully utilize the complex relationship between multiple modalities; and these systems lack the ability to dynamically learn and adaptively adjust the driver's individual behavior patterns, resulting in the inability to maintain high accuracy after long-term use. Summary of the invention
[0004] The main purpose of the present invention is to propose a driver status monitoring method, aiming to solve the problems of low recognition accuracy, slow response speed and lack of personalization in existing driver status monitoring systems.
[0005] To achieve the above object, the present invention provides a driver status monitoring method, the driver status monitoring method comprising:
[0006] Obtain modal data of multiple modes;
[0007] Perform feature extraction on modal data of multiple modes to obtain feature vectors of multiple modes;
[0008] Calculating attention weights of multiple modalities according to the feature vectors of the multiple modalities;
[0009] Calculating a global feature vector after fusion of multiple modalities according to the attention weights of the multiple modalities and the feature vectors of the multiple modalities;
[0010] The driver's state is determined according to the global feature vector.
[0011] In one embodiment, the feature vectors of the multiple modalities include: a facial feature vector, a voice feature vector, and a heart rate feature vector;
[0012] The step of extracting features from modal data of multiple modalities to obtain feature vectors of multiple modalities specifically includes:
[0013] Extracting features of facial images in the modal data of the multiple modalities to obtain facial feature vectors;
[0014] Extracting features of speech signals in the modal data of the multiple modalities to obtain speech feature vectors;
[0015] The modal data center frequencies of the multiple modes are extracted to obtain the heart rate feature vectors.
[0016] In one embodiment, the step of extracting features from facial images in the modal data of the multiple modalities to obtain facial feature vectors specifically includes:
[0017] A convolutional neural network is used to convolve the facial images in the modal data of the multiple modalities to obtain a facial feature vector.
[0018] In one embodiment, the step of extracting features from speech signals in the modal data of the multiple modalities to obtain speech feature vectors specifically includes:
[0019] Mel-frequency cepstral coefficients are used to extract the acoustic features of speech signals from modal data of multiple modes.
[0020] According to the acoustic features, a long short-term memory network is used to calculate a language feature vector.
[0021] In one embodiment, the step of extracting the modal data center heart rate of the multiple modalities to obtain the heart rate feature vector specifically includes:
[0022] Preprocessing the heart rate signals in the modal data of the multiple modalities;
[0023] extracting heart rate variability of the preprocessed heart rate signal;
[0024] According to the heart rate variability, a heart rate feature vector is calculated using a long short-term memory network.
[0025] In one embodiment, the step of calculating the attention weights of multiple modalities according to the feature vectors of the multiple modalities specifically includes:
[0026] According to the facial feature vector, the facial attention weight is calculated by the normalized exponential function;
[0027] According to the speech feature vector, the speech attention weight is calculated by a normalized exponential function;
[0028] According to the heart rate feature vector, the heart rate attention weight is calculated by the normalized exponential function.
[0029] In one embodiment, the step of determining the driver's state according to the global feature vector specifically includes:
[0030] Inputting the global feature vector into a fully connected layer;
[0031] Obtain the probability distribution vector of the driver’s state through the fully connected layer;
[0032] The driver's state is determined based on the probability distribution vector of the driver's state.
[0033] In addition, to achieve the above-mentioned purpose, the present invention also proposes a driver status monitoring system, which includes: a multimodal biometric recognition module, an intelligent learning and dynamic adjustment module, and a safety warning module;
[0034] The multimodal biometric module is used to collect multi-dimensional physiological and behavioral data of the driver;
[0035] The intelligent learning and dynamic adjustment module is used to strengthen the learning strategy and make real-time adjustments and warnings according to the driver's status;
[0036] The safety warning module is used to issue an alarm and take corresponding safety measures when it detects that the driver is tired, inattentive or emotional.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps applied to the driver status monitoring method as described above are implemented.
[0038] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the driver status monitoring method described above are implemented.
[0039] The present invention discloses a driver status monitoring method, system, medium and product, which relate to the field of biometric identification technology; the driver status monitoring method comprises: obtaining modal data of multiple modalities; extracting features of the modal data of multiple modalities to obtain feature vectors of multiple modalities; calculating the attention weights of multiple modalities according to the feature vectors of the multiple modalities; calculating the global feature vector after the fusion of multiple modalities according to the attention weights of the multiple modalities and the feature vectors of the multiple modalities; judging the driver's status according to the global feature vector. The present invention solves the problems of low recognition accuracy, slow response speed and lack of personalization in the existing driver status monitoring system by adopting a multimodal fusion algorithm and introducing an attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0041] Figure 1 It is a flowchart of a first embodiment of a driver status monitoring method of the present invention;
[0042] Figure 2 A schematic diagram of a flow chart of a second embodiment of a driver status monitoring method of the present invention;
[0043] Figure 3 A schematic diagram of a flow chart of a third embodiment of a driver status monitoring method of the present invention;
[0044] Figure 4 A schematic diagram of a flow chart of a fourth embodiment of a driver status monitoring method according to the present invention;
[0045] Figure 5 A schematic diagram of a fifth embodiment of a driver status monitoring method according to the present invention;
[0046] Figure 6 A schematic diagram of a sixth embodiment of a driver status monitoring method according to the present invention;
[0047] Figure 7 Schematic diagram of the seventh embodiment of the driver status monitoring method of the present invention.
[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.
[0051] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0052] Reference Figure 1 , Figure 1 Schematic diagram of the process of the first embodiment of the driver status monitoring method of the present invention.
[0053] In this embodiment, the driver status monitoring method includes steps S10 to S50;
[0054] S10: Acquire modal data of multiple modes.
[0055] S20: extracting features from modal data of multiple modes to obtain feature vectors of multiple modes.
[0056] S30: Calculate attention weights of multiple modalities according to the feature vectors of the multiple modalities.
[0057] S40: Calculate a global feature vector after fusion of multiple modalities according to the attention weights of the multiple modalities and the feature vectors of the multiple modalities.
[0058] S50: Determine the driver's state according to the global feature vector.
[0059] It should be noted that the multiple modes may be two modes, three modes, or any other number and type of modes, which is not limited in this embodiment.
[0060] It should be noted that the modal data may be text, image, audio, video, or heart rate or other types of data, which is not limited in this embodiment.
[0061] Understandably, the attention mechanism allows the model to dynamically adjust the weights of different modalities in order to pay more attention to important sources of information during the decision-making process.
[0062] In specific implementation, modal data of multiple modalities are acquired; features are extracted from the modal data of multiple modalities to obtain feature vectors of multiple modalities; attention weights of the multiple modalities are calculated based on the feature vectors of the multiple modalities; a global feature vector after fusion of the multiple modalities is calculated based on the attention weights of the multiple modalities and the feature vectors of the multiple modalities; and the driver's status is determined based on the global feature vector.
[0063] This embodiment solves the problems of low recognition accuracy, slow response speed and lack of personalization in the existing driver status monitoring system by adopting a multimodal fusion algorithm and introducing an attention mechanism.
[0064] Based on the first embodiment of the present invention, in the second embodiment of the present invention, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 Schematic diagram of the flow chart of the second embodiment of the driver status monitoring method of the present invention.
[0065] In this embodiment, the feature vectors of the multiple modalities include: a facial feature vector, a voice feature vector, and a heart rate feature vector; the step S20 includes steps S210 to S230:
[0066] S210: Extract features of facial images in the modal data of the multiple modalities to obtain facial feature vectors.
[0067] S220: Extract features of speech signals in the modal data of the multiple modalities to obtain speech feature vectors.
[0068] S230: Extract the modal data center frequencies of the multiple modes to obtain a heart rate feature vector.
[0069] It should be noted that the method for extracting features from facial images in the modal data of the multiple modalities may use color histograms, texture features, shape features, or may use convolutional networks or other image processing technologies, which is not limited in this embodiment.
[0070] It should be noted that the method of extracting features of speech signals in the modal data of the multiple modalities to obtain speech feature vectors may use Mel-frequency cepstral coefficients or short-time Fourier transform, or may use a variant of a recurrent neural network or a convolutional neural network, which is not limited in this embodiment.
[0071] It should be noted that the method of extracting the modal data center frequency of the multiple modes to obtain the heart rate feature vector can adopt time domain analysis, frequency domain analysis, nonlinear analysis, wavelet transform, Hilbert-Huang transform or other related methods, which is not limited in this embodiment.
[0072] In a specific implementation, feature extraction is performed on facial images in the modal data of the multiple modalities to obtain a facial feature vector; feature extraction is performed on voice signals in the modal data of the multiple modalities to obtain a voice feature vector; and heart rate is extracted from the modal data of the multiple modalities to obtain a heart rate feature vector.
[0073] Based on the second embodiment of the present invention, in the third embodiment of the present invention, the same or similar contents as those in the second embodiment can be referred to the above introduction, and will not be described in detail later. Figure 3 , Figure 3 Schematic diagram of the process of the third embodiment of the driver status monitoring method of the present invention.
[0074] In this embodiment, the feature vectors of the multiple modalities include: a facial feature vector, a voice feature vector, and a heart rate feature vector; the step S210 specifically includes step S211;
[0075] S211: Using a convolutional neural network to convolve facial images in the modal data of the multiple modalities to obtain facial feature vectors.
[0076] As you can understand, a convolutional neural network consists of multiple convolutional layers, pooling layers, and fully connected layers: The convolutional layer is used to extract features through convolution operations. Each convolutional layer usually includes multiple convolution kernels. Each convolution kernel performs a convolution operation on the input data to obtain a feature map. The pooling layer is used to reduce the size of the feature map by downsampling to enhance the robustness and feature extraction capabilities of the model. Commonly used pooling operations include maximum pooling and average pooling. The fully connected layer is used to flatten the feature map, integrate the information in the feature map through a fully connected method, and output the final classification result or regression value.
[0077] It is understandable that in the convolution layer of the convolutional neural network, each neuron is only connected to a part of the input data (i.e., the local receptive field), so that local features can be extracted and the feature extraction ability of the model can be enhanced. The weights of the same convolution kernel of the convolutional neural network at different positions are the same, which greatly reduces the model parameters and improves the generalization ability of the model. Due to the locality of the convolution operation and weight sharing of the convolutional neural network, the convolutional neural network has a certain robustness to the translation transformation of the input data. Convolutional neural networks can automatically learn features without manually designing feature extractors, reducing the difficulty and cost of feature engineering.
[0078] In this embodiment, the facial feature vector is calculated as follows: face =CNN(I); where F face is the facial feature vector, CNN is the convolutional neural network, and I is the facial image.
[0079] In a specific implementation, a convolutional neural network is used to convolve the facial images in the modal data of the multiple modalities to obtain the facial feature vectors.
[0080] Based on the third embodiment of the present invention, in the fourth embodiment of the present invention, the same or similar contents as those in the third embodiment can refer to the above introduction, and will not be repeated in the following. Figure 4 , Figure 4 Schematic diagram of the fourth embodiment of the driver status monitoring method of the present invention.
[0081] In this embodiment, the feature vectors of the multiple modalities include: a facial feature vector, a voice feature vector, and a heart rate feature vector; the step S220 specifically includes steps S221 to S222;
[0082] S221: Using Mel-frequency cepstral coefficients to extract speech signals from modal data of multiple modes to obtain acoustic features.
[0083] S222: Calculate language feature vectors using a long short-term memory network based on the acoustic features.
[0084] It can be understood that the Mel frequency cepstral coefficient is a feature extraction method widely used in speech signal processing. This method is based on the auditory characteristics of the human ear and converts complex audio signals into easy-to-process acoustic features through a series of signal processing steps.
[0085] It can be understood that the long short-term memory network is a special recurrent neural network. LSTM can effectively process variable-length audio signals and realize speech recognition and understanding.
[0086] In this embodiment, the calculation formula of the facial feature vector is: voice =LSTM(MFCC(S(t))); where F voice is the language feature vector, LSTM is the long short-term memory network, MFCC is the Mel frequency cepstral coefficient, and S(t) is the speech signal.
[0087] In a specific implementation, Mel-frequency cepstral coefficients are used to extract speech signals from modal data of multiple modes to obtain acoustic features; based on the acoustic features, a long short-term memory network is used to calculate language feature vectors.
[0088] Based on the fourth embodiment of the present invention, in the fifth embodiment of the present invention, the same or similar contents as those in the fourth embodiment can be referred to the above introduction, and will not be described in detail later. Figure 5 , Figure 5 Schematic diagram of the fifth embodiment of the driver status monitoring method of the present invention.
[0089] In this embodiment, the feature vectors of the multiple modalities include: facial feature vectors, voice feature vectors, and heart rate feature vectors; the step S230 specifically includes steps S231 to S233;
[0090] S231: Preprocess the heart rate signal in the modal data of the multiple modalities.
[0091] S232: Extracting the heart rate variability of the preprocessed heart rate signal.
[0092] S233: Calculate the heart rate feature vector using a long short-term memory network according to the heart rate variability.
[0093] It should be noted that the method of preprocessing the heart rate signal in the modal data of the multiple modalities may include data cleaning, signal denoising, signal normalization, data enhancement, storage and visualization, etc., which is not limited in this embodiment.
[0094] It should be noted that extracting the heart rate variability of the preprocessed heart rate signal may include time domain analysis, frequency domain analysis, nonlinear analysis, etc., which is not limited in this embodiment.
[0095] It can be understood that the long short-term memory network is suitable for processing heart rate signals with time series characteristics; it can solve the gradient vanishing or gradient exploding problems that are prone to occur in traditional RNNs when processing long sequences, and can be used to extract long-term dependencies in time series data.
[0096] In this embodiment, the calculation formula of the facial feature vector is: HR =LSTM(HRV(H(t))); where F HRis the language feature vector, LSTM is the long short-term memory network, HRV is the algorithm for extracting heart rate variability, and H(t) is the heart rate signal.
[0097] In a specific implementation, the heart rate signals in the modal data of the multiple modalities are preprocessed; the heart rate variability of the preprocessed heart rate signals is extracted; and according to the heart rate variability, the heart rate feature vector is calculated using a long short-term memory network.
[0098] Based on the fifth embodiment of the present invention, in the sixth embodiment of the present invention, the same or similar contents as those in the fifth embodiment can refer to the above introduction, and will not be described in detail later. Figure 6 , Figure 6 Schematic diagram of the sixth embodiment of the driver status monitoring method of the present invention.
[0099] In this embodiment, the feature vectors of the multiple modalities include: facial feature vectors, voice feature vectors, and heart rate feature vectors; the step S30 specifically includes steps S310 to S330;
[0100] S310: Calculating facial attention weights through a normalized exponential function according to the facial feature vector;
[0101] S320: Calculating a speech attention weight through a normalized exponential function according to the speech feature vector;
[0102] S330: Calculate the heart rate attention weight according to the heart rate feature vector using a normalized exponential function.
[0103] It can be understood that the normalized exponential function makes the prediction results of the model more intuitive and easier to understand by converting the output into the form of probability distribution.
[0104] In this embodiment, the calculation formula of the attention weight is: Where W i is a trainable parameter, F i Represents the feature vector of the i-th modality, i∈{face,voice,HR}.
[0105] In specific implementation, the facial attention weight is calculated by a normalized exponential function based on the facial feature vector; the speech attention weight is calculated by a normalized exponential function based on the speech feature vector; and the heart rate attention weight is calculated by a normalized exponential function based on the heart rate feature vector.
[0106] Based on the sixth embodiment of the present invention, in the seventh embodiment of the present invention, the same or similar contents as those in the sixth embodiment can refer to the above introduction, and will not be repeated in the following. Figure 7 , Figure 7Schematic diagram of the seventh embodiment of the driver status monitoring method of the present invention.
[0107] In this embodiment, the feature vectors of the multiple modalities include: facial feature vectors, voice feature vectors, and heart rate feature vectors; the step S50 specifically includes steps S510 to S530;
[0108] S510: Input the global feature vector to a fully connected layer.
[0109] S520: Obtain the probability distribution vector of the driver state through the fully connected layer.
[0110] S530: Determine the driver's state according to the probability distribution vector of the driver's state.
[0111] As you can understand, the main feature of the fully connected layer is that each of its nodes is connected to all the nodes of the previous layer, which is used to integrate and map the features extracted by the previous layers (such as convolutional layers, pooling layers, etc.). This layer calculates the connection weights between the input features and each neuron through matrix multiplication and bias addition operations to obtain the output result.
[0112] It should be noted that the driver's state includes but is not limited to the driver's concentration level, fatigue level, and emotional state, which are not limited in this embodiment.
[0113] In this embodiment, the calculation formula of the probability distribution vector is: state =softmax(W fusion ·F fusion +b); where P state is the probability distribution vector of the driver’s state, W fusion is a trainable parameter and b is the bias of the fully connected layer.
[0114] The present invention provides a driver status monitoring system, which includes: a multimodal biometric recognition module, an intelligent learning and dynamic adjustment module, and a safety warning module;
[0115] The multimodal biometric module is used to collect multi-dimensional physiological and behavioral data of the driver;
[0116] The intelligent learning and dynamic adjustment module is used to strengthen the learning strategy and make real-time adjustments and warnings according to the driver's status;
[0117] The safety warning module is used to issue an alarm and take corresponding safety measures when it detects that the driver is tired, inattentive or emotional.
[0118] The driver status monitoring system provided by the present invention adopts the driver status monitoring method in the above embodiment, which can solve the problems of low recognition accuracy, slow response speed and lack of personalization in the existing driver status monitoring system. Compared with the prior art, the beneficial effects of the driver status monitoring system provided by the present invention are the same as the beneficial effects of the driver status monitoring method provided by the above embodiment, and other technical features of the driver status monitoring system are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0119] It should be understood that the various parts disclosed in the present invention can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0120] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
[0121] The present invention provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the driver status monitoring method in the above-mentioned embodiment.
[0122] The computer-readable storage medium provided by the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0123] The computer-readable storage medium may be included in the driver status monitoring system; or may exist independently without being assembled into the driver status monitoring system.
[0124] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0126] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.
[0127] The readable storage medium provided by the present invention is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned driver status monitoring method, and can solve the problems of low recognition accuracy, slow response speed, and lack of personalization in the existing driver status monitoring system. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the driver status monitoring method provided by the above-mentioned embodiment, and will not be repeated here.
[0128] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the screen display method as described above are implemented.
[0129] The computer program product provided by the present invention can solve the problems of low recognition accuracy, slow response speed and lack of personalization in the existing driver status monitoring system. Compared with the prior art, the beneficial effects of the computer program product provided by the present invention are the same as the beneficial effects of the screen display method provided by the above embodiment, which will not be repeated here.
[0130] The above descriptions are only some embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the technical concept of the present invention, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.
Claims
1. A driver status monitoring method, characterized in that: The driver status monitoring method comprises: Obtain modal data of multiple modes; Perform feature extraction on modal data of multiple modes to obtain feature vectors of multiple modes; Calculating attention weights of multiple modalities according to the feature vectors of the multiple modalities; Calculating a global feature vector after fusion of multiple modalities according to the attention weights of the multiple modalities and the feature vectors of the multiple modalities; The driver's state is determined according to the global feature vector.
2. The driver status monitoring method according to claim 1, characterized in that: The feature vectors of the multiple modalities include: a facial feature vector, a voice feature vector, and a heart rate feature vector; The step of extracting features from modal data of multiple modalities to obtain feature vectors of multiple modalities specifically includes: Extracting features of facial images in the modal data of the multiple modalities to obtain facial feature vectors; Extracting features of speech signals in the modal data of the multiple modalities to obtain speech feature vectors; The modal data center frequencies of the multiple modes are extracted to obtain the heart rate feature vectors.
3. The driver status monitoring method according to claim 2, characterized in that: The step of extracting features from the facial images in the modal data of the multiple modalities to obtain facial feature vectors specifically includes: A convolutional neural network is used to convolve the facial images in the modal data of the multiple modalities to obtain a facial feature vector.
4. The driver status monitoring method according to claim 3, characterized in that: The step of extracting features from the speech signals in the modal data of the multiple modalities to obtain speech feature vectors specifically includes: Mel-frequency cepstral coefficients are used to extract the acoustic features of speech signals from modal data of multiple modes. According to the acoustic features, a long short-term memory network is used to calculate a language feature vector.
5. The driver status monitoring method according to claim 4, characterized in that: The step of extracting the modal data center frequencies of the multiple modes to obtain the heart rate feature vector specifically includes: Preprocessing the heart rate signals in the modal data of the multiple modalities; extracting heart rate variability of the preprocessed heart rate signal; According to the heart rate variability, a heart rate feature vector is calculated using a long short-term memory network.
6. The driver status monitoring method according to claim 5, characterized in that: The step of calculating the attention weights of multiple modalities according to the feature vectors of the multiple modalities specifically includes: According to the facial feature vector, the facial attention weight is calculated by the normalized exponential function; According to the speech feature vector, the speech attention weight is calculated by a normalized exponential function; According to the heart rate feature vector, the heart rate attention weight is calculated by the normalized exponential function.
7. The driver status monitoring method according to claim 6, characterized in that: The step of judging the driver's state according to the global feature vector specifically includes: Inputting the global feature vector into a fully connected layer; Obtain the probability distribution vector of the driver’s state through the fully connected layer; The driver's state is determined based on the probability distribution vector of the driver's state.
8. A driver status monitoring system, characterized in that: The driver status monitoring system includes: a multimodal biometric recognition module, an intelligent learning and dynamic adjustment module, and a safety warning module; The multimodal biometric module is used to collect multi-dimensional physiological and behavioral data of the driver; The intelligent learning and dynamic adjustment module is used to strengthen the learning strategy and make real-time adjustments and warnings according to the driver's status; The safety warning module is used to issue an alarm and take corresponding safety measures when it detects that the driver is tired, inattentive or emotional.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for driver status monitoring according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the driver state monitoring method according to any one of claims 1 to 7 are implemented.