Training method of fatigue driving detection model and fatigue driving detection method and device

Abstract

The invention provides a training method of a fatigue driving detection model and a fatigue driving detection method and device, and relates to the technical field of machine learning, and the method comprises the steps: obtaining training sample data; wherein the training sample data comprises an infrared image of the driver, an audio signal and a driver state, and the driver state comprises a fatigue state and a non-fatigue state; extracting feature vectors of the infrared image and the audio signal based on a deep convolutional neural network to obtain an infrared image sample and a spectrogram sample; and training a fatigue driving detection model based on the similarity of the infrared image sample and the spectrogram sample and the driver state to obtain a trained fatigue driving detection model. According to the invention, the precision and reliability of fatigue driving detection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a training method for a fatigue driving detection model, a fatigue driving detection method and a device. Background Art

[0002] With the rapid development of my country's economy, cars have become the most commonly used means of transportation for citizens. In recent years, the actual traffic volume on the road has been increasing rapidly. While cars provide convenience to life, they also bring a series of problems, such as the frequent occurrence of traffic accidents. According to reports, among the many causes of traffic accidents, fatigue driving is one of the factors with the highest number of fatalities. Therefore, fatigue driving has always been a problem that traffic control departments and drivers attach great importance to. If the driver can be reminded to rest in time when signs of driver fatigue driving occur, the occurrence of traffic accidents caused by driver fatigue driving can be effectively reduced.

[0003] In traditional fatigue detection methods, such as those based on visual data or physiological signals, the detection accuracy will drop significantly at night or in environments with strong light interference. Fatigue driving detection technology based on voice signals is difficult to cover all fatigue manifestations, especially in driving environments with large noise interference, the reliability of detection will be affected. In summary, the existing fatigue driving detection methods all have certain defects. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a training method for a fatigue driving detection model, a fatigue driving detection method and a device, so as to improve the accuracy and reliability of fatigue driving detection.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a training method for a fatigue driving detection model, comprising: obtaining training sample data; wherein the training sample data comprises: an infrared image of the driver, an audio signal and a driver state, and the driver state comprises: a fatigue state and a non-fatigue state; extracting feature vectors of the infrared image and the audio signal based on a deep convolutional neural network to obtain infrared image samples and spectrogram samples; training the fatigue driving detection model based on the similarity of the infrared image samples and the spectrogram samples and the driver state to obtain a trained fatigue driving detection model.

[0006] Optionally, obtaining training sample data includes: collecting an infrared image of the driver at a preset sampling rate through an infrared camera; collecting an audio signal of the driver through a microphone array; synchronously processing the infrared image and the audio signal, and obtaining the driver status corresponding to the synchronized infrared image and audio signal.

[0007] Optionally, the fatigue driving detection model is trained based on the similarity between the infrared image samples and the spectrogram samples and the driver status, including: calculating the similarity between the infrared image samples and the spectrogram samples based on the feature vector of the infrared image and the feature vector of the audio signal, and constructing an undirected graph based on the similarity between the infrared image samples and the spectrogram samples; wherein the nodes of the undirected graph are the infrared image samples or the spectrogram samples, and the edges of the undirected graph are the similarities between the infrared image samples or the spectrogram samples of the same driver status; determining an initial node feature matrix of the undirected graph, and determining a node feature matrix of each layer of the undirected graph based on the initial node feature matrix; constructing a first loss function based on the node feature matrix; constructing a second loss function based on the similarity between the infrared image samples and the spectrogram samples; determining the sum of the first loss function and the second loss function as the target loss function, and training the fatigue driving detection model by minimizing the target loss function.

[0008] Optionally, constructing a first loss function based on the node feature matrix includes: constructing the first loss function according to the following formula:

[0009]

[0010]

[0011]

[0012] in, represents the first loss function, Representation Node i The node feature matrix at the last layer of the undirected graph, Representation Node i and nodes j The weight between is the parameter used to improve the similarity between infrared image samples and spectrogram samples, is a parameter used to reduce the similarity between infrared image samples or spectrogram samples, represents the boundary threshold, and is a trade-off parameter.

[0013] Optionally, constructing a second loss function based on the similarity between the infrared image sample and the spectrogram sample includes: determining the spectrogram sample with the same driver state as the infrared image sample and the highest similarity as a positive sample, and determining the spectrogram sample with a different driver state from the infrared image sample and the highest similarity as a negative sample; constructing a second loss function based on the similarity between the infrared image sample and the positive sample and the negative sample; wherein the second loss function is:

[0014] in, Indicates i Infrared image samples, Indicates i The positive samples in the spectrogram samples corresponding to the infrared image samples, Indicates i Negative samples in the spectrogram samples corresponding to infrared image samples, N Indicates the number of samples, sim() indicates the cosine similarity calculation, Represents the temperature coefficient, which is used to control the smoothness of the cosine similarity.

[0015] In a second aspect, the present invention provides a fatigue driving detection method, comprising: acquiring a real-time infrared image and a real-time audio signal of a driver; extracting feature vectors of the real-time infrared image and the real-time audio signal based on a deep convolutional neural network; inputting the feature vectors of the real-time infrared image and the real-time audio signal into a pre-trained fatigue driving detection model to obtain the driver's driver state; wherein the fatigue driving detection model is trained by the training method of the fatigue driving detection model of any one of the items provided in the first aspect, and the driver state includes: a fatigue state and a non-fatigue state.

[0016] In a third aspect, the present invention provides a training device for a fatigue driving detection model, comprising: a sample acquisition module, used to acquire training sample data; wherein the training sample data includes: an infrared image of the driver, an audio signal and a driver state, and the driver state includes: a fatigue state and a non-fatigue state; a first feature extraction module, used to extract feature vectors of infrared images and audio signals based on a deep convolutional neural network, and obtain infrared image samples and spectrogram samples; a model training module, used to train the fatigue driving detection model based on the similarity of the infrared image samples and the spectrogram samples and the driver state, and obtain a trained fatigue driving detection model.

[0017] In a fourth aspect, the present invention provides a fatigue driving detection device, comprising: a data acquisition module, used to acquire a real-time infrared image and real-time audio signal of a driver; a second feature extraction module, used to extract feature vectors of the real-time infrared image and the real-time audio signal based on a deep convolutional neural network; a fatigue driving detection module, used to input the feature vectors of the real-time infrared image and the real-time audio signal into a pre-trained fatigue driving detection model to obtain the driver's driving state; wherein the fatigue driving detection model is trained based on the training method of the fatigue driving detection model of any one of the items provided in the first aspect, and the driver's state includes: fatigue state and non-fatigue state.

[0018] In a fifth aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect or the steps of the method provided in the second aspect.

[0019] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes any one of the steps of the method provided in the first aspect or the second aspect.

[0020] The present invention brings the following beneficial effects: The training method, fatigue driving detection method and device of the fatigue driving detection model provided by the present invention first obtain training sample data; wherein the training sample data includes: infrared image of the driver, audio signal and driver status, and the driver status includes: fatigue state and non-fatigue state; then, feature vectors of infrared image and audio signal are extracted based on deep convolutional neural network to obtain infrared image samples and spectrogram samples; finally, the fatigue driving detection model is trained based on the similarity of infrared image samples and spectrogram samples and the driver status to obtain a trained fatigue driving detection model. In the above method, by combining infrared image and spectrogram data and using deep convolutional neural network to extract features of multimodal data, the accuracy of fatigue driving detection is improved; at the same time, the above method can enhance the similarity of infrared image samples and spectrogram samples, and reduce the similarity of infrared image samples and spectrogram samples under different driver states, thereby improving the performance of fatigue driving detection model and further improving the accuracy and reliability of fatigue driving detection.

[0021] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the preferred embodiments are specifically mentioned below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a method for training a fatigue driving detection model provided by an embodiment of the present invention; Figure 2 A detection flow chart of fatigue driving detection provided by an embodiment of the present invention; Figure 3 A flowchart of a fatigue driving detection method provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a training device for a fatigue driving detection model provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a fatigue driving detection device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] At present, traditional fatigue detection methods, such as those based on visual data or physiological signals, include facial feature analysis, eye movement detection, etc., which use cameras to obtain facial images of drivers and analyze features such as eye closure frequency and yawning frequency to determine fatigue status. However, such methods are very sensitive to lighting conditions, and the detection accuracy will drop significantly at night or in environments with strong light interference. Infrared technology can capture clear images in completely dark or low-light environments, making up for the lack of visible light, but infrared technology detection methods still have the problem of missing information.

[0027] Fatigue driving detection technology based on voice signals can capture the differences in voice characteristics of drivers in different fatigue states by analyzing the changes in the frequency and energy distribution of voice signals. For example, voice in a fatigued state may show characteristics such as slower speech speed and narrower frequency range. Although spectrograms show certain advantages in analyzing voice fatigue characteristics, this method is also difficult to cover all fatigue manifestations, especially in driving environments with large noise interference, the reliability of detection will be affected.

[0028] Based on this, the embodiments of the present invention provide a training method for a fatigue driving detection model, a fatigue driving detection method and a device, which can improve the accuracy and reliability of fatigue driving detection.

[0029] To facilitate understanding of this embodiment, a training method for a fatigue driving detection model disclosed in an embodiment of the present invention is first introduced in detail. The method can be executed by an electronic device, such as a smart phone, a computer, a tablet computer, etc. Figure 1 The flowchart of a training method of a fatigue driving detection model shown in FIG. 1 illustrates that the method mainly includes the following steps S101 to S103: Step S101: Obtain training sample data.

[0030] The training sample data includes: infrared images of the driver, audio signals and driver status, and the driver status includes: fatigue state and non-fatigue state.

[0031] In one embodiment, an infrared image of the driver is first captured by an infrared camera at a preset sampling rate; then the driver's audio signal is captured by a microphone array; finally, the infrared image and audio signal are synchronously processed, and the driver's status corresponding to the synchronized infrared image and audio signal is obtained.

[0032] In the specific implementation, in order to ensure the reliability of the data, the time domain information of the microphone array and the infrared camera are synchronized in this embodiment. The sampling rate of the infrared camera is 1 frame / second (that is, the preset sampling rate) as a benchmark. Every time the infrared camera collects a frame of data, the audio signal from t-1s to the current time t collected by the microphone array is intercepted. That is, a frame of infrared image collected at the current time t and a 1s audio signal from t-1s to the current time t collected by the corresponding microphone array are a group of data; then the driver's state (fatigue state or non-fatigue state) corresponding to each group of infrared images and audio signals is determined.

[0033] Step S102: extracting feature vectors of infrared images and audio signals based on a deep convolutional neural network to obtain infrared image samples and spectrogram samples.

[0034] In one implementation, for an infrared image, the infrared image is first decoded and framed; then, the decoded and framed infrared image is preprocessed by filtering, denoising, cropping, and other preprocessing operations; finally, a deep convolutional neural network is used to extract a feature vector of the preprocessed infrared image to obtain an embedding of the infrared image modality A (i.e., an infrared image sample). ,in, i =1,2,…, N Indicates i Infrared image samples, d is the dimension of the embedding space.

[0035] For audio signals, the audio signals are first framed, windowed, and short-time discrete Fourier transformed to obtain the spectrogram of the audio signal; then a deep convolutional neural network is used to extract the feature vector of the spectrogram to obtain the embedding of the spectrogram mode B (i.e., the spectrogram sample). ,in, i =1,2,…, N Indicates i Spectrogram samples, d is the dimension of the embedding space.

[0036] Step S103: training the fatigue driving detection model based on the similarity between the infrared image samples and the spectrogram samples and the driver's state to obtain a trained fatigue driving detection model.

[0037] In one embodiment, in order to further improve the fatigue driving detection accuracy of the fatigue driving detection model in two different modes, infrared mode and audio mode, an adaptive joint graph embedding optimization method is proposed in this embodiment, which constructs an undirected graph through the similarity of infrared image samples and spectrogram samples and the driver's state, and optimizes the embedding of the undirected graph nodes by minimizing the loss function; in addition, this embodiment also provides a negative sample mining strategy to optimize fatigue states that are difficult to detect correctly, specifically, the positive samples and negative samples are determined by the similarity of infrared image samples and spectrogram samples, and the loss function is constructed according to the similarity between the positive samples and the negative samples, and the model is optimized by minimizing the loss function.

[0038] In the embodiment of the present invention, infrared image samples and spectrogram samples and their corresponding driver states are used to optimize and train the fatigue driving detection model through the aforementioned adaptive joint graph embedding optimization method and negative sample mining strategy to obtain a trained fatigue driving detection model.

[0039] The training method of the fatigue driving detection model provided by the present invention improves the accuracy of fatigue driving detection by combining infrared image and spectrogram data and utilizing a deep convolutional neural network to perform feature extraction of multimodal data; at the same time, the method can enhance the similarity between infrared image samples and spectrogram samples and reduce the similarity between infrared image samples and spectrogram samples under different driver states, thereby improving the performance of the fatigue driving detection model and further improving the accuracy and reliability of fatigue driving detection.

[0040] In one embodiment, for the aforementioned step S103, that is, when the fatigue driving detection model is trained based on the similarity between the infrared image sample and the spectrogram sample and the driver's state, the following methods may be adopted, including but not limited to the following steps (1) to (5): Step (1), calculating the similarity between the infrared image sample and the spectrogram sample based on the feature vector of the infrared image and the feature vector of the audio signal, and constructing an undirected graph based on the similarity between the infrared image sample and the spectrogram sample.

[0041] In the specific implementation, we first construct an undirected graph based on the similarity between the spectrogram samples and the infrared image samples. , where the nodes of the undirected graph are is an infrared image sample or a spectrogram sample, and the edge of the undirected graph is the similarity of infrared image samples or spectrogram samples of the same driver state.

[0042] Specifically, each node in an undirected graph Corresponding to a spectrogram sample or infrared image sample, the embedding vector (i.e., feature vector) generated by the spectrogram or infrared image is expressed as , for two nodes and , if their corresponding spectrogram samples or infrared image samples belong to the same category (i.e., the driver status is the same, the same fatigue state or the same non-fatigue state), then at node and Create an edge between , and use cosine similarity to calculate the edge weights .

[0043]

[0044] in, Representation Node The characteristic vector of Representation Node The characteristic vector of Representation Node and The weight of the edge between them.

[0045] Step (2), determining the initial node feature matrix of the undirected graph, and determining the node feature matrix of each layer of the undirected graph based on the initial node feature matrix.

[0046] In the specific implementation, it is assumed It is l The node feature matrix of the layer, represents the initial node feature matrix, then l The node feature matrix of layer +1 can be expressed as:

[0047] in, Indicates l +1 layer node feature matrix, Indicates l The node feature matrix of the layer, , L represents the number of layers of the undirected graph, represents the adjacency matrix of an undirected graph, , Indicates l The edge weight matrix of the layer, represents the degree matrix, Represents the sigmoid activation function.

[0048] Step (3), constructing the first loss function based on the node feature matrix.

[0049] In specific implementation, the first loss function can be constructed according to the following formula:

[0050]

[0051]

[0052]

[0053] in, represents the first loss function, Representation Node i The node feature matrix at the last layer of the undirected graph, Representation Node i and nodes j The weight between is the parameter used to improve the similarity between infrared image samples and spectrogram samples, is a parameter used to reduce the similarity between infrared image samples or spectrogram samples, represents the boundary threshold, and is a trade-off parameter.

[0054] In an embodiment of the present invention, the embedding representation of the node can be optimized by minimizing the loss function of the graph convolutional network, thereby enhancing the distinction between different samples (audio modality or infrared modality), and improving the effectiveness and robustness of the fatigue driving detection model for fatigue state detection.

[0055] Step (4), constructing a second loss function based on the similarity between the infrared image sample and the spectrogram sample.

[0056] In the specific implementation, in order to enable the fatigue driving detection model to better detect the fatigue state at night or in low-light environments, a negative sample mining strategy is designed in the cross-modal (audio mode and infrared mode) comparative learning in the embodiment of the present invention, which can be optimized specifically for fatigue states that are difficult to detect correctly. This method can significantly improve the model's ability to detect fatigue driving, especially in complex environments such as night or low light, it can better identify the characteristics of fatigue states, thereby improving overall performance. The calculation method is as follows: First, the cosine similarity is used to calculate the samples in the infrared image modality A With all samples in the spectrogram mode B Similarity:

[0057] Then, the spectrogram sample with the same driver state as the infrared image sample and the highest similarity is determined as a positive sample, and the spectrogram sample with a different driver state from the infrared image sample and the highest similarity is determined as a negative sample.

[0058] Specifically, choose The most similar samples but belonging to different driver states (such as "drowsy" or "not fatigued") As negative samples (That is, the spectrogram samples with different driver states from the infrared image samples and the highest similarity are determined as negative samples):

[0059] At the same time, choose The most similar samples that belong to the same driver status as a positive sample.

[0060] Finally, a second loss function is constructed based on the similarity between the infrared image sample and the positive sample and the negative sample; wherein the second loss function is:

[0061] in, Indicates i Infrared image samples, Indicates i The positive samples in the spectrogram samples corresponding to the infrared image samples, Indicates i Negative samples in the spectrogram samples corresponding to infrared image samples, N Indicates the number of samples, sim() indicates the cosine similarity calculation, Represents the temperature coefficient, which is used to control the smoothness of the cosine similarity.

[0062] In step (5), the sum of the first loss function and the second loss function is determined as the target loss function, and the fatigue driving detection model is trained by minimizing the target loss function.

[0063] In specific implementation, the target loss function is:

[0064] in, is a trade-off parameter.

[0065] In the embodiment of the present invention, the final objective loss function combines the negative sample mining loss with the cross-modal contrast loss, which can achieve comprehensive optimization of the fatigue driving detection model.

[0066] The above method provided by the embodiment of the present invention combines infrared images and spectrogram data, uses a deep learning model to extract features and fusion analysis of multimodal data, and achieves fast and accurate fatigue detection, overcoming the limitations of single-modal methods in complex driving environments, while avoiding dependence on external physical sensors and improving user experience. This multimodal fusion detection method provides an efficient, reliable and environmentally adaptable solution for fatigue driving monitoring.

[0067] For ease of understanding, the present invention also provides a detection process for fatigue driving detection, see Figure 2 As shown, it mainly includes the following steps 1 to 7: Step 1: Collect infrared images through the infrared camera and collect audio signals through the microphone array.

[0068] Step 2: In the image preprocessing module, the infrared image is first decoded and framed, and then preprocessing operations such as filtering, noise reduction, and cropping are performed.

[0069] Step 3: In the audio signal preprocessing module, the audio signal is framed, windowed, and short-time discrete Fourier transformed to obtain a spectrogram of the audio signal.

[0070] Step 4: Use a deep convolutional neural network in the image feature extraction module to extract the feature vector of the infrared image and obtain the embedding of the infrared image modality A. .

[0071] Step 5: Use a deep convolutional neural network in the audio feature extraction module to extract the feature vector of the spectrogram and obtain the embedding of the spectrogram modality B .

[0072] Step 6: Input the feature vector of the spectrogram and the feature vector of the infrared image into the joint embedding space module. In the joint embedding space module, adaptive joint graph embedding optimization and negative sample mining are performed respectively.

[0073] Specifically, the adaptive joint graph embedding optimization process refers to the aforementioned steps (1) to (3), and the negative sample discovery process refers to the aforementioned step (4), which will not be repeated here.

[0074] Step 7: Output the driver's fatigue status.

[0075] In the above method provided by the embodiment of the present invention, the fatigue state can be detected more efficiently in the shared joint embedding space through the adaptive joint graph embedding optimization method. Through this method, the similarity between the spectrogram (audio modality) and the infrared image (infrared modality) is enhanced, especially in the same fatigue state (such as "fatigue" or "non-fatigue"), while effectively reducing the similarity of cross-modal (infrared modality and audio modality) samples in different states, thereby improving the cross-modal learning ability of the model and significantly improving the detection accuracy and robustness of the fatigue state. Through the negative sample mining strategy, it can be optimized specifically for fatigue states that are difficult to detect correctly. This method can significantly improve the model's ability to detect fatigue driving, especially in complex environments such as night or low light, it can better identify the characteristics of fatigue states, thereby improving overall performance.

[0076] The present invention also provides a method for detecting fatigue driving. Figure 3 The flowchart of a fatigue driving detection method shown in FIG. 1 illustrates that the method mainly includes the following steps 301 to step S303: Step S301: Acquire the real-time infrared image and real-time audio signal of the driver.

[0077] Step S302: extracting feature vectors of real-time infrared images and real-time audio signals based on a deep convolutional neural network.

[0078] Step S303: Input the feature vectors of the real-time infrared image and the real-time audio signal into a pre-trained fatigue driving detection model to obtain the driver's state; wherein the fatigue driving detection model is trained based on the training method of the fatigue driving detection model provided in any one of the aforementioned embodiments, and the driver's state includes: fatigue state and non-fatigue state.

[0079] The above-mentioned fatigue driving detection method provided by the present invention uses a fatigue driving detection model to perform fatigue driving detection. By combining infrared images and spectrogram data, it can achieve stable detection under various lighting conditions (especially at night or in low-light environments); at the same time, the spectrogram is used to extract fatigue-related features from the driver's voice, and the voice frequency and energy changes are analyzed. The infrared image can capture the driver's facial temperature and expression changes, thereby providing more comprehensive physiological and behavioral information and improving the accuracy of fatigue state recognition.

[0080] Regarding the training method of the fatigue driving detection model provided in the above embodiment, the present invention also provides a training device for the fatigue driving detection model, see Figure 4 The structural diagram of a training device for a fatigue driving detection model shown in FIG. 1 shows that the device mainly includes the following parts: The sample acquisition module 401 is used to acquire training sample data; wherein the training sample data includes: infrared images of the driver, audio signals and driver status, and the driver status includes: fatigue state and non-fatigue state; A first feature extraction module 402 is used to extract feature vectors of infrared images and audio signals based on a deep convolutional neural network to obtain infrared image samples and spectrogram samples; The model training module 403 is used to train the fatigue driving detection model based on the similarity between the infrared image samples and the spectrogram samples and the driver's state to obtain a trained fatigue driving detection model.

[0081] The training device for the fatigue driving detection model provided by the present invention improves the accuracy of fatigue driving detection by combining infrared image and spectrogram data and using a deep convolutional neural network to extract features of multimodal data; at the same time, the above-mentioned device can enhance the similarity between infrared image samples and spectrogram samples, and reduce the similarity between infrared image samples and spectrogram samples under different driver states, thereby improving the performance of the fatigue driving detection model and further improving the accuracy and reliability of fatigue driving detection.

[0082] For the fatigue driving detection method provided in the above embodiment, the present invention also provides a fatigue driving detection device, see Figure 5 The schematic diagram of the structure of a fatigue driving detection device shown in the figure shows that the device mainly includes the following parts: The data acquisition module 501 is used to acquire the real-time infrared image and real-time audio signal of the driver; A second feature extraction module 502, for extracting feature vectors of real-time infrared images and real-time audio signals based on a deep convolutional neural network; The fatigue driving detection module 503 is used to input the feature vectors of the real-time infrared image and the real-time audio signal into a pre-trained fatigue driving detection model to obtain the driver's driving state.

[0083] The fatigue driving detection model is obtained by training based on the training method of the fatigue driving detection model provided in any one of the aforementioned embodiments, and the driver status includes: fatigue status and non-fatigue status.

[0084] The above-mentioned fatigue driving detection device provided by the present invention uses a fatigue driving detection model to perform fatigue driving detection. By combining infrared images and spectrogram data, it can achieve stable detection under various lighting conditions (especially at night or in low-light environments); at the same time, the spectrogram is used to extract fatigue-related features from the driver's voice and analyze the voice frequency and energy changes. The infrared image can capture the driver's facial temperature and expression changes, thereby providing more comprehensive physiological and behavioral information and improving the accuracy of fatigue state recognition.

[0085] It should be noted that the implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0086] An embodiment of the present invention further provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above implementation methods.

[0087] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0088] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0089] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0091] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0092] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0093] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0094] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A training method for a fatigue driving detection model, characterized in that: include: Acquire training sample data; wherein the training sample data includes: an infrared image of the driver, an audio signal and a driver state, wherein the driver state includes: a fatigue state and a non-fatigue state; Extracting feature vectors of the infrared image and the audio signal based on a deep convolutional neural network to obtain infrared image samples and spectrogram samples; The fatigue driving detection model is trained based on the similarity between the infrared image sample and the spectrogram sample and the driver state to obtain a trained fatigue driving detection model.

2. The training method according to claim 1, characterized in that: Get training sample data, including: The infrared image of the driver is collected by an infrared camera at a preset sampling rate; collecting an audio signal of the driver through a microphone array; The infrared image and the audio signal are processed synchronously, and the driver status corresponding to the synchronized infrared image and the audio signal is obtained.

3. The training method according to claim 1, characterized in that: The fatigue driving detection model is trained based on the similarity between the infrared image sample and the sound spectrogram sample and the driver state, including: The similarity between the infrared image sample and the spectrogram sample is calculated based on the feature vector of the infrared image and the feature vector of the audio signal, and an undirected graph is constructed based on the similarity between the infrared image sample and the spectrogram sample; wherein the nodes of the undirected graph are infrared image samples or spectrogram samples, and the edges of the undirected graph are similarities between infrared image samples or spectrogram samples of the same driver state; Determine an initial node feature matrix of the undirected graph, and determine a node feature matrix of each layer of the undirected graph based on the initial node feature matrix; Constructing a first loss function based on the node feature matrix; constructing a second loss function based on the similarity between the infrared image sample and the spectrogram sample; The sum of the first loss function and the second loss function is determined as a target loss function, and the fatigue driving detection model is trained by minimizing the target loss function.

4. The training method according to claim 3, characterized in that: Constructing a first loss function based on the node feature matrix includes: The first loss function is constructed according to the following formula: in, represents the first loss function, Representation Node i The node feature matrix at the last layer of the undirected graph, Representation Node i and nodes j The weight between is a parameter for improving the similarity between the infrared image sample and the spectrogram sample, is a parameter for reducing the similarity between the infrared image samples or the spectrogram samples, represents the boundary threshold, and is a trade-off parameter.

5. The training method according to claim 3, characterized in that: Constructing a second loss function based on the similarity between the infrared image sample and the spectrogram sample, comprising: Determine the sound spectrogram sample with the same driver status as the infrared image sample and the highest similarity as a positive sample, and determine the sound spectrogram sample with a different driver status as the infrared image sample and the highest similarity as a negative sample; A second loss function is constructed based on the similarity between the infrared image sample and the positive sample and the negative sample; wherein the second loss function is: in, Indicates i Infrared image samples, Indicates i The positive samples in the spectrogram samples corresponding to the infrared image samples, Indicates i Negative samples in the spectrogram samples corresponding to infrared image samples, N Indicates the number of samples, sim() indicates the cosine similarity calculation, Represents the temperature coefficient, which is used to control the smoothness of the cosine similarity.

6. A method for detecting fatigue driving, characterized in that: include: Acquire the driver's real-time infrared image and real-time audio signal; Extracting feature vectors of the real-time infrared image and the real-time audio signal based on a deep convolutional neural network; The feature vectors of the real-time infrared image and the real-time audio signal are input into a pre-trained fatigue driving detection model to obtain the driver's state; wherein the fatigue driving detection model is trained based on the training method of the fatigue driving detection model according to any one of claims 1 to 5, and the driver's state includes: fatigue state and non-fatigue state.

7. A training device for a fatigue driving detection model, characterized in that: include: A sample acquisition module, used to acquire training sample data; wherein the training sample data includes: an infrared image of the driver, an audio signal and a driver state, wherein the driver state includes: a fatigue state and a non-fatigue state; A first feature extraction module, used to extract feature vectors of the infrared image and the audio signal based on a deep convolutional neural network to obtain infrared image samples and spectrogram samples; The model training module is used to train the fatigue driving detection model based on the similarity between the infrared image sample and the spectrogram sample and the driver state to obtain a trained fatigue driving detection model.

8. A fatigue driving detection device, characterized in that: include: A data acquisition module, used to acquire the driver's real-time infrared image and real-time audio signal; A second feature extraction module, used for extracting feature vectors of the real-time infrared image and the real-time audio signal based on a deep convolutional neural network; A fatigue driving detection module is used to input the feature vectors of the real-time infrared image and the real-time audio signal into a pre-trained fatigue driving detection model to obtain the driver state of the driver; wherein the fatigue driving detection model is trained based on the training method of the fatigue driving detection model according to any one of claims 1 to 5, and the driver state includes: a fatigue state and a non-fatigue state.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 5 or the steps of the method according to claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 or the steps of the method according to claim 6 are executed.

Images