Fatigue driving intervention method, device, system and equipment based on facial recognition

Abstract

The invention discloses a fatigue driving intervention method, device, system and equipment based on facial recognition, and the method comprises the steps: obtaining a facial image sequence of a driver, and determining a corresponding local region image sequence; the facial image sequence and the local area image sequence are input into a preset facial expression recognition model, the facial expression category of the driver is obtained, in the preset facial expression recognition model, a local feature extraction module is used for conducting feature extraction on the local area image sequence based on a long-short-term memory neural network block, and the facial expression category of the driver is obtained; obtaining local dynamic features; the global feature extraction module is used for performing feature extraction on the facial image sequence based on deformable convolution to obtain global appearance features; and determining fatigue driving intervention measures according to the facial expression type and generating a corresponding execution instruction. According to the invention, the preset facial expression recognition model can accurately recognize the facial expression category of the driver in the changing scene, so that the timely response and execution of the fatigue driving intervention measure are realized.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to a fatigue driving intervention method, device, system and equipment based on facial recognition. Background Art

[0002] In road traffic safety, it is necessary to ensure that the driver remains awake while driving. For example, the driver's facial expressions can be monitored and analyzed in real time to determine whether he is in a state of fatigue driving, and the in-car environment can be automatically adjusted based on the monitoring and analysis results to implement fatigue driving intervention and ensure the driver's driving safety.

[0003] However, since traditional facial expression recognition (FER) usually only focuses on identifying facial expression categories based on the entire face, the robustness of the recognition results is poor and fails to take into account the impact of different lighting and occlusion conditions on the driver during vehicle driving. Therefore, it is difficult to accurately identify the driver's facial expression based on traditional FER, which in turn affects the timely response and execution of fatigue intervention measures. Summary of the invention

[0004] The main purpose of this application is to provide a fatigue driving intervention method, device, system and equipment based on facial recognition, aiming to solve the technical problem that the traditional FER has poor robustness and is difficult to recognize the driver's facial expressions in changing scenarios, resulting in untimely response of fatigue intervention measures.

[0005] To achieve the above objectives, the present application proposes a fatigue driving intervention method based on facial recognition, and the fatigue driving intervention method based on facial recognition includes:

[0006] Acquire a facial image sequence of the driver, and determine a corresponding local area image sequence based on the facial image sequence;

[0007] Inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used to perform feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used to perform feature extraction on the facial image sequence based on a deformable convolution to obtain global appearance features;

[0008] The corresponding fatigue driving intervention measure is determined according to the facial expression category, and an execution instruction of the fatigue driving intervention measure is generated.

[0009] In one embodiment, the preset facial expression recognition model further includes a shared shallow feature module and a fusion output module;

[0010] The step of inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the driver's facial expression category comprises:

[0011] Extracting basic features from the facial image sequence and the local area image sequence through the shared shallow feature module to obtain a number of low-level features;

[0012] The local feature extraction module uses a long short-term memory neural network block to extract features from each of the low-level features to obtain local dynamic features;

[0013] Using the global feature extraction module to extract features from each of the low-level features using deformable convolution, to obtain global appearance features;

[0014] The local dynamic features and the global appearance features are averaged through the fusion output module to obtain a fusion score, and the driver's facial expression category is determined according to the fusion score.

[0015] In one embodiment, the step of extracting features from each of the low-level features using a long short-term memory neural network block by the local feature extraction module to obtain local expression features comprises:

[0016] Extracting primary features of the local region image sequence associated with the local region image sequence through a long short-term memory neural network block to obtain time features corresponding to different local regions;

[0017] The temporal features corresponding to the different local regions are connected and integrated to obtain local dynamic features, and are determined as local high-level features.

[0018] In one embodiment, the step of extracting features from each of the low-level features using deformable convolution by the global feature extraction module to obtain global appearance features includes:

[0019] Based on the adaptive selection sampling method, deformable convolution is used to extract the primary features associated with the facial image sequence to obtain a number of spatial expression features, and each of the spatial expression features is integrated into a global appearance feature.

[0020] In one embodiment, the step of averaging the local dynamic features and the global appearance features through the fusion output module to obtain a fusion score, and determining the driver's facial expression category according to the fusion score includes:

[0021] Averaging the local dynamic features and the global appearance features to calculate fusion scores corresponding to different facial expression categories;

[0022] Determining a maximum fusion score among the fusion scores corresponding to the facial expression categories, and using the facial expression category corresponding to the maximum fusion score as the facial expression category of the driver;

[0023] The calculation formula for determining the maximum fusion score is:

[0024]

[0025] Where Ο is the facial expression category corresponding to the maximum fusion score, is the equilibrium parameter, P local (i) is the predicted probability of the local dynamic feature on facial expression category i, P global (i) is the predicted probability of the global appearance feature on facial expression category i.

[0026] In one embodiment, the step of determining the corresponding fatigue driving intervention measure according to the facial expression category and generating an execution instruction of the fatigue driving intervention measure includes:

[0027] Determining the current wakefulness state of the driver according to the facial expression category, and judging whether the driver is in a fatigue driving condition according to the current wakefulness state;

[0028] If so, the fatigue driving intervention measure corresponding to the current wakefulness state is queried in the preset fatigue driving intervention measure table, and an execution instruction of the fatigue driving intervention measure is generated.

[0029] In one embodiment, the fatigue driving intervention measures include: air conditioning adjustment measures and window control measures.

[0030] In addition, to achieve the above purpose, the present application also proposes a fatigue driving intervention device based on facial recognition, the device comprising:

[0031] An image processing module, used for acquiring a facial image sequence of the driver, and determining a corresponding local area image sequence based on the facial image sequence;

[0032] an expression recognition module, used for inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used for performing feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used for performing feature extraction on the facial image sequence based on a deformable convolution to obtain global appearance features;

[0033] The environment control module is used to determine the corresponding fatigue driving intervention measure according to the facial expression category and generate an execution instruction of the fatigue driving intervention measure.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a fatigue driving intervention system based on facial recognition, the system comprising: a camera, an environment response device and the fatigue driving intervention device based on facial recognition as described above, the fatigue driving adjustment device being electrically connected to the camera and the environment response device respectively;

[0035] The camera is used to collect the driver's facial image in real time, obtain a facial image sequence, and send the facial image sequence to the image processing module;

[0036] The environmental response device is used to receive the execution instruction sent by the environmental control module and execute corresponding fatigue driving intervention measures.

[0037] In addition, to achieve the above-mentioned objectives, the present application also proposes a facial recognition-based fatigue driving intervention device, which includes: a memory, a processor, and a facial recognition-based fatigue driving intervention program stored on the memory and executable on the processor, wherein the facial recognition-based fatigue driving intervention program, when executed by the processor, implements the steps of the facial recognition-based fatigue driving intervention method described above.

[0038] The present application provides a fatigue driving intervention method based on facial recognition, by acquiring a facial image sequence of a driver, and determining a corresponding local area image sequence based on the facial image sequence; inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used to extract features from the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used to extract features from the facial image sequence based on a deformable convolution to obtain global appearance features; according to the facial expression category, the corresponding fatigue driving intervention measures are determined, and the execution instructions of the fatigue driving intervention measures are generated. Since the local features and global features can be combined in the present application, the possible misjudgment caused by single feature analysis is reduced. And by using the long short-term memory network and the deformable convolution to extract features respectively, it can adapt to different lighting conditions, occlusion conditions and driving environments, ensuring that the driver's facial expression category can be accurately identified under various conditions, thereby realizing the timely response and implementation of the corresponding fatigue driving intervention measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A schematic diagram of the structure of a fatigue driving intervention device based on facial recognition in the hardware operating environment involved in the embodiment of the present application;

[0042] Figure 2 This is a flow chart of the first embodiment of the fatigue driving intervention method based on facial recognition of the present application;

[0043] Figure 3 This is a flow chart of the second embodiment of the fatigue driving intervention method based on facial recognition of the present application;

[0044] Figure 4 A schematic diagram of the framework structure of the facial expression recognition model preset for this application;

[0045] Figure 5 This is a flow chart of the third embodiment of the fatigue driving intervention method based on facial recognition of the present application;

[0046] Figure 6 This is a structural block diagram of the first embodiment of the fatigue driving intervention device based on facial recognition in this application.

[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0049] Reference Figure 1 , Figure 1 This is a structural diagram of a facial recognition-based fatigue driving intervention device in the hardware operating environment involved in an embodiment of the present application.

[0050] like Figure 1As shown, the fatigue driving intervention device based on facial recognition may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the fatigue driving intervention device based on facial recognition, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0052] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a fatigue driving intervention program based on facial recognition.

[0053] exist Figure 1 In the fatigue driving intervention device based on facial recognition shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the fatigue driving intervention device based on facial recognition of the present application can be set in the fatigue driving intervention device based on facial recognition, and the fatigue driving intervention device based on facial recognition calls the fatigue driving intervention program based on facial recognition stored in the memory 1005 through the processor 1001, and executes the fatigue driving intervention method based on facial recognition provided in the embodiment of the present application.

[0054] The present application embodiment provides a fatigue driving intervention method based on facial recognition, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the fatigue driving intervention method based on facial recognition of the present application. In this embodiment, the method includes steps S10 to S30:

[0055] Step S10: Acquire a facial image sequence of the driver, and determine a corresponding local area image sequence based on the facial image sequence.

[0056] It should be noted that the method of this embodiment can be applied to the scenario of assisting the driver to drive safely during vehicle driving. The execution subject of the method of this embodiment can be the central control module or the body control module in the vehicle, and can also be applied to other electronic devices inside the vehicle with the same or similar functions that can implement the fatigue driving intervention method based on facial recognition of this application. Here, the above-mentioned fatigue driving intervention device based on facial recognition (hereinafter referred to as "intervention device") is used as an example to explain the various embodiments of this application.

[0057] It should be understood that a camera may be installed on the dashboard of the vehicle to capture the driver's facial image in real time and send the generated facial image sequence to the intervention device.

[0058] It is understandable that the above facial image sequence reflects the overall facial area of ​​the driver, and a specific local area can be isolated from the overall facial area to obtain a local area image sequence. The specific local area can be: eyes, nose, mouth and other key facial areas for assisting in understanding facial expressions.

[0059] Step S20: Input the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the driver's facial expression category, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used to perform feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used to perform feature extraction on the facial image sequence based on deformable convolution to obtain global appearance features.

[0060] It should be noted that the key to determining facial expressions lies in the overall appearance of the face and the subtle dynamics of the local area, and the difference between the two expressions is sometimes vague, that is, the subtle dynamic changes in the local area will lead to different facial expression categories. Traditional FER relies too much on general facial appearance features and does not take into account the subtle dynamic differences of individuals in local areas, which leads to insufficient accuracy and poor robustness when identifying the facial expression categories of different individuals.

[0061] It should be understood that, based on the above-mentioned deficiencies in recognition of traditional FER, in this application, a facial image sequence reflecting the overall appearance of the face and a local area image sequence reflecting subtle dynamics are jointly input into a preset facial expression recognition model so that the model can capture features that are more important for understanding facial expressions.

[0062] It can be understood that the preset facial expression recognition model can be designed to include: a shared shallow feature extraction module, a local feature extraction module and a global feature extraction module.

[0063] The shared shallow feature extraction module can be used to learn low-level features from the input facial image sequence and the local region image sequence, so as to serve as the feature basis in the subsequent local feature extraction module and the global feature extraction module. The shared shallow feature extraction module can be constructed based on the pre-trained VGG-16 model, and can specifically be composed of the first 10 convolutional layers of the pre-trained VGG-16 model.

[0064] The local feature extraction module can combine the above extracted low-level features with the local area image sequence as input to further extract features associated with the micro-dynamics of the local area, such as eyes, nose and mouth. The local feature extraction module can be constructed based on a long short-term memory neural network block (LST block), which can effectively capture the temporal dynamics of expressions in the image sequence, and finally obtain local dynamic features based on the local area image sequence.

[0065] The global feature extraction module can also use the low-level features extracted above in combination with the facial image sequence as input to further extract features reflecting the overall appearance of the face. The global feature extraction module can be designed to use the peak frame as input, that is, to select the peak frame with the most obvious expression in the facial image sequence as input to focus on the spatial information of the overall facial expression. In order to extract effective spatial features, deformable convolution can be used in the global feature extraction module. Compared with uniform sampling, the sampling position in the deformable convolution can be adjusted and optimized to more effectively extract expression features, and finally obtain global appearance features.

[0066] It should be noted that the preset facial expression recognition model may also include a fusion output module for feature fusion and classification output. After obtaining the local dynamic features and global appearance features extracted by the above module, the global-local features can be fused through the fusion output module, and classified and outputted according to the fused features to obtain the target facial expression category, i.e., the driver's facial expression category.

[0067] It should also be noted that in the process of training and learning the above-mentioned preset facial expression recognition model, specific loss functions of the local feature extraction module and the global feature extraction module can be designed respectively and optimized during the training process to improve the accuracy of expression recognition. These specific loss functions can help model refinement by penalizing errors in expression representation and identity distinction.

[0068] Step S30: determining corresponding fatigue driving intervention measures according to the facial expression category, and generating execution instructions for the fatigue driving intervention measures.

[0069] It should be understood that several facial expression categories and facial expression category combinations symbolizing fatigue states and corresponding fatigue intervention measures can be preset. Then, when obtaining one or more facial expression categories of the driver, the facial expression category (combination) symbolizing fatigue states can be determined therein to obtain corresponding fatigue intervention measures.

[0070] It is understandable that the fatigue intervention measure can be a measure to prompt the driver by changing the voice, lighting, and in-car environment; it can also be a vehicle auxiliary driving measure triggered to prevent safety accidents, such as lane keeping assist, adaptive cruise control, etc.

[0071] In a specific implementation, the intervention device can determine the corresponding fatigue driving intervention measures according to the facial expression category of the fatigue state, and generate corresponding execution instructions to enable relevant equipment in the vehicle to execute the fatigue driving intervention measures.

[0072] The preset facial expression recognition model in this embodiment can combine local features and global features to obtain the driver's facial expression category, reducing the misjudgment that may be caused by single feature analysis. And the long short-term memory network and deformable convolution are used to extract features respectively, which can adapt to different lighting conditions, occlusions and driving environments, ensuring that the driver's facial expression category can be accurately identified under various conditions, thereby achieving the timely response and implementation of corresponding fatigue driving intervention measures.

[0073] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 This is a flow chart of the second embodiment of the fatigue driving intervention method based on facial recognition in this application.

[0074] In this embodiment, the preset facial expression recognition model also includes a shared shallow feature module and a fusion output module. In order to specifically illustrate the recognition process of the preset facial expression recognition model, step S20 specifically includes: steps S201 to S204:

[0075] Step S201: extracting basic features from the facial image sequence and the local area image sequence through the shared shallow feature module to obtain a number of low-level features.

[0076] It should be understood that the shared shallow feature module analyzes the images of the overall facial expression and the local region organs (the facial image sequence and the local region image sequence), and uses the shared knowledge to learn the general facial features as the preparation data for the more detailed analysis in the subsequent modules. The shared shallow feature module can select the first 10 convolutional layers of the pre-trained VGG-16 model, which can learn low-level features from the global and local facial regions.

[0077] Step S202: extracting features from each of the low-level features using a long short-term memory neural network block through the local feature extraction module to obtain local dynamic features.

[0078] It can be understood that first, the primary features of the image sequence associated with the local area can be extracted through the long short-term memory neural network block to obtain the time features corresponding to different local areas; the time features corresponding to the different local areas are connected and integrated to obtain local dynamic features, and determined as local high-level features.

[0079] It should be understood that the above-mentioned different regions may be eyes, nose and mouth. The LST block extracts temporal features from the different regions respectively, and connects the temporal features based on the same time to generate local dynamic features, which can be regarded as local high-level features.

[0080] Step S203: extracting features from each of the low-level features using deformable convolution through the global feature extraction module to obtain global appearance features.

[0081] It should be understood that the global feature extraction module is used to learn spatial features from frames with expression peaks, and extract spatial features reflecting the overall appearance of the face by taking the peak frames as input. Specifically, the global feature extraction module uses deformable convolution to extract expression features by using an adaptive sampling position selection scheme, that is, obtaining global appearance features.

[0082] Step S204: The local dynamic features and the global appearance features are averaged by the fusion output module to obtain a fusion score, and the facial expression category of the driver is determined according to the fusion score.

[0083] It should be understood that the feature vector output of the local feature extraction module, namely the local dynamic feature, can be expressed as f local , the feature vector output of the global feature extraction module, that is, the global appearance feature, is represented as fglobal Each feature vector is sent to the fusion output module and is used in the fusion output module to generate the feature vector. local =[a 1 , a 2 , ...a C ] indicates f local The output score of P global =[b 1 , b 2 , ...b C ] indicates f global The output score of .

[0084] It can be understood that the local dynamic features and the global appearance features can first be averaged to calculate the fusion scores corresponding to different facial expression categories; then the maximum fusion score can be determined among the fusion scores corresponding to each of the facial expression categories, and the facial expression category corresponding to the maximum fusion score can be used as the facial expression category of the driver.

[0085] The calculation formula for determining the maximum fusion score is:

[0086]

[0087] Where Ο is the facial expression category corresponding to the maximum fusion score; As a balance parameter, it can be set to 0.5 according to experience; P local (i) is the predicted probability of the local dynamic feature on facial expression category i, P global (i) is the predicted probability of the global appearance feature on facial expression category i, and i represents the category index of different facial expression categories.

[0088] In addition, you can also refer to Figure 4 The framework of the preset facial expression recognition model used in the fatigue driving intervention method based on facial recognition in this application is described. Figure 4 Schematic diagram of the framework structure of the facial expression recognition model preset for this application.

[0089] Figure 4 In the proposed method, the shared shallow feature module (SSM) is used to perform preliminary feature extraction from facial image sequences (Image sequence) and local area image sequences (Part Extraction), and the obtained low-level features are used as preparation data for more detailed analysis in subsequent modules.

[0090] After the preliminary feature extraction, the local feature extraction module (PBM) processes the key local areas (eyes, nose and mouth), extracts temporal features from these different areas through the LST block, and connects the temporal features to generate local dynamic features.

[0091] The global feature extraction module (GFM) can learn spatial features from image frames with expression peaks and use deformable convolution to extract expression features, that is, to obtain global appearance features.

[0092] The above-mentioned PBM and GFM are respectively designed with specific loss functions (Loss) to optimize the parameters of the models within the modules based on the specific Loss during the training process.

[0093] Figure 4 The specific steps of a multi-task learning method based on video sequences are shown. Figure 4 After PBM and GFM output local dynamic features and global appearance features respectively, a fusion output module can be designed to average the outputs of the above two modules to generate a fusion score. The fusion score determines the facial expression category corresponding to the input, thereby completing the facial expression recognition task.

[0094] In this embodiment, the local dynamic features and the global appearance features are averaged by the fusion output module to obtain a fusion score, and the driver's facial expression category is determined according to the fusion score. Since the preset facial expression recognition model can be constructed based on the multi-task network learning concept in this embodiment, where the multi-task network is a shared model that simultaneously learns the fusion network of the overall facial expression and the local facial expression, due to the sharing of local facial expressions, irrelevant change factors can be gradually filtered out, accelerating the learning process and the effectiveness of detection, thereby further improving the recognition accuracy of the preset facial expression recognition model obtained by training.

[0095] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 5 , Figure 5 This is a flow chart of the third embodiment of the fatigue driving intervention method based on facial recognition in this application.

[0096] In this embodiment, in order to specifically illustrate how to perform corresponding fatigue intervention measures based on the facial expression recognition results, step S30 specifically includes: steps S301 to S302.

[0097] Step S301: determining the current wakefulness state of the driver according to the facial expression category, and judging whether the driver is in a fatigue driving condition according to the current wakefulness state.

[0098] It should be understood that the driver's facial expression categories identified by the preset facial expression recognition model may be one or more, such as smiling, frowning, closing eyes, yawning, etc.

[0099] In a specific implementation, the current awake state of the driver can be analyzed based on the facial expression category. For example, frequent yawning or closed eyes can be considered as fatigue. The yawning threshold and eye closure threshold within a certain interval can be specifically set to determine whether the driver is in a fatigue driving state.

[0100] Step S302: If yes, query the fatigue driving intervention measure corresponding to the current wakefulness state in the preset fatigue driving intervention measure table, and generate an execution instruction of the fatigue driving intervention measure.

[0101] It should be understood that the preset fatigue driving intervention measures table may include fatigue driving intervention measures that can be executed by relevant equipment in the vehicle under different pre-set wakefulness states, such as the lighting system in the vehicle can adjust the lighting, and the voice system in the vehicle can output voice warnings; it may also be safe driving assistance measures that can be executed by relevant equipment in the vehicle under different pre-set wakefulness states, such as the vehicle's advanced driving assistance system for lane assistance, adaptive cruise control, or reducing vehicle speed.

[0102] It should also be noted that the above-mentioned fatigue driving intervention measures may also include: air conditioning adjustment measures and window control measures.

[0103] Exemplarily, when the driver is driving fatigued, the intervention device can send air conditioning adjustment instructions to the air conditioning system in the vehicle, so that the vehicle-mounted air conditioning unit can turn on the cold air and increase the air volume to lower the temperature in the vehicle; or it can also send a lowering instruction to the window unit to achieve the same purpose of stimulating the driver to wake up as the above-mentioned lowering of the temperature in the vehicle.

[0104] In this embodiment, the corresponding fatigue driving intervention measures are determined in the preset fatigue driving intervention measures table according to the different emotional states of the driver, and then the execution instructions of the fatigue driving intervention measures are generated, wherein the fatigue driving intervention measures may include air conditioning adjustment measures and window control measures, so that the above fatigue driving intervention measures can be executed by controlling the on-board air conditioning unit and the window unit to remind and improve the driver's fatigue state and reduce the probability of traffic accidents.

[0105] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the fatigue driving intervention method based on facial recognition of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0106] The present application also provides a fatigue driving intervention device based on facial recognition, referring to Figure 6 , Figure 6 This is a schematic diagram of the module structure of the fatigue driving intervention device based on facial recognition in this application, and the fatigue driving intervention device based on facial recognition includes:

[0107] An image processing module 601 is used to obtain a facial image sequence of the driver and determine a corresponding local area image sequence based on the facial image sequence;

[0108] An expression recognition module 602 is used to input the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used to perform feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used to perform feature extraction on the facial image sequence based on a deformable convolution to obtain global appearance features;

[0109] The environment control module 603 is used to determine the corresponding fatigue driving intervention measure according to the facial expression category and generate an execution instruction of the fatigue driving intervention measure.

[0110] Furthermore, the preset facial expression recognition model also includes a shared shallow feature module and a fusion output module;

[0111] The expression recognition module 602 is also used to extract basic features from the facial image sequence and the local area image sequence through the shared shallow feature module to obtain a number of low-level features; to extract features from each of the low-level features using a long short-term memory neural network block through the local feature extraction module to obtain local dynamic features; to extract features from each of the low-level features using deformable convolution through the global feature extraction module to obtain global appearance features; to average the local dynamic features and the global appearance features through the fusion output module to obtain a fusion score, and to determine the driver's facial expression category based on the fusion score.

[0112] Furthermore, the expression recognition module 602 is also used to extract the primary features of the associated local area image sequence through the long short-term memory neural network block to obtain the time features corresponding to different local areas; connect and integrate the time features corresponding to the different local areas to obtain local dynamic features, and determine them as local high-level features.

[0113] Furthermore, the expression recognition module 602 is also used to extract the primary features associated with the facial image sequence based on an adaptive sampling method using deformable convolution to obtain a plurality of spatial expression features, and integrate each of the spatial expression features into a global appearance feature.

[0114] Furthermore, the expression recognition module 602 is further used to average the local dynamic features and the global appearance features to calculate fusion scores corresponding to different facial expression categories; determine a maximum fusion score among the fusion scores corresponding to the facial expression categories, and use the facial expression category corresponding to the maximum fusion score as the facial expression category of the driver;

[0115] The calculation formula for determining the maximum fusion score is:

[0116]

[0117] Where Ο is the facial expression category corresponding to the maximum fusion score, is the equilibrium parameter, P local (i) is the predicted probability of the local dynamic feature on facial expression category i, P global (i) is the predicted probability of the global appearance feature on facial expression category i.

[0118] Furthermore, the environmental control module 603 shown is also used to determine the current wakefulness state of the driver according to the facial expression category, and judge whether the driver is in a fatigue driving state according to the current wakefulness state; if so, query the fatigue driving intervention measures corresponding to the current wakefulness state in the preset fatigue driving intervention measures table, and generate execution instructions for the fatigue driving intervention measures.

[0119] Among them, the fatigue driving intervention measures include: air conditioning adjustment measures and window control measures.

[0120] The preset facial expression recognition model in this embodiment can combine local features and global features to obtain the driver's facial expression category, reducing the misjudgment that may be caused by single feature analysis. And the long short-term memory network and deformable convolution are used to extract features respectively, which can adapt to different lighting conditions, occlusions and driving environments, ensuring that the driver's facial expression category can be accurately identified under various conditions, thereby achieving the timely response and implementation of corresponding fatigue driving intervention measures.

[0121] In addition, to achieve the above-mentioned purpose, the embodiment of the present application further provides a fatigue driving intervention system based on facial recognition, and the fatigue driving intervention system based on facial recognition includes:

[0122] A camera, an environment response device and the fatigue driving intervention device based on facial recognition as described above, wherein the fatigue driving adjustment device is electrically connected to the camera and the environment response device respectively.

[0123] The camera is used to collect the driver's facial image in real time, obtain a facial image sequence, and send the facial image sequence to the image processing module.

[0124] It should be noted that the camera can be installed on the dashboard of a vehicle to obtain the driver's facial image in real time and send the generated facial image sequence to the image processing module in the intervention device.

[0125] The environmental response device is used to receive the execution instruction sent by the environmental control module and execute corresponding fatigue driving intervention measures.

[0126] It is understandable that the environmental response device can be a window control unit and an air conditioning system in the vehicle. Then, when receiving the execution instruction sent by the environmental control module in the intervention device, the corresponding fatigue driving intervention measures are executed, such as lowering the window or turning on the air conditioning cold air, increasing the air conditioning air volume, etc., so as to stimulate the driver to stay awake, remind and improve the driver's fatigue state, and reduce the probability of traffic accidents.

[0127] Other embodiments or specific implementation methods of the facial recognition-based fatigue driving intervention system device described in this application can refer to the above-mentioned method embodiments and will not be repeated here.

[0128] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0129] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A fatigue driving intervention method based on facial recognition, characterized in that: The method comprises: Acquire a facial image sequence of the driver, and determine a corresponding local area image sequence based on the facial image sequence; Inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used to perform feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used to perform feature extraction on the facial image sequence based on a deformable convolution to obtain global appearance features; The corresponding fatigue driving intervention measure is determined according to the facial expression category, and an execution instruction of the fatigue driving intervention measure is generated.

2. The method according to claim 1, characterized in that The preset facial expression recognition model also includes a shared shallow feature module and a fusion output module; The step of inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the driver's facial expression category comprises: Extracting basic features from the facial image sequence and the local area image sequence through the shared shallow feature module to obtain a number of low-level features; The local feature extraction module uses a long short-term memory neural network block to extract features from each of the low-level features to obtain local dynamic features; Using the global feature extraction module to extract features from each of the low-level features using deformable convolution, to obtain global appearance features; The local dynamic features and the global appearance features are averaged through the fusion output module to obtain a fusion score, and the driver's facial expression category is determined according to the fusion score.

3. The method according to claim 2, characterized in that The step of extracting features from each of the low-level features using a long short-term memory neural network block through the local feature extraction module to obtain local expression features comprises: Extracting primary features of the local region image sequence associated with the local region image sequence through a long short-term memory neural network block to obtain time features corresponding to different local regions; The temporal features corresponding to the different local regions are connected and integrated to obtain local dynamic features, and are determined as local high-level features.

4. The method according to claim 2, characterized in that The step of extracting features from each of the low-level features using deformable convolution by the global feature extraction module to obtain global appearance features comprises: Based on the adaptive selection sampling method, deformable convolution is used to extract the primary features associated with the facial image sequence to obtain a number of spatial expression features, and each of the spatial expression features is integrated into a global appearance feature.

5. The method according to claim 2, characterized in that The step of averaging the local dynamic features and the global appearance features through the fusion output module to obtain a fusion score, and determining the driver's facial expression category according to the fusion score includes: Averaging the local dynamic features and the global appearance features to calculate fusion scores corresponding to different facial expression categories; Determining a maximum fusion score among the fusion scores corresponding to the facial expression categories, and using the facial expression category corresponding to the maximum fusion score as the facial expression category of the driver; The calculation formula for determining the maximum fusion score is: Where Ο is the facial expression category corresponding to the maximum fusion score, is the equilibrium parameter, P local (i) is the predicted probability of the local dynamic feature on facial expression category i, P global (i) is the predicted probability of the global appearance feature on facial expression category i.

6. The method according to claim 1, characterized in that The step of determining the corresponding fatigue driving intervention measure according to the facial expression category and generating an execution instruction of the fatigue driving intervention measure includes: Determining the current wakefulness state of the driver according to the facial expression category, and judging whether the driver is in a fatigue driving condition according to the current wakefulness state; If so, the fatigue driving intervention measure corresponding to the current wakefulness state is queried in the preset fatigue driving intervention measure table, and an execution instruction of the fatigue driving intervention measure is generated.

7. The method according to claim 6, characterized in that The fatigue driving intervention measures include: air conditioning adjustment measures and window control measures.

8. A fatigue driving intervention device based on facial recognition, characterized in that: The device comprises: An image processing module, used for acquiring a facial image sequence of the driver, and determining a corresponding local area image sequence based on the facial image sequence; an expression recognition module, used for inputting the facial image sequence and the local area image sequence into a preset facial expression recognition model to obtain the facial expression category of the driver, wherein the preset facial expression recognition model includes a local feature extraction module and a global feature extraction module, the local feature extraction module is used for performing feature extraction on the local area image sequence based on a long short-term memory neural network block to obtain local dynamic features, and the global feature extraction module is used for performing feature extraction on the facial image sequence based on a deformable convolution to obtain global appearance features; The environment control module is used to determine the corresponding fatigue driving intervention measure according to the facial expression category and generate an execution instruction of the fatigue driving intervention measure.

9. A fatigue driving intervention system based on facial recognition, characterized in that: The system comprises: a camera, an environment response device and the fatigue driving intervention device based on facial recognition as claimed in claim 8, wherein the fatigue driving adjustment device is electrically connected to the camera and the environment response device respectively; The camera is used to collect the driver's facial image in real time, obtain a facial image sequence, and send the facial image sequence to the image processing module; The environmental response device is used to receive the execution instruction sent by the environmental control module and execute corresponding fatigue driving intervention measures.

10. A fatigue driving intervention device based on facial recognition, characterized in that: The facial recognition-based fatigue driving intervention device includes: a memory, a processor, and a facial recognition-based fatigue driving intervention program stored in the memory and executable on the processor. When the facial recognition-based fatigue driving intervention program is executed by the processor, the steps of the facial recognition-based fatigue driving intervention method as described in any one of claims 1 to 7 are implemented.

Images