A cross-domain driver fatigue driving detection method and device, a terminal and a storage medium

Abstract

The application provides a cross-domain driver fatigue driving detection method and device, a terminal and a storage medium, wherein the method comprises the following steps: constructing a detection model; obtaining face image samples of source domain data sets and target domain data sets in a driver fatigue public data set and performing preprocessing; inputting the preprocessed face image samples of the source domain data sets and the target domain data sets into the detection model, and training the detection model by adopting a dynamic structured attention feedback and a priori semantic semi-supervised mechanism to obtain a trained detection model; collecting a face image of a target driver and performing preprocessing; inputting the preprocessed face image of the target driver into the trained detection model; and obtaining a detection result output by the detection model. The application provides a cross-domain driver fatigue driving detection method based on a dynamic structured attention feedback and a priori semantic semi-supervision, which can accurately identify the fatigue state and grade of a driver in different scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a cross-domain driver fatigue detection method, device, terminal and storage medium. Background Technology

[0002] Due to the lack of fatigue detection systems in advanced driver assistance systems, a large number of drivers and pedestrians suffer serious injuries due to driver fatigue. According to the Central Road Research Institute (CRRI), fatigued drivers who fall asleep at the wheel account for approximately 40% of all traffic injuries and fatalities.

[0003] Fatigue driving generally refers to the muscle relaxation and mental fatigue experienced by a driver after prolonged periods of intense concentration while driving, leading to decreased reaction and predictive abilities in the hands and feet, and consequently, sluggish movements. In recent years, fatigue driving detection has become an important research area. According to recent research, fatigue detection technologies are divided into three main categories: fatigue detection based on physiological characteristics, fatigue detection based on vehicle behavior, and fatigue detection based on facial features. First, physiological characteristics depend on physical factors such as heart rate, blood oxygen saturation, and pulse. Generally, electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and electromyography (EMG) are commonly used physiological signals for assessing a driver's physical condition. Among these, EEG is considered the "gold standard" in the field of fatigue driving. However, fatigue detection methods based on physiological characteristics require the driver to wear sensors, which may compromise driver convenience and comfort. Second, fatigue detection methods based on vehicle behavior detect fatigue by observing vehicle behavior, such as steering wheel movement, random braking, and speed changes. The main drawback of using fatigue detection methods based on vehicle behavior is that vehicle behavior can change due to adverse weather and road conditions. Finally, fatigue detection methods based on facial features utilize machine learning and computer vision (CV) to observe facial expressions and movements to detect fatigue. Vision-based facial feature fatigue detection methods have attracted widespread attention due to their advantages of being non-contact, easily obtaining real-time driver facial information, and being cost-effective.

[0004] Existing algorithms for facial expression detection include traditional algorithms such as Viola Jones (Haarcascade), Canny edge detection, and Support Vector Machines (SVM), as well as neural network algorithms such as CNNs, ANNs, Naive Bayes classifiers, and GANs for fatigue detection. However, these algorithms still have shortcomings in fatigue detection based on facial features. They use handcrafted features and fail to learn the complex facial features of the human face, leading to a significant reduction in detection accuracy. Summary of the Invention

[0005] In view of this, the present invention aims to propose a cross-domain driver fatigue detection method, device, terminal and storage medium to solve the problem of poor accuracy of existing algorithms for detecting facial expressions when applied to fatigue driving detection.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] In a first aspect, the present invention provides a method for detecting driver fatigue driving across domains, comprising:

[0008] A detection model is constructed, comprising two parallel sub-networks. Each sub-network utilizes a sequentially connected intra-domain graph convolutional neural network and an inter-domain graph convolutional neural network to mine intra-domain relationships and learn inter-domain feature invariance, respectively. Each sub-network has two inputs: one input consists of the global and local features extracted by ResNet in the detection model; the other input is a feature vector obtained by initializing the global and local features with the semantic feature distribution of the face image through a structured attention feedback mechanism.

[0009] Obtain face image samples from the source and target domain datasets of the publicly available driver fatigue dataset, and perform preprocessing.

[0010] The preprocessed face image samples from the source domain dataset and the target domain dataset are input into the detection model, and the detection model is trained using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model.

[0011] Collect facial images of the target driver and perform preprocessing;

[0012] The preprocessed target driver's face image is input into the trained detection model;

[0013] Obtain the detection results output by the detection model.

[0014] Furthermore, the specific steps of the preprocessing include:

[0015] Use YOLOv5 to detect faces in the image and crop it to a size of 224*224;

[0016] The Dilb facial landmark locator was used to annotate key points on the face; the key points included the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth.

[0017] Furthermore, the step of inputting preprocessed face image samples from the source domain dataset and the target domain dataset into the detection model, and training the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain a trained detection model includes:

[0018] ResNet is used to extract features from face image samples in the preprocessed source and target domain datasets based on keypoint coordinates, and global and local features are extracted respectively.

[0019] By using a structured attention feedback mechanism, global and local features are initialized from the semantic feature distributions of face image samples in the source and target domain datasets, respectively, to obtain feature vectors.

[0020] The global and local features, as well as the feature vectors, are input into each sub-network. The intra-domain graph convolutional neural network and inter-domain graph convolutional neural network in each sub-network are used to learn the relationship between global and local features within the domain and to mine the feature invariance between domains to obtain cross-domain collaborative adaptation. Each sub-network mines and learns the weight values ​​of the connections between nodes through nodes with structured attention and graph convolutional neural networks to obtain the relationship between regions within the domain and the collaborative adaptation between domains.

[0021] Obtain the classification feature vector output by each sub-network;

[0022] The classification feature vector is input into the classifier for classification, and the classification result is optimized by using a prior semantic semi-supervised mechanism to promote the feature distribution of the source domain dataset and the target domain dataset, thus obtaining the trained detection model.

[0023] Furthermore, the feature vector is obtained by initializing global and local features from the semantic feature distributions of face image samples in the source and target domain datasets using a structured attention feedback mechanism, including:

[0024] For the face image samples corresponding to both the source and target domain datasets, a dynamic structured attention feedback mechanism is used to divide the face image samples into C clusters, and the attention-enabled feature vector for each cluster is calculated. The calculation formula is shown below:

[0025]

[0026]

[0027] Among them, f k (·) is the feature extractor for region k; k is the set of global and local features, k∈{h,le,re,no,lm,rm} representing global features, left eye, right eye, nose, left corner of mouth, and right corner of mouth, respectively; It is the total number of samples in a certain cluster C of the source domain dataset(s). It is the total number of samples in a certain cluster C of the target domain dataset (t); It is the i-th sample in the cluster of the source domain dataset(s); It is the j-th sample in the cluster of the target domain dataset (t);

[0028] A dynamic structured attention feedback mechanism is employed to iteratively update the statistical distribution in a progressive manner. For each batch iteration, the distance between each face image sample in each domain and each cluster distribution is calculated to group face image samples into the cluster with the smallest distance. Then, feature vectors with structured attention are calculated and updated across all face image samples in the same cluster. The update formula is shown below:

[0029]

[0030]

[0031] Where α is a balance parameter;

[0032] The face image samples are re-clustered to obtain a new statistical distribution for each cluster;

[0033] Iterative epoch-level re-aggregation and iteration-level updates are performed to obtain the final statistical distribution, as well as feature vectors of global and local features.

[0034] Furthermore, the step of inputting the classification feature vector into the classifier for classification, and using a prior semantic semi-supervised mechanism to back-promote the feature distribution of the source domain dataset and the target domain dataset to optimize the classification result, thereby obtaining the trained detection model, includes:

[0035] The classification feature vector is input into the classifier for classification;

[0036] A multi-hot prior semantic semi-supervised mechanism is employed to inversely promote the feature distribution of each domain and optimize the classification results. The process of the prior semantic semi-supervised mechanism is shown in the following formula:

[0037]

[0038] in, It is the prior semantic feature distribution of face image samples in the source domain dataset or the target domain dataset;

[0039] The trained detection model is obtained.

[0040] Furthermore, the sub-network includes an intra-domain graph convolutional neural network and an inter-domain graph convolutional neural network, as shown in the following formulas:

[0041] G in_tra =(V,A in_tra ), G in_ter =(V,A in_ter ),

[0042] in These are the corresponding nodes obtained from global or local features extracted from the source domain dataset, or the corresponding nodes obtained from global or local features extracted from the target domain dataset; A in_tra A is the adjacency matrix within the field. in_tra It includes two connection methods: local to global, and local to local; A in_ter It is the adjacency matrix between fields, A in_ter It includes three connection methods: local to local, global to local, and global to global.

[0043] Secondly, the present invention also provides a cross-domain driver fatigue detection device, comprising:

[0044] Build modules are used to construct detection models;

[0045] The acquisition module is used to acquire face image samples from the source domain dataset and the target domain dataset in the publicly available driver fatigue dataset, and to perform preprocessing.

[0046] The training module is used to input face image samples from the preprocessed source domain dataset and target domain dataset into the detection model, and to train the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model.

[0047] The acquisition module is used to acquire facial images of the target driver and perform preprocessing.

[0048] The processing module is used to input the preprocessed target driver face image into the trained detection model;

[0049] The module is used to obtain the detection results output by the detection model.

[0050] Thirdly, the present invention also provides a terminal, comprising:

[0051] One or more processors;

[0052] Storage device for storing one or more programs;

[0053] A camera is used to capture images;

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement the detection method provided in the above embodiments.

[0055] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the detection method provided in the above embodiments.

[0056] Compared with existing technologies, the cross-domain driver fatigue detection method, device, terminal, and storage medium described in this invention have the following advantages:

[0057] (1) This invention proposes a cross-domain driver fatigue detection method based on dynamic structured attention feedback and prior semantic semi-supervision, which can accurately identify the driver's fatigue state and level in different scenarios.

[0058] (2) The detection method of this invention inputs preprocessed face image samples from the source domain dataset and the target domain dataset into a detection model, and trains the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism, thus obtaining a detection model that can effectively monitor driver fatigue. Then, by using the trained detection model to detect the face image of the target driver, the current fatigue state of the target driver can be obtained, enabling the determination of whether the driver is driving while fatigued. This facilitates subsequent monitoring or reminders to the target driver, and helps ensure the safety of the target driver during driving.

[0059] (3) The detection method described in this invention is also based on a dynamic structured attention feedback mechanism. By constructing a graph neural network with attention nodes, it mines the relationship between regions within the domain and the invariant features between domains to eliminate feature transfer between domains. It also adjusts the distribution of the source domain and the target domain through a feedback mechanism with class information.

[0060] (4) The detection method described in this invention also utilizes a priori semantic semi-supervised mechanism to further classify the feature distribution between different domains using multi-hot encoded information to obtain accurate classification, which is beneficial to enhancing the robustness and generalization ability of the detection model. Attached Figure Description

[0061] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0062] Figure 1 This is a flowchart of a cross-domain driver fatigue detection method according to Embodiment 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of the detection model in the cross-domain driver fatigue driving detection method described in Embodiment 1 of the present invention;

[0064] Figure 3This is a schematic diagram of the structure of a cross-domain driver fatigue detection device according to Embodiment 2 of the present invention;

[0065] Figure 4 This is a schematic diagram of the structure of a terminal provided in Embodiment 3 of the present invention. Detailed Implementation

[0066] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0067] Example 1

[0068] Figure 1 This is a flowchart of a cross-domain driver fatigue detection method according to Embodiment 1 of the present invention. This detection method can be used for cross-domain driver fatigue detection by detecting the driver's facial image information to determine whether the driver is fatigued. See also Figure 1 The specific steps of this method are as follows:

[0069] Step 101: Construct a detection model; the detection model includes two parallel sub-networks, each of which is constructed by sequentially connecting an intra-domain graph convolutional neural network and an inter-domain graph convolutional neural network to mine intra-domain relationships and learn inter-domain feature invariance; each sub-network has two inputs, one of which is the global and local features extracted by ResNet in the detection model, and the other of which is the feature vector obtained after initializing the global and local features with the semantic feature distribution of the face image through a structured attention feedback mechanism.

[0070] Traditional supervised learning typically requires a large amount of labeled data for training, and it's essential to ensure that the data distributions in the training and test sets are similar. If the training and test sets have different distributions, the trained classifier will not perform well on the test set. The method described in this embodiment is primarily based on an adversarial cross-domain mechanism. This method learns domain-invariant features through adversarial mechanisms; that is, the goal of the feature extractor is to generate transferable features that can deceive the domain discriminator, while the goal of the domain discriminator is to distinguish face image samples between the source and target domain datasets.

[0071] Step 102: Obtain face image samples from the source domain dataset and the target domain dataset in the driver fatigue public dataset, and perform preprocessing.

[0072] The dataset in the cross-domain driver fatigue detection framework consists of a source domain dataset. and target domain dataset Composition. Specifically, the source domain dataset can be the publicly available driver fatigue dataset UTA-RLDD, and the target domain dataset can be the publicly available driver fatigue dataset DROZY. Each source domain dataset contains face image samples. They all have a label (i.e., alert, early fatigue, severe fatigue), but the target domain samples have no labels.

[0073] In practical applications, the following preprocessing steps can be used to process face image samples: First, use YOLOv5 to detect faces in the image and crop it to a size of 224*224; second, use the Dilb face landmark locator to annotate the key points of the face; among which, the key points include the left eye, right eye, nose, left corner of mouth, and right corner of mouth.

[0074] Step 103: Input the preprocessed face image samples from the source domain dataset and the target domain dataset into the detection model, and train the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model.

[0075] The detection model in this method consists of two parallel sub-networks, each with two inputs. The first input to each sub-network is the global and local features extracted by the feature extraction network (ResNet, Residual Neural Network) from the source and target domain datasets based on keypoint coordinates. This embodiment uses ResNet as the feature extraction network because it is a backbone network in the visual domain and has superior performance. The second input to each sub-network is the feature vector obtained by initializing the global and local features using a structured attention feedback mechanism, respectively, based on the semantic feature distribution of face image samples in the source and target domain datasets.

[0076] In practical applications, when training the detection model using face image samples from the preprocessed source domain dataset and target domain dataset, the face image samples in the two datasets can be divided into a training set and a test set for training and testing the detection model, with the training set and test set divided in a 7:3 ratio.

[0077] For example, the specific training process of the detection model can be carried out by referring to the following steps:

[0078] Step 1031: Using ResNet, feature extraction is performed on the preprocessed source and target domain face image samples based on keypoint coordinates, obtaining global and local features for both datasets. Specifically, the global and local feature vectors for each class with attention in the source domain are estimated as follows: The global and local feature vectors with attention for each class in the target domain are as follows:

[0079] Step 1032: Using a structured attention feedback mechanism, initialize global and local features from the semantic feature distribution of face image samples in the source and target domain datasets to obtain feature vectors.

[0080] Integrating class information plays a crucial role in enabling finer-grained intra-class interactions and adaptations. Specifically, global and local feature vectors are first initialized from the feature distributions of the source and target datasets. Then, by clustering the samples, a statistical distribution is obtained every E periods, followed by iterative-level updates. Each iteration updates the statistical distribution, resulting in updated global and local feature vectors.

[0081] In practical applications, the face image samples corresponding to both the source and target domain datasets can be divided into C clusters using a dynamic structured attention feedback mechanism. The attention-enabled feature vector for each cluster is then calculated, as shown in the following formula:

[0082]

[0083]

[0084] Among them, f k (·) is the feature extractor for region k; k is the set of global and local features, k∈{h,le,re,no,lm,rm} representing global features, left eye, right eye, nose, left corner of mouth, and right corner of mouth, respectively; It is the total number of samples in a certain cluster C of the source domain dataset(s). It is the total number of samples in a certain cluster C of the target domain dataset (t); It is the i-th sample in the cluster of the source domain dataset(s); It is the j-th sample in the cluster of the target domain dataset (t).

[0085] Then, a dynamic structured attention feedback mechanism is used to iteratively update the statistical distribution in a progressive manner. For each batch iteration, the distance between each face image sample in each domain and each cluster distribution is calculated to group the face image samples into the clusters with the smallest distance. Subsequently, feature vectors with structured attention are calculated and updated across all face image samples in the same cluster. The update formula is shown below:

[0086] Here, α is a balancing parameter. To avoid distribution shifts, the update process must be repeated in each epoch.

[0087] The face image samples were then re-clustered to obtain a new statistical distribution for each cluster.

[0088] Specifically, re-clustering can be performed using the following formula, as shown below:

[0089]

[0090]

[0091] Finally, as the training process continues, epoch-level re-aggregation and iteration-level updates are performed iteratively to obtain the final statistical distribution, as well as feature vectors of global and local features.

[0092] By constructing a graph neural network with attention nodes based on a dynamic structured attention feedback mechanism, the relationships between regions within a domain and the invariant features between domains can be mined, which helps to eliminate feature transfer between domains. Furthermore, the distribution of the source and target domains can be adjusted through a feedback mechanism with class information.

[0093] Step 1033: Input the global features, local features, and feature vectors into each sub-network. Utilize the intra-domain graph convolutional neural network and inter-domain graph convolutional neural network in each sub-network to learn the relationship between global and local features within the domain, and mine inter-domain feature invariance to obtain cross-domain collaborative adaptation. Each sub-network mines and learns the weight values ​​of each node connection through nodes with structured attention and graph convolutional neural networks to obtain the relationship between regions within the domain and inter-domain collaborative adaptation.

[0094] After initializing the global and local features in step 1032 above, it is necessary to construct two types of graphs: intra-domain and inter-domain. Messages are then propagated through a graph convolutional neural network to learn the relationships between global and local features within the same domain and to mine feature invariance between domains to obtain cross-domain collaborative adaptation. Therefore, this method utilizes sub-networks to process the feature vectors after initializing the global and local features.

[0095] Specifically, the sub-networks include intra-domain graph convolutional neural networks and inter-domain graph convolutional neural networks, as shown in the following formulas:

[0096] G in_tra =(V,A in_tra ), G in_ter =(V,A in_ter ),

[0097] in These are the corresponding nodes obtained from global or local features extracted from the source domain dataset, or the corresponding nodes obtained from global or local features extracted from the target domain dataset; A in_tra A is the adjacency matrix within the field. in_tra It includes two connection methods: local to global, and local to local; A in_ter It is the adjacency matrix between fields, A in_ter It includes three connection methods: local to local, global to local, and global to global.

[0098] By using node-based and graph-based convolutional neural networks with structured attention to mine and learn the weight values ​​of each node connection, the relationships between regions within the domain and the collaborative adaptation between domains are obtained, ultimately yielding a classification feature vector for classification.

[0099] Step 1034: After obtaining the classification feature vector output by each sub-network, input the classification feature vector into the classifier for classification, and use the prior semantic semi-supervised mechanism to back-promote the feature distribution of the source domain dataset and the target domain dataset to optimize the classification results, thus obtaining the trained detection model.

[0100] In practical applications, the classification feature vectors can be input into a classifier for classification. Then, a multi-hot prior semantic semi-supervised mechanism is used to backpropagate the feature distribution of each domain to optimize the classification results. The process of the prior semantic semi-supervised mechanism is shown in the following formula:

[0101]

[0102] in, This refers to the prior semantic feature distribution of face image samples in the source or target domain dataset. Finally, the trained detection model can be obtained.

[0103] Specifically, this work employs intra-domain and inter-domain graph convolutional neural networks (GCNs) to mine intra-domain relationships and learn inter-domain feature invariance, respectively, while utilizing a prior semantic semi-supervised mechanism. Since the sub-networks construct two graphs with structured attention, each sub-network performs message propagation through the intra-domain graph to explore interactions with the global and local features of each domain, and performs message propagation through the inter-domain graph to achieve global-local feature adaptation. The graph convolutional neural networks (GCNs) effectively update the node features of the graph structure data by iteratively propagating node information to neighboring nodes. In this work, two stacked GCNs are used to propagate messages through the two graphs, while dynamic structured attention feedback and prior semantic semi-supervised updating of the feature distributions of the source and target domains are employed.

[0104] Furthermore, since the class information feedback mechanism is a one-hot encoding form, it is difficult to capture multi-hot encoded information. Therefore, employing multi-hot semantic semi-supervision can overcome the above shortcomings. Multi-hot encoded information can further classify the feature distributions between different domains to obtain accurate classification, which is beneficial to enhancing the robustness and generalization ability of the detection model.

[0105] Step 104: Acquire the facial image of the target driver and perform preprocessing.

[0106] In practical applications, facial images of the target driver can be captured by a camera to detect the driver's fatigue level. Specifically, the captured facial images of the target driver can be processed using the following preprocessing steps: First, use YOLOv5 to detect the face in the image and crop it to a size of 224*224; second, use the Dilb facial landmark locator to annotate the key points of the face; among which, the key points include the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth.

[0107] Step 105: Input the preprocessed target driver's face image into the trained detection model.

[0108] By inputting the preprocessed target driver's face image into the trained detection model, the trained detection model can recognize and detect the face and obtain the detection result, i.e. the classification result output by the classifier, including sober, early fatigue, and severe fatigue.

[0109] Step 106: Obtain the detection results output by the detection model.

[0110] The detection results output by the detection model can be used to remind the target driver to rest in time, avoid driving while fatigued, and improve the driver's safety while driving.

[0111] This embodiment proposes a cross-domain driver fatigue detection method based on dynamic structured attention feedback and prior semantic semi-supervised learning, which can accurately identify the driver's fatigue state and level in different scenarios. By inputting preprocessed face image samples from the source and target domain datasets into the detection model, and training the model using dynamic structured attention feedback and prior semantic semi-supervised learning, a detection model capable of effectively monitoring driver fatigue levels is obtained. Subsequently, by using the trained detection model to detect the target driver's face image, the current fatigue state of the target driver can be determined, enabling the judgment of whether the driver is driving while fatigued. This facilitates subsequent monitoring or alerting of the target driver, ensuring the driver's safety during driving.

[0112] In a preferred embodiment of this example, after acquiring and preprocessing the facial image of the target driver, the following step can be added: uploading the facial image of the target driver to the monitoring server via the Internet, and storing and backing up the facial image of the target driver using the built-in or external storage device of the monitoring server for later tracing or verification.

[0113] Example 2

[0114] Figure 3 This is a schematic diagram of the structure of a cross-domain driver fatigue detection device according to Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the detection device includes:

[0115] Module 201 is used to build the detection model.

[0116] The acquisition module 202 is used to acquire face image samples from the source domain dataset and the target domain dataset in the driver fatigue public dataset, and to perform preprocessing.

[0117] The training module 203 is used to input face image samples from the preprocessed source domain dataset and target domain dataset into the detection model, and to train the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model.

[0118] The acquisition module 204 is used to acquire the facial image of the target driver and perform preprocessing.

[0119] The processing module 205 is used to input the preprocessed target driver face image into the trained detection model.

[0120] The module 206 is used to obtain the detection results output by the detection model.

[0121] The detection device provided in this embodiment constructs a detection model based on dynamic structured attention feedback and prior semantic semi-supervised learning, and trains the detection model using face image samples from the source domain dataset and target domain dataset in the driver fatigue public dataset. The trained detection model can then be used to achieve accurate identification and detection of the fatigue level of the target driver.

[0122] The detection device provided in the embodiments of the present invention can execute the detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0123] Example 3

[0124] Figure 4 This is a schematic diagram of the structure of a terminal provided in Embodiment 3 of the present invention. Figure 4A block diagram is shown of an exemplary terminal 12 suitable for implementing embodiments of the present invention. Figure 4 The terminal 12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0125] like Figure 4 As shown, terminal 12 is presented in the form of a general-purpose computing device. The components of terminal 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0126] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0127] Terminal 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by terminal 12, including volatile and non-volatile media, removable and non-removable media.

[0128] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Terminal 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0129] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0130] Terminal 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the terminal 12, and / or with any device that enables the terminal 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, terminal 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of terminal 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the detection method provided in the embodiments of the present invention.

[0132] Example 4

[0133] Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform any of the detection methods provided in the above embodiments.

[0134] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0135] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0136] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0137] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for detecting driver fatigue across domains, characterized in that, include: A detection model is constructed; the detection model includes two parallel sub-networks, each of which is constructed by sequentially connecting an intra-domain graph convolutional neural network and an inter-domain graph convolutional neural network to mine intra-domain relationships and learn inter-domain feature invariance, respectively; wherein, each sub-network has two inputs, one of which is the global and local features extracted by ResNet in the detection model, and the other of which is the feature vector obtained after initializing the global and local features with the semantic feature distribution of the face image through a structured attention feedback mechanism; Obtain face image samples from the source and target domain datasets of the publicly available driver fatigue dataset, and perform preprocessing. The preprocessed face image samples from the source domain dataset and the target domain dataset are input into the detection model, and the detection model is trained using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model. Collect facial images of the target driver and perform preprocessing; The preprocessed target driver's face image is input into the trained detection model; Obtain the detection results output by the detection model; The process involves inputting preprocessed face image samples from the source and target domain datasets into the detection model, and training the model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model, including: ResNet is used to extract features from face image samples in the preprocessed source and target domain datasets based on keypoint coordinates, and global and local features are extracted respectively. By using a structured attention feedback mechanism, global and local features are initialized from the semantic feature distributions of face image samples in the source and target domain datasets, respectively, to obtain feature vectors. The global and local features, as well as the feature vectors, are input into each sub-network. The intra-domain graph convolutional neural network and inter-domain graph convolutional neural network in each sub-network are used to learn the relationship between global and local features within the domain and to mine the feature invariance between domains to obtain cross-domain collaborative adaptation. Each sub-network mines and learns the weight values ​​of the connections between nodes through nodes with structured attention and graph convolutional neural networks to obtain the relationship between regions within the domain and the collaborative adaptation between domains. Obtain the classification feature vector output by each sub-network; The classification feature vector is input into the classifier for classification, and the classification result is optimized by using a prior semantic semi-supervised mechanism to promote the feature distribution of the source domain dataset and the target domain dataset, thus obtaining the trained detection model. The feature vector is obtained by initializing global and local features from the semantic feature distributions of face image samples in the source and target domain datasets using a structured attention feedback mechanism, including: For the face image samples corresponding to both the source and target domain datasets, a dynamic structured attention feedback mechanism is used to divide the face image samples into... There are 10 clusters, and the feature vector with attention for each cluster is calculated as follows: in, For the region Feature extractor; It is a collection of global and local features. These represent the global features, left eye, right eye, nose, left corner of mouth, and right corner of mouth, respectively. It is the source domain dataset ( s A certain cluster The total number of samples, It is the target domain dataset ( t A certain cluster The total number of samples; It is the source domain dataset ( s ) The first cluster One sample; It is the target domain dataset ( t ) The first cluster One sample; A dynamic structured attention feedback mechanism is employed to iteratively update the statistical distribution in a progressive manner. For each batch iteration, the distance between each face image sample in each domain and each cluster distribution is calculated to group the face image samples into the clusters with the smallest distances. Then, feature vectors with structured attention are calculated and updated across all face image samples in the same cluster. The update formula is shown below: ， ， in It is a balance parameter; The face image samples are re-clustered to obtain a new statistical distribution for each cluster; Iterative epoch-level re-aggregation and iteration-level update are performed to obtain the final statistical distribution, as well as the feature vectors of global and local features; The process involves inputting the classification feature vector into a classifier for classification, and using a prior semantic semi-supervised mechanism to back-promote the feature distribution of the source and target domain datasets to optimize the classification results, thereby obtaining a trained detection model, including: The classification feature vector is input into the classifier for classification; A multi-hot prior semantic semi-supervised mechanism is employed to inversely promote the feature distribution of each domain and optimize the classification results. The process of the prior semantic semi-supervised mechanism is shown in the following formula: in, It is the prior semantic feature distribution of face image samples in the source domain dataset or the target domain dataset; The trained detection model is obtained.

2. The method according to claim 1, characterized in that, The specific steps of the preprocessing include: Use YOLOv5 to detect faces in the image and crop it to a size of 224*224; The Dilb facial landmark locator was used to annotate key points on the face; the key points included the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth.

3. The method according to claim 1, characterized in that, The sub-network includes an intra-domain graph convolutional neural network and an inter-domain graph convolutional neural network, as shown in the following formula: ， ， ， in These are the corresponding nodes obtained from global or local features extracted from the source domain dataset, or the corresponding nodes obtained from global or local features extracted from the target domain dataset. It is the adjacency matrix within the domain. It includes two connection methods: local to global, and local to local; It is the adjacency matrix between domains. It includes three connection methods: local to local, global to local, and global to global.

4. A cross-domain driver fatigue detection device, used to perform the method as described in claim 1, characterized in that, include: Build modules are used to construct detection models; The acquisition module is used to acquire face image samples from the source domain dataset and the target domain dataset in the publicly available driver fatigue dataset, and to perform preprocessing. The training module is used to input face image samples from the preprocessed source domain dataset and target domain dataset into the detection model, and to train the detection model using a dynamic structured attention feedback and prior semantic semi-supervised mechanism to obtain the trained detection model. The acquisition module is used to acquire facial images of the target driver and perform preprocessing. The processing module is used to input the preprocessed target driver face image into the trained detection model; The module is used to obtain the detection results output by the detection model.

5. A terminal, characterized in that, include: One or more processors; Storage device for storing one or more programs; A camera is used to capture images; When the one or more programs are executed by the one or more processors, the one or more processors implement the detection method as described in any one of claims 1-3.

6. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the detection method as described in any one of claims 1-3.

Images