Face fatigue state detection method based on three-channel graph neural network

Abstract

The invention provides a face fatigue state detection method based on a three-channel graph neural network, and the method is based on a multi-branch graph network detection model, and comprises the steps: converting the feature point information of each region of a face into graph node structure information, and inputting the graph node structure information into the multi-branch graph network detection model; according to the graph node structure information, extracting node features of a corresponding region through a graph convolutional network; the mapping relation between the node features and the facial fatigue is learned; self-adaptively learning the weight of each region through a self-adaptive learning weight network; the method comprises the following steps of: obtaining a probability detection result P, multiplying by corresponding regional features, splicing the three regional features, and regressing to obtain a probability detection result P through a multi-layer sensing mechanism; according to the method, the fatigue features corresponding to the node features are learned through the graph convolutional network, so that generalization is greatly improved; through adaptive learning of the real weight of each branch feature, the proportion of the occlusion area in the total feature is effectively adjusted, the detection precision is ensured, the single-frame detection time is reduced, and the real-time requirement of actual driving detection is met.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving recognition and detection, and in particular to a method for detecting facial fatigue status using a three-channel graph neural network. Background Art

[0002] In today's society, fatigue driving is one of the important causes of serious traffic accidents and has a significant impact on public safety. According to statistics, traffic accidents caused by fatigue driving account for about 20% of all traffic accidents each year, and such accidents account for more than 40% of serious traffic accidents. It is difficult for drivers to self-detect before entering a fatigue state, so it is of great significance to study high-performance driving fatigue detection methods. Due to the rapid development of machine vision technology, image-based fatigue detection has become one of the research hotspots in recent years.

[0003] The fatigue detection method based on visual features mainly uses a camera to shoot the driver in real time and extract the driver's facial features from the video. The eyes and mouth can provide information about the closing time, blinking frequency, or yawning, thereby providing information about the driver's sleepiness or fatigue. A fatigue driving detection method combining improved YOLOv5s and DeepSort combines facial feature detection results with the number of consecutive closed eye frames, the number of consecutive yawning frames, and the pupil closure percentage (PERCLOS) score to build a fatigue driving detection model. There is also a CNN-LSTM robust method for advanced detection of driver sleepiness, including yawning, eye movement detection, and head movement, with a model accuracy of 98.46%. The eye aspect ratio (EAR) is used to find the degree of eye opening to determine the driver's sleepiness, and the mouth aspect ratio (MAR) is used to find the degree of mouth opening to determine yawning. A driver fatigue detection method based on EAR is proposed, which corrects the EAR by head posture estimation to reduce the false alarm rate, which is 5.09% higher than the traditional method.

[0004] In the current driving process, many drivers are accustomed to wearing sunglasses and masks, and the state information of eyes and mouth is the main feature of fatigue driving. These problems make it difficult to extract fatigue features from driving images, thereby reducing the accuracy of fatigue detection. An improved facial fatigue sign detection algorithm based on DS evidence theory uses improved DS for fusion decision-making and determines the fatigue level, effectively solving the problems of face illumination and partial occlusion in complex environments, but still has the problem of low generalization in different facial features. The MC-Facenet model uses a multi-task cascade convolutional neural network to complete partial occlusion detection, and then introduces MobilenetV1 and attention mechanism to solve the problem of partially occluded face recognition, but the method only improves the detection accuracy of the model on the face. The DNN-based face detection method captures the landmark information of the face when the face is occluded by a large area, and improves the detection robustness by calculating the kinetic energy of facial organs based on the improved optical flow method, but the method still calculates the PERCLOS value to determine whether it is in a fatigue state, and the generalization is low. Summary of the invention

[0005] The purpose of the present invention is to provide a method for detecting facial fatigue with high generalization and accuracy.

[0006] To achieve the above object, the present invention proposes a three-channel graph neural network face fatigue state detection method, the detection method is based on a multi-branch graph network detection model, and includes the following steps:

[0007] S1: After the feature point information extracted from each facial region is converted into graph node structure information, it is input into the multi-branch graph network detection model;

[0008] S2: In the multi-branch graph network detection model, the node features of the corresponding area are extracted through the graph convolutional network according to the graph node structure information; and the mapping relationship between the node features and facial fatigue is learned;

[0009] S3: Adaptively learn the weights of each region through the adaptive learning weight network module; multiply it with the corresponding regional features, then concatenate the three regional features, and regress the probability detection result P through the multi-layer perception mechanism.

[0010] Furthermore, the multi-branch graph network detection model includes a detection module, an extraction module, an adaptive learning weight network module and a training module:

[0011] Detection module: used to detect feature points in each area of ​​the face image, and convert the position coordinates of the feature points and the distances between adjacent feature points into the extraction module to construct the nodes required for the graph convolution network and the relationship between the nodes, namely the node feature matrix and the adjacency matrix;

[0012] Extraction module: extract node features of the corresponding area through graph convolutional network, and learn the mapping relationship between node features and facial fatigue;

[0013] Adaptive Learning Weight Network Module: Through the Adaptive Learning Weight Network (ALW-Net), the matching degree between the detection node and the area shown in the image is detected, and the true weight of each branch feature is adaptively learned to enhance the features of the unobstructed area of ​​the face, while reducing the influence of the features of the feature points in the occluded area on the overall features;

[0014] Training module: used for training data synthesis, manually adding facial occluders through synthesis.

[0015] Furthermore, the training method of the multi-branch graph network detection model is: collecting personal driving data sets of different driving scenes, different perspectives, different times and different countries, and randomly extracting discontinuous images to record corresponding fatigue states. At the same time, randomly selecting images from unobstructed images, and manually adding facial occluders, such as sunglasses and masks, through synthesis.

[0016] Furthermore, the data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, and the size is uniformly adjusted to 256×256×3. After the data set is normalized, it is input into the multi-branch graph network detection model.

[0017] Furthermore, in step S1, feature point information of the left eye, right eye and mouth area of ​​the face in the image is extracted by Dlib facial feature point detection technology, and the feature point information of the left eye, right eye and mouth is converted into graph node structure information. The specific steps are as follows:

[0018] 1.1: Mark the position coordinates of the feature points in each region as N i =(x i ,y i ), and used as input information for each graph node; the feature points are feature marker points extracted from the face image by the Dlib face feature point detection algorithm, and the graph nodes are logical points in the graph convolutional neural network;

[0019] 1.2: According to the results of the Dlib algorithm, the eye area morphology is marked by 6 feature points, and the mouth area morphology is marked by 20 feature points;

[0020] 1.3: Use the position coordinates of the 6 feature points in the eye area to build a 2×6 eye area node feature matrix, recorded as: X eye =(N1,N2,…,N6);

[0021] Use the position coordinates of the 20 feature points in the mouth area to build a 2×20 mouth area node feature matrix, recorded as: Xmouth =(N1,N2,…,N 20 );

[0022] 1.4: In order to capture the overall structure and characteristics of each regional graph node, the Euclidean distance D between adjacent feature points is calculated by the position coordinates of adjacent feature points i,j Build the corresponding adjacency matrix A. Since the distance from a node to itself is zero and the graph nodes are also undirected, the adjacency matrix A is a symmetric matrix.

[0023] The Euclidean distance calculation formula is:

[0024] The expression of the adjacency matrix A is:

[0025] Among them, x i and i Respectively represent the horizontal and vertical coordinates of node i; x j and j Respectively represent the horizontal and vertical coordinates of node j; D i,j represents the Euclidean distance between node i and node j; n represents the number of nodes in the adjacency matrix, D nn It represents the distance of the nth node itself to be zero;

[0026] 1.5: After normalizing the node feature matrix and adjacency matrix of each region, input them into the corresponding node feature extraction network.

[0027] Furthermore, in step S2, the multi-branch graph network detection model has three branch graph channels, each of which passes through a graph convolutional network to detect the left eye G eye1 =(X eye1 ,A eye1 ), right eye G eye2 =(X eye2 ,A eye2 ) and mouth G mouth =(X mouth ,A mouth ) area map for feature extraction, where X eye1 is the node feature matrix of the left eye area, A eye1 is the left eye adjacency matrix; X eye2 is the node feature matrix of the right eye region, A eye2 is the right eye adjacency matrix; X mouth is the feature matrix of the mouth area nodes, A mouth is the mouth adjacency matrix. Graph convolution extracts facial fatigue features by extracting the node feature matrix and adjacency matrix of each node, extracting the position change of the node and the distance change between adjacent nodes. The calculation process formula of graph convolution feature extraction is:

[0028]

[0029] Among them, H l is the feature matrix of the l-th layer node, W (l) is the weight matrix of the lth layer, is the adjacency matrix with self-loops added, is the degree matrix, σ is the activation function, H l+1 is the feature of the l+1th layer.

[0030] Furthermore, the graph convolutional network performs feature extraction through two GCN layers, and then accesses global pooling to aggregate node-level information into global information, ensuring that the outputs of the three-branch GCN are global feature vectors of fixed size.

[0031] Furthermore, the adaptive learning weight network includes a spatial attention module, which is used to splice and convolve the results of maximum pooling and average pooling of image features, then calculate the attention value of the image target area through a Sigmoid function, and finally strengthen the image features by multiplication.

[0032] Furthermore, step S3 is specifically as follows:

[0033] 3.1: Inputting the image features extracted by the graph convolutional network into the adaptive learning weight network;

[0034] 3.2: Using the spatial attention module SAM technology, the attention value of the target area of ​​the image is calculated to strengthen the features of the area of ​​interest;

[0035] 3.3: Unify the feature image size by adopting the spatial pyramid pooling structure SPP;

[0036] 3.4: After flattening, normalizing and linearly transforming the node feature matrix, it is concatenated with the image feature matrix and a value of (0, 1) is obtained through softmax as the corresponding branch weight;

[0037] 3.5: Multiply the three regional features by the weights of each branch and then concatenate them, and regress the probability detection result P through the multi-layer perceptron. The P value range is [0,1]. When it is greater than 0.5, it is considered to be a fatigue state, otherwise it is a normal state.

[0038] Furthermore, in the graph convolutional network, eye contour features are extracted through 3×3 convolution and Relu activation function; the number of convolution channels is 32 and the step size is 1.

[0039] Furthermore, considering that the sizes of the cropped area images are different in different situations, a spatial pyramid pooling structure is used to unify the size of the feature map. The spatial pyramid pooling structure SPP uses three pooling layers of 1×1, 2×2, and 4×4 to calculate the image features, and outputs a feature vector of a fixed size after splicing.

[0040] Furthermore, the data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, and the size is uniformly adjusted to 256×256×3. After the data set is normalized, it is input into the multi-branch graph network detection model.

[0041] Compared with the prior art, the advantages of the present invention are:

[0042] 1. The present invention uses feature points as the main data for detection, effectively avoiding the interference of other features in the image; at the same time, compared with the method of calculating a single aspect ratio, the present invention extracts node features of the corresponding area through a graph convolutional network and learns the fatigue features corresponding to the node features, which greatly improves the generalization.

[0043] 2. In the process of fatigue state detection, the present invention introduces an adaptive learning weight network module, which adaptively learns the real weight of each branch feature, effectively adjusts the proportion of the occluded area in the total feature, reduces its impact on classification accuracy, ensures detection accuracy, effectively improves accuracy, reduces single-frame detection time, and meets the real-time requirements of actual driving detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method for detecting facial fatigue state according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a single-channel branch graph network structure of a multi-branch graph network detection model in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the structure of an adaptive learning weight network (ALW-Net) in an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of the structure of the spatial attention module in an embodiment of the present invention;

[0048] Figure 5 is a diagram of a spatial pyramid pooling network structure in an embodiment of the present invention; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0050] This embodiment proposes a three-channel graph neural network method for detecting facial fatigue state. The method is based on a multi-branch graph network detection model, which includes a detection module, an extraction module, an adaptive learning weight network module and a training module, wherein the detection module is used to detect feature point information of each area of ​​the face, and convert the coordinates of the feature points and the distances between each point into the node feature matrix and adjacency matrix required in the extraction module; the extraction module extracts node features of the corresponding area through a graph convolutional network, and learns the mapping relationship between node features and facial fatigue; the adaptive learning weight network module detects the degree of matching between the node and the area shown in the image through an adaptive learning weight network, adaptively learns the true weight of each branch feature, enhances the features of the unobstructed area of ​​the face, and reduces the influence of the feature points of the obstructed area on the overall feature; the training module is used for training data synthesis, and facial occlusions are manually added through synthesis.

[0051] Based on the above multi-branch graph network detection model, the face fatigue state detection method proposed in this embodiment is as follows: Figure 1 As shown, the steps are as follows:

[0052] Step 1: Using the detection module, extract the facial right eye feature point positioning information, left eye feature point positioning information and mouth feature point positioning information in the source image through the Dlib facial feature point detection technology, and convert them into corresponding graph node structure information, namely right eye graph node, left eye graph node and mouth graph node, and then normalize the converted graph node information and input it into the extraction module as the input data of the multi-branch graph network detection model; the feature points in this embodiment are the feature marker points extracted by the Dlib facial feature point detection algorithm in the face image, and the graph nodes are the logical points in the graph convolutional neural network; this step 1 is specifically as follows:

[0053] 1.1: Mark the position coordinates of the feature points in each region as N i =(x i ,y i ) and serves as the input information for each node;

[0054] 1.2: According to the results of the Dlib algorithm, the eye area morphology is marked by 6 feature points, and the mouth area morphology is marked by 20 feature points;

[0055] 1.3: Use the position coordinates of the 6 feature points in the eye area to build a 2×6 eye area node feature matrix, recorded as: X eye =(N1,N2,…,N6);

[0056] Use the position coordinates of the 20 feature points in the mouth area to build a 2×20 mouth area node feature matrix, recorded as: X mouth =(N1,N2,…,N 20);

[0057] 1.4: In order to capture the overall structure and characteristics of each regional graph node, the Euclidean distance D between each feature point is calculated by the feature point position coordinates i,j And build the corresponding adjacency matrix A. Because the distance from a node to itself is zero and the graph nodes are also undirected, the adjacency matrix A is a symmetric matrix.

[0058] The Euclidean distance calculation formula is:

[0059] The expression of the adjacency matrix A is:

[0060] Among them, x i and i Respectively represent the horizontal and vertical coordinates of node i; x j and j Respectively represent the horizontal and vertical coordinates of node j; D i,j represents the Euclidean distance between node i and node j; n represents the number of nodes in the adjacency matrix, D nn It means that the distance of the nth node itself is zero.

[0061] 1.5: In order to ensure that the input weight of each node in the graph convolution operation is relatively stable, the node feature matrix and adjacency matrix of each region are normalized and then input into the corresponding node feature extraction network in the extraction module.

[0062] Step 2: Through the extraction module in the multi-branch graph network detection model, the graph convolution network is used to extract the node features of the corresponding area according to the graph node structure information; in the fatigue state, there is a specific change in the position between the mouth feature points and the eye feature points. The change is that when yawning, the position of the mouth feature points will expand outward, and the eyelids may become more relaxed, causing the position of the eye feature points to be close; once the feature point position changes, the graph node feature matrix composed of the feature point position information will also have corresponding changes. Therefore, the graph network can learn facial fatigue features through position changes in the node feature matrix. At the same time, in the fatigue state, after the position of the eye feature points is close to the outer contour of the mouth feature points, the distance between the eye feature points and the adjacent eye feature points will become smaller, and the distance between the mouth feature points and the adjacent mouth feature points will also become larger. Therefore, the graph network can learn facial fatigue features through distance changes in the adjacency matrix. Among them, the multi-branch graph network detection model has three branch graph channels, and the extraction module is composed of three single-channel branch graph networks, such as Figure 2As shown, the node feature matrix and adjacency matrix of the graph nodes are input into the corresponding GCN network, and the two GCN layers of the corresponding graph convolutional network are used for feature extraction. After feature extraction, global pooling is connected to aggregate node-level information into global information to ensure that the outputs of the three branch GCNs are all fixed-size, global feature vectors.

[0063] In this embodiment, each branch channel passes through the graph convolution network to respectively process the left eye G eye1 =(X eye1 ,A eye1 ), right eye G eye2 =(X eye2 ,A eye2 ) and mouth G mouth =(X mouth ,A mouth ) area map for feature extraction, where X eye1 is the node feature matrix of the left eye area, A eye1 is the left eye adjacency matrix; X eye2 is the node feature matrix of the right eye region, A eye2 is the right eye adjacency matrix; X mouth is the feature matrix of the mouth area nodes, A mouth is the adjacency matrix of the mouth.

[0064] Graph convolution extracts facial fatigue features by extracting the node feature matrix and adjacency matrix of each node, extracting the position change of the node and the distance change between adjacent nodes. The calculation process formula of graph convolution feature extraction is:

[0065]

[0066] Among them, H l is the feature matrix of the l-th layer node, H l+1 is the feature of the l+1th layer, W (l) is the weight matrix of the lth layer, is the adjacency matrix with self-loops added, is the degree matrix, and σ is the activation function. .

[0067] Step 3: Adaptively learn the weights of each region through the adaptive learning weight network module; and multiply them with the corresponding regional features to obtain the right eye state features, left eye state features and mouth state features respectively; then concatenate the state features of the three regions, and regress the fatigue probability P through the multi-layer perception mechanism (fully connected layer), where the P value range is [0,1]. When it is greater than 0.5, it is considered to be a fatigue state; otherwise, it is a normal state;

[0068] In this embodiment, the adaptive learning weight network includes a spatial attention module, such as Figure 4As shown in the figure, the spatial attention module is used to concatenate and convolve the results of the maximum pooling and average pooling of the image features, then calculate the attention value of the image target area through the Sigmoid function, and finally strengthen the image features by multiplication.

[0069] like Figure 3 As shown, step 3 is specifically as follows:

[0070] 3.1: Input the image features extracted by the graph convolutional network into the adaptive learning weight network module;

[0071] 3.2: Using the spatial attention module SAM technology, the attention value of the target area of ​​the image is calculated to strengthen the features of the area of ​​interest;

[0072] 3.3: Unify the feature image size by adopting the spatial pyramid pooling structure SPP;

[0073] 3.4: At the same time, the node feature matrix constructed by each branch is flattened and normalized, and then expanded by a linear layer to make the dimension consistent with (C×21). Finally, the output of the node feature is element-wise added to the output of the image feature to achieve the fusion and splicing of the node feature and the image feature. Then, a value of (0, 1) is obtained through softmax as the corresponding branch weight.

[0074] 3.5: Multiply the three regional features by the corresponding branch weights and then concatenate them, and regress the probability detection result P through the multi-layer perceptron; where the P value range is [0,1]. When it is greater than 0.5, it is considered to be a fatigue state, otherwise it is a normal state.

[0075] In this embodiment, the eye area and mouth area images are captured by the outermost coordinate position of the feature point in the image, and are used as the main input of the adaptive learning weight network (ALW-Net); in the graph convolution network of this embodiment, the eye contour and mouth contour features are extracted through 3×3 convolution and Relu activation function; the number of convolution channels is 32, and the step size is 1.

[0076] In addition, considering that the sizes of the captured area images are different in different cases, a spatial pyramid pooling structure is used to unify the size of the feature map, such as Figure 5 As shown in FIG. 1 , the spatial pyramid pooling structure SPP uses three pooling layers of 1×1, 2×2, and 4×4 to calculate image features, and outputs a fixed-size feature vector (C×21) after splicing.

[0077] In this embodiment, training is performed through a training module of a multi-branch graph network detection model, and the specific method is as follows:

[0078] 1) Collect personal driving data sets in different driving scenarios, different perspectives, different times and different countries. This embodiment collects three public data sets: YAWDD, NTHU-DDD and Drozy. Among them, the YAWDD data set as a whole contains drivers of multiple different races and drivers wearing glasses, sunglasses and no glasses. Its data records driving scenes such as normal driving, yawning, slow blinking and falling asleep under various lighting conditions; the NTHU-DDD data set and the Drozy data set cover personal driving record videos in different scenarios, different perspectives, different times and different countries, totaling about 460 video data, and each data set contains a large amount of face occlusion data, such as colored glasses and masks.

[0079] 2) Randomly extract discontinuous images to record the corresponding fatigue state. In this embodiment, 100 discontinuous frames of images are randomly extracted from each video and the corresponding fatigue state is recorded, with a total of about 46,000 data.

[0080] 3) At the same time, in order to further enhance the anti-occlusion ability of the model, 5,000 images were randomly selected from the unobstructed images in the above dataset, and facial occluders such as sunglasses and masks were manually added through synthesis.

[0081] In this embodiment, the data set is divided into a training set, a validation set, and a test set in an 8:1:1 manner, and the size is uniformly adjusted to 256×256×3. After the data set is normalized, it is input into the multi-branch graph network detection model.

[0082] The facial fatigue state detection method of this embodiment has an accuracy rate of more than 97%, and the single-frame detection time is only 0.05 seconds, which not only ensures the detection accuracy but also meets the real-time requirements of actual driving detection.

[0083] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A three-channel graph neural network method for detecting facial fatigue state, characterized in that: Based on the multi-branch graph network detection model, the following steps are included: S1: After the feature point information extracted from each facial region is converted into graph node structure information, it is input into the multi-branch graph network detection model; S2: In the multi-branch graph network detection model, according to the graph node structure information, the node features of the corresponding area are extracted through the graph convolution network; And learn the mapping relationship between node features and facial fatigue; S3: Adaptively learn the weights of each region through the adaptive learning weight network; multiply it with the corresponding regional features, then concatenate the three regional features, and regress the probability detection result P through the multi-layer perception mechanism.

2. According to the three-channel graph neural network face fatigue state detection method of claim 1, the multi-branch graph network detection model comprises: Detection module: used to detect feature points in each area of ​​the face image, and convert the position coordinates of the feature points and the distances between adjacent feature points into the extraction module. Construct the nodes and the relationships between the nodes required for the graph convolutional network, corresponding to the node feature matrix and the adjacency matrix respectively; Extraction module: extract node features of the corresponding area through graph convolutional network, and learn the mapping relationship between node features and facial fatigue; Adaptive learning weight network module: Through the adaptive learning weight network, the matching degree between the detection node and the area shown in the image is detected, and the real weight of each branch feature is adaptively learned to enhance the features of the unobstructed area of ​​the face, while reducing the influence of the features of the feature points in the obstructed area on the overall features; Training module: used for training data synthesis, manually adding facial occluders through synthesis.

3. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 1, characterized in that: The multi-branch graph network detection model is used to synthesize training data through a training module. The training method is as follows: personal driving data sets of different driving scenes, different perspectives, different times and different countries are collected, and discontinuous images are randomly extracted to record corresponding fatigue states. At the same time, images are randomly selected from unobstructed images, and facial occluders are manually added by synthesis; the data set is divided into a training set, a validation set and a test set in an 8:1:1 manner, and the size is uniformly adjusted to 256×256×3. After the data set is normalized, it is input into the multi-branch graph network detection model.

4. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 1, characterized in that: In step S1, the feature point information of the left eye, right eye and mouth area of ​​the face in the image is extracted by Dlib face feature point detection technology, and the feature point information of the left eye, right eye and mouth is converted into graph node structure information. The specific steps are as follows: 1.1: Mark the position coordinates of the feature points in each region as N i =(x i ,y i ), and used as input information for each graph node; the feature points are feature marker points extracted from the face image by the Dlib face feature point detection algorithm, and the graph nodes are logical points in the graph convolutional neural network; 1.2: According to the results of the Dlib algorithm, the eye area morphology is marked by 6 feature points, and the mouth area morphology is marked by 20 feature points; 1.3: Use the position coordinates of the 6 feature points in the eye area to build a 2×6 eye area node feature matrix, recorded as: X eye =(N1,N2,…,N6); Use the position coordinates of the 20 feature points in the mouth area to build a 2×20 mouth area node feature matrix, recorded as: X mouth =(N1,N2,…,N 20 ); 1.4: Calculate the Euclidean distance D between adjacent feature points through the position coordinates of adjacent feature points i,j Construct a corresponding adjacency matrix A, wherein the adjacency matrix A is a symmetric matrix; The Euclidean distance calculation formula is: The expression of the adjacency matrix A is: Among them, x i and i Respectively represent the horizontal and vertical coordinates of node i; x j and j Respectively represent the horizontal and vertical coordinates of node j; D i,j represents the Euclidean distance between node i and node j; n represents the number of nodes in the adjacency matrix, D nn It represents the distance of the nth node itself to be zero; 1.5: After normalizing the node feature matrix and adjacency matrix of each region, input them into the corresponding node feature extraction network.

5. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 1, characterized in that: In step S2, the multi-branch graph network detection model has three branch graph channels, each of which passes through a graph convolutional network to detect the left eye G and eye1 =(X eye1 ,A eye1 ), right eye G eye2 =(X eye2 ,A eye2 ) and mouth G mouth =(X mouth ,A mouth ) neural network graph for feature extraction, the calculation formula is: Among them, H l is the feature matrix of the nodes in layer I, W (l) is the weight matrix of layer I, is the adjacency matrix with self-loops added, is the degree matrix, σ is the activation function, H l+1 is the feature of the l+1th layer.

6. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 5, characterized in that: The graph convolutional network performs feature extraction through two GCN layers, and then accesses global pooling to aggregate node-level information into global information, ensuring that the outputs of the three-branch GCN are global feature vectors of fixed size.

7. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 1, characterized in that: The adaptive learning weight network is used to adaptively learn the matching degree between the detection node and the area shown in the image, so as to learn the proportional weight of the corresponding branch feature in the total feature; the adaptive learning weight network includes a spatial attention module, which is used to splice and convolve the results of the maximum pooling and average pooling of the image features, and then calculate the attention value of the image target area through the Sigmoid function, and finally strengthen the image features by multiplication.

8. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 7, characterized in that: Step S3 is specifically as follows: 3.1: Inputting the image features extracted by the graph convolutional network into the adaptive learning weight network; 3.2: Using the spatial attention module SAM technology, the attention value of the target area of ​​the image is calculated to strengthen the features of the area of ​​interest; 3.3: Unify the feature image size by adopting the spatial pyramid pooling structure SPP; 3.4: After flattening, normalizing and linearly transforming the node feature matrix, it is concatenated with the image feature matrix and a value of (0, 1) is obtained through softmax as the corresponding branch weight; 3.5: The three regional features are multiplied by the weights of each branch and then spliced, and the probability detection result P is regressed through a multi-layer perceptron; the range of the P value is [0,1], and when the P value is greater than 0.5, it is considered to be a fatigue state, otherwise it is a normal state.

9. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 8, characterized in that: In the graph convolutional network, eye contour features are extracted through 3×3 convolution and Relu activation function; the number of convolution channels is 32 and the step size is 1.

10. The method for detecting facial fatigue state using a three-channel graph neural network according to claim 1, characterized in that: The spatial pyramid pooling structure SPP uses three pooling layers of 1×1, 2×2, and 4×4 to calculate image features, and outputs a feature vector of a fixed size after splicing.

Images