Mining driver fatigue detection method based on SinGAN and capsule network

Abstract

The mining driver fatigue detection method based on the SinGAN and the capsule network comprises the following steps that 11, a mining vehicle-mounted terminal is directly connected with a network camera, and the network camera is installed in a cab of a mining rubber-tyred vehicle and right faces a driver; 12, the network camera shoots the face state of the driver in the driving process in real time and sends a face state image to the mining vehicle-mounted terminal in real time; 13, the mining vehicle-mounted terminal judges whether the driver is in a fatigue state or not in real time by using a capsule network pre-trained through a data set generated by the improved SinGAN; when fatigue driving of the driver is detected, a sound prompt is given out. The mining driver fatigue detection method based on the SinGAN and the capsule network has remarkable advantages in the aspects of improving detection accuracy, enhancing robustness and adaptability, realizing real-time detection, non-invasiveness, usability and the like.

Description

Technical Field

[0001] The invention relates to the technical field of SinGAN data enhancement and capsule network image classification, and in particular to a mining driver fatigue detection method based on SinGAN and capsule network. Background Art

[0002] SinGAN data augmentation technology is an innovative image processing method that uses the multi-scale architecture of generative adversarial networks (GANs) to extract details from a single image and generate high-quality, diverse images. This technology is particularly suitable for data augmentation scenarios and can generate more realistic and varied samples in computer vision and deep learning training. SinGAN can generate high-resolution and diverse images by training single pixel-level data without relying on a large number of training samples. The design of its multi-scale generator and discriminator enables the model to learn and capture detailed information of images at multiple resolution levels, thereby generating realistic image data. In the field of data augmentation, SinGAN not only increases the diversity of samples, but also retains the realism of the input image, which helps to improve the generalization ability and recognition accuracy of deep learning models.

[0003] Capsule Networks is an emerging neural network model that shows unique advantages in the field of image classification. Compared with traditional convolutional neural networks (CNNs), capsule networks can capture the posture and spatial relationship information of objects in images by introducing the concept of "capsules", thereby improving the accuracy of image classification. The capsules in the capsule network contain not only existence information, but also specific attributes of objects, such as position, size, and direction. This design makes the capsule network more robust when dealing with complex scenes or overlapping objects, and can more accurately identify objects in images. In addition, the capsule network also uses a dynamic routing algorithm to enable capsules at different levels to collaborate with each other to complete the task of classifying images. This feature makes the capsule network have higher potential and application value in the field of image classification.

[0004] Since facial fatigue data generated when the driver is driving fatigued is a low-probability event, it is difficult to obtain facial fatigue data. The number of fatigue data obtained by obtaining the driver's facial fatigue status in real time through a network camera is small, and the number of non-fatigue data is large, resulting in a large gap between positive and negative samples, making the training data set an unbalanced data set. When processing such an unbalanced data set, minimizing the empirical risk as the training goal will make the classification model biased towards the non-fatigue class, and the recognition performance of a few fatigue classes is poor. Therefore, sufficient training data with balanced category distribution is the prerequisite for ensuring the effective operation of the capsule network algorithm. In response to the problems caused by the above-mentioned unbalanced data, traditional solutions use two different solutions: undersampling and oversampling. However, the undersampling method will cause some information loss, resulting in poor model performance fitting effect. The oversampling method often does not take into account the characteristics of the overall distribution of the model, so the improvement of model performance is limited. There is also a simple method to expand the data set through affine transformation to improve the model data performance. However, this method only changes the angle, size and other properties of the fatigue data, the sample expansion category is limited, and the model is prone to overfitting. Generative adversarial network (GAN) is a new deep network model developed in recent years. It can obtain the distribution law of data samples and generate new artificial samples through unsupervised learning. It is mostly used for artificial data enhancement of unbalanced data sets, but it has the problems of difficult training process, too long training time and low quality of generated data for fatigue data sets. In view of the shortcomings of enhancing fatigue data sets by undersampling, oversampling and affine transformation, and the influence of unbalanced data sets generated by using general data sets in fatigue recognition models, the traditional GAN ​​training process is difficult, too long training time and low quality of generated data.

[0005] The fatigue detection method for mining rubber-tyred truck drivers generally uses the traditional convolutional neural network training method, which requires a large number of fatigue and non-fatigue data sets. It is difficult to obtain driver facial fatigue and non-fatigue data. If an insufficient number of training data sets are used, the traditional convolutional neural network model will be overfitted. In addition, the traditional convolutional neural network has poor spatial recognition of fatigue data sets. If the data set is greatly rotated or flipped due to the shooting angle when acquiring the data, the model detection ability will be greatly reduced. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0007] To this end, the present invention proposes a mining driver fatigue detection method based on SinGAN and capsule network, which has significant advantages in improving detection accuracy, enhancing robustness and adaptability, realizing real-time detection, as well as non-invasiveness and ease of use.

[0008] A mining driver fatigue detection method based on SinGAN and capsule network according to an embodiment of the present invention comprises the following steps:

[0009] Step 11: The mining vehicle terminal is directly connected to the network camera, and the network camera is installed in the driving cabin of the mining rubber-wheeled vehicle and faces the driver;

[0010] Step 12: The network camera takes real-time photos of the driver's facial state while driving, and sends the facial state image to the mining vehicle terminal in real time;

[0011] Step 13: The mining vehicle terminal uses the capsule network pre-trained by the data set generated by the improved SinGAN to determine in real time whether the driver is in a fatigue state; when the driver's fatigue driving is detected, a sound prompt is issued; when the driver's fatigue driving is not detected, it returns to step 12.

[0012] The beneficial effects of the present invention include the following advantages:

[0013] 1. Improve detection accuracy: ① The SinGAN network is good at learning rich image statistics from a single image and generating high-quality image samples. In driver fatigue detection, by improving SinGAN, it can more accurately capture subtle changes in the driver's face, such as blurred eyes and increased blinking frequency, which are important indicators for judging fatigue status; ② The Capsule Network encapsulates low-level features into "capsules" to retain more spatial and directional information, allowing the model to more accurately understand the complex relationships in the image. In driver fatigue detection, this feature helps to more accurately identify the driver's head posture, eye status, etc., thereby improving detection accuracy;

[0014] 2. Enhanced robustness and adaptability: ① The working environment of rubber-wheeled mining vehicles is usually harsh. Problems such as unstable lighting conditions and flying dust may have a significant impact on the quality of images collected by the camera. By improving the SinGAN network, the model can be more robust to interference factors such as lighting changes and image blur, thereby maintaining high detection performance in various environments; ② At the same time, the dynamic routing mechanism of the capsule network enables the model to adaptively adjust the weights between different features to cope with individual differences and different driving states of different drivers, further enhancing the adaptability of the model;

[0015] 3. Real-time detection: In mining environments, real-time detection of driver fatigue is crucial. Based on the improved SinGAN and capsule network detection methods, by optimizing the algorithm structure and reducing the computational complexity, rapid response and real-time detection can be achieved while ensuring detection accuracy, ensuring that drivers can be reminded in time when signs of fatigue appear;

[0016] 4. Non-intrusiveness and ease of use: This detection method is non-contact detection, and does not require any sensors or equipment to be installed on the driver, and will not cause any discomfort or interference to the driver. At the same time, this method processes and analyzes images collected by the camera, and has the advantages of being easy to deploy and use.

[0017] 5. Data-driven and optimizable: With the development of big data and deep learning technology, the detection method based on improved SinGAN and capsule network can continuously improve its performance through training and optimization. By collecting more driver fatigue data and using it for model training, the model can more accurately identify various fatigue states and adapt to different driving scenarios and conditions.

[0018] These advantages make this method have broad application prospects and important practical value in the field of fatigue detection of mining rubber-tyred truck drivers.

[0019] According to one embodiment of the present invention, the improved SinGAN model is obtained by combining the SinGAN model, the affine transformation unit, and the size transformation unit. The specific improvement process is as follows:

[0020] After SinGAN generates a large amount of deformation data, the output results are input into the affine transformation unit to achieve the angle transformation of the generated data, and the data is input into the size transformation unit to achieve the size transformation of the data. Finally, a large amount of noise interference is filtered out through the non-local average algorithm, and finally an original data set with a fitting degree suitable for the capsule network model is generated.

[0021] According to one embodiment of the present invention, the affine transformation is set to 45 degree, 135 degree, 225 degree, 315 degree rotation, translation in four dimensions of up, down, left and right, vertical flip in four dimensions of up, down, left and right, and horizontal flip in four dimensions of up, down, left and right.

[0022] According to an embodiment of the present invention, the size transformation is set to enlarge the original data by 1.3 times, 1.6 times, 2 times, reduce it by 1.3 times, reduce it by 1.6 times, and reduce it by 2 times.

[0023] According to one embodiment of the present invention, the improved SinGAN back propagation training process is as follows:

[0024] Step 21: First, for the coarsest precision GAN NFor training, the input is a coarse precision noise, through the generator G N Then generate the generated data and send it to the discriminator D N Among them, GAN N represents the Nth generative adversarial network; G N represents the generator in the Nth generative adversarial network; D N Represents the discriminator in the Nth generative adversarial network;

[0025] Step 22: Then, do not fix GAN N G N Parameters, using G N The generated data features are input into the next precision GAN N-1 Training is performed, and G is trained at the same time N , G N-1 , D N-1 Among them, GAN N-1 represents the N-1th generative adversarial network; G N-1 represents the generator in the N-1th generative adversarial network; D N-1 Represents the discriminator in the N-1th generative adversarial network;

[0026] Step 23, and so on to the fourth precision GAN N-3 , the learning rate in the process is reduced by one tenth respectively; among them, GAN N-3 represents the N-3th generative adversarial network;

[0027] Step 24: From the fifth precision GAN N-4 Start, fix G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 Among them, GAN N-4 represents the N-4th generative adversarial network; G N-2 represents the generator in the N-2th generative adversarial network; G N-3 represents the generator in the N-3th generative adversarial network; G N-4 represents the generator in the N-4th generative adversarial network;

[0028] Step 25, and so on. In the subsequent training process, each time the accuracy increases by a level, the front-end one of the trained generators is fixed, and the subsequent generators participate in the training until the accuracy meets the same size as the original input data, and the entire training process is stopped.

[0029] According to one embodiment of the present invention, the Gaussian noise z is N Through the generator GN Mapping Generates Data Images or data features As shown in formula (1) and formula (2):

[0030]

[0031]

[0032] The symbols in formula (1) and formula (2) mean:

[0033] z N Represents the input noise of this layer;

[0034] If the data image is generated Then its input goes into the discriminator D N ;

[0035] If the data characteristics are generated According to formula (3), the Gaussian noise of the next precision layer is combined with the Gaussian noise of the next precision layer and then enters the generator of the next precision layer:

[0036] y N-1 =δ N (z N +(y N+1 )↑ N+1 ) (3)

[0037] The symbols in formula (3) mean:

[0038] y N+1 Represents the lower layer generator G N+1 Output data characteristics;

[0039] ↑ N+1 Indicates the reconstruction process from layer N+1 to layer N;

[0040] δ N It represents the 4-layer convolution operation of the improved main capsule layer;

[0041] y N-1 Represents the upper generator input.

[0042] According to one embodiment of the present invention, the total training loss of the Nth generator of the improved SinGAN includes an overall loss function of a generator loss and a reconstruction loss, as shown in formula (4):

[0043] LOSS G =LOSS(D,G)+LOSS rec (G N ) (4)

[0044] The symbols in formula (4) mean:

[0045] LOSS G Represents the total training loss of the generator;

[0046] LOSS(D,G) represents the generator training loss;

[0047] LOSS rec (G N ) represents the reconstruction training loss.

[0048] According to one embodiment of the present invention, the reconstruction loss is as shown in formula (6):

[0049]

[0050] The symbols in formula (6) mean:

[0051] LOSS rec is the reconstruction loss function;

[0052] G N is the generation model at the current accuracy;

[0053] y0 is the data feature at the current accuracy;

[0054] x N is the original image at the current precision.

[0055] According to one embodiment of the present invention, the training loss of the Nth discriminator of SinGAN is improved, as shown in formula (7):

[0056]

[0057] The symbols in formula (7) mean:

[0058] LOSS G is the discriminator loss;

[0059] z is the Gaussian noise input;

[0060] I is the total number of discriminations.

[0061] According to one embodiment of the present invention, both the generator training and the discriminator training use a gradient ascent algorithm to update parameters, and the algorithm is shown in formula (8):

[0062]

[0063] The symbols in formula (8) mean:

[0064] θ t Indicates the current value of the parameter;

[0065] θt+1 Indicates the updated value of the parameter;

[0066] α represents the learning rate (step size), which controls the magnitude of the update;

[0067] Denotes the objective function f(θ t ) in the parameter θ t The gradient at .

[0068] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 This is a schematic diagram of the hardware architecture of fatigue detection for a driver of a mining rubber-tyred vehicle according to the present invention;

[0072] Figure 2 Schematic diagram of the improved SinGAN model of the present invention;

[0073] Figure 3 Schematic diagram of the improved SinGAN back-propagation training process of the present invention;

[0074] Figure 4 It is a schematic diagram of the single-precision generator structure of the improved SinGAN of the present invention;

[0075] Figure 5 is a schematic diagram of a deep feature extraction unit of the present invention;

[0076] Figure 6 is a schematic diagram of the improved main capsule layer of the present invention;

[0077] Figure 7 It is a schematic diagram of the improved dynamic routing structure of the present invention;

[0078] Figure 8 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The following describes in detail a mining driver fatigue detection method based on SinGAN and capsule network according to an embodiment of the present invention with reference to the accompanying drawings.

[0081] The present invention proposes a generative adversarial network based on an improved SinGAN, which improves its data generation quality by improving its structure, shortens the training time, and completes the enhancement of the data set under the condition that the number of fatigue data sets is too small. It can effectively solve the detection problem of the fatigue detection model for the unbalanced sample data set, improve its data generation quality, shorten the model training time, and can be used as an important data preprocessing link of the fatigue detection method for the driver of the rubber-wheeled mining truck. At the same time, the present invention uses a capsule network structure to adapt to the built-in network camera of the rubber-wheeled mining truck at different shooting angles and different shooting environments, and the data set is a small sample of the fatigue detection of the driver of the rubber-wheeled mining truck, which effectively solves the problems of poor spatial recognition and excessive training sample requirements of the traditional convolutional neural network algorithm, improves the average accuracy of the fatigue detection of the driver of the rubber-wheeled mining truck, and enables the data sets collected by the network camera under different external environments to be used in the capsule network structure, improves the efficiency of the use of the data collected by the network camera, and realizes efficient automated fatigue detection of the driver of the rubber-wheeled mining truck.

[0082] See Figure 8 The present invention provides a method for detecting fatigue of mining drivers based on SinGAN and capsule network, comprising the following steps:

[0083] Step 11: Hardware architecture for detecting fatigue of drivers of mining rubber-tyred vehicles, i.e., the connection of hardware devices, such as Figure 1 As shown, the mining vehicle terminal is directly connected to the network camera, and the network camera is installed in the driving cabin of the mining rubber-wheeled vehicle and faces the driver;

[0084] Step 12: The network camera takes real-time photos of the driver's facial state while driving, and sends the facial state image to the mining vehicle terminal in real time;

[0085] Step 13: The mining vehicle terminal uses the capsule network pre-trained by the data set generated by the improved SinGAN to determine in real time whether the driver is in a fatigued state; when the driver is detected to be driving fatigued, a sound prompt is issued.

[0086] As for the installation position of the network camera, if the installation position has deviations, different shooting angles during the sampling process will lead to different angles of fatigue data. If only the data of inherent angles are used when training the detection model, it will lead to overfitting of the model; different shooting distances during the camera sampling process will lead to different sizes of fatigue data. If only affine transformation of fatigue data is performed and the size factor is ignored, it will lead to overfitting of the model; during the camera sampling process, due to the influence of the driver's driving position and different appearances, the data will also be deformed to varying degrees. If only affine transformation and size transformation are used and the deformation factor of the data is ignored, it will also lead to overfitting of the model. Therefore, the SinGAN model that adapts to the deformation of fatigue data is combined with the affine transformation unit and the size transformation unit to solve the above problems. The improved SinGAN model is as follows: Figure 2 As shown. Figure 2 It can be seen that the improved SinGAN is obtained by combining the SinGAN model, the affine transformation unit, and the size transformation unit. The specific improvement process is as follows:

[0087] After SinGAN generates a large amount of deformation data (including a small amount of size transformation data), the output result is input into the affine transformation unit to achieve the angle transformation of the generated data, and the data is input into the size transformation unit to achieve the size transformation of the data. Finally, the non-local mean algorithm (Non-LocalMeans) is used to filter out the large amount of noise interference generated in the previous stages, and finally the original data set with a degree of fitting suitable for the capsule network model is generated. Among them, the affine transformation is set to 45 degrees, 135 degrees, 225 degrees, 315 degrees rotation, translation in four dimensions of up, down, left, and right, vertical flip in four dimensions of up, down, left, and right, and horizontal flip in four dimensions of up, down, left, and right. The size transformation is set to enlarge the original data by 1.3 times, 1.6 times, 2 times, reduce by 1.3 times, reduce by 1.6 times, and reduce by 2 times.

[0088] Depend on Figure 2It can be seen that the traditional training method fixes the parameters of each GAN after the training is completed, and then sends the generated sample data to the next layer. This serial operation method will result in a small amount of data running in a single CPU storage unit, which cannot fully exert the performance of the CPU. In addition, only one level of GAN can be trained in a single given time, resulting in greater training difficulty, long training time and low quality of generated samples. Since the fatigue detection and classification model of mining rubber-wheeled vehicle drivers mostly uses small-resolution data sets, mainly 224*224*3 image data, the main body of the data is the driver and the vehicle cab background, and the data background complexity is not high, the overly complex training method in the traditional SinGAN is improved. In response to the above problems, the data samples generated by the generator are replaced with data features, that is, no images are generated, and the GAN that can only train one level at a time is replaced by batch training with 4 levels as a unit to improve the CPU utilization efficiency and shorten the training time. The specific training process diagram is shown below. Figure 3 As shown. Figure 3 It can be seen that the improved SinGAN back propagation training process is as follows:

[0089] Step 21: First, for the coarsest precision GAN N For training, the input is a coarse precision noise, through the generator G N Then generate the generated data and send it to the discriminator D N , then the generator G N The learning rate of GAN is 0.1; N represents the Nth generative adversarial network; G N represents the generator in the Nth generative adversarial network; D N Represents the discriminator in the Nth generative adversarial network;

[0090] Step 22: Then, do not fix GAN N G N Parameters, using G N The generated data features (non-image data) are input into the next precision GAN N-1 Training is performed, and G is trained at the same time N , G N-1 , D N-1 , at this time G N The learning rate is 0.01, G N-1 The learning rate of GAN is 0.1; N-1 represents the N-1th generative adversarial network; G N-1 represents the generator in the N-1th generative adversarial network; D N-1 Represents the discriminator in the N-1th generative adversarial network;

[0091] Step 23, and so on to the fourth precision GAN N-3 , the learning rate in the process is reduced by one tenth respectively; among them, GAN N-3 represents the N-3th generative adversarial network; and so on, the content included is that the GAN is not fixed N-1 G N-1 Parameters, using G N-1 The generated data features (non-image data) are input into the next precision GAN N-2 Training is performed, and G is trained at the same time N-1 , G N-2 , D N-2 , at this time G N-1 The learning rate is 0.001, G N-2 The learning rate is 0.01; GAN is not fixed N-2 G N-2 Parameters, using G N-2 The generated data features (non-image data) are input into the next precision GAN N-3 Training is performed, and G is trained at the same time N-2 , G N-3 , D, at this time G N-2 The learning rate is 0.0001, G N-3 The learning rate is 0.001; GAN is not fixed N-3 G N-3 Parameters, using G N-3 The generated data features (non-image data) are input into the next precision GAN N-4 Training is performed, and G is trained at the same time N-3 , G N-4 , D N-4 , at this time G N-3 The learning rate is 0.00001, G N-4 The learning rate is 0.0001;

[0092] Step 24: From the fifth precision GAN N-4 Start, fix G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 Among them, GAN N-4 represents the N-4th generative adversarial network; G N-2 represents the generator in the N-2th generative adversarial network; G N-3 represents the generator in the N-3th generative adversarial network; G N-4 represents the generator in the N-4th generative adversarial network;

[0093] Step 25, and so on. In the subsequent training process, each time the accuracy increases by one level, the most advanced generator is fixed from the trained generators, and the subsequent generators participate in the training until the accuracy meets the original input data size. The entire training process is stopped, and the learning rate of the discriminator with different accuracies in the whole process is 0.01, the training speed is faster and the generated data effect is best. Among them, the content of this and so on is that the sixth precision GAN N-5 , fixed G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 , G N-5 ; The seventh precision GAN N-6 , fixed G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 , G N-5 , G N-6 ; Eighth precision GAN N-7 , fixed G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 , G N-5 , G N-6 , G N-7 ;etc.

[0094] The 5 convolutions in the single-precision generator structure of SinGAN are reduced to 4, and each convolution block is reduced to 16 kernels at the coarsest precision, and then the number of kernels is increased by 2 times every 4 scales. The improved structure is as follows Figure 4 As shown, the Nth generator G N The input is the Gaussian noise Z of the current level N And upsampled data, through the Nth generator G N , the output is the generator generated data containing noise and upsampled data.

[0095] At different precisions, the Gaussian noise z N Through the generator G N Mapping Generates Data Images or data features (depending on the generator being located at Figure 5 Which part of the equation is shown in formula (1) and formula (2):

[0096]

[0097]

[0098] The symbols in formula (1) and formula (2) mean:

[0099] z N Represents the input noise of this layer;

[0100] If the data image is generated Then its input goes into the discriminator D N ;

[0101] If the data characteristics are generated According to formula (3), the Gaussian noise of the next precision layer is combined with the Gaussian noise of the next precision layer and then enters the generator of the next precision layer:

[0102] y N-1 =δ N (z N +(y N+1 )↑ N+1 ) (3)

[0103] The symbols in formula (3) mean:

[0104] y N+1 Represents the lower layer generator G N+1 Output data characteristics;

[0105] ↑ N+1 Indicates the reconstruction process from layer N+1 to layer N;

[0106] δ N It represents the 4-layer convolution operation of the improved main capsule layer;

[0107] y N-1 Represents the upper generator input.

[0108] The total training loss of the Nth generator of the improved SinGAN includes an overall loss function of generator loss and reconstruction loss, as shown in formula (4):

[0109] LOSS G =LOSS(D,G)+LOSS rec (G N ) (4)

[0110] The symbols in formula (4) mean:

[0111] LOSS G Represents the total training loss of the generator;

[0112] LOSS(D,G) represents the generator training loss;

[0113] LOSS rec (G N ) represents the reconstruction training loss.

[0114] The total training loss of the Nth generator, the traditional training loss method, is shown in formula (5):

[0115]

[0116] The symbols in formula (5) mean:

[0117] LOSS(D,G) represents the generator training loss;

[0118] V(D,G) represents the loss function;

[0119] min and max represent the optimization process, trying to minimize the loss of the generator and maximize the loss of the discriminator;

[0120] D(y i ) represents the output of the discriminator, y i represents the discriminator input;

[0121] D(G(z i )) indicates that Gaussian input noise is first fed into the generator, and then the generated data is fed into the discriminator output;

[0122] G represents the generator;

[0123] D represents the discriminator;

[0124] m represents the number of generated data;

[0125] z represents Gaussian noise input;

[0126] y represents the discriminator input;

[0127] i represents the total number of generated items.

[0128] The total training loss of the Nth generator is shown in formula (6):

[0129]

[0130] The symbols in formula (6) mean:

[0131] LOSS rec is the reconstruction loss function;

[0132] G N is the generation model at the current accuracy;

[0133] y0 is the data feature at the current accuracy;

[0134] x N is the original image (Real) at the current precision.

[0135] Improve the training loss of the Nth discriminator of SinGAN, as shown in formula (7):

[0136]

[0137] The symbols in formula (7) mean:

[0138] LOSS G is the discriminator loss;

[0139] z is the Gaussian noise input;

[0140] I is the total number of discriminations.

[0141] The above generator training and discriminator training both use the gradient ascent algorithm to update parameters. The algorithm is shown in formula (8):

[0142]

[0143] The symbols in formula (8) mean:

[0144] θ t Indicates the current value of the parameter;

[0145] θ t+1 Indicates the updated value of the parameter;

[0146] α represents the learning rate (step size), which controls the magnitude of the update;

[0147] Denotes the objective function f(θ t ) in the parameter θ t The gradient at

[0148] Capsule network adaptive contribution pooling layer: The capsule network was first applied in the handwritten digit dataset, and its classification target was 10 categories. In order to be suitable for the binary classification of fatigue detection of mining rubber-wheeled vehicle drivers, the capsule network input was changed to 112*112*1 grayscale data, and a deep feature extraction unit was added. The deep feature extraction unit is as follows: Figure 5 As shown. Figure 5 It can be seen that for the 112*112*1 data, 128 112*112 feature maps are first obtained through 128 3*3 convolution kernels with a step size of 1, and then 128 56*56 feature maps are obtained through a 2*2 capsule network adaptive contribution pooling layer with a step size of 1. Then, 256 3*3 convolution kernels with a step size of 1 are obtained to obtain 256 56*56 feature maps. Finally, 256 28*28 deep feature extraction unit outputs are obtained through a 2*2 capsule network adaptive contribution pooling layer with a step size of 1. The activation function of the convolution layer uses ReLU.

[0149] Capsule network main capsule layer: The output of the deep feature extraction unit is first passed through 8 groups of 32 3*3 convolution kernels with a step size of 1 to obtain 8 groups of 32 26*26 feature maps. Secondly, it is passed through a capsule network adaptive contribution pooling layer to obtain 8 groups of 32 13*13 feature maps. Finally, the pooling results are flattened and then combined at corresponding positions to obtain 5408 capsules (a1, a2...a5408). Each capsule contains a 1*8 vector. The schematic diagram of the improved main capsule layer (capsule network main capsule layer) is shown below. Figure 6 As shown in the figure, adding adaptive contribution pooling can make the data set become 13*13, realizing deep feature data compression. By retaining the main feature information, the amount of data and training parameters are reduced, speeding up the model training process.

[0150] Improved dynamic routing structure: Since the final result of fatigue detection of mining rubber-wheeled vehicle drivers is binary classification, the 5408 capsules (a1, a2...a5408) generated by the main capsule layer are Figure 7 The parameters are updated by dynamic routing in a . The output of the main capsule layer is connected to the dynamic routing unit in a fully connected manner, with a total of 5408*2 weights, each of which is a 8*16 vector. The dynamic routing unit has two layers, each of which has 5408 calculation factors, equivalent to a high-level feature and corresponding to the final binary classification results. Figure 7 a represents the main capsule layer data, u represents the dynamic travel capsule data, V1 represents the fatigue probability value, and V2 represents the non-fatigue probability value.

[0151] In dynamic routing, the u+1 layer capsule is calculated from the u layer capsule. In the initial stage of dynamic routing, the u+1 layer capsule and the u layer capsule can be considered to be fully connected, and the probability of the u layer capsule connecting to the u+1 layer capsule is shown in formula (9):

[0152]

[0153] The symbols in formula (9) mean:

[0154] c ij It represents the probability of connecting the u-layer capsule to the u+1-layer capsule, and also represents the connection probability between capsule i and capsule j;

[0155] b ij Represents the prior probability of capsule i being connected to capsule j, and its value is initialized to 0;

[0156] The output capsule output of the final layer (v layer) of u is shown in formula (10):

[0157]

[0158] The symbols in formula (10) mean:

[0159] u(v j ) represents the output of capsule j at layer u+1;

[0160] s j represents u(v j )-layer capsule’s total input.

[0161] In the routing update phase, the connection probability c of capsule i and capsule j in the two layers is iteratively trained ij First, calculate the output prediction of capsule i in layer u to capsule j in layer u+1, and sum the products of all capsules in layer u and their corresponding connection probabilities as the input of capsule j in layer u to capsule j in layer u+1, as shown in formula (11) and formula (12):

[0162]

[0163]

[0164] The symbols in formula (11) and formula (12) mean:

[0165] Represents the set of output prediction probabilities of the u-layer capsule to the u+1-layer capsule;

[0166] W ij represents the routing matrix;

[0167] represents u(v j )The total input set of the layer capsule;

[0168] c ij Represents the connection probability of capsule i and capsule j in the iterative training of two layers.

[0169] Finally, the output prediction probability of the u-layer capsule for the u+1-layer capsule is set Update b together with the u+1 layer output vector ij , thereby updating the weight b ij , the update process is shown in formula (13):

[0170]

[0171] The symbols in formula (13) mean:

[0172] b ij represents the network weight;

[0173] v j Represents the probability value of each classification prediction result.

[0174] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A mining driver fatigue detection method based on SinGAN and capsule network, characterized in that: The following steps are involved: Step 11: The mining vehicle terminal is directly connected to the network camera, and the network camera is installed in the driving cabin of the mining rubber-wheeled vehicle and faces the driver; Step 12: The network camera takes real-time photos of the driver's facial state while driving, and sends the facial state image to the mining vehicle terminal in real time; Step 13: The mining vehicle terminal uses the capsule network pre-trained by the dataset generated by improving SinGAN to determine in real time whether the driver is in a fatigue state; When driver fatigue driving is detected, a sound prompt is issued; when driver fatigue driving is not detected, the process returns to step 12.

2. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 1 is characterized in that: The improved SinGAN model is obtained by combining the SinGAN model, the affine transformation unit, and the size transformation unit. The specific improvement process is as follows: After SinGAN generates a large amount of deformation data, the output results are input into the affine transformation unit to achieve the angle transformation of the generated data, and the data is input into the size transformation unit to achieve the size transformation of the data. Finally, a large amount of noise interference is filtered out through the non-local average algorithm, and finally an original data set with a fitting degree suitable for the capsule network model is generated.

3. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 2 is characterized in that: The affine transformation is set to 45 degree, 135 degree, 225 degree, 315 degree rotation, translation in four dimensions of up, down, left, and right, vertical flip in four dimensions of up, down, left, and right, and horizontal flip in four dimensions of up, down, left, and right.

4. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 2 is characterized in that: The size conversion is set to enlarge the original data by 1.3 times, 1.6 times, 2 times, reduce it by 1.3 times, reduce it by 1.6 times, and reduce it by 2 times.

5. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 1, characterized in that: The improved SinGAN back propagation training process is as follows: Step 21: First, for the coarsest precision GAN N For training, the input is a coarse precision noise, through the generator G N Then generate the generated data and send it to the discriminator D N Among them, GAN N represents the Nth generative adversarial network; G N represents the generator in the Nth generative adversarial network; D N Represents the discriminator in the Nth generative adversarial network; Step 22: Then, do not fix GAN N G N Parameters, using G N The generated data features are input into the next precision GAN N-1 Training is performed, and G is trained at the same time N , G N-1 , D N-1 Among them, GAN N-1 represents the N-1th generative adversarial network; G N-1 represents the generator in the N-1th generative adversarial network; D N-1 Represents the discriminator in the N-1th generative adversarial network; Step 23, and so on to the fourth precision GAN N-3 , the learning rate in the process is reduced by one tenth respectively; among them, GAN N-3 represents the N-3th generative adversarial network; Step 24: From the fifth precision GAN N-4 Start, fix G N Parameters of training G N-1 , G N-2 , G N-3 , G N-4 Among them, GAN N-4 represents the N-4th generative adversarial network; G N-2 represents the generator in the N-2th generative adversarial network; G N-3 represents the generator in the N-3th generative adversarial network; G N-4 represents the generator in the N-4th generative adversarial network; Step 25, and so on. In the subsequent training process, each time the accuracy increases by a level, the front-end one of the trained generators is fixed, and the subsequent generators participate in the training until the accuracy meets the same size as the original input data, and the entire training process is stopped.

6. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 5 is characterized in that: At different precisions, the Gaussian noise z N Through the generator G N Mapping Generates Data Images or data features As shown in formula (1) and formula (2): The symbols in formula (1) and formula (2) mean: z N Represents the input noise of this layer; If the data image is generated Then its input goes into the discriminator D N ; If the data characteristics are generated According to formula (3), the Gaussian noise of the next precision layer is combined with the Gaussian noise of the next precision layer and then enters the generator of the next precision layer: and N-1 =δ N (z N +(and N+1 )↑ N+1 ) (3) The symbols in formula (3) mean: y N+1 Represents the lower layer generator G N+1 Output data characteristics; ↑ N+1 Indicates the reconstruction process from layer N+1 to layer N; δ N It represents the 4-layer convolution operation of the improved main capsule layer; y N-1 Represents the upper generator input.

7. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 5, characterized in that: The total training loss of the Nth generator of the improved SinGAN includes an overall loss function of generator loss and reconstruction loss, as shown in formula (4): LOSS G =LOSS(D,G)+LOSS rec (G N ) (4) The symbols in formula (4) mean: LOSS G Represents the total training loss of the generator; LOSS(D,G) represents the generator training loss; LOSS rec (G N ) represents the reconstruction training loss.

8. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 7 is characterized in that: The reconstruction loss is shown in formula (6): The symbols in formula (6) mean: LOSS rec is the reconstruction loss function; G N is the generation model at the current accuracy; y0 is the data feature at the current accuracy; x N is the original image at the current precision.

9. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 8, characterized in that: Improve the training loss of the Nth discriminator of SinGAN, as shown in formula (7): The symbols in formula (7) mean: LOSS G is the discriminator loss; z is the Gaussian noise input; I is the total number of discriminations.

10. The method for detecting fatigue of mining drivers based on SinGAN and capsule network according to claim 9, characterized in that: Both generator training and discriminator training use the gradient ascent algorithm to update parameters. The algorithm is shown in formula (8): The symbols in formula (8) mean: θ t Indicates the current value of the parameter; θ t+1 Indicates the updated value of the parameter; α represents the learning rate (step size), which controls the magnitude of the update; Denotes the objective function f(θ t ) in the parameter θ t The gradient at .

Images