Neural network-based safe psychological stress recognition method and system

Abstract

The present application relates to the technical field of image recognition, and in particular to a safety psychological stress recognition method and system based on a neural network, comprising: collecting a first image and a second image; obtaining key point coordinates of the first image and the second image; obtaining a high-quality first image; determining an expression performance degree of the key points; using the expression performance degree of the key points to obtain an improved loss function of a discriminator; based on the improved loss function, obtaining an augmented image; using the augmented image and the high-quality first image as inputs, training an expression recognition neural network using a cross-entropy loss function to obtain a trained expression recognition neural network; inputting a third image into the trained expression recognition neural network to output a probability sequence of miner expressions; and based on the probability sequence of the miner expressions, determining a stress level of the miner. The present application can train an emotion recognition network with better recognition capability to obtain more accurate stress recognition results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a neural network-based safe psychological stress recognition method and system. BACKGROUND

[0002] The underground environment of a coal mine is special, with problems such as insufficient illumination, continuous noise, high temperature and humidity, dust accumulation, and harmful gas accumulation. The production process is complex, shift work is frequent, and the actual working time often exceeds the system working hours. These factors bring continuous work pressure to miners. Long-term high-pressure state can easily cause physical and mental discomfort and behavioral abnormalities, such as insomnia, anxiety, job burnout, and increased unsafe behavior, which not only affects the personal health of miners, but also threatens the safety production and overall operational efficiency of the coal mine. Therefore, timely and accurate recognition of the psychological stress of miners is an important link to ensure the safety of underground work and the health of personnel.

[0003] Existing methods usually collect miners' facial images and use neural networks to recognize stress. Since psychological stress is often indirectly reflected through facial expressions, such methods generally first perform expression recognition and then infer the stress level based on the expression, rather than directly recognizing the stress state.

[0004] Due to poor lighting conditions and low visibility in the underground environment of a coal mine, the contrast between the face region and the background in the collected images is weak, and the miners' facial skin color is dark, resulting in a limited number of high-quality image samples for various expressions. Therefore, existing methods usually use data augmentation techniques to enhance training data, i.e., augmenting a small number of high-quality expression images. Currently, generative adversarial networks (GAN) are commonly used for data augmentation, but traditional GANs do not consider the contribution difference of different facial regions to expression recognition during data augmentation, resulting in poor augmented images and affecting the performance of subsequent expression recognition models. SUMMARY

[0005] The present application provides a neural network-based safe psychological stress recognition method and system to solve the existing problems.

[0006] The neural network-based safe psychological stress recognition method and system of the present application adopts the following technical solutions:

[0007] One embodiment of the present application provides a neural network-based safe psychological stress recognition method, which includes the following steps:

[0008] Collecting first and second images; wherein the first image represents the face image of each miner in the underground environment of a coal mine, and the second image represents the neutral expression image of each miner when not entering the underground environment of a coal mine;

[0009] input the first image and the second image into a face key point detection model respectively to obtain key point coordinates of the first image and the second image;

[0010] obtain a high-quality first image based on the key point coordinates of the first image;

[0011] determine an expression performance degree of each key point based on the key point coordinates of the high-quality first image and the second image;

[0012] obtain an improved loss function of a cGAN network discriminator using the expression performance degree of each key point;

[0013] obtain an augmented image based on the improved loss function of the cGAN network discriminator;

[0014] train an expression recognition neural network using a cross-entropy loss function by taking the augmented image and the high-quality first image as inputs, to obtain a trained expression recognition neural network;

[0015] input a third image into the trained expression recognition neural network to output a probability sequence of various expressions of the miner; the third image represents a real-time collected face image of the miner;

[0016] determine a stress level of the miner based on the probability sequence of various expressions of the miner.

[0017] Further, obtaining a high-quality first image based on the key point coordinates of the first image includes the following specific steps:

[0018] position each key point set corresponding to each facial feature in each first image based on the key point coordinates of each first image;

[0019] construct a minimum circumscribed rectangle for each facial feature based on the key point set corresponding to each facial feature, and determine the minimum circumscribed rectangle as a region corresponding to each facial feature;

[0020] crop a region identical to the region corresponding to each facial feature from the first image, and determine each cropped region as a local image block of each facial feature;

[0021] extract a number of local image blocks from all local image blocks of each facial feature, and determine a minimum variance in Laplacian variance of each extracted local image block as a variance of the local image block of each facial feature;

[0022] determine a minimum variance in variances of all local image blocks of the five facial features as a variance threshold;

[0023] determine a first image in which the Laplacian variance of all local image blocks of the five facial features is greater than the variance threshold as a high-quality first image.

[0024] Further, the expression performance degree of each key point is determined based on the key point coordinates of the high-quality first image and the second image, and the specific steps include the following:

[0025] Based on the key point coordinates of the high-quality first image and the second image, a first face key point graph and a second face key point graph are constructed by using a triangulation method, wherein the first face key point graph represents the face key point graph of the high-quality first image, and the second face key point graph represents the face key point graph of the second image.

[0026] According to the first face key point graph and the second face key point graph, the density change relationship and the change direction value of each key point are determined.

[0027] The associated key point of each key point is obtained by using the density change relationship and the change direction value of each key point.

[0028] Based on the key point coordinates of the high-quality first image and the second image, the texture descriptor and the neutral texture descriptor of each key point are obtained.

[0029] The expression performance degree of each key point is obtained by using the texture descriptor, the neutral texture descriptor, and the associated key point of each key point.

[0030] Further, the expression performance degree of each key point is determined based on the key point coordinates of the high-quality first image and the second image, and the specific steps include the following:

[0031] For each key point in the first face key point graph, the number of nodes directly connected to the key point and the area surrounded by the nodes directly connected to the key point are obtained, and the ratio of the area to the number of nodes is determined as the density of each key point in the first face key point graph.

[0032] For each key point in the second face key point graph, the number of nodes directly connected to the key point and the area surrounded by the nodes directly connected to the key point are obtained, and the ratio of the area to the number of nodes is determined as the density of each key point in the second face key point graph.

[0033] For the same key point in the first face key point graph and the second face key point graph, the frequency that the density of the key point in the first face key point graph is greater than the density of the key point in the second face key point graph, and the frequency that the density of the key point in the first face key point graph is less than the density of the key point in the second face key point graph are counted, and the density relationship corresponding to the frequency greater than the preset frequency threshold in the two frequencies is determined as the density change relationship of each key point.

[0034] For each miner corresponding to the same key point in the high-quality first image and the second image, the motion direction of the key point from the position in the second image to the position in the high-quality first image is obtained, and the angle of the motion direction is determined as the direction value of each key point;

[0035] The frequency of each direction value of each key point is counted, and the direction value corresponding to the frequency greater than the preset frequency threshold is determined as the change direction value of each key point.

[0036] Further, the use of the density change relationship and the change direction value of each key point obtains the associated key point of each key point, which includes the following specific steps:

[0037] For any two key points in all key points, compare whether the density change relationship of the two key points is consistent;

[0038] In the case that the density change relationship of the two key points is consistent, the number of times that the change direction values of the two key points appear in the same high-quality first image is counted, and the sum value of the number of times that the change direction values of the two key points appear in the high-quality first image is calculated.

[0039] The ratio of the number of times that the change direction values of the two key points appear in the same high-quality first image to the sum value of the number of times that the change direction values of the two key points appear in the high-quality first image is determined as the association of the two key points.

[0040] When the association of the two key points is greater than the preset association threshold, the two key points are determined as the associated key points.

[0041] Based on the obtained all associated key points, the associated key points of each key point are determined.

[0042] Further, the use of the key point coordinates of the high-quality first image and the second image obtains the texture descriptor and the neutral texture descriptor of each key point, which includes the following specific steps:

[0043] For the high-quality first image, a fixed size image region is intercepted with each key point coordinate as the center, and the intercepted image region is determined as the corresponding region of each key point in the high-quality first image.

[0044] The LBP histogram of the corresponding region of each key point in the high-quality first image is calculated, the corresponding LBP feature vector is generated, and the LBP feature vector is determined as the texture descriptor of each key point.

[0045] For the second image, a fixed size image region is intercepted with each key point coordinate as the center, and the intercepted image region is determined as the corresponding region of each key point in the second image.

[0046] The LBP histogram of the region corresponding to each key point in the second image is calculated, a corresponding LBP feature vector is generated, and the LBP feature vector is determined as the neutral texture descriptor of each key point.

[0047] Further, the expression performance degree of each key point is obtained by using the texture descriptor, the neutral texture descriptor and the associated key point of each key point, and the specific steps include the following:

[0048] The cosine similarity between the texture descriptor of each key point and the neutral texture descriptor thereof is calculated for the same key point in the high-quality first image and the second image corresponding to each miner, and the cosine similarity is determined as the similarity of each key point;

[0049] The mean of all similarities of each key point is calculated, and the difference between 1 and the mean is determined as the self-performance degree of each key point;

[0050] The mean of the self-performance degrees of the associated key points of each key point is obtained;

[0051] The product of the ratio of the number of associated key points of each key point to 68 and the mean of the self-performance degrees of the associated key points of each key point is determined as the associated key point performance degree of each key point;

[0052] The mean of the self-performance degree of each key point and the associated key point performance degree of each key point is determined as the performance degree of each key point to the current expression;

[0053] The mean of the performance degrees of each key point to each expression is determined as the expression performance degree of each key point.

[0054] Further, the improved loss function of the cGAN network discriminator is obtained by using the expression performance degree of each key point, and the specific steps include the following:

[0055] The key point coordinates of the input face image of the cGAN network discriminator are obtained;

[0056] The influence range of each key point is determined by multiplying the expression performance degree of each key point by 20 and then taking the nearest positive integer;

[0057] The circular region with each key point coordinate as the center and the influence range of each key point as the radius is determined as the influence area of each key point; wherein the weight in the influence area of each key point is the expression performance degree of each key point;

[0058] The influence areas of all key points are superimposed to form a weight map with the same size as the input face image of the cGAN network discriminator;

[0059] obtain an improved loss function of the cGAN network discriminator based on the weight map;

[0060] The improved loss function of the cGAN network discriminator is calculated in the following manner: the square of the difference between the high-quality first image and the generated image is calculated pixel by pixel, then multiplied by the weight value at the corresponding position in the weight map, and finally the weighted results of all pixels are summed.

[0061] Further, the improved loss function of the cGAN network discriminator is used to obtain an augmented image, including the following specific steps:

[0062] identify and label the expression of each high-quality first image;

[0063] use the improved loss function of the cGAN network discriminator to train the cGAN network by taking the high-quality first image and its label as input, and obtain a trained cGAN network;

[0064] obtain an augmented image based on the trained cGAN network.

[0065] An embodiment of the present application provides a neural network-based safe psychological stress recognition system, which comprises the following modules:

[0066] The acquisition module is used to acquire first images and second images, wherein the first images represent the face images of each miner in the coal mine, and the second images represent the neutral expression images of each miner when not entering the coal mine;

[0067] The analysis module is used to input the first images and the second images into a face key point detection model respectively to obtain the key point coordinates of the first images and the second images, obtain high-quality first images based on the key point coordinates of the first images, determine the expression performance of each key point based on the key point coordinates of the high-quality first images and the second images, obtain an improved loss function of the cGAN network discriminator by using the expression performance of each key point, and obtain an augmented image based on the improved loss function of the cGAN network discriminator;

[0068] The recognition module is used to train an expression recognition neural network by taking the augmented image and the high-quality first image as input and using a cross-entropy loss function, obtain a trained expression recognition neural network, input a third image into the trained expression recognition neural network to output a probability sequence of various expressions of the miner, wherein the third image represents a real-time acquired face image of the miner, and determine the stress level of the miner based on the probability sequence of various expressions of the miner.

[0069] The beneficial effects of the technical solutions of the present application are: the safety psychological pressure recognition method and system based on a neural network are provided in the embodiments of the present application, high-quality data is obtained by screening the collected face data, the high-quality data is analyzed to obtain the expression performance degree of each key point, the loss function of the discriminator in the existing cGAN network is improved based on the expression performance degree of each key point, greater loss function weight is given to these high-discrimination key points, and then the expansion image containing more features that are helpful for expression recognition is obtained, that is, the quality of the expanded face data is higher, so that a mood recognition network with better recognition ability is trained to obtain more accurate pressure recognition results. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0071] Figure 1 The step flow chart of the safety psychological pressure recognition method based on a neural network of the present application is shown in the figure.

[0072] Figure 2 The cGAN network architecture diagram of the present application is shown in the figure.

[0073] Figure 3 The block diagram of the safety psychological pressure recognition system based on a neural network of the present application is shown in the figure. DETAILED DESCRIPTION

[0074] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the safety psychological pressure recognition method and system based on a neural network according to the present application, its specific implementation, structure, features and effects are described in detail as follows by combining the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0076] The specific scheme of the safety psychological pressure recognition method and system based on a neural network provided by the present application is specifically described below in combination with the drawings.

[0077] The present application collects the face data of the miners in the coal mine by the installed camera, then selects the data with better quality in different expressions from the face data, and the data is expanded by the cGAN network to obtain the training data, and the expression recognition network is trained to obtain the mapping relationship between the expression data and the stress degree. For the actual expression data of the miners, the expression recognition neural network is input to obtain the probability of different expression categories, and then the stress degree of the miners is obtained according to the mapping relationship.

[0078] The present application improves the loss function of the discriminator, so that the images retained by the discriminator contain more features that help to recognize expressions, thereby improving the usability of the expanded data.

[0079] Please refer to Figure 1 , which shows the step flow chart of the safety psychological stress recognition method based on the neural network provided by an embodiment of the present application, and the method comprises the following steps:

[0080] Step S001: collecting first images and second images; wherein the first images represent the face images of the miners in the coal mine, and the second images represent the neutral expression images of the miners when they do not enter the coal mine.

[0081] It should be noted that: the neutral expression image represents that the miner's heart is in a calm and relaxed state at the moment, and the miner's face shows no obvious emotional expression characteristics.

[0082] Specifically, in the key parts of the working area in the coal mine, such as the front side of the tunneling mechanical operation table, the inside and entrance of the loading and transportation vehicle cab, and the surrounding of the engineering vehicle operation room, install the intrinsically safe or explosion-proof high-resolution camera. These cameras are responsible for capturing the facial expressions of the miners when operating equipment, shift change, and short rest from the third person's perspective, especially capturing their natural expressions in the state of concentrating on work, thereby collecting a plurality of first images. In a quiet environment on the ground, guide the miners to be in a relaxed state, collect the natural expression images of their faces facing the camera, uniform illumination and no obvious emotional expression, that is, collect a plurality of second images.

[0083] Step S002: input the first image and the second image into the face key point detection model respectively to obtain the key point coordinates of the first image and the second image.

[0084] It should be noted that the face key point detection model is a computer vision-based deep learning tool, which mainly functions to automatically locate and output a series of semantically meaningful facial feature point coordinates from an input face image. The face key point detection model of the present embodiment is a Dlib model. The present embodiment obtains 68 key point coordinates, among which numbers 1-17 correspond to the face contour, numbers 18-27 correspond to the eyebrows, numbers 28-36 correspond to the nose, numbers 37-42 correspond to the left eye, numbers 43-48 correspond to the right eye, and numbers 49-68 correspond to the mouth.

[0085] Specifically, the two face images (the first image and the second image) are respectively input into the Dlib model. The model first detects and locates the face region in the image, and then automatically identifies the two-dimensional coordinates of a total of 68 key points such as eyes, eye corners, eyebrow arches, nose tips, nose wings, mouth corners, lip peaks, and face contours according to the pixel of the region. The Dlib model is a known technology, and will not be described in detail here.

[0086] Step S003: obtaining a high-quality first image based on the key point coordinates of the first image.

[0087] It should be noted that, considering that there is less high-quality data in each type of expression data collected, high-quality data in each type of expression data is first screened out, and then data augmentation is performed through a cGAN (conditional generative adversarial network) network.

[0088] Step S003 further includes steps S0031-S0036:

[0089] Step S0031: locating each facial feature corresponding key point set in each first image based on the key point coordinates of each first image.

[0090] Specifically, based on the numbering rules of the face key point model, the key point sets respectively corresponding to the left eye, the right eye, the eyebrows, the nose, and the mouth are extracted from the detected 68 key point coordinates.

[0091] Step S0032: constructing the minimum circumscribed rectangle of each facial feature based on each facial feature corresponding key point set, and determining the minimum circumscribed rectangle as the corresponding region of each facial feature.

[0092] Exemplarily, for the key point set of the left eye, the minimum circumscribed rectangle thereof is constructed and the minimum circumscribed rectangle is taken as the corresponding region of the left eye: first, the minimum and maximum values of all key point coordinates in the horizontal and vertical directions are obtained to determine the coordinates of the upper left corner and the lower right corner of the rectangle, and boundary expansion can be optionally performed to finally obtain the rectangular region corresponding to the left eye. The construction of the minimum circumscribed rectangle is a known technology, and will not be described in detail here.

[0093] Step S0033: cropping a region corresponding to each facial feature from the first image, and determining the cropped region as a local image block of each facial feature.

[0094] It should be noted that the local image block of each facial feature includes 5 local image blocks of left eye, right eye, eyebrows, nose and mouth, i.e. each first image contains 5 local image blocks.

[0095] For example, a region corresponding to the left eye is cropped from the first image, and the cropped region is determined as the local image block of the left eye.

[0096] Step S0034: extracting a plurality of local image blocks from all local image blocks of each facial feature, and determining the minimum variance in the Laplacian variance of each extracted local image block as the variance of the local image block of each facial feature.

[0097] It should be noted that the Laplacian variance is an evaluation index for quantifying the definition of an image. The variance of the pixel value of the image after the Laplacian operator is calculated to reflect the sharpness of the image edge and texture, and the higher the value, the clearer the image.

[0098] The Laplacian variance of each extracted local image block is calculated. First, each local image block is converted into a grayscale image. Then, a Laplacian operator (a 3x3 convolution kernel) is applied to the grayscale image to obtain a Laplacian response map, which highlights the edge and high-frequency texture region in the image. Then, the variance of all pixel values in the response map is calculated, and the variance value is the Laplacian variance of the local image block. The calculation of the Laplacian variance is a known technology, which will not be described in detail here.

[0099] Step S0035: determining the minimum variance in the variances of the local image blocks of all facial features as the variance threshold.

[0100] Step S0036: determining the first image whose Laplacian variances of the local image blocks of all facial features are greater than the variance threshold as the high-quality first image.

[0101] For example, there are 2000 first images, and each first image has 5 local image blocks. Therefore, there are 2000 local image blocks for each facial feature, and 1000 local image blocks are randomly extracted from the 2000 local image blocks of the left eye. The Laplacian variances of the 1000 extracted local image blocks are calculated, and the minimum variance is selected as the variance of the local image block of the left eye.

[0102] According to the above method, the variances of the local image blocks corresponding to the left eye, right eye, eyebrows, nose and mouth can be obtained, and the minimum variance is selected from the 5 variances as the variance threshold.

[0103] Then, it is judged whether the Laplacian variance of the 5 local image blocks in each first image is greater than the variance threshold value, and if so, the first image is determined to be a high-quality first image.

[0104] Step S004: Based on the key point coordinates of the high-quality first image and the second image, the expression performance degree of each key point is determined.

[0105] It should be noted that the facial muscles work in coordination. We focus on the point groups that will pull each other and link in expression changes. Therefore, a group of neighbor key points with direct muscle connection or motion association is defined for each key point.

[0106] Step S004 further includes steps S0041-S0045:

[0107] Step S0041: Based on the key point coordinates of the high-quality first image and the second image, a first face key point map and a second face key point map are constructed using a triangulation method; wherein the first face key point map represents the face key point map of the high-quality first image, and the second face key point map represents the face key point map of the second image.

[0108] It should be noted that triangulation is a geometric partitioning method that connects a set of points in a plane into a non-overlapping triangular mesh, aiming to cover and describe the overall shape and topology of the target area with the simplest geometric unit (triangle). The embodiment adopts the Delaunay triangulation method.

[0109] Using the triangulation method, the first face key point map and the second face key point map are constructed: the 68 key points in the high-quality first image or the second image are regarded as a set of points in a plane, and a set of triangles with these points as vertices and completely covering the face area is automatically generated by the triangulation algorithm, wherein the edges of each triangle connect spatially adjacent key points, thereby forming a mesh map that describes the geometric topology of the face, i.e. a triangular mesh structure, and the triangular mesh structure is recorded as a face key point map. Triangulation is a well-known technology and will not be described in detail here.

[0110] Step S0042: According to the first face key point map and the second face key point map, the density change relationship and change direction value of each key point are determined.

[0111] It should be noted that the embodiment obtains the neighbor key points associated with the motion of each key point by judging whether the density of the key points in the miner's face image changes uniformly, and whether the key points always change in a fixed direction. For example: the density uniformly increases, corresponding to the frown in the case of greater pressure; the density uniformly decreases, corresponding to the relaxation of the brow in the case of less pressure.

[0112] The key points with the same number in different expressions are the key points of the same position of the face.

[0113] Step S0042 further comprises steps S0421-S0425:

[0114] Step S0421: For each key point in the first face key point map, the number of nodes directly connected to the key point and the area surrounded by the nodes directly connected to the key point are obtained, and the ratio of the area to the number of nodes is determined as the density of each key point in the first face key point map.

[0115] It should be noted that the nodes directly connected to the key point are other key points connected to the key point. The number of nodes directly connected to the key point is denoted as n, and the area surrounded by the nodes directly connected to the key point is denoted as h. Then the density of each key point in the first face key point map is denoted as m1: m1=h / n.

[0116] Step S0422: For each key point in the second face key point map, the number of nodes directly connected to the key point and the area surrounded by the nodes directly connected to the key point are obtained, and the ratio of the area to the number of nodes is determined as the density of each key point in the second face key point map.

[0117] Specifically, the density of each key point in the second face key point map is also denoted as m2.

[0118] Step S0423: For the same key point in the first face key point map and the second face key point map, the frequency that the density of the key point in the first face key point map is greater than the density of the key point in the second face key point map is counted, and the frequency that the density of the key point in the first face key point map is less than the density of the key point in the second face key point map is counted. The density relationship corresponding to the frequency greater than a preset frequency threshold in the two frequencies is determined as the density change relationship of each key point.

[0119] It should be noted that the preset frequency threshold is mainly based on the statistical confidence required by the sample size (the number of miners), the strictness requirement for the key point density change relationship judgment, and the motion direction judgment of the key point. For example, when the sample size is sufficient (such as more than 30 people) and it is desired to exclude individual accidental differences, the threshold can be set to 0.8 to ensure that only when most miners show the same density change trend (contraction or expansion) at the same key point, the key point is determined to have a stable density change relationship. The embodiment is preferably 0.8.

[0120] Specifically, for the same key point in the first face key point map and the second face key point map of the miner A, if the frequency of occurrence of m1>m2 and m1<m2 is greater than 0.8, it is considered that the key point of the underground expression is changed in this way, and the relationship between m1 and m2 is obtained, which is denoted as the density change relationship (here, the density change relationship is relative to the neutral expression). When there is no frequency greater than 0.8, it means that there is no density change relationship for the key point.

[0121] For example, for the same key point (No. 12) in the first face key point map and the second face key point map of the miner A, if the frequency of occurrence of m1>m2 is greater than 0.8, the density change relationship of the key point No. 12 is m1>m2.

[0122] Step S0424: For the same key point in the high-quality first image and the second image corresponding to each miner, the motion direction of the key point from the position in the second image to the position in the high-quality first image is obtained, and the angle of the motion direction is determined as the direction value of each key point.

[0123] Specifically, for each miner, the two-dimensional coordinates of the same key point in the high-quality first image (underground expression image) and the second image (neutral expression image) are extracted respectively. The displacement vector of the key point from the position in the second image to the position in the first image is calculated to represent the spatial motion direction of the key point during the expression change. The angle of the displacement vector (usually 0° to the right horizontally, and counterclockwise rotation as the positive direction) is quantified as the direction value. Thus, the motion direction of each key point is obtained.

[0124] For convenience of calculation, in this embodiment, the degree of the closest multiple of 5 to the angle of the obtained displacement vector is taken as the direction value, for example, the angle of the displacement vector is 43°, and the degree of the closest multiple of 5 to the angle is 45°, then the direction value is 45°. Therefore, 72 types of direction values are finally obtained according to this calculation.

[0125] Step S0425: The frequency of occurrence of each direction value of each key point is counted, and the direction value corresponding to the frequency greater than the preset frequency threshold is determined as the change direction value of each key point.

[0126] It should be noted that for a certain key point, the direction values calculated in the high-quality first image and the second image corresponding to different miners can be the same or different, so the probability of occurrence of each direction value of a certain key point is counted. The preset frequency threshold is the same as the frequency threshold in step S0423.

[0127] For each key point, obtain the orientation value of the key point of all miners, and count the frequency of each direction value to obtain the change direction value corresponding to the frequency greater than 0.8, that is, when the expression of a person changes from neutral to non-neutral, the key point always moves in the direction of the change direction value.

[0128] Step S0043: Obtain the associated key point of each key point using the density change relationship and the change direction value of each key point.

[0129] Step S0043 further comprises steps S0431-S0435:

[0130] Step S0431: For any two key points in all key points, compare whether the density change relationship of the two key points is consistent.

[0131] Specifically, for any two key points in the 68 key points, first determine whether the density change relationship of the two key points is consistent. If the density change relationship of one of the key points does not exist, directly use the direction value of the two key points to determine, that is, directly proceed to step S0432.

[0132] Step S0432: In the case where the density change relationship of the two key points is consistent, count the number of times the change direction value of the two key points appears in the same high-quality first image, and calculate the sum of the number of times the change direction value of the two key points appears in the high-quality first image.

[0133] Step S0433: The ratio of the number of times the change direction value of the two key points appears in the same high-quality first image to the sum of the number of times the change direction value of the two key points appears in the high-quality first image is determined as the association of the two key points.

[0134] Specifically, for any two key points, to determine whether they are motion-associated points, first compare the density change relationship, and if the size relationship of m1 and m2 in the density change relationship is consistent, then compare the change direction:

[0135] The ratio v of the number of times the change direction value of the two key points appears in the same high-quality first image to the sum of the number of times the change direction value of the two key points appears in the high-quality first image is recorded as the association of the two key points.

[0136] Step S0434: When the association of the two key points is greater than a predetermined association threshold, the two key points are determined to be associated key points.

[0137] It should be noted that the greater the value of v, the more the two key points are always moving, and if v is greater than 0.8, the two key points are recorded as the associated key points.

[0138] The setting of the preset association threshold is based on the significance requirement of population data statistics and the physiological priori knowledge of facial muscle coordinated movement. The embodiment is preferably 0.8.

[0139] Step S0435: Based on all the acquired associated key points, the associated key points of each key point are determined.

[0140] It should be noted that the comparison of the density change relationship and the change direction value of any two key points can obtain all the associated key points, and then the associated key points of each key point are obtained.

[0141] The associated key points correspond to expressions, for example, when smiling, the key points of the mouth and the eyes and the corners of the eyes are easy to form associated key points, corresponding to the coordinated and pleasant expression of "smiling corners, slightly squinting eyelids, and fish tail wrinkles appearing at the corners of the eyes" when smiling; when surprised, the key points of the eyebrows and the mouth are easy to form associated key points, corresponding to the instantaneous shocked expression of "eyebrows raised and uplifted, eyelids widely open, and mouth involuntarily open and round" when surprised.

[0142] The associated key points of each key point are obtained by calculation, that is, the point group that will pull and link each other in expression changes. Then, the performance degree of each key point to the expression is obtained through the change of each key point itself and the change of the associated key points in each underground expression (the first image corresponding to the expression with high quality).

[0143] Step S0044: Based on the key point coordinates of the high-quality first image and the second image, the texture descriptor and the neutral texture descriptor of each key point are obtained.

[0144] It should be noted that when a person is under great psychological pressure, the facial expression will present a composite state of "tension, inhibition and exhaustion", which is specifically embodied in the typical characteristics of the three regions:

[0145] Eyebrow and eye area (serious and absent-minded): eyebrows are tightly locked, eyes are dull or wandering, and muscles around the eyes are tense.

[0146] Lip area (inhibition and tension): lips are pinched, corners of the mouth are drooping, and the lower jaw is stiff with slight tremors.

[0147] Overall state (rigid and tired): facial muscles are rigid and lack natural micro-movement, and the skin color may be abnormal, as if wearing a "mask".

[0148] The expression under high pressure is not a single emotion, but a complex state of frowning in the center of the eyebrows, dim eyes, sinking corners of the mouth, and rigid face, reflecting that the individual is in the process of tension, consumption and self-restraint. On this basis, the embodiment describes these stress features through the texture of each face key point, obtains the texture of each key point among the 68 key points of each person, and compares other expressions with the neutral expression to determine which key point texture and texture relationship has changed, and then takes the key point with large changes as the key point representing the expression feature, for example: when angry, the eyebrows are often frowning, so the key point in the eyebrow area is taken as the key point representing the expression feature. When the discriminator discriminates the generated image of the generator, the attention to these key points is increased.

[0149] Comparing other expressions with the neutral expression is equivalent to comparing with the standard expressionless state to determine which changes have occurred in the texture of the features of the other expressions, and then obtaining the expression features of the face.

[0150] Step S0044 further includes steps S0441-S0444:

[0151] Step S0441: For the high-quality first image, a fixed-size image region is intercepted with each key point coordinate as the center, and the intercepted image region is determined as the region corresponding to each key point in the high-quality first image.

[0152] It should be noted that the setting of the fixed size includes: the normalized size of the face detection frame, the local range of the physiological structure of each key point, and the demand of the subsequent texture feature operator (such as LBP) for the effective neighborhood. In this embodiment, it is set to a 7x7 pixel region, which can achieve a good balance between feature discrimination and computational efficiency.

[0153] Exemplarily, for the high-quality first image, the key point numbered 12, a 7x7 pixel image region is intercepted as the region corresponding to the key point with the key point coordinate as the center.

[0154] Step S0442: The LBP histogram of each key point corresponding region in the high-quality first image is calculated, the corresponding LBP feature vector is generated, and the LBP feature vector is determined as the texture descriptor of each key point.

[0155] It should be noted that the LBP histogram (Local Binary Pattern Histogram) is a statistical representation method for describing the local texture features of an image. It forms a fixed-dimension histogram vector (LBP feature vector) by counting the distribution of LBP code values generated by comparing each pixel point and its neighborhood in the image, so as to effectively represent the texture pattern and structural features of the image region. The LBP histogram is a known technology, and will not be described in detail here.

[0156] Obtaining the texture descriptor of each key point: after converting the area corresponding to each key point into a gray-scale image, a uniform LBP (Local Binary Pattern) operator is used to calculate the texture coding image thereof, and then the LBP value distribution of all pixels in the coding image is counted to form a 256-dimensional histogram. The histogram is normalized to finally obtain a standard LBP feature vector representing the local texture structure of the key point. The vector is the texture descriptor of the key point.

[0157] Step S0443: for the second image, a fixed-size image area is intercepted with each key point coordinate as the center, and the intercepted image area is determined as the area corresponding to each key point in the second image.

[0158] It should be noted that the fixed size in this step is the same as the fixed size in step S0441.

[0159] Step S0444: the LBP histogram of the area corresponding to each key point in the second image is calculated, the corresponding LBP feature vector is generated, and the LBP feature vector is determined as the neutral texture descriptor of each key point.

[0160] Step S0045: the expression performance degree of each key point is obtained by using the texture descriptor, the neutral texture descriptor and the associated key point of each key point.

[0161] Step S0045 further includes steps S0451-S0456:

[0162] Step S0451: for the same key point in the high-quality first image and the second image corresponding to each miner, the cosine similarity between the texture descriptor of the key point and the neutral texture descriptor thereof is calculated, and the cosine similarity is determined as the similarity of each key point.

[0163] It should be noted that the cosine similarity is an index for measuring the consistency of the directions of two vectors, and the cosine value of the included angle between them is calculated to evaluate their similarity. The cosine similarity is a known technology, and will not be described in detail here.

[0164] Exemplarily, for the miner A, the high-quality first image corresponding thereto represents the expression state of the miner in the coal mine, and the second image corresponding thereto represents the state of the miner being calm and relaxed, and then for the key point numbered 12, the similarity between the texture features of the key point in the two images is calculated. The smaller the similarity s is, the greater the change of the key point is, and the greater the role of the key point in identifying the expression in the high-quality first image corresponding to the expression.

[0165] Calculate the cosine similarity between the texture descriptor of the key point and the neutral texture descriptor: calculate the cosine similarity between the two LBP feature vectors.

[0166] Step S0452: Calculate the mean of all similarities of each key point, and determine the difference between 1 and the mean as the self-performance degree of each key point.

[0167] For example, for miner A, the similarity of key point No. 12 is s1; for miner B, the similarity of key point No. 12 is s2; then the self-performance degree c of key point No. 12 is: c=1-(s1+s2) / 2.

[0168] Step S0453: Obtain the mean of the self-performance degrees of the associated key points of each key point.

[0169] For example, the self-performance degree of key point No. 12 is c1, and its associated key points are No. 13 and No. 11, the self-performance degree of key point No. 13 is c2, and the self-performance degree of key point No. 11 is c3, then the mean w of the self-performance degrees of the associated key points of key point No. 12 is: w=(c2+c3) / 2.

[0170] Step S0454: Determine the product of the ratio of the number of associated key points of each key point to 68 and the mean of the self-performance degrees of the associated key points of each key point as the associated key point performance degree of each key point.

[0171] For example, the number of associated key points of key point No. 12 is 2, then the associated key point performance degree q of key point No. 12 is: q=(2 / 68)×w.

[0172] Step S0455: Determine the mean of the self-performance degree of each key point and the associated key point performance degree of each key point as the performance degree of each key point to the current expression.

[0173] For example, the performance degree d of key point No. 12 to the current expression (the expression corresponding to the high-quality first image) is: d=(c1+q) / 2.

[0174] Step S0456: Determine the mean of the performance degree of each key point to each expression as the expression performance degree of each key point.

[0175] It should be noted that the expression performance degree of each key point represents the importance of the key point in identifying the expression.

[0176] Exemplarily, the expression performance degree of the key point No. 12 to the happy expression is d1, the expression performance degree of the key point No. 12 to the angry expression is d2, and the expression performance degree of the key point No. 12 to the disappointed expression is d3, and then the expression performance degree k of the key point No. 12 is k=(d1+d2+d3) / 3.

[0177] Step S005: obtaining the improved loss function of the cGAN network discriminator by using the expression performance degree of each key point.

[0178] It should be noted that when the discriminator discriminates, first, 68 key points of the generated image are obtained through key point recognition, and then the high-quality data set is calculated to obtain the loss weight of each key point and each surrounding key point as the loss weight of the improved discriminator.

[0179] After obtaining the expression performance degree of each key point, the part of the loss function of the existing discriminator for identifying the image forged by the generator is improved, and the specific improvement is as follows:

[0180] Step S005 further includes steps S0051-S0055:

[0181] Step S0051: obtaining the key point coordinates of the face image input into the cGAN network discriminator.

[0182] It should be noted that the embodiment generates images with different expressions by using a conditional GAN network, i.e., cGAN.

[0183] Figure 2 The cGAN network architecture of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, in the cGAN, the inputs of the generator and the discriminator are additionally increased with a conditional information, i.e., a vector representation of a category label (for example, happy, sad, surprised, etc.).

[0184] Input of the generator: random noise z+ conditional label u; output: specific expression image corresponding to the label u.

[0185] The generator hopes that the image manufactured according to a certain conditional label can be mistaken for a real image when input into the discriminator (with the same label) and identified by the discriminator, so as to give a very high score.

[0186] Therefore, the loss of the generator is defined as the negative logarithm of the score given by the discriminator to the forged image. Through training, the generator strives to make the loss value smaller, which means that the score given by the discriminator to the image generated by the generator is higher and higher.

[0187] Input of the discriminator: image x+ conditional label u; output: judging whether the image is real and consistent with the expression described by the label.

[0188] The goal of the discriminator is to maximize its discrimination ability, and its loss is composed of two parts:

[0189] The first part (identifying real images): when the input is a real image and is accompanied by a correct condition label, the discriminator should give a very high score, indicating that "this is undoubtedly a real image that meets the label". This part of the loss calculates the logarithm of the discriminator's score for these "real correct" samples. The discriminator wants this value to be as large as possible.

[0190] The second part (recognizing fake images): when the input is an image generated by the generator according to a certain condition label, the discriminator should give a very low score, indicating that "this is a fake image". This part of the loss calculates the logarithm of the complement of the discriminator's score for these "fake" samples. Similarly, the discriminator wants this value to be as large as possible. The complement is the difference between 1 and the score.

[0191] The overall goal of the discriminator is to improve its judgment in both aspects through training, that is, to increase the sum of the two part loss values.

[0192] Specifically, first perform face key point recognition on the face image input to the discriminator to obtain 68 key points.

[0193] Step S0052: Multiply the expression performance of each key point by 20 and determine the nearest positive integer as the influence range of each key point.

[0194] Step S0053: Determine the circular area centered at each key point coordinate with the influence range of each key point as the radius as the influence area of each key point; wherein the weight in each key point's influence area is the expression performance of each key point.

[0195] Step S0054: Superimpose the influence areas of all key points to form a weight map that is the same size as the face image input to the cGAN network discriminator.

[0196] It should be noted that: the larger the weight of a pixel point, the greater the importance of that position to expression recognition.

[0197] Step S0055: Based on the weight map, obtain the improved loss function of the cGAN network discriminator.

[0198] The calculation method of the improved loss function of the cGAN network discriminator is: calculate the square of the difference between the high-quality first image and the generated image pixel by pixel, multiply it by the weight value at the corresponding position in the weight map, and finally sum the weighted results of all pixels.

[0199] Specifically, the loss function of the discriminator for identifying the image generated by the generator is changed from the square sum of (real image - generated image) to the square sum of weight map x (real image feature - generated image feature), and the improved loss function expression is:

[0200] ;

[0201] wherein, represents the improved loss function value of the cGAN network discriminator; H and W are the height and width of the image, respectively; represents the pixel value of position (i, j) in the real image (high-quality first image), represents the pixel value of position (i, j) in the generated image; represents the weight value of position (i, j) in the weight map.

[0202] Step S006: obtaining an augmented image based on the improved loss function of the cGAN network discriminator.

[0203] Step S006 further includes steps S0061-S0063:

[0204] Step S0061: identifying the expression of each high-quality first image and labeling.

[0205] It should be noted that the expression categories in the embodiment are divided into 7 categories: anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0206] Specifically, the embodiment is manually labeled.

[0207] Step S0062: taking the high-quality first image and its label as input, training the cGAN network using the improved loss function of the cGAN network discriminator, and obtaining the trained cGAN network.

[0208] Specifically, the high-quality first image and its corresponding expression category label are taken as training samples and input into the conditional generative adversarial network (cGAN), wherein the generator receives random noise and condition labels to attempt to synthesize a face image of the corresponding expression, and the discriminator is optimized based on the improved weighted loss function: the loss function gives higher weight to the pixel difference between the real image and the generated image in the expression key area (such as the inter-brow and the corner of the mouth) by introducing the weight map generated by the key point expression. During the adversarial training process, the discriminator minimizes the weighted loss to more accurately identify the authenticity of the real and synthesized images in the key expression features, while the generator is constantly optimized according to the feedback of the discriminator, gradually generating synthesized images with more distinctive key expression area features and higher quality, and finally obtaining a trained cGAN network.

[0209] Step S0063: Based on the trained cGAN network, an augmented image is obtained.

[0210] Specifically, when generating an augmented image using the trained cGAN network, first, the generator is input with a conditional label of a specified expression category (such as "happy") and a random noise vector conforming to a preset distribution. The generator synthesizes a face image of the corresponding category with clear expression features according to the underground miner face feature distribution and the generation rules of key expression regions learned in the training. Through batch generation and screening, a large number of augmented images with high quality and significant features in key expression regions are finally obtained.

[0211] Step S007: The augmented image and the high-quality first image are input, and the expression recognition neural network is trained using a cross-entropy loss function to obtain a trained expression recognition neural network.

[0212] Specifically, each generated image is discriminated by the improved discriminator loss function, and the generated image obtained thereby contains more features that can identify expressions.

[0213] All images (augmented images) generated by the cGAN network and the high-quality first image are input into the expression recognition neural network to obtain a trained expression recognition neural network, and the training process is as follows:

[0214] Input and processing: all images generated by the cGAN network and the original high-quality image are input, and the ResNet50 model is used as the architecture of the expression recognition neural network.

[0215] Output and comparison: the model outputs a 7-dimensional probability, representing the possible degree of anger, disgust, fear, happiness, sadness, surprise, and neutrality, respectively. The system compares this result with the true expression label and calculates the difference (cross-entropy loss).

[0216] Learning and optimization: based on the calculated difference, the Adam optimizer automatically adjusts the millions of parameters inside the model. This process is repeated hundreds of times on hundreds of thousands of images until the model can accurately identify emotions.

[0217] Step S008: The third image is input into the trained expression recognition neural network, and a probability sequence of various expressions of the miner is output; wherein the third image represents a real-time collected face image of the miner in the coal mine.

[0218] It should be noted that the third image represents the expression image of the miner in the coal mine in real time, i.e., the image to be identified.

[0219] Step S009: Based on the probability sequence of various expressions of the miner, the stress level of the miner is determined.

[0220] Specifically, for the real-time obtained face image of the miner, according to the expression recognition neural network, a probability sequence of various emotions of the miner is obtained, and the stress level is judged according to the judgment rule library preset by the safety expert:

[0221] If the emotion of “fear” or “anger” is more than 40%, it is immediately recorded as a high-risk event.

[0222] If the emotion of “disappointment” plus “fear” is more than half (50%), and this state lasts more than 5 minutes, it is determined as a sustained high pressure.

[0223] Every half minute, a “low, medium, high” stress level result is given according to the above situation.

[0224] The corresponding warning is executed:

[0225] The system automatically triggers different actions according to the stress level, and the relevant personnel need to cooperate:

[0226] If it is low stress, the system only records and does not act.

[0227] If it is medium stress, the system sends an instruction to vibrate the miner's bracelet, prompting “pay attention to adjust”, and sends a message to the foreman: team member XXX, state fatigue, please pay attention. The foreman needs to confirm the team member's situation within 10 minutes and click “handled” on the device.

[0228] If it is high stress, the system sends a command to the miner's bracelet to vibrate strongly and voice broadcast: high risk, please stop working immediately and evacuate! The dispatch center big screen flashes red light, sounds alarm, displays the location of the personnel, and automatically notifies the safety officer. The dispatcher immediately prompts evacuation with the broadcast and sends a safety officer to the scene. The safety officer assists until the situation is resolved. The processing result is recorded in the system afterwards.

[0229] In the key point-based expression recognition model, the information contribution degree of different facial regions is significantly different. This embodiment evaluates this difference through quantitative analysis: if the texture pattern of a certain key point region (such as the glabella, the corner of the mouth) shows greater difference in low, medium, and high pressure states, it indicates that the key point has stronger discrimination ability for stress state, and its information entropy is higher. Therefore, when training the discriminator of the cGAN network, greater loss function weight should be given to these high discrimination key points, so as to guide the model to pay more attention to these information-rich regions.

[0230] When using images generated by cGAN as training data, the facial expression recognition network is forced to learn how to extract effective features from images that may be blurry but whose facial features are discernible. This training enables the model to achieve good stress expression recognition even when faced with realistic, poorly lit faces of underground miners, as long as the core facial features are identified, thanks to its learned ability to "focus on key discriminative regions." This significantly improves the system's robustness and practicality in the underground coal mine environment.

[0231] Please see Figure 3 The diagram illustrates a block diagram of a neural network-based safety psychological stress recognition system according to an embodiment of the present invention. The system includes the following modules:

[0232] The acquisition module 100 is used to acquire a first image and a second image; wherein the first image represents the facial image of each miner in the coal mine, and the second image represents the neutral expression image of each miner before entering the coal mine.

[0233] The analysis module 200 is used to input the first image and the second image into the face key point detection model respectively to obtain the key point coordinates of the first image and the second image; based on the key point coordinates of the first image, obtain a high-quality first image; based on the key point coordinates of the high-quality first image and the second image, determine the expression degree of each key point; use the expression degree of each key point to obtain the improved loss function of the cGAN network discriminator; and based on the improved loss function of the cGAN network discriminator, obtain the augmented image.

[0234] The recognition module 300 is used to take the augmented image and the high-quality first image as input, and train the facial expression recognition neural network using the cross-entropy loss function to obtain the trained facial expression recognition neural network; input the third image into the trained facial expression recognition neural network, and output the probability sequence of various facial expressions of the miner; wherein the third image represents the real-time acquired facial image of the miner; based on the probability sequence of various facial expressions of the miner, the stress level of the miner is determined.

[0235] In summary, in this embodiment of the invention, the collected facial data is first filtered to obtain high-quality data. Then, the high-quality data is analyzed to obtain the expression performance of each key point. Based on the expression performance of each key point, the loss function of the discriminator in the existing cGAN network is improved, and these high-discrimination key points are given greater weights in the loss function. This results in an expanded image containing more features that are helpful for expression recognition. In other words, the quality of the expanded facial data is higher, thereby training an emotion recognition network with better recognition capabilities and obtaining more accurate stress recognition results.

[0236] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for identifying psychological stress related to safety based on neural networks, characterized in that, The method includes the following steps: Acquire a first image and a second image; wherein the first image represents the facial image of each miner underground in the coal mine, and the second image represents the neutral facial expression image of each miner before entering the coal mine. The first image and the second image are respectively input into the face key point detection model to obtain the key point coordinates of the first image and the second image. Based on the keypoint coordinates of the first image, a high-quality first image is obtained, specifically including: Based on the key point coordinates of each first image, locate the set of key points corresponding to each facial feature in each first image; Based on the set of key points corresponding to each facial feature, construct the minimum bounding rectangle of each facial feature, and determine the region corresponding to each facial feature; The same region corresponding to each facial feature is cropped from the first image, and each cropped region is determined as a local image block for each facial feature; Several local image blocks are extracted from all local image blocks of each facial feature, and the minimum variance among the Laplacian variances of each extracted local image block is determined as the variance of the local image block of each facial feature. The minimum variance among the variances of all local image patches of the facial features is determined as the variance threshold. The first image in which the Laplacian variance of all local image patches of the facial features is greater than the variance threshold is identified as a high-quality first image. Based on the keypoint coordinates of high-quality first and second images, the expressiveness of each keypoint is determined, specifically including: Based on the key point coordinates of the high-quality first and second images, a triangulation method is used to construct a first face key point map and a second face key point map; wherein the first face key point map represents the face key point map of the high-quality first image, and the second face key point map represents the face key point map of the second image. Based on the first and second facial landmark maps, determine the density variation relationship and direction of change for each landmark, specifically including: For each key point in the first face key point map, obtain the number of nodes directly connected to the key point and the area enclosed by the nodes directly connected to the key point, and determine the density of each key point in the first face key point map by the ratio of the area to the number of nodes. For each key point in the second face key point map, obtain the number of nodes directly connected to the key point and the area enclosed by the nodes directly connected to the key point, and determine the density of each key point in the second face key point map by the ratio of the area to the number of nodes. For the same key point in the first face key point map and the second face key point map, the frequency of the key point having a higher density in the first face key point map than its density in the second face key point map, and the frequency of the key point having a lower density in the first face key point map than its density in the second face key point map are counted. The density relationship corresponding to the frequency of the two frequencies that is greater than a preset frequency threshold is determined as the density change relationship of each key point. For the same key point in the high-quality first and second images corresponding to each miner, obtain the direction of motion of the key point from its position in the second image to its position in the high-quality first image, and determine the angle of the direction of motion as the direction value of each key point; The frequency of each direction value of each key point is counted, and the direction value corresponding to the frequency greater than the preset frequency threshold is determined as the change direction value of each key point; By utilizing the density variation relationship and direction value of each keypoint, the associated keypoints of each keypoint are obtained, specifically including: For any two key points among all key points, compare whether the density change relationship between the two key points is consistent; When the density change relationship between two key points is consistent, count the number of times the change direction values ​​of the two key points appear in the same high-quality first image, and calculate the sum of the number of times the change direction values ​​of the two key points appear in the high-quality first image respectively; The correlation between two keypoints is determined by the ratio of the number of times the change direction values ​​of two keypoints appear in the same high-quality first image to the sum of the number of times the change direction values ​​of the two keypoints appear in each high-quality first image. When the correlation between two key points is greater than a preset correlation threshold, the two key points are identified as related key points. Based on all the associated key points obtained, determine the associated key points of each key point; Based on the keypoint coordinates of the high-quality first and second images, the texture descriptor and neutral texture descriptor of each keypoint are obtained, specifically including: For a high-quality first image, a fixed-size image region is cropped with the coordinates of each key point as the center, and the cropped image region is determined as the region corresponding to each key point in the high-quality first image; Calculate the LBP histogram of the region corresponding to each key point in the high-quality first image, generate the corresponding LBP feature vector, and determine the texture descriptor of each key point using the LBP feature vector; For the second image, a fixed-size image region is cropped with the coordinates of each key point as the center, and the cropped image region is determined as the region corresponding to each key point in the second image; Calculate the LBP histogram of the region corresponding to each key point in the second image, generate the corresponding LBP feature vector, and determine the LBP feature vector as the neutral texture descriptor for each key point; Using the texture descriptor, neutral texture descriptor, and associated keypoints for each keypoint, the expressive power of each keypoint is obtained, specifically including: For the same key point in the high-quality first and second images corresponding to each miner, calculate the cosine similarity between the texture descriptor of the key point and its neutral texture descriptor, and determine the cosine similarity as the similarity of each key point; Calculate the mean of all similarities for each keypoint, and determine the self-expression of each keypoint by the difference between 1 and the mean. Obtain the average performance of the associated keypoints for each keypoint; The performance of each keypoint's associated keypoints is determined by multiplying the ratio of the number of associated keypoints to 68 and the average performance of each keypoint's associated keypoints. The average of the self-expression of each keypoint and the expression of each keypoint's associated keypoints is used to determine the expression of each keypoint to the current expression. The average value of the expressiveness of each key point for each expression is used to determine the expressiveness of each key point. By utilizing the facial expression performance at each key point, the improved loss function of the cGAN network discriminator is obtained, specifically including: Obtain the key point coordinates of the face image input to the cGAN network discriminator; The nearest positive integer obtained by multiplying the facial expression score of each key point by 20 is used to determine the range of influence for each key point. The influence area of ​​each key point is defined as a circular region centered on the coordinates of each key point and with the radius of the influence range of each key point as the radius; the weight of each key point's influence area is the expression degree of each key point. The influence areas of all key points are superimposed to form a weight map of the same size as the face image input to the cGAN network discriminator. Based on the weight graph, obtain the improved loss function of the cGAN network discriminator; The improved loss function of the cGAN network discriminator is calculated as follows: the squared difference between the high-quality first image and the generated image is calculated pixel by pixel, then multiplied by the weight value of the corresponding position in the weight map, and finally the weighted result of all pixels is summed. An augmented image is obtained based on a loss function improved from the cGAN network discriminator. Using the augmented image and the high-quality first image as input, the facial expression recognition neural network is trained using the cross-entropy loss function to obtain the trained facial expression recognition neural network. The third image is input into the trained facial expression recognition neural network, which outputs a probability sequence of various facial expressions of the miner; where the third image represents a real-time captured image of the miner's face. The stress level of miners is determined based on the probability sequence of various facial expressions.

2. The method for identifying psychological stress based on neural networks according to claim 1, characterized in that, The specific steps for obtaining the augmented image using the improved loss function based on the cGAN network discriminator are as follows: Identify and label the facial expressions in each high-quality first image; Using a high-quality first image and its label as input, the cGAN network is trained using the improved loss function of the cGAN network discriminator to obtain the trained cGAN network. Based on the trained cGAN network, augmented images are obtained.

3. A neural network-based safety psychological stress recognition system, used in the neural network-based safety psychological stress recognition method as described in claim 1 or 2, characterized in that, The system includes the following modules: The acquisition module is used to acquire a first image and a second image; wherein the first image represents the facial image of each miner in the coal mine, and the second image represents the neutral expression image of each miner before entering the coal mine. The analysis module is used to input the first image and the second image into the face key point detection model respectively, obtain the key point coordinates of the first image and the second image; and obtain a high-quality first image based on the key point coordinates of the first image. Based on the key point coordinates of high-quality first and second images, the expressiveness of each key point is determined. By utilizing the facial expression intensity of each key point, the improved loss function of the cGAN network discriminator is obtained; An augmented image is obtained based on a loss function improved from the cGAN network discriminator. The recognition module is used to train the facial expression recognition neural network by taking the augmented image and the high-quality first image as input and using the cross-entropy loss function to obtain the trained facial expression recognition neural network. The third image is input into the trained facial expression recognition neural network, which outputs a probability sequence of various facial expressions of the miner. The third image represents a real-time captured image of a miner's face; the miner's stress level is determined based on the probability sequence of various facial expressions.

Images