Underground miner emotion recognition method based on facial expression and motion trail multi-modal fusion

Abstract

The invention relates to the technical field of computer visual recognition and data processing, and discloses an underground miner emotion recognition method based on facial expression and motion trail multi-modal fusion. Comprising the steps of obtaining an initial face image and a track data set in the production operation process of the underground miner, reducing noise of the initial face image and the track data set, extracting facial expression features and motion track features of the underground miner, and training a multi-modal information fusion recognition model of the emotion of the underground miner. And performing multi-modal information fusion identification on the emotion of the underground miner and the like. The underground miner emotion recognition process and method are provided, and the robustness and accuracy of recognition are greatly improved. According to the method, emotion recognition of the underground miner is supplemented through multi-modal information fusion of the facial expression and the motion trail, and the accuracy of emotion recognition of the underground miner can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision recognition and data processing, in particular to a mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion. BACKGROUND

[0002] Coal is the main energy source for the development and operation of our country, and as a "ballast" it ensures national energy security, but the coal mine operating environment is complex and changeable, and the operating process is complex and mostly cross-acting, with high safety production risk. Among them, relevant research shows that coal mine accidents caused by human unsafe behavior account for more than 90% of all accidents, and the emotion of underground miners is one of the main factors leading to unsafe behavior. Therefore, accurate and real-time recognition of the emotions of underground miners is an effective means to prevent and control the generation of unsafe behavior, which is conducive to reducing the occurrence of accidents in the production process of the coal industry. However, existing emotion recognition is mostly performed through facial feature recognition, and the robustness and accuracy of its recognition model are often low. And due to the low light intensity and easy occlusion of the coal mine operating environment, the emotion recognition of underground miners often has problems such as large information noise, distorted information source, and lag, and cannot accurately, reliably and in real time predict the emotional state of underground miners. Personnel motion trajectory is an easily observable indicator and performs well in predicting personnel emotions. For example, chaotic and disordered motion direction usually represents an unpleasant personnel emotion, and the greater the variance of the motion amplitude, the stronger the emotion. However, due to the need for walking and moving at each operating post in the coal mine, there is often a large error in predicting the emotional state of personnel only through their motion trajectory. Therefore, how to overcome the problems of single-modal information recognition in predicting the emotional results of underground miners, reduce the limitations of emotional information, the low accuracy of the algorithm, and the imbalance of source data, and realize multi-modal fusion prediction of miner emotions is a major challenge in current emotion recognition. SUMMARY

[0003] In view of the above shortcomings of the prior art, the purpose of the present application is to provide a mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion, which can effectively solve the above-mentioned pain points and difficulties, and the miner emotion recognition accuracy of the method is significantly higher than that of single-modal information emotion recognition. To achieve the above purpose, the technical scheme adopted by the present application is as follows: A mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion, characterized in that it comprises the following steps: Step 1: Obtain the video data set of the underground miners in the production process by using the fixed camera and personnel positioning system of each operation surface in the underground coal mine, and obtain the frame sequence image of the video data set by video frame extraction, and obtain the initial face image and trajectory data set in the production process of the underground miners; Step 2: The initial face image and trajectory data set are stored in the storage space, and the image is enhanced, cropped, corrected, etc., to complete the noise reduction of the initial face image and trajectory data set; Step 3: The facial expression features and motion trajectory features of the underground miners are extracted by combining top-down and bottom-up, and are substituted into the pre-trained facial expression and motion trajectory emotion recognition model to obtain the emotion recognition results based on facial expression and motion trajectory respectively; Step 4: Based on the emotion recognition results of the underground miners based on facial expression and motion trajectory, multi-modal data decision-level fusion is carried out to obtain the probability recognition result of the emotion of the underground miners. For step 1, the frame extraction of the video data set is carried out by using the equal length cutting method to obtain the frame sequence image, preferably, the frame extraction time interval can be selected as 3 seconds. For step 2, the noise reduction of the initial face image and trajectory includes 4 operations of gray scale transformation, light compensation, geometric correction and filter sharpening. Preferably, the gray scale transformation adopts linear transformation method, defines the gray value D=f(x,y) of the pixel of the initial face image and trajectory, and the gray range is [a,b], the gray value D'=g(x,y) of the pixel of the processed image, and the gray range is [c,d], then the gray linear transformation calculation formula of the image is as follows: Preferably, the light compensation adopts the algorithm based on the reference white, adjusts the R, G, B component values to the maximum 255, thereby reducing the influence of light on the image definition, and the specific operation is as follows: First, count the number of pixels of each gray value, and obtain the top 5% pixel gray value as the reference white through loop extraction, then the brightness average value aveGray of the reference white pixel is: aveGray=Gray ref / Gray refNum Second, calculate the light compensation coefficient coe, the calculation formula is as follows: coe=255 / aveGray Finally, multiply the original pixel value of the image by the light compensation coefficient to obtain the pixel value after light compensation. In the formula, Gray ref is the reference white gray value, and Gray refNum is the reference white pixel. Preferably, the geometric correction can utilize Halcon to correct the image geometry. Preferably, the filter sharpening can utilize a sharpening filter to sharpen the image by a Laplace operator with a 4-neighborhood convolution kernel, and the initial face image and trajectory are defined as f(x, y) and g(x, y) represents the sharpened image, and the filter sharpening calculation formula is: g(x, y) = 5f(x, y) - f(x+1, y) - f(x-1, y) - f(x, y+1) - f(x, y-1) For the face expression feature extraction described in step 3, MTCNN algorithm and deep neural network model VGGFace algorithm are used to automatically extract face features through face key point positioning. The motion trajectory feature extraction includes dense trajectory feature extraction and trajectory shape feature descriptor feature extraction. Preferably, the dense trajectory feature extraction adopts a trajectory tracking method based on an optical flow field, and defines a certain frame image time point as I t , the next frame image time point is I t+1 , and the position coordinates of the two frames of miner optical flow field are (x t , y t ) and (x t+1 , y t+1 ), and the position calculation formula of the frame is as follows: In the formula, M represents a median filter, and the trajectory (Pt, Pt+1…, Pt+L) is the dense trajectory of the underground miner. Preferably, the trajectory shape feature descriptor feature extraction adopts a method based on a linear deformation variable for description, and defines a certain frame image time point as I t , the next frame image time point is I t+1 , and the position coordinates of the two frames of miner trajectory are (x t , y t ) and (x t+1 , y t+1 ), and the trajectory feature descriptor calculation formula is as follows: For the pre-trained face expression and motion trajectory emotion recognition model described in step 3, it refers to gathering different emotional crowds through emotion stimulation in the laboratory, using the face expression feature extraction and motion trajectory feature extraction methods and processes of claim 4, taking the stimulated emotion as the result variable, and taking the face expression and motion trajectory features as the previous cause variable to be input into the deep learning convolutional neural network for training, to obtain the mapping model between the face expression and motion trajectory features and the emotion. Preferably, the deep learning convolutional neural network is an Xception network. The calculation formula of the decision-level fusion of the multi-modal data described in step 4 is as follows: In the formula, f(x) represents the emotional probability evaluation result of the underground miner based on the multi-modal feature data of facial expression and motion trajectory; M represents the number of multi-modal data categories, that is, M = 3; w m (x) represents the emotional recognition weight under the modal data x; f m (x) represents the emotional recognition result under the modal data x. To improve the emotional recognition accuracy of the underground miner, the emotional recognition of the underground miner is performed by the multi-modal fusion method, the emotional recognition result robustness and the recognition inaccuracy caused by the single-modal information are eliminated, and the emotional recognition accuracy and real-time performance of the underground miner are greatly improved. On the other hand, the initial facial image and trajectory data set in the production operation process of the underground miner are denoised by four operations of gray scale transformation, light compensation, geometric correction and filter sharpening, and the problems of poor facial expression recognition accuracy of the underground miner caused by the low illumination and easy shielding of the coal mine industry operation environment are effectively eliminated. BRIEF DESCRIPTION OF DRAWINGS Figure 1 is the overall flowchart of the present application; Figure 2 is the multi-modal fusion recognition framework of the emotional of the underground miner; DETAILED DESCRIPTION To make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below. To achieve the above object, the present application adopts the following technical scheme: A kind of emotional recognition method of underground miner based on facial expression and motion trajectory multi-modal fusion, it is characterized in that, including the following steps: Step 1: the fixed camera and personnel positioning system of each operation surface in coal mine are used to obtain the video data set in the production operation process of the underground miner, and the frame sequence image of the video data set is obtained by video frame extraction, to obtain the initial facial image and trajectory data set in the production operation process of the underground miner; The video data set frame extraction is to obtain frame sequence image by equal-length cutting method, preferably, the frame extraction time interval can be selected as 3 seconds. Step 2: the initial facial image and trajectory data set are stored space compressed and enhanced, cropped, corrected and the like, to complete the noise reduction of the initial facial image and trajectory data set; The noise reduction of the initial face image and trajectory includes four operations of gray scale transformation, light compensation, geometric correction and filter sharpening. Preferably, the light compensation adopts a reference white-based algorithm, and the R, G and B component values are all adjusted to 255, thereby reducing the influence of light on image clarity, and the specific operation is as follows: First, the number of pixels of each gray value is counted, and the top 5% of pixel gray values are obtained through a loop as reference white, and the brightness average value aveGray of the reference white pixel is: aveGray = Gray ref / Gray refNum Secondly, the light compensation coefficient coe is calculated, and the calculation formula is as follows: coe = 255 / aveGray Finally, the pixel value after light compensation is obtained by multiplying the original pixel value of the image by the light compensation coefficient. In the formula, Gray ref is the reference white gray value, and Gray refNum is the reference white pixel. Preferably, the geometric correction can be performed by Halcon. Preferably, the filter sharpening can be performed by a sharpening filter through a Laplace operator, and a 4-neighborhood convolution kernel is used for image sharpening, and the initial face image and trajectory are defined as f(x, y), and g(x, y) represents the sharpened image. The filter sharpening calculation formula is: g(x, y) = 5f(x, y)-f(x+1, y)-f(x-1, y)-f(x, y+1)-f(x, y-1) The face expression feature extraction of step 3 is performed by using MTCNN algorithm and deep neural network model VGGFace algorithm, and the face features are automatically extracted by face key point positioning. The motion trajectory feature extraction includes two feature extractions of dense trajectory feature extraction and trajectory shape feature descriptor feature extraction. Preferably, the dense trajectory feature extraction adopts a trajectory tracking method based on an optical flow field, and defines a certain frame image time point as I t , the next frame image time point as I t+1 , and the position coordinates of the two frames of miner optical flow field as (x t, y t , y t+1 , y t+1 , then the position calculation formula of the frame is as follows: In the formula, M represents a median filter, and the trajectory (Pt, Pt+1…, Pt+L) is a dense trajectory level of the underground miner. Preferably, the trajectory shape feature descriptor feature extraction adopts a linear deformation variable-based method for description, and the certain frame image time point is defined as I t , the next frame image time point is I t+1 , and the position coordinates of the miner trajectories of the two frames are (x t , y t ) and (x t+1 , y t+1 ), respectively. Step 3: The facial expression features and motion trajectory features of the underground miner are extracted by combining the top-down and bottom-up methods, and are substituted into the pre-trained facial expression and motion trajectory emotion recognition model to obtain the emotion recognition results based on the facial expression and motion trajectory, respectively. The pre-trained facial expression and motion trajectory emotion recognition model refers to the following process: in the laboratory, different emotional crowds are gathered by emotional stimulation, the facial expression feature extraction and motion trajectory feature extraction method process of claim 4 are used, the stimulated emotion is used as the result variable, the facial expression and motion trajectory features are used as the previous cause variable, and the deep learning convolutional neural network is trained to obtain the mapping model between the facial expression and motion trajectory features and the emotion. Preferably, the deep learning convolutional neural network is an Xception network. Step 4: Based on the emotion recognition results of the underground miner based on the facial expression and motion trajectory, multi-modal data decision-level fusion is carried out to obtain the probability recognition result of the emotion of the underground miner. The calculation formula of the multi-modal data decision-level fusion is as follows: In the formula, f(x) represents the underground miner emotion probability evaluation result based on the facial expression and motion trajectory multi-modal feature data; M represents the number of multi-modal data categories, that is, M=3; w m (x) represents the emotion recognition weight under the modal data x; f m (x) represents the emotion recognition result under the modal data x. The experiments prove that the single-mode information prediction accuracy of the emotion recognition of the underground miners based on the facial expression features is 85.23%, the single-mode information prediction accuracy of the emotion recognition of the underground miners based on the dense trajectory features is 72.51%, the single-mode information prediction accuracy of the emotion recognition of the underground miners based on the trajectory shape feature descriptor features is 77.68%, and the accuracy of predicting the emotion of the underground miners based on the multi-mode information of the facial expression and the motion trajectory features is 91.01%.

Claims

1. A downhole miner emotion recognition method based on facial expression and motion trajectory multi-modal fusion, characterized in that, Comprising the following steps: Step 1: Obtain the video data set of the underground miner production operation process by using the fixed camera and personnel positioning system of each operation surface in the coal mine, and obtain the frame sequence image of the video data set by video frame extraction, and obtain the initial face image and trajectory data set of the underground miner production operation process; Step 2: The initial face image and trajectory data set is stored in the storage space, and the image is enhanced, cropped, corrected, etc., and the noise reduction of the initial face image and trajectory data set is completed; Step 3: The facial expression features and motion trajectory features of the underground miner are extracted by combining top-down and bottom-up, and are substituted into the pre-trained facial expression and motion trajectory emotion recognition model to obtain the emotion recognition results based on facial expression and motion trajectory respectively; Step 4: Based on the emotion recognition results of the underground miner based on facial expression and motion trajectory, multi-modal data decision-level fusion is carried out to obtain the probability recognition result of the emotion of the underground miner.

2. The mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion according to claim 1, characterized in that: The frame extraction of the video data set in step 1 is to obtain the frame sequence image by using the equal-length cutting method, preferably, the frame extraction time interval can be selected as 3 seconds.

3. The mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion according to claim 1, characterized in that: The noise reduction of the initial face image and trajectory in step 2 includes 4 operations of gray scale transformation, light compensation, geometric correction and filter sharpening. Preferably, the gray scale transformation adopts linear transformation method, defines the gray value D=f(x,y) of the initial face image and trajectory pixel, the gray range is [a,b], the gray value D'=g(x,y) of the image pixel after processing, the gray range is [c,d], and the gray linear transformation calculation formula of the image is as follows: Preferably, the light compensation adopts the algorithm based on the reference white, adjusts the R, G and B component values to 255, thereby reducing the influence of light on the image definition, and the specific operation is as follows: First, count the number of pixels of each gray value, and extract the top 5% of pixel gray values as reference white through loop, then the brightness average value aveGray of the reference white pixel is: aveGray = Gray ref / Gray refNum Second, calculate the light compensation coefficient coe, the calculation formula is as follows: coe=255 / aveGray Finally, multiply the original pixel value of the image by the light compensation coefficient to obtain the pixel value after light compensation. Gray ref Gray refNum Gray Preferably, the geometric correction can be carried out by using Halcon to correct the image geometry; Preferably, the filter sharpening can be carried out by using a sharpening filter through a Laplace operator, using a 4-neighborhood convolution kernel to sharpen the image, defining the initial face image and trajectory as f(x,y), and g(x,y) represents the sharpened image, then the filter sharpening calculation formula is: g(x,y)=5f(x,y)-f(x+1,y)-f(x-1,y)-f(x,y+1)-f(x,y-1) 4. The mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion according to claim 1, characterized in that: The facial expression feature extraction in step 3 is carried out by using MTCNN algorithm and deep neural network model VGGFace algorithm, through face key point positioning, and automatically extracting facial features. The motion trajectory feature extraction includes dense trajectory feature extraction and trajectory shape feature descriptor feature extraction. Preferably, the dense trajectory feature extraction adopts a trajectory tracking method based on an optical flow field, defines the certain frame image time point as I t , the next frame image time point as I t+1 , the position coordinates of the two frames of the miner optical flow field as (x t , y t ) and (x t+1 , y t+1 ), and the position calculation formula of the frame is as follows: In the formula, M represents a median filter, and the track (Pt, Pt+1…, Pt+L) is a dense track level of the miner underground. Preferably, the trajectory shape feature descriptor feature extraction adopts a method based on linear deformation variable to describe, defining the certain frame image time point as I t , the next frame image time point as I t+1 , the two frame miner trajectory position coordinates as (x t , y t ) and (x t+1 , y t+1 ), then the trajectory feature this descriptor calculation formula is as follows:

5. The method of claim 1, wherein the method is based on a multi-modal fusion of facial expressions and motion trajectories for underground miner emotion recognition. The pre-trained facial expression and motion track emotion recognition model of step 3 refers to the following: in the laboratory, different emotional crowds are gathered through emotional stimulation, the facial expression feature extraction and motion track feature extraction method process of claim 4 is used, the stimulated emotion is taken as a result variable, the facial expression and motion track features are taken as previous cause variables, and a deep learning convolutional neural network is used for training, so that a mapping model between the facial expression and motion track features and the emotion is obtained. Preferably, the deep learning convolutional neural network is an Xception network.

6. The mine worker emotion recognition method based on facial expression and motion trajectory multi-modal fusion according to claim 1, characterized in that: The calculation formula of the decision-level fusion of the multi-modal data of step 4 is as follows: In the formula, f(x) represents the emotional probability evaluation result of the underground miner based on the multi-modal feature data of facial expressions and motion trajectories; M represents the number of multi-modal data categories, that is, M=3; w m (x) represents the emotional recognition weight under the modal data x; f m (x) represents the emotional recognition result under the modal data x.

Images