Method for identifying expression state of underground coal mine personnel and computer program product

Abstract

The invention provides an underground coal mine personnel expression state identification method and a computer program product, and the method comprises the steps: obtaining an initial video of an underground coal mine, and the initial video at least comprises a target object; the initial video is preprocessed, a target video is obtained, and the preprocessing mode at least comprises image enhancement and / or image denoising; extracting face feature information in the target video by adopting a convolution algorithm; and adopting a YOLOv5s algorithm to recognize an expression state corresponding to the face feature information. According to the scheme, the problem that in the prior art, the emotion condition recognition accuracy of the personnel under the coal mine is poor is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a method for identifying the facial expressions of personnel in underground coal mines, a device for identifying the facial expressions of personnel in underground coal mines, a computer program product, and an facial expression recognition system in underground coal mines. Background Art

[0002] Since coal miners usually work in a difficult environment, such as long hours of heavy physical labor, narrow and humid mine tunnels, etc., they are in a state of high psychological and physiological stress, which makes them more likely to have bad emotions. Various emotions can directly affect underground operations and thus reflect the different working conditions of underground personnel.

[0003] There are some solutions that can recognize the facial expressions of workers and determine their emotional status. However, the accuracy of the current solutions in recognizing the facial expressions of people in coal mines is relatively poor. Summary of the invention

[0004] The main purpose of the present application is to provide a method for identifying the facial expression status of personnel underground in a coal mine, a device for identifying the facial expression status of personnel underground in a coal mine, a computer program product and a facial expression recognition system underground in a coal mine, so as to at least solve the problem of poor accuracy in identifying the facial expression status of personnel underground in a coal mine in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for identifying the facial expression state of personnel in coal mines is provided. An initial video in a coal mine is obtained, wherein the initial video at least includes a target object; the initial video is preprocessed to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising; a convolution algorithm is used to extract facial feature information in the target video; and a YOLOv5s algorithm is used to identify the facial expression state corresponding to the facial feature information.

[0006] Optionally, the initial video is preprocessed to obtain a target video, including: extracting a dark environment area in the initial video, wherein the dark environment area is an area with brightness less than a preset brightness threshold; and at least performing histogram equalization processing on the dark environment area to obtain the target video.

[0007] Optionally, the initial video is preprocessed to obtain a target video, including: extracting edge points in the initial video through an edge detection algorithm; deleting the edge points in the initial video to obtain an intermediate video; and highlighting the outline of the target object in the intermediate video to obtain the target video.

[0008] Optionally, after preprocessing the initial video to obtain the target video, the method further includes: obtaining a first grayscale value of a first frame in the target video; obtaining a second grayscale value of a second frame in the target video, wherein the number of frames between the first frame and the second frame is a predetermined number of frames, and the first frame is earlier than the second frame; calculating the difference between the first grayscale value and the second grayscale value to obtain a grayscale differential value; and determining that the second frame is a key frame when the grayscale differential value is greater than or equal to a preset differential threshold.

[0009] Optionally, a convolution algorithm is used to extract facial feature information in the target video, including: constructing a feature extraction model, wherein the feature extraction model is trained by a convolution algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical target videos acquired within a historical time period and historical facial feature information corresponding to the historical target videos; inputting the target video into the feature extraction model to obtain the facial feature information corresponding to the target video.

[0010] Optionally, a YOLOv5s algorithm is used to identify the expression state corresponding to the facial feature information, including: constructing an expression recognition model, wherein the expression recognition model is trained by the YOLOv5s algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical facial feature information obtained within a historical time period, and historical expression states corresponding to the historical facial feature information; inputting the facial feature information into the expression recognition model to obtain the expression state corresponding to the facial feature information.

[0011] Optionally, after adopting the YOLOv5s algorithm to identify the expression state corresponding to the facial feature information, the method further includes: when the currently identified expression state is characterized as a negative state, continuously identifying the expression state; when the duration of the expression state characterized as the negative state is greater than or equal to the target duration, generating an alarm message.

[0012] According to another aspect of the present application, a device for identifying the facial expression state of people in a coal mine is provided, comprising: a first acquisition unit, used to acquire an initial video in the coal mine, wherein the initial video at least includes a target object; a processing unit, used to preprocess the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising; an extraction unit, used to extract facial feature information in the target video using a convolution algorithm; and a first recognition unit, used to recognize the facial expression state corresponding to the facial feature information using a YOLOv5s algorithm.

[0013] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to perform any one of the steps of the method for identifying the facial expression state of personnel underground in a coal mine.

[0014] According to another aspect of the present application, a coal mine underground facial expression recognition system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the above-described recognition methods for the facial expression states of personnel underground in a coal mine.

[0015] By applying the technical solution of the present application, image preprocessing can be performed on videos in coal mines to improve the quality of imaging. The convolution algorithm has an excellent effect on image feature extraction, so accurate facial features can be identified. The YOLOv5s algorithm is fast and has high detection accuracy. Therefore, the YOLOv5s algorithm can identify accurate facial expressions, thereby solving the problem of poor accuracy in identifying the facial expressions of people underground in coal mines in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for identifying facial expressions of personnel in underground coal mines provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a flow chart of a method for identifying facial expressions of personnel in underground coal mines provided according to an embodiment of the present application is shown;

[0019] Figure 3 A flow chart of a method for analyzing the working status of underground personnel based on facial expression processing is shown;

[0020] Figure 4 The schematic diagram of the architecture of this solution is shown;

[0021] Figure 5 A schematic diagram of the key frame extraction process is shown;

[0022] Figure 6 A schematic diagram of the expression recognition model of the present invention is shown;

[0023] Figure 7 A structural block diagram of a device for identifying the facial expression state of personnel in underground coal mines provided according to an embodiment of the present application is shown.

[0024] The above drawings include the following reference numerals:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] The reasons for the slow detection speed and poor recognition accuracy in the existing technology include: Manually designed features: Traditional face detection algorithms usually rely on manually designed feature extractors, such as Haar features, HOG features, etc. These feature extractors require manual selection and design, and may not be able to fully capture complex facial features, resulting in low detection accuracy. Representation ability: The features used by traditional algorithms usually have limited representation capabilities and it is difficult to effectively distinguish different facial features. In contrast, neural network-based methods are able to learn higher-level and more complex feature representations, thereby improving detection accuracy. Lack of contextual information: Traditional algorithms usually only consider local information and ignore global contextual information. Neural networks can effectively utilize contextual information through mechanisms such as convolutional layers, thereby improving detection accuracy. Traditional algorithms are not very robust (they compare image features based on a given template during calculation, and the given template can only preset some specific image scenes, such as light and shadow change points, color difference gradient points, and the threshold of change is given). In addition: relying solely on traditional algorithms, each step is spliced ​​with traditional algorithms, which will lead to serial execution of calculations. Neural networks can perform efficient parallel calculations with the support of hardware accelerators (such as GPU, TPU, etc.), which can improve processing speed.

[0030] As introduced in the background technology, the prior art has the problem of poor accuracy in identifying the emotional status of personnel underground in a coal mine. To solve the above problem, the embodiments of the present application provide a method for identifying the facial expression status of personnel underground in a coal mine, a device for identifying the facial expression status of personnel underground in a coal mine, a computer program product and an facial expression recognition system underground in a coal mine.

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0032] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal for a method for identifying facial expressions of personnel in coal mines according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0033] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] In this embodiment, a method for identifying the facial expression state of people underground in a coal mine is provided, which runs on a mobile terminal, a computer terminal or a similar computing device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 2 FIG. 1 is a flow chart of a method for identifying facial expressions of personnel in coal mines according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0036] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0037] Specifically, a video of an underground coal mine may be acquired. An initial video may be acquired at an initial moment, when workers or working equipment are working underground. Therefore, the initial video at least includes a target object and, of course, may also include an image of the environment in the coal mine.

[0038] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0039] Specifically, the initial video may be preprocessed by image enhancement and / or image denoising to improve the imaging quality.

[0040] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0041] Specifically, the convolution algorithm can be used for learning, and then the facial feature information can be extracted from the target video.

[0042] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0043] Specifically, the YOLOv5s algorithm can be used for learning to determine the expression state corresponding to the facial feature information. The YOLOv5s algorithm has the characteristics of fast recognition speed and high recognition accuracy, and is very suitable for real-time detection tasks.

[0044] Through this embodiment, image preprocessing can be performed on the video in the coal mine to improve the imaging quality. The convolution algorithm has an excellent effect on image feature extraction, so accurate facial features can be recognized. The YOLOv5s algorithm is fast and has high detection accuracy. Therefore, the YOLOv5s algorithm can recognize accurate facial expressions, thereby solving the problem of poor accuracy in recognizing the facial expressions of people underground in coal mines in the prior art.

[0045] Specifically, Figure 3 As shown in the figure, this solution integrates image preprocessing, key frame extraction, and target detection to achieve human expression recognition based on image processing, and then analyzes the working status of personnel based on expression. The main architecture is as follows Figure 4 As shown in the figure, the camera provides real-time monitoring video, and the image quality is improved through image preprocessing (i.e., image enhancement and image denoising of the real-time monitoring video). During calculation, redundant frames with small pixel differences before and after the video are ignored, and only key frames are retained to reduce the amount of model calculation. Convolutional neural network is used to extract facial features and directly regress the face area to reduce subsequent interference from non-target areas. Then, the facial area is compared for expression, and the facial expression (happy, sad, surprised, afraid, disgusted, angry, neutral) and duration are determined through YOLOv5s model training and comparison. The positivity and negativity are analyzed based on facial expressions, and the work status of employees is analyzed.

[0046] This solution belongs to the field of image processing technology, and relates to a facial expression data processing based on image processing, specifically a method for analyzing the working status of underground personnel based on facial expression processing. Based on the existing densely distributed camera environment underground and visual neural network algorithm, this solution proposes a method for analyzing the working status of underground personnel in coal mine scenes based on the mine tunnel video monitoring system. This solution combines digital image processing technology and convolutional neural networks, aiming to improve detection speed and recognition accuracy, especially face recognition in dim scenes.

[0047] Based on the existing mine tunnel video surveillance system, this scheme proposes a method for analyzing the working status of underground personnel based on facial expression processing. In view of the problems of slow detection speed, poor recognition accuracy and difficulty in face recognition in dim scenes of traditional algorithms, this scheme combines digital image processing technology with convolutional neural networks. The image quality is improved by image enhancement and denoising and other processing technologies, the key frame extraction method is used to reduce the model calculation amount, and the lightweight neural network model YOLOv5s is used to achieve fast facial expression recognition. By recognizing the facial expressions of the characters, the working status of the employees is further analyzed. This scheme gives full play to the speed advantage of the traditional algorithm and improves the recognition accuracy by combining the neural network. The research results show that by adopting the above method, the problems of slow speed, poor recognition accuracy and difficulty in face recognition in dim scenes of traditional algorithms can be effectively solved, and the working status analysis of underground personnel can be realized. This research result has important application value in the field of face recognition and provides new ideas and methods for research in related fields.

[0048] During the specific implementation process, the above-mentioned initial video is preprocessed to obtain the target video, which can be achieved through the following steps: extracting the dark environment area in the above-mentioned initial video, wherein the above-mentioned dark environment area is an area with brightness less than a preset brightness threshold; at least performing histogram equalization processing on the above-mentioned dark environment area to obtain the above-mentioned target video.

[0049] In this solution, dark areas, i.e. areas with darker brightness, are extracted from the initial video. The brightness of these areas is lower than the preset brightness threshold. These dark areas are processed by histogram equalization to make the brightness distribution of these areas more uniform, thereby improving the quality and clarity of the image. Histogram equalization is a method used to enhance the contrast of an image by redistributing the brightness values ​​of pixels to increase the dynamic range of the image, making the details in the image more prominent. When processing dark areas, histogram equalization can make the details of these areas more clearly visible, thereby improving the quality of the overall image.

[0050] Specifically, the solution includes video image preprocessing. The video image preprocessing includes dark environment image enhancement processing and blurred scene image enhancement. In the dark environment image enhancement processing, the real-time video data collected by the downhole camera is image enhanced to adjust the video frame contrast, balance the image brightness, and improve the image quality of the subsequent face recognition image. First, histogram equalization is performed on each small block separately. The implementation process is as follows:

[0051] Step 1: Count the number of gray levels in the small area n i (0≤i≤L-1, L is 256), n i Represents the number of the i-th gray level.

[0052] Step 2: Calculate the probability distribution of the area occupied by each gray level. The probability calculation formula for the kth gray level is as follows: k Represents the k-th grayscale value, k[0,L-1], and N represents the total number of pixel values:

[0053] P r (r k ) represents the probability value of the k-level gray value,

[0054] Step 3: Calculate the cumulative distribution function:

[0055] S k Represents the cumulative distribution function value of the k-level grayscale value (the probability of being less than or equal to the k-level grayscale value, the value range is between [0,1]),

[0056] Step 4: Calculate the equalized grayscale value and map it back to the coordinate value of the original pixel coordinate.

[0057] Step 5: The above 4 steps complete the histogram equalization of a small block. Then the histogram is cropped, and the grayscale levels that exceed the contrast upper limit in the small block are evenly distributed to each grayscale level through contrast limitation.

[0058] Step 6: Perform the above operation for each small block. In order to avoid the layering of the boundaries between small blocks, traverse each image block and perform bilinear interpolation between blocks to suppress pixel mutations between blocks.

[0059] Specifically, this solution does not only solve the difficulty of face recognition in dim scenes through image preprocessing, but improves the accuracy of face recognition in dim scenes through the designed whole set of processes. The image preprocessing is to quickly, efficiently and highly applicablely calculate each picture to perform preliminary image preprocessing work, reduce noise, and improve image perception and image quality, because high-quality pictures can reduce the amount of calculation of subsequent neural networks, so that the calculation results are faster and more precise.

[0060] In some embodiments, the initial video is preprocessed to obtain a target video, which can be achieved through the following steps: extracting edge points in the initial video through an edge detection algorithm; deleting the edge points in the initial video to obtain an intermediate video; highlighting the outline of the target object in the intermediate video to obtain the target video.

[0061] In this solution, the edge points of the target object can be extracted from the original video through the edge detection algorithm, and these edge points can be deleted to obtain an intermediate video. The outline of the target object in the intermediate video is highlighted, and finally a video with the outline of the target object is obtained. In this way, the outline of the target object can be clearly displayed, reducing the interference of other messy information in the video.

[0062] Specifically, the image preprocessing of the video also includes image enhancement of blurred scenes. In view of the phenomenon of heavy moisture and blurred vision in the well, the edges of the read video frames are sharpened to highlight the foreground information of the image, which can improve the impact of blurred scenes on the subsequent facial expression detection. This paper uses Canny edge detection to find edge information (the brightness of the edge area changes significantly, and the maximum value of the first derivative of the variation function or the zero crossing of the second derivative can be used to judge the points where the brightness changes suddenly. These points are used as the basis for edge detection), and then the edge information is superimposed with the original image to compensate for the contour information. First, Canny edge detection is performed on the read video frame, and the following steps are performed:

[0063] Step 1: Use Gaussian filtering to remove image noise. Edge detection is sensitive to noise, so we first use Gaussian filtering to remove noise from the video frame. Use a 55-degree Gaussian filter to smooth the input frame.

[0064] Step 2: Calculate the horizontal and vertical gradient information of the image. Use the Sobel operator to calculate the first-order derivatives (Gx and Gy) in the horizontal and vertical directions of the smoothed video frame image, and obtain two gradient feature maps in the two directions. For the two returned gradient feature maps (Gx and Gy), calculate the gradient E(G) and direction Angle (θ) of the boundary respectively. Convolve the image I with the template of odd size to obtain the sum. The specific formula is:

[0065]

[0066]

[0067]

[0068]

[0069] If a pixel in an image is an edge, its gradient direction is always perpendicular to the edge. Gradient directions are classified into four categories: vertical, horizontal, and two diagonal directions.

[0070] Step 3: Filter edge pixels by non-maximum suppression. After obtaining the gradient direction and magnitude of the image, scan the entire image to remove non-boundary points. The specific operation is to check each pixel of the image to see whether the gradient of the point has the same gradient direction as the surrounding pixels and has the largest value. If it is a maximum value, it is retained, otherwise it is suppressed.

[0071] Step 4: Hysteresis threshold to determine the true boundary. Set two thresholds: minVal and maxVal. When the grayscale gradient of the edge pixel of the image is higher than maxVal, it is determined as a boundary pixel. If it is lower than minVal, it is directly discarded. If it is between the two, it depends on whether the pixel is connected to a pixel that has been determined as a boundary. If it is connected, it is determined as a boundary point, otherwise it is discarded.

[0072] Step 5: Use OpenCV to highlight the contour information in the original image through information superposition to achieve image enhancement of blurred scenes.

[0073] After image preprocessing, the image quality will be significantly improved and the actual viewing experience of the image will be enhanced.

[0074] In a specific implementation process, after preprocessing the initial video to obtain the target video, the method further includes the following steps: obtaining a first grayscale value of a first frame in the target video; obtaining a second grayscale value of a second frame in the target video, wherein the number of frames between the first frame and the second frame is a predetermined number of frames, and the first frame is earlier than the second frame; calculating the difference between the first grayscale value and the second grayscale value to obtain a grayscale differential value; and determining that the second frame is a key frame when the grayscale differential value is greater than or equal to a preset differential threshold.

[0075] In this scheme, key frames in the video can be effectively extracted, that is, the frames containing the most representative and information, which can save storage space and speed up video retrieval and processing. By extracting key frames, the video can be analyzed and processed more quickly, further improving efficiency and accuracy.

[0076] Specifically, this solution also includes video key frame extraction. Figure 5As shown in the figure, in the video sequence, the content of adjacent frames is very similar. If the content of the current frame has no substantial changes compared with the previous frame, the previous frame image can continue to be used for subsequent image calculations. This paper uses 30-frame intervals as the judgment standard, calculates the grayscale value of each frame, and compares the average difference between the current frame and the previous 30 interval frames. If it exceeds the specified threshold (the average difference of 30 frames is used as the threshold, and the threshold size can be manually adjusted according to actual needs), it means that the content of the current frame has substantially changed and is determined to be a key frame, otherwise it is ignored. Based on the frame difference method, key frame screening is implemented. For the input video stream, the following steps are performed:

[0077] Step 1: Grayscale the image. Grayscale can reduce the amount of data calculation. Use OpenCV built-in functions to read the frame and perform grayscale operations. The grayscale conversion formula is as follows:

[0078] Gray(i,j)=0.30×R(i,j)+0.59×G(i,j)+0.11×B(i,j),

[0079] Step 2: The key frame determination starts after the initialization of 30 frames, and the sliding window is set to 30 frames. The difference value of two consecutive frames in the first 30 frames (the absolute value of the grayscale difference between the two frames) is calculated, and the average of the difference values ​​between the first 30 frames is taken as the initial threshold A. The start frame and the end frame of the 30 consecutive frames are automatically included in the key frame sequence H(t). The calculation formula of the difference value D(t) between two frames is:

[0080] D(t)=|I(t)-I(t-1)|, t represents the time,

[0081] The calculation formula of the initial threshold A is:

[0082]

[0083] Wherein represents the gray value of the image at time t, and D(t) represents the frame differential value at time t.

[0084] Step 3: After the initial 30 frames, calculate the difference value of the following frames and compare it with the initial threshold. If it is higher than the threshold, add it to the key frame sequence H(t), otherwise ignore it. Move the sliding window back one position and reset the threshold. Read the next frame, calculate the difference value with the previous frame, and compare it with the reset threshold. Repeat the threshold reset and threshold comparison operations. The implementation formula is:

[0085]

[0086] H(t) = 1 means that the frame read at time t is a key frame, and H(t) = 0 means that it is ignored in subsequent calculations. After the key frames of the video are extracted, redundant frames can be ignored in subsequent calculations, reducing the amount of model calculations and speeding up model recognition.

[0087] In some embodiments, a convolution algorithm is used to extract facial feature information from the target video, which can be achieved through the following steps: constructing a feature extraction model, wherein the feature extraction model is trained by a convolution algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical target videos acquired within a historical time period and historical facial feature information corresponding to the historical target videos; inputting the target video into the feature extraction model to obtain the facial feature information corresponding to the target video.

[0088] In this solution, multiple sets of training data (including historical target videos and corresponding facial feature information) are used to train with a convolution algorithm to obtain a feature extraction model. The target video is input into the feature extraction model to obtain the facial feature information corresponding to the video. By training multiple sets of data, the feature extraction model can better capture the feature differences between different faces, thereby further improving the accuracy of face recognition.

[0089] In the specific implementation process, the YOLOv5s algorithm is used to identify the expression state corresponding to the above-mentioned facial feature information, which can be achieved through the following steps: constructing an expression recognition model, wherein the above-mentioned expression recognition model is trained by the YOLOv5s algorithm using multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes historical facial feature information obtained within a historical time period and historical expression states corresponding to the above-mentioned historical facial feature information; the above-mentioned facial feature information is input into the above-mentioned expression recognition model to obtain the above-mentioned expression state corresponding to the above-mentioned facial feature information.

[0090] In this scheme, by using multiple sets of training data (including historical facial feature information and corresponding historical expression states), the YOLOv5s algorithm is used for training to obtain an expression recognition model. The facial feature information is input into the expression recognition model to obtain the expression state corresponding to the facial feature information. The YOLOv5s algorithm is fast and has a relatively high accuracy rate, and can obtain relatively accurate recognition results.

[0091] Specifically, this solution also includes expression recognition. Figure 6As shown in the figure, after the preliminary image preprocessing and key frame extraction, the video frame to be processed is obtained, and then the target detection is performed to detect the facial expression in the image. This stage is one of the core tasks of this paper, and all subsequent operations are based on the detected object. Through relevant research, due to the disadvantages of traditional face detection: low recognition rate, difficulty in manual feature design and susceptibility to environmental influences, the convolutional neural network has excellent effect on image feature extraction. Based on this, this paper selects a face detection algorithm based on a convolutional neural network. YOLOv5 is a single-stage general target detection algorithm based on a convolutional neural network based on deep learning. It converts the target detection problem into a regression problem. It does not need to extract candidate boxes, but directly regresses the input image to detect the coordinate information of the object. This method has a fast detection speed and is easy to operate as an end-to-end target detection algorithm.

[0092] The steps of facial expression recognition can be summarized as: detecting the facial position, generating a rectangular frame, and then classifying the detected facial expressions according to the categories marked in the data set. The specific implementation process is as follows:

[0093] Step 1: It is too rough to directly find the location information of an object from the perspective of the whole image. In actual detection, by dividing the input image into grids and detecting faces based on each grid, the target frame can be located more accurately. Divide the input image into S×S grids. It should be noted that usually an image is divided into more than one set of grids, and multiple sets of S×S grids of different sizes are often designed to accommodate facial targets of different sizes. For example, three sets of grids are selected: 20×20, 40×40, and 80×80, which are used to detect large-size, medium-size, and small-target facial objects respectively.

[0094] Step 2: Predict the facial position information in the grid. If the center point of the facial object is in the grid, the grid can be used to regress the facial position information through a convolutional neural network. Each grid corresponds to a 5-dimensional target vector, and the neural network outputs the predicted position information (x, y, w, h, p). Among them, (x, y) is the offset of the center of the prediction box relative to the grid, and (w, h) is the ratio of the prediction box to the entire image. The confidence p indicates whether the current grid contains the center point of the face. The value of p is 0 to 1. If p is 0, it means that the center point of the face is not in the grid, and the facial position information predicted by the grid is meaningless.

[0095] Step 3: Train the neural network and output (x, y, w, h, p). By calculating the loss function between the predicted box output by the convolutional neural network and the actual labeled box when the data set was created, the network parameters are adjusted by the gradient descent method to reduce the loss function. The predicted position information returned by the network is more accurate. The face position information in all grids in the image is predicted, and the vector dimension of the final convolutional neural network output is S×S×(4+1).

[0096] Step 4: Facial expressions can be classified by adding category information based on the original convolutional neural network output prediction position information. A single grid additionally predicts the category based on the predicted position, and the output vector dimension of the entire image at this time is S×S×B×(4+1+C). C represents the category information corresponding to the prediction box, which is determined by the number of categories in the labeled data set. B is the number of target objects that can be predicted for each grid. The data set in this article is divided into seven states: happy, sad, surprised, afraid, disgusted, angry, and neutral. The value of C is 7, and then the vector value returned by the prediction box contains the predicted probability values ​​of the seven categories of c1, c2, c3, c4, c5, c6, and c7, and outputs the probability values ​​of the seven categories, thereby realizing facial expression judgment.

[0097] Specifically, deep neural network models generally include convolution operations, which can speed up image calculations and highlight image features.

[0098] Specifically, the detection speed is improved for traditional algorithms and some two-stage target detection algorithms. Here, the actual applicable scenarios must meet real-time video calculations (low calculation delay), based on the compromise between calculation speed and accuracy. In the field of computer vision, commonly used target detection algorithms can be divided into two types: one-stage target detection (YOLO series) and two-stage target detection (Faster R-CNN series). Candidate boxes are a common sign of two-stage target detection algorithms. The two-stage target detection algorithm first generates candidate areas, and then classifies and locates the candidate areas. The one-stage target detection algorithm is suitable for scenarios with high speed requirements (high accuracy), while the two-stage target detection algorithm is suitable for scenarios with high accuracy requirements.

[0099] Specifically, this solution also includes employee status analysis. Studies have found that human emotions can directly affect employee work status. The most intuitive feedback of human emotions is facial expressions, and negative states are often associated with negative emotions. The employee work status is referenced to the following two conditions: If it is identified as a sad, disgusted, angry, or fearful expression and the confidence level exceeds 0.5, it is determined to be a negative state. If it is identified as a happy, neutral (calm) expression and the confidence level exceeds 0.5, it is determined to be a positive state; the comprehensive duration exceeds 6s.

[0100] If the confidence level of the expression detected is less than 0.6, the detection model will give a reminder. If the confidence level is between 0.6 and 0.8, the system will remind and record it. If the confidence level exceeds 0.8, the system will record the level and pop up an alert:

[0101]

[0102] Specifically, the grid internal fitting calculation is for the YOLO algorithm, which is the core idea of ​​the YOLO algorithm. The final result is to recognize the face for this frame of image and return the probability values ​​of seven expressions based on the face.

[0103] In some embodiments, after using the YOLOv5s algorithm to identify the expression state corresponding to the above-mentioned facial feature information, the above-mentioned method also includes the following steps: when the currently identified expression state is characterized as a negative state, continuously identifying the above-mentioned expression state; when the above-mentioned expression state is characterized as a duration of the above-mentioned negative state greater than or equal to the target duration, generating an alarm message.

[0104] In this solution, when someone's expression is identified as negative, this state will be continuously monitored. If this negative state lasts longer than the set target time, an alarm message will be automatically generated. This can quickly and accurately identify negative emotions and help monitoring personnel to promptly discover possible problems or dangerous situations.

[0105] Specifically, this solution has the following features: key frame screening based on the frame difference method is used to extract key frames from video frames, and redundant video frames are screened to achieve video acceleration; employee work status is determined based on facial expression recognition.

[0106] Specifically, the technical solution of this scheme obtains the user's facial picture and facial expression features by reading the real-time camera underground, and combines the neural network algorithm to recognize facial expressions, and analyzes the employee's work status based on facial expressions and comprehensive duration. Compared with traditional methods, this scheme has the following technical effects: In terms of image preprocessing, the traditional image enhancement algorithm has the characteristics of simple algorithm, fast recognition speed and strong robustness. This scheme adopts the combination of traditional image preprocessing method and convolutional neural network, which not only fully utilizes the speed advantage of traditional algorithm, but also superimposes neural network to improve recognition accuracy. The video frame after image preprocessing can significantly improve the image quality; in terms of key frame extraction, the frame rate of modern cameras is very fast, for example, 30 frames per second, and the content similarity between adjacent frames of the video to be processed is very high. Under the premise of ensuring the integrity of the main content of the original video, key frame extraction can remove redundant images in the video data and retain key frames that can provide a summary of the video content. Subsequent processing only requires operations on key frames, reducing the amount of calculation and improving recognition accuracy and speed. In terms of facial expression recognition based on YOLOv5s, the one-stage target detection algorithm YOLOv5s is selected to implement facial expression detection. This model has the characteristics of fast recognition speed and high recognition accuracy, and is very suitable for real-time detection tasks. In terms of periodic monitoring and analysis, users are monitored and analyzed within the set period. Through periodic monitoring, the user's emotional changes and trends can be understood in real time, providing data support for subsequent emotional recognition and work status analysis.

[0107] The embodiment of the present application also provides a device for identifying the facial expression state of personnel in coal mines. It should be noted that the device for identifying the facial expression state of personnel in coal mines in the embodiment of the present application can be used to execute the method for identifying the facial expression state of personnel in coal mines provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0108] The following is an introduction to the device for identifying the facial expression status of personnel underground in a coal mine provided in an embodiment of the present application.

[0109] Figure 7 It is a structural block diagram of a device for identifying the facial expression state of personnel in coal mines according to an embodiment of the present application.

[0110] like Figure 7 As shown, the device comprises:

[0111] A first acquisition unit 10 is used to acquire an initial video of the coal mine, wherein the initial video at least includes a target object;

[0112] The processing unit 20 is used to preprocess the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0113] An extraction unit 30, configured to extract facial feature information from the target video using a convolution algorithm;

[0114] The first recognition unit 40 is used to use the YOLOv5s algorithm to recognize the expression state corresponding to the above-mentioned facial feature information.

[0115] Through this embodiment, image preprocessing can be performed on the video in the coal mine to improve the imaging quality. The convolution algorithm has an excellent effect on image feature extraction, so accurate facial features can be recognized. The YOLOv5s algorithm is fast and has high detection accuracy. Therefore, the YOLOv5s algorithm can recognize accurate facial expressions, thereby solving the problem of poor accuracy in recognizing the facial expressions of people underground in coal mines in the prior art.

[0116] During the specific implementation process, the processing unit includes a first extraction module and a first processing module. The first extraction module is used to extract the dark environment area in the above-mentioned initial video, wherein the above-mentioned dark environment area is an area with brightness less than a preset brightness threshold; the first processing module is used to at least perform histogram equalization processing on the above-mentioned dark environment area to obtain the above-mentioned target video.

[0117] In this solution, dark areas, i.e. areas with darker brightness, are extracted from the initial video. The brightness of these areas is lower than the preset brightness threshold. These dark areas are processed by histogram equalization to make the brightness distribution of these areas more uniform, thereby improving the quality and clarity of the image. Histogram equalization is a method used to enhance the contrast of an image by redistributing the brightness values ​​of pixels to increase the dynamic range of the image, making the details in the image more prominent. When processing dark areas, histogram equalization can make the details of these areas more clearly visible, thereby improving the quality of the overall image.

[0118] In some embodiments, the processing unit also includes a second extraction module, a deletion module and a second processing module. The second extraction module is used to extract edge points in the above-mentioned initial video through an edge detection algorithm; the deletion module is used to delete the above-mentioned edge points in the above-mentioned initial video to obtain an intermediate video; the second processing module is used to highlight the outline of the above-mentioned target object in the above-mentioned intermediate video to obtain the above-mentioned target video.

[0119] In this solution, the edge points of the target object can be extracted from the original video through the edge detection algorithm, and these edge points can be deleted to obtain an intermediate video. The outline of the target object in the intermediate video is highlighted, and finally a video with the outline of the target object is obtained. In this way, the outline of the target object can be clearly displayed, reducing the interference of other messy information in the video.

[0120] In the specific implementation process, the device further includes a second acquisition unit, a third acquisition unit, a calculation unit and a determination unit. The second acquisition unit is used to obtain the first grayscale value of the first frame in the target video after preprocessing the initial video to obtain the target video; the third acquisition unit is used to obtain the second grayscale value of the second frame in the target video, wherein the number of frames between the first frame and the second frame is a predetermined number of frames, and the first frame is earlier than the second frame; the calculation unit is used to calculate the difference between the first grayscale value and the second grayscale value to obtain a grayscale differential value; the determination unit is used to determine that the second frame is a key frame when the grayscale differential value is greater than or equal to a preset differential threshold.

[0121] In this scheme, key frames in the video can be effectively extracted, that is, the frames containing the most representative and information, which can save storage space and speed up video retrieval and processing. By extracting key frames, the video can be analyzed and processed more quickly, further improving efficiency and accuracy.

[0122] In some embodiments, the extraction unit further includes a first construction module and a third processing module. The construction module is used to construct a feature extraction model, wherein the feature extraction model is obtained by training with a convolution algorithm using multiple sets of training data, each of which includes a historical target video acquired within a historical time period and historical facial feature information corresponding to the historical target video; the third processing module is used to input the target video into the feature extraction model to obtain the facial feature information corresponding to the target video.

[0123] In this solution, multiple sets of training data (including historical target videos and corresponding facial feature information) are used to train with a convolution algorithm to obtain a feature extraction model. The target video is input into the feature extraction model to obtain the facial feature information corresponding to the video. By training multiple sets of data, the feature extraction model can better capture the feature differences between different faces, thereby further improving the accuracy of face recognition.

[0124] During the specific implementation process, the first recognition unit includes a second construction module and a fourth processing module. The second construction module includes constructing an expression recognition model, wherein the above-mentioned expression recognition model is obtained by training through the YOLOv5s algorithm using multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes historical facial feature information obtained within a historical time period and historical expression states corresponding to the above-mentioned historical facial feature information; the fourth processing module includes inputting the above-mentioned facial feature information into the above-mentioned expression recognition model to obtain the above-mentioned expression state corresponding to the above-mentioned facial feature information.

[0125] In this scheme, by using multiple sets of training data (including historical facial feature information and corresponding historical expression states), the YOLOv5s algorithm is used for training to obtain an expression recognition model. The facial feature information is input into the expression recognition model to obtain the expression state corresponding to the facial feature information. The YOLOv5s algorithm is fast and has a relatively high accuracy rate, and can obtain relatively accurate recognition results.

[0126] In some embodiments, the above-mentioned device also includes a second recognition unit and a generation unit. The second recognition unit is used to, after using the YOLOv5s algorithm to identify the expression state corresponding to the above-mentioned facial feature information, continue to recognize the above-mentioned expression state when the currently recognized expression state is characterized as a negative state; the generation unit is used to generate an alarm message when the duration of the above-mentioned expression state characterized as the above-mentioned negative state is greater than or equal to the target duration.

[0127] In this solution, when someone's expression is identified as negative, this state will be continuously monitored. If this negative state lasts longer than the set target time, an alarm message will be automatically generated. This can quickly and accurately identify negative emotions and help monitoring personnel to promptly discover possible problems or dangerous situations.

[0128] The above-mentioned device for identifying the facial expression state of personnel in underground coal mines includes a processor and a memory. The above-mentioned first acquisition unit, processing unit, extraction unit and first identification unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0129] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of poor accuracy in identifying the emotional status of personnel in coal mines in the prior art can be solved by adjusting kernel parameters.

[0130] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0131] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for identifying the facial expression state of personnel underground in a coal mine.

[0132] Specifically, the method for identifying the facial expression state of personnel in coal mines includes:

[0133] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0134] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0135] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0136] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0137] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the program executes the method for recognizing the facial expression state of personnel underground in a coal mine when running.

[0138] Specifically, the method for identifying the facial expression state of personnel in coal mines includes:

[0139] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0140] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0141] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0142] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0143] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are implemented to identify the facial expression state of personnel in a coal mine:

[0144] Specifically, the method for identifying the facial expression state of personnel in coal mines includes:

[0145] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0146] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0147] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0148] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0149] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0150] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for identifying the facial expression state of underground personnel in a coal mine described in each embodiment of the present application are implemented.

[0151] Specifically, the method for identifying the facial expression state of personnel in coal mines includes:

[0152] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0153] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0154] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0155] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0156] The present application also provides a coal mine underground facial expression recognition system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned one or more processors, and the above-mentioned one or more programs include a method for executing any one of the above-mentioned recognition methods of the facial expression states of personnel underground in the coal mine.

[0157] Specifically, the method for identifying the facial expression state of personnel in coal mines includes:

[0158] Step S201, obtaining an initial video of a coal mine, wherein the initial video at least includes a target object;

[0159] Step S202, preprocessing the initial video to obtain a target video, wherein the preprocessing method at least includes image enhancement and / or image denoising;

[0160] Step S203, using a convolution algorithm to extract facial feature information from the target video;

[0161] Step S204: using the YOLOv5s algorithm to identify the expression state corresponding to the above facial feature information.

[0162] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0172] 1) A method for identifying the facial expressions of people underground in a coal mine according to the present application can perform image preprocessing on videos in the coal mine to improve the quality of imaging. The convolution algorithm has an excellent effect on image feature extraction, so that accurate facial features can be identified. The YOLOv5s algorithm is fast and has high detection accuracy. Therefore, the YOLOv5s algorithm can identify accurate facial expressions, thereby solving the problem of poor accuracy in identifying the facial expressions of people underground in coal mines in the prior art.

[0173] 2) A state determination device for an isolating switch of the present application can perform image preprocessing on videos in coal mines to improve the quality of imaging. The convolution algorithm has an excellent effect on image feature extraction, so that accurate facial features can be recognized. The YOLOv5s algorithm is fast and has high detection accuracy. Therefore, the YOLOv5s algorithm can recognize accurate facial expressions, thereby solving the problem of poor accuracy in the recognition of facial expressions of underground coal mine personnel in the prior art.

[0174] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying facial expressions of people in coal mines, characterized in that: include: Acquire an initial video of a coal mine, wherein the initial video at least includes a target object; Preprocessing the initial video to obtain a target video, wherein the preprocessing method includes at least image enhancement and / or image denoising; Using a convolution algorithm to extract facial feature information in the target video; The YOLOv5s algorithm is used to identify the expression state corresponding to the facial feature information.

2. The method according to claim 1, characterized in that Preprocessing the initial video to obtain a target video includes: Extracting a dark environment area in the initial video, wherein the dark environment area is an area whose brightness is less than a preset brightness threshold; At least the dark environment area is subjected to histogram equalization processing to obtain the target video.

3. The method according to claim 1, characterized in that Preprocessing the initial video to obtain a target video includes: Extracting edge points in the initial video by using an edge detection algorithm; Deleting the edge points in the initial video to obtain an intermediate video; The contour of the target object in the intermediate video is highlighted to obtain the target video.

4. The method according to claim 1, characterized in that: After preprocessing the initial video to obtain the target video, the method further includes: Obtaining a first grayscale value of a first frame in the target video; Acquire a second grayscale value of a second frame in the target video, wherein the number of frames between the first frame and the second frame is a predetermined number of frames, and the first frame is earlier than the second frame; Calculating a difference between the first grayscale value and the second grayscale value to obtain a grayscale difference value; When the grayscale difference value is greater than or equal to a preset difference threshold, the second frame is determined to be a key frame.

5. The method according to claim 1, characterized in that A convolution algorithm is used to extract facial feature information in the target video, including: Constructing a feature extraction model, wherein the feature extraction model is trained by a convolution algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes a historical target video acquired within a historical time period and historical facial feature information corresponding to the historical target video; The target video is input into the feature extraction model to obtain the facial feature information corresponding to the target video.

6. The method according to claim 1, characterized in that The YOLOv5s algorithm is used to identify the facial expression state corresponding to the facial feature information, including: Constructing an expression recognition model, wherein the expression recognition model is trained by a YOLOv5s algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical facial feature information obtained within a historical time period and historical expression states corresponding to the historical facial feature information; The facial feature information is input into the expression recognition model to obtain the expression state corresponding to the facial feature information.

7. The method according to any one of claims 1 to 6, characterized in that: After using the YOLOv5s algorithm to identify the expression state corresponding to the facial feature information, the method further includes: When the currently recognized facial expression state is characterized as a negative state, continue to recognize the facial expression state; When the duration of the negative state characterized by the facial expression state is greater than or equal to the target duration, an alarm message is generated.

8. A device for identifying facial expressions of people in coal mines, characterized in that: include: A first acquisition unit is used to acquire an initial video of the coal mine, wherein the initial video at least includes a target object; A processing unit, configured to preprocess the initial video to obtain a target video, wherein the preprocessing method includes at least image enhancement and / or image denoising; An extraction unit, used to extract facial feature information from the target video using a convolution algorithm; The first recognition unit is used to use the YOLOv5s algorithm to recognize the expression state corresponding to the facial feature information.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for recognizing the facial expression state of personnel underground in a coal mine as described in any one of claims 1 to 7 are implemented.

10. A coal mine underground facial expression recognition system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the facial expression state recognition method of personnel underground in a coal mine as described in any one of claims 1 to 7.

Images