A coal mine underground personnel action recognition method applied in a low-illumination environment
By transforming and extracting video features in low-light environments using improved CycleGAN and I3D networks, and combining self-attention layers and action classification modules, the problems of low accuracy and high cost in action recognition of underground coal mine personnel in low-light environments are solved, achieving efficient action recognition.
Patent Information
- Application Number
- CN202211631477.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-19
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-12-19
AI Technical Summary
In low-light environments, the accuracy of personnel movement recognition in coal mines is low, making it difficult to accurately identify the movements of multiple workers. Existing methods are computationally expensive and time-consuming.
The style transfer module uses an improved CycleGAN network to convert low-light videos into normal-light videos. Combined with a global spatiotemporal feature extraction module, an object detection module, and an action classification module, feature extraction and action recognition are performed through an I3D network and a self-attention layer. The model is trained using a designed loss function.
It improves the accuracy of motion recognition in low-light environments, reduces the difficulty of data acquisition and computational costs, and achieves efficient end-to-end human motion recognition.
Smart Images
Figure CN115984958B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video recognition, specifically a method for recognizing the actions of personnel in coal mines under low-light conditions. Background Technology
[0002] Coal mines are characterized by a highly variable and complex environment, numerous work processes, a large number of workers, and massive, concentrated equipment, resulting in a high overall risk factor. Safety hazards can arise at any moment, making accidents prone to occur. Due to the extremely complex and harsh environment of coal mines, workers must endure long periods in this difficult and dangerous environment, and multiple complex factors can significantly impact their work. Therefore, without strong self-safety skills, coping with such a dangerous and challenging working environment will be extremely difficult.
[0003] The biggest challenge in recognizing personnel actions underground in coal mines lies in the low accuracy of action recognition under low lighting conditions and the poor performance in recognizing the actions of multiple workers. Essentially, personnel action recognition in coal mines can be viewed as an organic unification of video low-light enhancement and action recognition tasks. By enhancing the video in low light, richer semantic features can be obtained, thereby better completing the action recognition task.
[0004] Traditional dark lighting enhancement methods, such as deep networks based on Retinex theory and deep networks based on image fusion, can achieve dark lighting enhancement through illumination decomposition and multi-feature fusion of images. However, most of them require supervised learning with paired data, which requires a large amount of paired labeled data for model training, thus increasing the time and labor costs of model training.
[0005] Traditional action recognition methods, such as two-stream networks and 3D networks, can achieve action recognition through convolutional networks. However, two-stream networks require extracting the optical flow of video frames as input to the temporal flow network, which is a time-consuming and computationally resource-intensive task. 3D networks are effective in capturing short-range patterns within local spatiotemporal regions, but they cannot model spatiotemporal dependencies beyond their receptive domain. To address this issue, 3D networks need to apply a large number of 3D convolutional filters to all spatiotemporal locations in the video, which significantly increases the computational cost of 3D networks. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention proposes a method for recognizing personnel movements in coal mines under low-light conditions, solving the problem of inaccurate and rapid recognition of personnel movements caused by low lighting, large number of personnel, and complex environment in coal mines.
[0007] The objective of this invention is achieved as follows: a method for recognizing the actions of personnel in coal mines under low-light conditions, utilizing video data of personnel working underground in coal mines to achieve action recognition under low-light conditions; the action recognition method includes: a style transfer module, a global feature extraction module, a target detection module, an action classification module, and a regression module;
[0008] After resampling the video data, the low-light video data is converted to normal-light video data through the style conversion module; then the global feature extraction module extracts the global spatiotemporal features of the video frame sequence; then the object detection module detects people in the video and extracts the feature representation of the people's action pipeline from the video frame sequence; finally, the pipeline features are passed to the regression module and the action classification module respectively to calculate the position of each pipeline and the probability of belonging to each action.
[0009] Furthermore, the specific steps are as follows:
[0010] Step 1: Collect video data from underground cameras in the coal mine, resample the data, and obtain preprocessed video data;
[0011] Step 2: Pass the preprocessed video data obtained in Step 1 through the style conversion module to obtain normal lighting video data;
[0012] Step 3: Extract global spatiotemporal features from the normal lighting video data obtained in Step 2 using the global spatiotemporal feature extraction module;
[0013] Step 4: Pass the global spatiotemporal features obtained in Step 3 through the target detection module to obtain a set of specific pipeline features;
[0014] Step 5: Calculate the position of each pipeline and the probability of it belonging to each action by using the specific pipeline features obtained in Step 4 through the action classification module and the regression module;
[0015] Step 6: Divide the normal light video data obtained in Step 2 into training samples and test samples, and use the designed loss function to train the action recognition model of the application in low light environment.
[0016] Furthermore, in step 1, the acquired video data is resampled;
[0017] The method for resampling video data is as follows:
[0018] The acquired video data, used It means that l v u is the total number of frames in the video. t For the t-th RGB frame of video U, i.e. The video U is sampled into frames of size H0×W0×C, with T frames and C channels.
[0019] Furthermore, in step 2, the style conversion module is used to perform style conversion on the preprocessed video data U0 obtained in step 1, so as to convert the video under low light into video data U1 under normal light.
[0020] The style transfer module consists of an improved CycleGAN. This module is used to perform style transfer on video data, converting videos from low light conditions to videos from normal light conditions, enriching the information of people and scenes in the videos, and making detection and classification tasks more accurate.
[0021] CycleGAN is a method for learning to translate images from a source domain to a target domain. CycleGAN consists of two sets of generative adversarial networks (GANs); each GAN contains a generator and a discriminator; the generator and discriminator together form a translator, translating the image from the X domain to the Y domain and vice versa. It is implemented using two sets. The first set of GANs is a network that generates video images under normal lighting conditions from low-light video images from underground coal mines, including a generator network G. vt and discriminator network D vt The second group of generative adversarial networks generates video images of underground coal mines under dim light from video images under normal light conditions, including a generator network G. tv and discriminator network D tv ;
[0022] In the CycleGAN network structure, the generator's encoder-decoder network uses the feature extraction layer of the ResNet50 network as the encoder and the corresponding deconvolutional network as the decoder; the discriminator adopts the VGG19 network structure; the network output includes a branch for judging whether the image is real or fake and a branch for classification, which are used to generate images of the corresponding real modalities without changing the identity labels.
[0023] The loss function of the CycleGAN network consists of two parts, namely Loss = Loss GAN +Loss cycle Loss GAN This ensures that the generator and discriminator evolve together, thereby guaranteeing that the generator produces more realistic images; Loss cycle Ensure that the generator's output image has a different style than the input image, but the same content; Loss GAN and Loss cycle Specifically:
[0024]
[0025]
[0026] To ensure that the transferred image maintains similarity to the original image in terms of color and edge texture structure, a structural loss is added to the original CycleGAN network loss function, expressed as follows:
[0027] The structural loss is expressed as:
[0028]
[0029] The Here, is the structural similarity loss function, which measures the similarity between two images, and is expressed as:
[0030]
[0031] Where, μ x μ y σ is the expected value of the pixel. x , σ y σ is the variance of the pixel values. xy Here, c1 and c2 are covariances, and they are constants.
[0032] The This is a multi-scale structural similarity loss function that preserves image edges and details, expressed as:
[0033]
[0034] Where β m γ m This indicates the relative importance between two items.
[0035] Furthermore, in step 3, the normal light video data U1 obtained in step 2 is processed by the global spatiotemporal feature extraction module to extract global spatiotemporal features;
[0036] The global spatiotemporal feature extraction module uses an I3D network pre-trained on the Kinetics-400 dataset as its backbone network; spatiotemporal features are extracted from video data via this backbone network. Where T′ is the time dimension and C′ is the feature dimension.
[0037] Furthermore, in step 4, the global spatiotemporal features obtained in step 3 are processed by the target detection module to obtain a set of specific pipeline features;
[0038] The specific pipeline features are a sequence of predicted personnel boundaries detected frame by frame in the video, used to describe the predicted bounding boxes of personnel in each frame.
[0039] The target detection module consists of an encoder group and a decoder group;
[0040] The encoder group consists of M encoders connected in series. Its function is to perform spatiotemporal modeling of the global spatiotemporal features in step 3 and further extract spatiotemporal features. The first encoder input in the encoder group is the global spatiotemporal feature F from step 3. b The linear mapping superimposed with 3D positional encoding is represented as The input of the i-th encoder The output of the (i-1)th encoder The output of the last encoder is F en ;
[0041] The decoder group consists of M decoders connected in series, and its function is to generate a set of pipeline features F of the current video data. tub N is the number of pipes; the first decoder input of the decoder group is the output F of the encoder group. en and a set of randomly initialized query vectors Where N is the total number of query vectors, T out It's TQ i Number of frames included The input to the i-th decoder is the output F of the encoder group. en and the output of the (i-1)th decoder The output of the last decoder is F tub .
[0042] Furthermore, each encoder in the encoder group consists of a self-attention layer, two normalization layers, and a feedforward neural network;
[0043] Taking the i-th encoder as an example, where i = {1, ..., M}, the specific processing flow is as follows:
[0044] Step 4-1-1, input sequence e (i) First, a normalization layer is applied, mapping the data to three different matrices Q, K, and V.
[0045] Q = σ q (LN(e (i) ))
[0046] K = σ k (LN(e (i) ))
[0047] V = σ v (LN(e (i) ))
[0048] Where σ(*) is a linear transformation;
[0049] Step 4-1-2 involves inputting Q, K, and V into the self-attention layer to calculate the dot product attention, specifically as follows:
[0050]
[0051] Step 4-1-3: Perform residual join on SA(Q,K,V) to obtain z. ′(i) Specifically:
[0052] z ′(i) =SA(Q,K,V)+e (i)
[0053] Step 4-1-4, z ′(i) The output z of the i-th encoder is obtained through a multilayer perceptron. (i) Specifically:
[0054] z (i) =MLP(z) ′(i) )+z ′(i)
[0055] MLP(*) is a multilayer perceptron.
[0056] Each decoder in the decoder group consists of a spatiotemporal self-attention layer and a cross self-attention layer. The spatiotemporal self-attention layer consists of a spatial multi-head self-attention layer and a temporal multi-head self-attention layer.
[0057] Taking the i-th decoder as an example, where i = {1, ..., M}, the spatial multi-head self-attention layer is used to extract the spatial relationships of the input sequence. Its specific processing flow is as follows:
[0058] Step 4-2-1, input sequence d (i) These are mapped to three different matrices Q, K, and V, respectively:
[0059]
[0060]
[0061]
[0062] Where t={1,…,T out}, n={1,…,N}, a={1,…,A} are the indices of the multi-head self-attention heads;
[0063] Step 4-2-2, will The input is fed into the self-attention layer to compute the dot product attention, specifically as follows:
[0064]
[0065]
[0066]
[0067] Where α is the attention coefficient and m is the attention vector;
[0068] Step 4-2-3, will The output of the spatial multi-head self-attention layer is obtained through a multilayer perceptron. Specifically:
[0069]
[0070] The temporal multi-head self-attention layer is used to extract the temporal relationship of the input sequence, and its specific processing flow is as follows:
[0071] Step 4-3-1, sequence They are mapped to three different matrices Q, K, and V, respectively:
[0072]
[0073]
[0074]
[0075] Step 4-3-2, will The input is fed into the self-attention layer to compute the dot product attention, specifically as follows:
[0076]
[0077]
[0078]
[0079] Step 4-3-3, s ′(i) The output s of the temporal multi-head self-attention layer is obtained through a multi-layer perceptron. (i) Specifically:
[0080] s (i) =MLP(LN(s) ′(i) ))+s ′(i)
[0081] The input to the cross-self-attention layer is F. en s (i) The specific process can be represented as follows:
[0082]
[0083] r ′(i) =CA(F en ,F q )+s (i)
[0084] r (i) =MLP(r ′(i) )+r ′(i)
[0085] Where CA(*,*) is the cross-self-attention function, r (i) It is the output of the i-th decoder.
[0086] Furthermore, in step 5, the specific pipe feature F obtained in step 4 is... tub The position of each pipe and the probability of belonging to each action are calculated using the action classification module and the regression module, specifically:
[0087] The action classification module consists of a temporal pooling layer, a self-attention layer, a cross-attention layer, and a feedforward neural network. Its specific process is as follows:
[0088] Step 5-1, convert the global spatiotemporal features F b Through the self-attention layer, i.e., SA(F) b );
[0089] Step 5-2, for F tub Perform time pooling operations, i.e., Pool t (F tub );
[0090] Step 5-3, SA(F) b ) and Pool t (F tub Intermediate features are obtained through cross-self-attention layers. Specifically:
[0091] F c =CA(SA(F) b Pool t (F tub ))+Pool t (F tub )
[0092] Step 5-4, F c Through a feedforward neural network, we obtain Where y class This represents the score on L possible action labels.
[0093] The regression module consists of two feedforward neural networks, with specific pipeline features F as input. tub This is used to output the coordinates of personnel boundaries and exclude the influence of non-action boundaries on action classification. The specific processing flow is as follows:
[0094]
[0095]
[0096] Where N is the number of pipe vectors, T out It is the time dimension of the pipeline, y coor The coordinates of the personnel boundary, y action Indicates whether any action has occurred within the boundary.
[0097] Further, in step 6, the normal lighting video data obtained in step 2 is divided into training samples and test samples, and the designed loss function is used to train the action recognition model for the application in low-light environments, specifically as follows:
[0098] The training and test samples are divided as follows: the normal light video data obtained in step 2 is mixed and shuffled, and then appropriately divided into training and test samples according to the ratio.
[0099] The loss function includes the loss function for action classification tasks and the loss function for regression tasks, expressed as:
[0100]
[0101] Where y is the output prediction and Y is the actual data;
[0102] The action classification loss function It is the cross-entropy loss function, expressed as:
[0103]
[0104] Where G is the total number of action categories, g = {1, ..., G};
[0105] The loss function To remove the influence of non-action boundaries in the pipeline on action classification, a binary cross-entropy loss function is used.
[0106] The loss function of the regression task Composed of the L1 loss function and the IOU loss function, it is expressed as:
[0107]
[0108] Where n = {1, ..., N}, Let be the crossover ratio loss function, which represents The degree of overlap.
[0109] Beneficial effects: Due to the adoption of the above scheme, which includes five modules: style transfer module, spatiotemporal feature extraction module, object detection module, action classification module, and regression module. In the style transfer module, an improved CycleGAN network is used to perform style transfer on the video data, transforming the low-light videos from underground coal mines to normal lighting conditions. This enriches the information contained in the videos, making subsequent action classification and object detection tasks more accurate and eliminating the need for paired data training, thus reducing the difficulty of data acquisition. In the object detection module, object detection in the video is transformed into a problem of predicting a set of data, requiring no prior knowledge or subsequent processing, achieving end-to-end detection of people in the video. Attached Figure Description
[0110] Figure 1 This is a flowchart of a method for recognizing the movements of personnel in coal mines under low-light conditions, according to the present invention.
[0111] Figure 2 This is a structural diagram of a method for recognizing the movements of underground personnel in coal mines under low-light conditions, according to the present invention. Detailed Implementation
[0112] A method for recognizing personnel actions in coal mines under low-light conditions: This method utilizes video data of personnel working underground in coal mines to achieve personnel action recognition under low-light conditions; the action recognition method includes: a style transfer module, a global feature extraction module, a target detection module, an action classification module, and a regression module.
[0113] After resampling the video data, the low-light video data is converted into normal-light video data through the style conversion module; then the global feature extraction module extracts the global spatiotemporal features of the video frame sequence; then the object detection module detects people in the video and extracts the feature representation of the people's action pipeline from the video frame sequence; finally, the pipeline features are passed through the regression module and the action classification module to calculate the position of each pipeline and the probability of belonging to each action.
[0114] The specific steps are as follows:
[0115] Step 1: Collect video data from underground cameras in the coal mine, resample the data, and obtain preprocessed video data;
[0116] Step 2: Pass the preprocessed video data obtained in Step 1 through the style conversion module to obtain video data under normal lighting conditions;
[0117] Step 3: Extract global spatiotemporal features from the normal lighting video data obtained in Step 2 using the global spatiotemporal feature extraction module;
[0118] Step 4: Pass the global spatiotemporal features obtained in Step 3 through the target detection module to obtain a set of specific pipeline features;
[0119] Step 5: Calculate the position of each pipeline and the probability of it belonging to each action by using the specific pipeline features obtained in Step 4 through the action classification module and the regression module;
[0120] Step 6: Divide the normal light video data obtained in Step 2 into training samples and test samples, and use the designed loss function to train the action recognition model of the application in low light environment.
[0121] In step 1, the acquired video data is resampled;
[0122] The method for resampling video data is as follows:
[0123] The acquired video data, used It means that l v u is the total number of frames in the video. t For the t-th RGB frame of video U, i.e. The video U is sampled into frames of size H0×W0×C, with T frames and C channels.
[0124] In step 2, the style conversion module is used to perform style conversion on the preprocessed video U0 obtained in step 1, so as to convert the video under low light into the video U1 under normal light.
[0125] The style transfer module consists of an improved CycleGAN, which is a recurrent generative adversarial network. This module is used to perform style transfer on video data, converting videos from low light conditions to videos from normal light conditions, enriching the information of people and scenes in the videos, and making detection and classification tasks more accurate.
[0126] CycleGAN is a method for learning to translate images from a source domain to a target domain. CycleGAN consists of two sets of generative adversarial networks (GANs). Each GAN contains a generator and a discriminator. The generator and discriminator together form a translator, translating the image from the X domain to the Y domain and vice versa. It is implemented using two sets. The first set of GANs is a network that generates video images under normal lighting conditions from low-light video images from underground coal mines. This network includes a generator network G. vt and discriminator network D vt The second group of generative adversarial networks generates video images of underground coal mines under dim light from video images under normal light conditions, including a generator network G. tv and discriminator network D tv ;
[0127] In the CycleGAN network structure, the generator's encoder-decoder network uses the feature extraction layer of the ResNet50 network as the encoder and the corresponding deconvolutional network as the decoder; the discriminator adopts the VGG19 network structure; the network output includes a branch for judging whether the image is real or fake and a branch for classification, which are used to generate images of the corresponding real modalities without changing the identity labels.
[0128] The loss function of the CycleGAN network consists of two parts, namely Loss = Loss GAN +Loss cycle Loss GAN This ensures that the generator and discriminator evolve together, thereby guaranteeing that the generator produces more realistic images; Loss cycle Ensure that the generator's output image has a different style than the input image, but the same content; Loss GAN and Loss cycle Specifically:
[0129]
[0130]
[0131] To ensure that the transferred image maintains similarity to the original image in terms of color and edge texture structure, a structural loss is added to the original CycleGAN network loss function, expressed as follows:
[0132] The structural loss can be expressed as:
[0133]
[0134] The The structural similarity loss function measures the similarity between two images and can be expressed as:
[0135]
[0136] Where, μ x μ y σ is the expected value of the pixel. x , σ y σ is the variance of the pixel values. xy Let c1 and c2 be the covariances.
[0137] The The multi-scale structural similarity loss function preserves the edges and details of an image and can be expressed as:
[0138]
[0139] Where βm γ m This indicates the relative importance between two items.
[0140] In step 3, the normal light video data U1 obtained in step 2 is processed by the global spatiotemporal feature extraction module to extract global spatiotemporal features.
[0141] The global spatiotemporal feature extraction module uses an I3D network pre-trained on the Kinetics-400 dataset as its backbone network. Spatiotemporal features are extracted from video data using this backbone network. Where T′ is the time dimension and C′ is the feature dimension.
[0142] In step 4, the global spatiotemporal features obtained in step 3 are processed by the target detection module to obtain a set of specific pipeline features.
[0143] The specific pipeline features are a sequence of predicted personnel boundaries detected frame by frame in the video, used to describe the predicted bounding boxes of personnel in each frame.
[0144] The target detection module consists of an encoder group and a decoder group.
[0145] The encoder group consists of M encoders connected in series. Its function is to perform spatiotemporal modeling of the global spatiotemporal features in step 3 and further extract spatiotemporal features. The first encoder input in the encoder group is the global spatiotemporal feature F from step 3. b The linear mapping superimposed with 3D positional encoding is represented as The input of the i-th encoder The output of the (i-1)th encoder The output of the last encoder is F en .
[0146] The decoder group consists of M decoders connected in series, and its function is to generate a set of pipeline features F of the current video data. tub N is the number of pipes. The first decoder input in the decoder group is the output F of the encoder group. en and a set of randomly initialized query vectors Where N is the total number of query vectors, T out It's TQ i Number of frames included The input to the i-th decoder is the output F of the encoder group. en and the output of the (i-1)th decoder The output of the last decoder is F tub .
[0147] Each encoder in the encoder group consists of a self-attention layer, two normalization layers, and a feedforward neural network.
[0148] Taking the i-th encoder as an example, where i = {1, ..., M}, the specific processing flow is as follows:
[0149] Step 4-1-1, input sequence e (i) First, a normalization layer is applied, mapping the data to three different matrices Q, K, and V.
[0150] Q = σ q (LN(e (i) ))
[0151] K = σ k (LN(e (i) ))
[0152] V = σ v (LN(e (i) ))
[0153] Where σ(*) is a linear transformation;
[0154] Step 4-1-2 involves inputting Q, K, and V into the self-attention layer to calculate the dot product attention, specifically as follows:
[0155]
[0156] Step 4-1-3: Perform residual join on SA(Q,K,V) to obtain z′. (i) Specifically:
[0157] z′ (i) =SA(Q,K,V)+e (i)
[0158] Step 4-1-4, z′ (i) The output z of the i-th encoder is obtained through a multilayer perceptron. (i) Specifically:
[0159] z (i) =MLP(z′) (i) )+z ′(i)
[0160] MLP(*) is a multilayer perceptron.
[0161] Each decoder in the decoder group consists of a spatiotemporal self-attention layer and a cross self-attention layer, wherein the spatiotemporal self-attention layer is composed of a spatial multi-head self-attention layer and a temporal multi-head self-attention layer.
[0162] Taking the i-th decoder as an example, where i = {1, ..., M}, the spatial multi-head self-attention layer is used to extract the spatial relationships of the input sequence. Its specific processing flow is as follows:
[0163] Step 4-2-1, input sequence d (i) These are mapped to three different matrices Q, K, and V, respectively:
[0164]
[0165]
[0166]
[0167] Where t={1,…,T out}, n={1,…,N}, a={1,…,A} are the indices of the multi-head self-attention heads;
[0168] Step 4-2-2, will The input is fed into the self-attention layer to compute the dot product attention, specifically as follows:
[0169]
[0170]
[0171]
[0172] Where α is the attention coefficient and m is the attention vector;
[0173] Step 4-2-3, will The output of the spatial multi-head self-attention layer is obtained through a multilayer perceptron. Specifically:
[0174]
[0175] The temporal multi-head self-attention layer is used to extract the temporal relationship of the input sequence, and its specific processing flow is as follows:
[0176] Step 4-3-1, sequence They are mapped to three different matrices Q, K, and V, respectively:
[0177]
[0178]
[0179]
[0180] Step 4-3-2, will The input is fed into the self-attention layer to compute the dot product attention, specifically as follows:
[0181]
[0182]
[0183]
[0184] Step 4-3-3, s′ (i) The output s of the temporal multi-head self-attention layer is obtained through a multi-layer perceptron. (i) Specifically:
[0185]
[0186] The input to the cross-self-attention layer is F. en s (i) The specific process can be represented as follows:
[0187]
[0188] r′ (i) =CA(F en ,F q )+s (i)
[0189] r (i) =MLP(r′) (i) )+r′ (i)
[0190] Where CA(*,*) is the cross-self-attention function, r (i) It is the output of the i-th decoder.
[0191] In step 5, the specific set of pipe features F obtained in step 4 are... tub The position of each pipe and the probability of belonging to each action are calculated using the action classification module and the regression module, specifically:
[0192] The action classification module consists of a temporal pooling layer, a self-attention layer, a cross-attention layer, and a feedforward neural network. Its specific process is as follows:
[0193] Step 5-1, convert the global spatiotemporal features F b Through the self-attention layer, i.e., SA(F) b );
[0194] Step 5-2, for F tub Perform time pooling operations, i.e., Pool t (F tub );
[0195] Step 5-3, SA(F) b ) and Poolt (F tub Intermediate features are obtained through cross-self-attention layers. Specifically:
[0196] F c =CA(SA(F) b Pool t (F tub ))+Pool t (F tub )
[0197] Step 5-4, F c Through a feedforward neural network, we obtain Where y class This represents the score on L possible action labels.
[0198] The regression module consists of two feedforward neural networks, with F as the input. tub This is used to output the coordinates of personnel boundaries and exclude the influence of non-action boundaries on action classification. The specific processing flow is as follows:
[0199]
[0200]
[0201] Where N is the number of pipe vectors, T out It is the time dimension of the pipeline, y coor The coordinates of the personnel boundary, y action Indicates whether any action has occurred within the boundary.
[0202] In step 6, the normal lighting video data obtained in step 2 is divided into training samples and test samples. The designed loss function is used to train the action recognition model for the application in low-light environments. Specifically:
[0203] The training and test samples are divided as follows: the normal light video data obtained in step 2 is mixed and shuffled, and then appropriately divided into training and test samples according to the ratio.
[0204] The loss function includes the loss function for action classification tasks and the loss function for regression tasks, expressed as:
[0205]
[0206] Where y is the output prediction and Y is the actual data.
[0207] The action classification loss function It is the cross-entropy loss function, expressed as:
[0208]
[0209] Where G is the total number of action categories, g = {1, ..., G}.
[0210] The loss function To remove the influence of non-action boundaries in the pipeline on action classification, a binary cross-entropy loss function is used.
[0211] The loss function of the regression task Composed of the L1 loss function and the IOU loss function, it is expressed as:
[0212]
[0213] Where n = {1, ..., N}, Let be the crossover ratio loss function, which represents The degree of overlap.
[0214] The following is in conjunction with the appendix Figure 1-2 Specific embodiments of the present invention are described to enable those skilled in the art to understand the invention. However, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0215] A method for recognizing personnel movements in coal mines under low-light conditions, using a real underground video dataset from a coal mine, involves collecting 45 days of video data, with the collected video files totaling 800 GB in size. The method includes the following steps:
[0216] Step 1: Collect video data from underground cameras in the coal mine, resample the data, and obtain preprocessed video data;
[0217] Step 2: Pass the video data obtained in Step 1 through the style conversion module to obtain video data under normal lighting conditions;
[0218] Step 3: Extract spatiotemporal features from the video data obtained in Step 2 using the global spatiotemporal feature extraction module;
[0219] Step 4: Pass the global spatiotemporal features obtained in Step 3 through the target detection module to obtain a set of specific pipeline features;
[0220] Step 5: Calculate the position of each pipeline and the probability of it belonging to each action by using the specific pipeline features obtained in Step 4 through the action classification module and the regression module;
[0221] Step 6: Divide the video data into training samples and test samples in a 7:3 ratio, and use the designed loss function to train the action recognition model for the application in low-light environments.
[0222] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A coal mine underground personnel action recognition method applied in a low-illumination environment, characterized in that: The method uses video data of personnel operation in a coal mine to realize personnel action recognition in a low-illumination environment; the action recognition method comprises a style conversion module, a global feature extraction module, a target detection module, an action classification module and a regression module; After resampling the video data, the low-illumination video data is converted into normal-light video data through the style conversion module; then the global spatio-temporal features of the video frame sequence are extracted through the global feature extraction module; then the personnel in the video are detected through the target detection module, and the feature representation of the personnel action pipeline is extracted from the video frame sequence; finally, the pipeline features are respectively input into the regression module and the action classification module to calculate the position of each pipeline and the probability of belonging to each action; The specific steps are as follows: Step 1: collect the video data collected by the camera in the coal mine, resample the video data to obtain preprocessed video data; Step 2: the preprocessed video data obtained in step 1 is input into the style conversion module to obtain normal-light video data; Step 3: the normal-light video data obtained in step 2 is input into the global spatio-temporal feature extraction module to extract global spatio-temporal features; Step 4: the global spatio-temporal features obtained in step 3 are input into the target detection module to obtain a specific pipeline feature; Step 5: the specific pipeline feature obtained in step 4 is input into the action classification module and the regression module to calculate the position of each pipeline and the probability of belonging to each action; Step 6: the normal-light video data obtained in step 2 is divided into training samples and test samples, and a designed loss function is used to train the action recognition model for the application in the low-illumination environment; In step 2, the preprocessed video data U0 obtained in step 1 is input into the style conversion module to convert the dark-light video into normal-light video U1; The style conversion module is composed of an improved CycleGAN, which is used to convert the dark-light video into normal-light video, enrich the information of the characters and scenes in the video, and make the detection task and the classification task more accurate; CycleGAN is a method for learning to convert images from a source domain to a target domain; CycleGAN comprises two sets of generative adversarial networks; each set of generative adversarial networks comprises a generator and a discriminator; the generator and the discriminator constitute a translator for translating images from an X domain to a Y domain, and vice versa; two sets are adopted; the first set of generative adversarial networks is a network for generating video images under normal light from video images under dark light in a coal mine, comprising a generator network G vt and a discriminator network D vt ; the second set of generative adversarial networks is a network for generating video images under dark light in a coal mine from video images under normal light, comprising a generator network G tv and a discriminator network D tv ; In the network structure of CycleGAN, the encoding-decoding structure network of the generator uses the feature extraction layer of ResNet50 network as the encoding network and uses the corresponding deconvolution network as the decoding network; the discriminator adopts the VGG19 network structure; the output of the network includes a branch for judging true and false and a branch for classification, which is used to generate real corresponding modal images and the identity label does not change; The loss function of the CycleGAN network is composed of two parts, i.e. Loss = Loss GAN + Loss cycle , wherein Loss GAN ensures the mutual evolution of the generator and the discriminator, thereby ensuring that the generator can generate more realistic pictures; Loss cycle ensures that the output picture of the generator is different in style and the same in content with the input picture; Loss GAN and Loss cycle are specific to: In order to ensure that the migrated image and the original image are similar in color and edge texture structure after migration, a structure loss is added to the original CycleGAN network loss function, and is represented as The structure loss can be expressed as: The is a structural similarity loss function, which can measure the similarity of two images, and can be expressed as: where μ x , σ y are the mean, σ x , σ y are the variance, σ xy is the covariance, and c1, c2 are constants. The For multi-scale structural similarity loss function, it can preserve the edges and details of the image, which can be expressed as: where β m , γ m denote the relative importance between the two terms.
2. The mine underground personnel action recognition method applied in a low-illumination environment according to claim 1, characterized in that: In step 1, the collected video data is resampled; The method for resampling the video data is as follows: The acquired video data, used It means that l v u is the total number of frames in the video. t For the t-th RGB frame of video U, i.e. The video U is sampled into frames of size H0×W0×C, with T frames and C channels.
3. The mine personnel action recognition method applied in a low-illumination environment according to claim 1, characterized in that: In step 3, the normal-light video data U1 obtained in step 2 is input into the global spatio-temporal feature extraction module to extract global spatio-temporal features; The global space-time feature extraction module uses a backbone network which is an I3D network pre-trained on a Kinectics-400 dataset; video data is subjected to space-time feature extraction by the backbone network where T' is the time dimension and C' is the feature dimension.
4. The mine personnel action recognition method applied in a low-illumination environment according to claim 1, characterized in that: In step 4, the global spatio-temporal features obtained in step 3 are input into the target detection module to obtain a specific pipeline feature; The specific pipeline feature is a sequence composed of personnel boundary prediction detected from video frame images, which is used to describe the boundary box prediction of the personnel in each frame; The target detection module is composed of an encoder group and a decoder group; The encoder group consists of M encoders in series, which acts on the global spatio-temporal features in step 3 to model the spatio-temporal features and further extract the spatio-temporal features The input of the first encoder of the encoder group is the global spatio-temporal feature F in step 3 b The linear mapping of the global spatio-temporal feature F in step 3 is superimposed with the 3D position encoding, denoted as The input of the i-th encoder The output of the i-th encoder is The output of the last encoder is F en ; The decoder set consists of M decoders in series, whose role is to generate a set of pipe features F for the current video data tub , N is the number of pipes; the input of the first decoder of the decoder set is the output of the encoder set F en and a set of randomly initialized query vectors where N is the total number of query vectors, T out is the number of frames contained in TQ i , The input of the i-th decoder is the output of the encoder set F en and the output of the i-1-th decoder The output of the last decoder is F tub .
5. The coal mine underground personnel action recognition method applied in a low-illumination environment according to claim 4, characterized in that: Each encoder of the encoder group is composed of a self-attention layer, two normalization layers and a feedforward neural network; Taking the i-th encoder as an example, i = {1,..., M}, the specific processing procedure is as follows: Step 4-1-1, map the input sequence e (i) First, through the normalization layer, respectively mapped to three different matrix Q, K, V, Q = σ q (LN(e (i) )) K = σ k (LN(e (i) )) V = σ v (LN(e (i) )) Where σ (*) is a linear transformation; Step 4-1-2, input Q, K and V into the self-attention layer to calculate the dot product attention, specifically: Step 4-1-3, residual connection of SA(Q, K, V) to get z' (i) Specifically, z' (i) = SA(Q, K, V) + e (i) Step 4-1-4, z' = z + a * (z - z') (i) The output z of the i-th encoder is obtained by a multi-layer perception (i) , specifically: z (i) = MLP(z' + z (i) )+z' (i) Where MLP (*) is a multi-layer perceptron; Each decoder of the decoder group is composed of a space-time self-attention layer and a cross-attention layer, and the space-time self-attention layer is composed of a spatial multi-head self-attention layer and a temporal multi-head self-attention layer; Taking the i-th decoder as an example, i = {1,..., M}, the spatial multi-head self-attention layer is used to extract the spatial relationship of the input sequence, and the specific processing procedure is as follows: Step 4-2-1, the input sequence d (i) , is mapped into three different matrices Q, K, V, respectively, as follows: where t = {1,..., T out}, n = {1,..., N}, a = {1,..., A} is the sequence number of multi-head self-attention heads. Step 4 - 2 - 2, to Input to the self-attention layer computes dot-product attention, specifically: Where a is the attention coefficient, and m is the attention vector; Step 4-2-3, to By the multi-layer perception, the output of the spatial multi-head self-attention layer is obtained Specifically: The temporal multi-head self-attention layer is used to extract the temporal relationship of the input sequence, and the specific processing procedure is as follows: Step 4-3-1, the sequence are mapped to three different matrices Q, K, V, respectively, as follows: Step 4 - 3 - 2, to Input to the self-attention layer computes dot-product attention, specifically: Step 4-3-3, s' = s + s (i) Through the multi-layer perception, the output s of the time multi-head self-attention layer is obtained (i) , Specifically: s (i) = MLP(LN(s' + s) + s') + s' (i) (i) The input of the cross self-attention layer is F en , s (i) , and the specific flow can be represented as: r' (i) = CA(F en , F q ) + s (i) r (i) = MLP(r' - r) + r' (i) )+ r' (i) where CA(*,*) is the cross-attention function, r (i) is the output of the i-th decoder.
6. The mine personnel action recognition method applied in a low-illumination environment according to claim 1, characterized in that: The step 5, the specific pipe feature F obtained from step 4 tub Through the action classification module and the regression module, the position of each pipe and the probability belonging to each action are calculated, specifically: The action classification module is composed of a temporal pooling layer, a self-attention layer, a cross-attention layer and a feedforward neural network, and the specific procedure is as follows: Step 5-1, global spatio-temporal features F b by a self-attention layer, i.e. SA(F b ); Step 5-2, to F tub Perform a temporal pooling operation, i.e. Pool t (F tub ); Step 5-3, SA(F b ) and Poll t (F tub ) pass through a cross self-attention layer to obtain intermediate features Specifically: F c = CA(SA(F b ), Pool t (F tub ))+ Pool t (F tub ) Step 5 - 4, F c By the feedforward neural network, obtain where y class denotes the score on L possible action labels; The regression module consists of two feed-forward neural networks, with inputs being specific pipe features F tub , for outputting the coordinates of the person boundary and excluding the influence of non-motion boundary on motion classification, and the specific processing procedure is as follows: where N is the number of pipe vectors, T out is the time dimension of the pipe, y coor represents the coordinates of the person boundary, y action represents whether an action occurred in the boundary.
7. The coal mine underground personnel action recognition method applied in a low-illumination environment according to claim 1, characterized in that: Step 6, the normal light video data obtained in step 2 is divided into training samples and test samples, and the designed loss function is used to train the action recognition model applied in a low-illumination environment, specifically: The training sample and test sample division method is: the normal light video data obtained in step 2 is shuffled, and is divided into training samples and test samples in proportion; The loss function includes the loss function of the action classification task and the loss function of the regression task, and is expressed as: Where y is the output prediction, and Y is the true data; The action classification loss function is a cross-entropy loss function, denoted as: Where G is the total number of action categories, g = {1,..., G}; The loss function To remove the influence of non-action boundary in the pipeline on action classification, a binary cross-entropy loss function is adopted. a loss function of the regression task consisting of a L1 loss function and an IOU loss function, denoted as: where n = {1,..., N}, is the intersection over union loss function, which represents the degree of overlap of the two bounding boxes.