A multi-modal contrast learning guided coal mine underground skeletal action recognition method

Abstract

The application discloses a kind of multi-modal contrast learning guide's coal mine underground skeletal action recognition method, it is related to computer vision technical field.The method obtains the visible light and infrared double-mode video data of coal mine underground synchronous acquisition, utilizes double-flow OpenPose network to extract visible light skeleton sequence and infrared skeleton sequence;Two isomeric views of the same action instance are formed by double-mode skeleton sequence, modeling is carried out by double-flow time sequence encoder based on Mamba, and multi-modal InfoNCE contrast loss function is constructed after mapping by non-linear projection head, and the time sequence encoder is pre-trained on unlabeled data;Double-mode features are extracted by loading pre-training encoder, cross attention fusion is carried out with infrared features as query, visible light features as key and value, after splicing classification mark, it is interacted by Transform coding layer, and input classification head is completed action classification.The application solves the problem of low recognition accuracy of coal mine underground skeletal action under small sample condition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for recognizing skeletal movements in underground coal mines. Background Technology

[0002] Personnel motion recognition in coal mines is a crucial technology for ensuring safety during underground operations. Existing skeletal motion recognition methods are mostly based on convolutional neural networks or graph convolutional networks, achieving good recognition results under normal lighting conditions. However, they face significant challenges in the limited-sample learning environment of underground coal mines. On one hand, specific dangerous actions occur infrequently underground and are difficult to collect, resulting in an extremely limited number of labeled action samples. On the other hand, action labeling requires professional safety personnel to calibrate frame by frame, which is costly and time-consuming. Under limited-sample conditions, deep learning models using fully supervised training are prone to overfitting, exhibiting severely degraded generalization ability and failing to adapt to the changing actions of different workers and personnel in different underground work areas. Existing methods typically employ data augmentation or transfer learning to alleviate the limited-sample problem, but data augmentation generates samples with limited diversity, and transfer learning relies on large-scale source domain data and suffers from domain shift issues. Neither approach fundamentally solves the performance bottleneck caused by the scarcity of underground action samples.

[0003] Therefore, there is an urgent need for a technical solution that can achieve high-precision skeletal motion recognition with a small amount of labeled data. Summary of the Invention

[0004] This invention provides a multimodal contrastive learning-guided method for skeletal action recognition in coal mines. The method includes acquiring visible light and infrared dual-modal video data simultaneously collected in a coal mine; using a dual-stream OpenPose network to infer the visible light and infrared images respectively to obtain visible light and infrared skeleton sequences; and further includes the following steps: constructing the visible light and infrared skeleton sequences into two heterogeneous views of the same action instance; performing long-term temporal modeling on the visible light and infrared skeleton sequences respectively using a Mamba-based dual-stream temporal encoder to extract visible light and infrared skeleton feature sequences; mapping the visible light and infrared skeleton feature sequences to a low-dimensional latent feature space using a nonlinear projection head to obtain visible light and infrared low-dimensional latent features; constructing a multimodal InfoNCE contrastive loss function; and comparing the visible light low-dimensional latent features with the infrared low-dimensional latent features of the same action instance. The latent features in each batch are positive samples, and the remaining low-dimensional latent features in the batch are negative samples. Bidirectional contrast loss is calculated for the visible-to-infrared direction and the infrared-to-visible direction. A dual-stream temporal encoder is pre-trained by minimizing the total contrast loss. After pre-training, the dual-stream temporal encoder is retained, and the nonlinear projection head is removed. The pre-trained dual-stream temporal encoder is loaded, and visible light and infrared skeleton feature sequences are extracted separately. Using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, feature fusion is performed through a cross-attention mechanism. After normalization and weighted aggregation, residual connections are superimposed to generate an enhanced infrared feature sequence. A learnable classification label is concatenated at the beginning of the enhanced infrared feature sequence. The feature sequence after concatenation with the classification label is input into the Transformer encoding layer for feature interaction. The interaction-derived classification label is input into the classification head, and the action category probability distribution is output. Supervised fine-tuning is performed using the cross-entropy loss function.

[0005] Preferably, the step of using a dual-stream OpenPose network to infer visible light and infrared images respectively to obtain visible light skeleton sequences and infrared skeleton sequences includes the following steps: using a visible light OpenPose network and an infrared OpenPose network with mutually independent weights to process the visible light video stream and the infrared video stream respectively; both the visible light OpenPose network and the infrared OpenPose network use ResNet as a deep convolutional backbone network to extract multi-scale feature maps and output a two-dimensional confidence map and a partial affinity domain; calculating line integrals based on the partial affinity domain to measure the correlation strength between candidate keypoints; assembling discrete keypoints into a human skeleton structure through a greedy parsing algorithm and bipartite graph matching, and outputting visible light skeleton sequences and infrared skeleton sequences.

[0006] More preferably, the Mamba-based dual-stream temporal encoder includes a visible light Mamba encoder and an infrared Mamba encoder. Both the visible light Mamba encoder and the infrared Mamba encoder are composed of N cascaded Mamba blocks. Each Mamba block includes a layer normalization unit, a first linear transformation layer, a one-dimensional convolutional layer, a first activation function layer, a state space model unit, a second linear transformation layer, a second activation function layer, and a third linear transformation layer. After being normalized by the layer normalization unit, the input sequence is divided into two branches. The first branch is processed sequentially by the first linear transformation layer, the one-dimensional convolutional layer, and the first activation function layer before being input to the state space model unit. The state space model unit generates selective scanning parameters based on the input sequence and performs temporal modeling on the input sequence using a selective scanning algorithm. The second branch is processed by the second linear transformation layer and the second activation function layer, and then multiplied element-wise with the output of the state space model unit. The multiplication result is transformed by the third linear transformation layer and then residual-connected with the input sequence to output a feature sequence.

[0007] Further preferably, the construction of the multimodal InfoNCE contrast loss function, using the visible light low-dimensional latent features and infrared low-dimensional latent features of the same action instance as positive sample pairs, and the remaining low-dimensional latent features in the batch as negative samples, calculates the bidirectional contrast loss in the visible light to infrared direction and the infrared to visible light direction. The pre-training of the dual-stream temporal encoder by minimizing the total contrast loss includes the following steps: For a training batch containing N action samples, the visible light low-dimensional latent features and infrared low-dimensional latent features of the i-th action sample form a positive sample pair, and the remaining 2(N-1) low-dimensional latent features in the batch are used as negative samples; the contrast loss in the visible light to infrared direction is calculated, using the visible light low-dimensional latent features of the i-th action sample as the benchmark, with the numerator being the temperature coefficient normalized similarity between the visible light low-dimensional latent features and the corresponding infrared low-dimensional latent features of the i-th action sample. The exponent value is calculated as follows: the denominator is the sum of the temperature-normalized exponent values ​​of the similarity between the visible low-dimensional latent features of the i-th action sample and all infrared low-dimensional latent features in the batch; the contrast loss in the infrared-to-visible direction is calculated using the infrared low-dimensional latent features of the i-th action sample as a benchmark, the numerator is the temperature-normalized exponent value of the similarity between the infrared low-dimensional latent features of the i-th action sample and the corresponding visible low-dimensional latent features, and the denominator is the sum of the temperature-normalized exponent values ​​of the similarity between the infrared low-dimensional latent features of the i-th action sample and all visible low-dimensional latent features in the batch; the total contrast loss is the average of the contrast loss in the visible-to-infrared direction and the contrast loss in the infrared-to-visible direction, and the total contrast loss is minimized through backpropagation and gradient update, and a pre-trained dual-stream temporal encoder is used; the temperature coefficient is a hyperparameter used to adjust the smoothness of the feature distribution.

[0008] More preferably, the nonlinear projection head includes a visible light projection head and an infrared projection head, with independent weights for the two. Both the visible light projection head and the infrared projection head consist of a first fully connected layer, an activation function layer, and a second fully connected layer. The first fully connected layer maps the skeletal feature sequence to intermediate-dimensional features, the activation function layer performs a nonlinear transformation on the intermediate-dimensional features, and the second fully connected layer maps the nonlinearly transformed intermediate-dimensional features to a low-dimensional latent feature space, outputting low-dimensional latent features.

[0009] Further preferably, the step of using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, performing feature fusion through a cross-attention mechanism, and generating an enhanced infrared feature sequence after normalization, weighted aggregation, and residual connection, includes the following steps: linearly transforming the infrared skeleton feature sequence into a query matrix using a first learnable weight matrix; linearly transforming the visible light skeleton feature sequence into a key matrix using a second learnable weight matrix; linearly transforming the visible light skeleton feature sequence into a value matrix using a third learnable weight matrix; calculating the matrix product of the transpose of the query matrix and the key matrix, dividing the matrix product by a scaling factor, and inputting it into a Softmax function for normalization to obtain an attention weight matrix; multiplying the attention weight matrix with the value matrix to obtain a cross-attention output feature sequence; and performing a residual connection between the cross-attention output feature sequence and the original infrared skeleton feature sequence to generate an enhanced infrared feature sequence; the cross-attention mechanism is a unidirectional cross-attention mechanism, retrieving the visible light skeleton feature sequence only using the infrared skeleton feature sequence as the query; the scaling factor is the square root of the feature dimension of the key matrix.

[0010] Further preferably, the step of splicing a learnable classification label at the beginning of the enhanced infrared feature sequence and inputting the feature sequence after splicing the classification label into the Transformer encoding layer for feature interaction includes the following steps: initializing a learnable classification label with the same feature dimension as the enhanced infrared feature sequence; updating the parameters of the learnable classification label through backpropagation during model training; splicing the learnable classification label at the beginning of the time dimension of the enhanced infrared feature sequence to form an input sequence containing the classification label and the enhanced infrared feature sequence; inputting the input sequence into the Transformer encoding layer, which consists of a multi-head self-attention sub-layer, a feedforward neural network sub-layer, layer normalization, and residual connections; the classification label interacts with the enhanced infrared features at all time steps in the input sequence through the multi-head self-attention sub-layer, and outputting a feature sequence containing the interacted classification label.

[0011] Further preferably, the step of extracting the post-interaction classification label as input to the classification head, outputting the action category probability distribution, and using the cross-entropy loss function for supervised fine-tuning includes the following steps: extracting the post-interaction classification label located at the beginning of the time dimension from the feature sequence output by the Transformer encoding layer; inputting the post-interaction classification label into the classification head composed of fully connected layers; the fully connected layers map the feature dimension of the classification label to the action category number dimension; and outputting the probability distribution of each action category after normalization by the Softmax function; calculating the cross-entropy loss based on the labeled action category label and the predicted action category probability distribution; and performing supervised fine-tuning of the dual-stream temporal encoder, the cross-attention fusion module, the Transformer encoding layer, and the classification head through backpropagation and gradient update.

[0012] A further preferred embodiment includes a low-confidence recognition result correction step: using a supervised fine-tuned model to infer the bimodal skeletal sequence to be recognized, outputting the probability distribution and corresponding confidence of each action category; when the confidence of the recognition result is lower than a preset threshold, the selective scanning mechanism of the Mamba encoder is activated, dynamically weighting the skeletal sequence of multiple consecutive frames according to the action context information, and inferring the movement trend of the occluded part by analyzing the motion trajectory changes of key joints on the time axis; using the infrared skeletal feature sequence as the query and the visible light skeletal feature sequence as the key and value, the feature is re-fused through a cross-attention mechanism, and the cross-attention output feature sequence is residually fused with the original infrared skeletal feature sequence to recalculate the fused feature sequence; the classification label is extracted from the recalculated fused feature sequence, and the corrected action category probability distribution is input into the classification head again; if the corrected confidence increases to above the preset threshold, the correction result is adopted; if the corrected confidence does not reach the preset threshold, it is marked as awaiting manual review.

[0013] Further preferably, the acquisition of visible light and infrared dual-modal video data synchronously collected underground in the coal mine includes the following steps: synchronously acquiring visible light video streams and infrared thermal imaging video streams using a dual-spectrum camera deployed in the underground working area of ​​the coal mine, aligning the visible light video streams and infrared thermal imaging video streams on the time axis; performing frame decoding on the acquired visible light video streams and infrared thermal imaging video streams, judging the image quality frame by frame, removing invalid frames with missing images or environmental noise exceeding the noise threshold, and retaining valid frames to form a visible light image sequence and an infrared image sequence.

[0014] Technical effects: This invention constructs two heterogeneous views of the same action instance using visible light and infrared skeleton sequences. A Mamba-based dual-stream temporal encoder is employed for long-term temporal modeling, and self-supervised pre-training on unlabeled data is performed using the multimodal InfoNCE contrastive loss function. This enables the encoder to learn robust action representations consistent across modalities, effectively mitigating the performance degradation problem of supervised learning under small sample conditions. In the downstream classification stage, a one-way cross-attention fusion is performed using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value. Residual connections are combined to generate enhanced infrared feature sequences with complementary features. Finally, global contextual information is aggregated through learnable classification labels to complete action classification, improving the accuracy and robustness of skeleton action recognition in the complex environment of underground coal mines. Attached Figure Description

[0015] Figure 1 A flowchart of a method for skeletal motion recognition in underground coal mines; Figure 2 This invention relates to a multimodal contrastive learning-guided skeletal motion recognition algorithm for underground coal mines.

[0016] Figure 3 This is a diagram of the self-supervised multimodal contrastive pre-training stage.

[0017] Figure 4 This is a diagram of the downstream feature fusion and action classification stages. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Current methods suffer from localization drift under single-modal illumination, and labeling small samples is costly.

[0020] Please see Figures 1-4This invention provides a method for skeletal action recognition in coal mines. It constructs two heterogeneous views of the same action instance using visible light and infrared skeletal sequences. A Mamba-based dual-stream temporal encoder is used to perform long-term temporal modeling on both sequences, extracting visible light and infrared skeletal feature sequences. A nonlinear projection head maps these sequences to low-dimensional latent feature spaces, obtaining visible light and infrared low-dimensional latent features. A multimodal InfoNCE contrastive loss function is constructed, using the visible light and infrared low-dimensional latent features of the same action instance as positive sample pairs and other low-dimensional latent features within the batch as negative samples. The visible light to infrared direction and the infrared to visible light direction are then calculated. The bidirectional contrast loss is used to pre-train a dual-stream temporal encoder by minimizing the total contrast loss. After pre-training, the dual-stream temporal encoder is retained while the nonlinear projection head is removed. The pre-trained dual-stream temporal encoder is loaded, and visible light skeleton feature sequences and infrared skeleton feature sequences are extracted separately. Using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, feature fusion is performed through a cross-attention mechanism. After normalization and weighted aggregation, residual connections are superimposed to generate an enhanced infrared feature sequence. A learnable classification label is concatenated at the beginning of the enhanced infrared feature sequence. The feature sequence after concatenation with the classification label is input into the Transformer encoding layer for feature interaction. The classification label after interaction is extracted and input into the classification head, and the action category probability distribution is output. The cross-entropy loss function is used for supervised fine-tuning.

[0021] The method is implemented by a processor deployed in an underground coal mine monitoring system. After receiving video data from a dual-spectrum camera, the processor sequentially performs three stages of computation: skeleton extraction, contrast pre-training, and fusion classification. In the dual-modal skeleton extraction stage, the processor inputs visible light and infrared images into a dual-stream OpenPose network. The dual-stream OpenPose network contains two independent inference branches. Each branch transforms the input pixel-level image into a skeleton sequence composed of two-dimensional joint coordinates. The visible light skeleton sequence is denoted as... The infrared skeleton sequence is denoted as ,in For sequence length, and The first The visible light skeleton point coordinate vectors and infrared skeleton point coordinate vectors are used to represent the skeleton sequences of the two modalities, which are corresponding one-to-one according to the frame number. During the self-supervised contrastive pre-training phase, the processor uses the visible light skeleton sequences of the same action instance... and infrared skeleton sequence Input-based Mamba dual-stream timing encoder, visible light Mamba encoder right Perform long-term modeling and output visible light skeleton feature sequences. Infrared Mamba encoder right Perform long-term modeling and output infrared skeleton feature sequences , and The feature dimensions are all The visible light skeleton feature sequence and infrared skeletal feature sequences Input visible light projection head respectively and infrared projection head The projection head maps high-dimensional features to a low-dimensional latent feature space, outputting visible light low-dimensional latent features. and infrared low-dimensional potential features , and All dimensions are The processor constructs a multimodal InfoNCE contrastive loss function, for which... The training batch of the nth action sample, the nth one sample and Forming positive sample pairs, the rest within the batch Using low-dimensional latent features as negative samples, calculate the contrast loss from visible light to infrared light respectively. Contrast loss from infrared to visible light The total contrastive loss is the average of the losses in the two directions. The processor calculates the gradient of the total contrastive loss with respect to the encoder and projector parameters using the backpropagation algorithm, updates the parameters using gradient descent, and iterates until the loss converges to complete pre-training. After pre-training, the processor retains... and Weight, delete and In the downstream fusion and classification stage, the processor loads the pre-trained... and The visible light skeleton sequence and infrared skeleton sequence to be classified were extracted separately. and ,Will As a query, The cross-attention fusion module takes the key and value as inputs, calculates the similarity between the query and the key to obtain attention weights, and then aggregates the values ​​using these attention weights. Residual connection, outputting enhanced infrared feature sequence The processor is in Time-dimension head-seam learning classification tags The concatenated sequence is input into the Transformer encoding layer, which uses multi-head self-attention to... With all time steps The system performs information interaction and outputs the resulting classification labels. The processor inputs these labels into a fully connected classification head, performs Softmax normalization, outputs the action category probability distribution, and fine-tunes the network parameters based on the labeled cross-entropy loss. This scheme achieves underground action recognition in coal mines under small sample conditions through contrastive pre-training of dual-modal skeleton sequences and infrared-guided cross-attention fusion.

[0022] Existing OpenPose features intermodal feature interference when sharing weights.

[0023] Based on this, the method of using a dual-stream OpenPose network to infer visible light and infrared images respectively to obtain visible light skeleton sequences and infrared skeleton sequences includes the following steps: using a visible light OpenPose network and an infrared OpenPose network with mutually independent weights to process the visible light video stream and the infrared video stream respectively. Both the visible light OpenPose network and the infrared OpenPose network use ResNet as a deep convolutional backbone network to extract multi-scale feature maps and output a two-dimensional confidence map and a partial affinity domain; calculating line integrals based on the partial affinity domain to measure the correlation strength between candidate keypoints; assembling discrete keypoints into a human skeleton structure through a greedy parsing algorithm and bipartite graph matching, and outputting visible light skeleton sequences and infrared skeleton sequences.

[0024] The visible light OpenPose network and the infrared OpenPose network are two structurally identical but parameter-independent convolutional neural networks. Each network receives frame images from visible light and infrared video streams, respectively, as input. Visible light images reflect the distribution of reflected light intensity in a scene, and texture and edge information are significantly affected by lighting conditions. Infrared thermal imaging images reflect the distribution of thermal radiation in a scene; when there is a temperature difference between the human body and the background, the outline is clear, but texture details are lacking. The differences in the imaging mechanisms of the two modalities lead to different image statistical distributions. If convolution weights are shared, the network parameters cannot simultaneously adapt to both distributions. Therefore, a dual-stream architecture with independent weights is adopted, allowing each branch to independently learn the feature extraction capabilities of its corresponding modality. Each network uses ResNet as its deep convolutional backbone. ResNet consists of multiple cascaded residual blocks, each containing two convolutional layers and a cross-layer connection. The cross-layer connection directly adds the input to the convolutional output, alleviating the gradient vanishing problem during deep network training. The backbone network progressively maps the input image from high resolution to low resolution multi-scale feature maps through multiple convolutions and downsampling. Shallow feature maps retain low-level information such as edges and corners, while deep feature maps contain high-level information such as parts and semantics. On the feature maps output by the backbone network, the network predicts a two-dimensional confidence map and a partial affinity domain in parallel. Each channel of the two-dimensional confidence map corresponds to a keypoint type, and the value at each position on the confidence map represents the probability that the position belongs to the corresponding keypoint; a higher value indicates a greater likelihood of the position being a keypoint. The partial affinity domain is a set of two-dimensional vector fields, where each vector represents the connection direction and degree of association between two adjacent keypoints. During inference, the processor extracts the coordinates of candidate keypoints on the confidence map using non-maximum suppression, resulting in a set of discrete keypoint candidates. For each pair of candidate keypoints with a connection relationship, the processor performs a line integral calculation on the partial affinity domain along the line path between the two points. A higher line integral value indicates a stronger association between the two keypoints. The formula for calculating the line integral is: ,in For the first The partial affinity field vector field corresponding to the class connection, The parameterized path between two candidate points is defined. The processor constructs a bipartite graph matching problem with all candidate joints and their association strengths. The left node is a candidate joint of one class, and the right node is a candidate joint of another class. The edge weight is the association strength obtained by the line integral. A greedy parsing algorithm selects matching pairs in descending order of association strength, gradually assembling discrete joints into a complete human skeleton structure. The above reasoning is performed on each frame of the video stream, and the skeleton structure of each frame is arranged by frame number to obtain visible light skeleton sequences and infrared skeleton sequences. Each frame in each skeleton sequence contains the two-dimensional coordinates of a fixed number of joints.

[0025] Existing RNNs and GCNs have limited receptive fields for temporal modeling.

[0026] Based on this, the Mamba-based dual-stream temporal encoder includes a visible light Mamba encoder and an infrared Mamba encoder. Both the visible light Mamba encoder and the infrared Mamba encoder are composed of N cascaded Mamba blocks. Each Mamba block includes a layer normalization unit, a first linear transformation layer, a one-dimensional convolutional layer, a first activation function layer, a state space model unit, a second linear transformation layer, a second activation function layer, and a third linear transformation layer. After being normalized by the layer normalization unit, the input sequence is divided into two branches. The first branch is processed sequentially by the first linear transformation layer, the one-dimensional convolutional layer, and the first activation function layer before being input to the state space model unit. The state space model unit generates selective scanning parameters based on the input sequence and performs temporal modeling on the input sequence using a selective scanning algorithm. The second branch is processed by the second linear transformation layer and the second activation function layer, and then multiplied element-wise with the output of the state space model unit. The multiplication result is transformed by the third linear transformation layer and then residual-connected with the input sequence to output a feature sequence.

[0027] The visible light Mamba encoder and the infrared Mamba encoder receive visible light skeleton sequences and infrared skeleton sequences as input, respectively. Each encoder consists of... Mamba blocks are concatenated sequentially, with the output of one Mamba block serving as the input to the next. Through multi-layer stacking, temporal features, from local inter-frame relationships to global action patterns, are extracted progressively. The input to a single Mamba block is a shape... The feature sequence, where For the number of time steps, The feature dimension is defined as follows: The input sequence enters the layer normalization unit. The layer normalization unit calculates the mean and variance of the feature vector at each time step, normalizing the features to zero mean and unit variance. Then, an affine transformation is performed using learnable scaling and offset parameters. The formula for layer normalization is: ,in and The mean and variance are the feature dimensions. and For learnable parameters, To prevent division by zero, the normalized features are divided into two branches. The first branch is input to the first linear transformation layer, which uses matrix multiplication to reduce the feature dimension from zero. Transform to The output shape is The characteristics of the first linear transform layer are as follows: the output of the first linear transform layer is input to a one-dimensional convolutional layer, and the one-dimensional convolutional layer performs local convolution along the time dimension. The kernel size is [missing information]. For each channel continuous Weighted summation is performed at each time step, and one-dimensional convolution enables the model to capture local temporal dependencies. The output of the one-dimensional convolutional layer is input to the first activation function layer, which uses the SiLU function, calculated as follows: ,in The Sigmoid and SiLU functions introduce nonlinear transformations. The output of the first activation function layer is input to the state-space model unit, which projects the input sequence into three sets of parameters. , and and the state transition matrix ,in For step size parameters, For the input matrix, For the output matrix, Here, the parameters are dynamically generated from the input sequence through a linear transformation, hence the term selective scan. The discretization recursive formula for the state-space model is: , , in , , The selective scanning algorithm recursively calculates the hidden state in the order of time steps. The entire sequence is modeled in linear time complexity. The second branch input undergoes dimensionality transformation in the second linear transformation layer, followed by SiLU nonlinear transformation in the second activation function layer. The output of the second branch is multiplied element-wise with the output of the state-space model unit to implement a gating mechanism, allowing the model to selectively retain or suppress state information based on the input content. The multiplication result is then processed by the third linear transformation layer to reduce the dimension from... Transform back The output of the third linear transform layer is residually concatenated with the original input sequence of the Mamba block, i.e. Residual connections enable direct backpropagation of gradients, stabilizing the training of deep networks. After... After cascading several Mamba blocks, the output shape remains the same. The skeletal feature sequence.

[0028] Existing one-way contrast loss is not sufficiently aligned across modes.

[0029] Based on this, the construction of the multimodal InfoNCE contrast loss function, using the visible light low-dimensional latent features and infrared low-dimensional latent features of the same action instance as positive sample pairs, and the remaining low-dimensional latent features in the batch as negative samples, calculates the bidirectional contrast loss in the visible light to infrared direction and the infrared to visible light direction. The pre-training of the dual-stream temporal encoder by minimizing the total contrast loss includes the following steps: For a training batch containing N action samples, the visible light low-dimensional latent features and infrared low-dimensional latent features of the i-th action sample are used as positive sample pairs, and the remaining 2(N-1) low-dimensional latent features in the batch are used as negative samples; the contrast loss in the visible light to infrared direction is calculated, using the visible light low-dimensional latent features of the i-th action sample as the benchmark, and the numerator being the temperature coefficient-normalized index of the similarity between the visible light low-dimensional latent features and the corresponding infrared low-dimensional latent features of the i-th action sample. The numerical values ​​are calculated as follows: the denominator is the sum of the temperature-normalized exponents of the similarity between the visible low-dimensional latent features of the i-th action sample and all infrared low-dimensional latent features in the batch; the contrast loss from infrared to visible light is calculated using the infrared low-dimensional latent features of the i-th action sample as a benchmark, the numerator is the temperature-normalized exponent of the similarity between the infrared low-dimensional latent features of the i-th action sample and the corresponding visible low-dimensional latent features, and the denominator is the sum of the temperature-normalized exponents of the similarity between the infrared low-dimensional latent features of the i-th action sample and all visible low-dimensional latent features in the batch; the total contrast loss is the average of the contrast loss from visible to infrared and the contrast loss from infrared to visible light, and the total contrast loss is minimized through backpropagation and gradient update, and a pre-trained dual-stream temporal encoder is used; the temperature coefficient is a hyperparameter used to adjust the smoothness of the feature distribution.

[0030] The training batch includes Each of the 10 motion samples consists of a visible light skeleton sequence and an infrared skeleton sequence acquired simultaneously, with both sequences corresponding to the same motion instance. After processing by the Mamba encoder and projector, the 10th motion sample... Each sample yielded visible light low-dimensional latent features. and infrared low-dimensional potential features In the contrastive learning framework, and They are positive sample pairs because they represent the same action; the rest within the batch All low-dimensional latent features are negative samples because they represent distinct action instances. Similarity is calculated using cosine similarity, with the formula: The range of cosine similarity values ​​is: A larger value indicates that the two feature directions are closer. The contrast loss from visible light to infrared uses visible light features as the query and retrieves matching positive samples from the infrared feature set. The formula for calculating the loss of a sample is: ,in The formula uses a temperature coefficient. The numerator is the temperature-coefficient-scaled exponential value of the similarity between positive sample pairs, and the denominator is the sum of the temperature-coefficient-scaled exponential values ​​of the similarity between the query feature and all infrared features. The entire fraction represents the probability of selecting a positive sample from all infrared candidates. Taking the negative logarithm yields the loss value. Temperature coefficient Controlling the smoothness of the probability distribution, When the size is smaller, the distribution is sharper, and the model pays more attention to hard-to-distinguish negative samples; When the value is larger, the distribution is smoother, and the model treats all negative samples more evenly. Symmetrically, the contrast loss from infrared to visible light uses infrared features as the query and retrieves positive samples from the visible light feature set; the calculation formula is as follows. The losses in two directions constrain cross-modal feature alignment from the perspectives of the visible light and infrared modes, respectively. Calculating only the unidirectional loss leads to features from one mode converging towards the other, while the reverse direction lacks constraint. The bidirectional symmetric loss ensures that the feature spaces of the two modes are aligned. The total batch loss is the average of the losses in both directions for all samples, calculated using the following formula: The processor calculates using the backpropagation algorithm. The gradients of the Mamba encoder parameters and projector parameters are used to update the parameters using the Adam optimizer, with a learning rate set to [value missing]. Batch size Set to 32, temperature coefficient Set the value to 0.07, and iterate until the loss converges to complete the pre-training.

[0031] The mapping space does not match when existing projection heads share weights.

[0032] Based on this, the nonlinear projection head includes a visible light projection head and an infrared projection head. The weights of the visible light projection head and the infrared projection head are independent of each other. Both the visible light projection head and the infrared projection head are composed of a first fully connected layer, an activation function layer, and a second fully connected layer. The first fully connected layer maps the skeletal feature sequence to intermediate dimension features. The activation function layer performs a nonlinear transformation on the intermediate dimension features. The second fully connected layer maps the nonlinearly transformed intermediate dimension features to a low-dimensional latent feature space and outputs low-dimensional latent features.

[0033] The visible light projection head Receive visible light skeletal feature sequence output from visible light Mamba encoder As input, infrared projection head Receive infrared skeleton feature sequence output from infrared Mamba encoder As input, the visible light skeleton feature sequence and the infrared skeleton feature sequence, although corresponding to the same action, have different feature space distributions after passing through their respective independent Mamba encoders due to the different imaging mechanisms of the two modalities. The geometric structures and feature clustering methods of the visible light and infrared feature spaces are also different. If the two projectors share weights, the same linear transformation matrix would need to simultaneously map the two different feature distributions to the same contrast space, making it impossible to learn the optimal mapping parameters for each modality. Therefore, the visible light projector and the infrared projector use independent weights and each has independent fully connected layer parameters, learning the mapping function from the corresponding modal feature space to the contrast latent space. The mapping relationship is as follows: , Each projection head consists of two fully connected layers and one activation function layer connected in series. The first fully connected layer reduces the dimension of the input features from... Linear transformation to intermediate dimension The calculation formula is: ,in This is the first layer weight matrix. This is the bias vector for the first layer. The first fully connected layer's input and output are the activation function layers, which use the ReLU function, calculated as follows: The ReLU function sets negative values ​​to zero and retains positive values, introducing nonlinear transformation capability. This allows the projector to fit complex nonlinear mappings. If only linear layers are used, the projector degenerates into a linear transformation and cannot map features to nonlinear manifolds suitable for contrastive learning. The output of the activation function layer is input to the second fully connected layer, which then modulates the feature dimension from... Linear transformation to the low-dimensional latent space required for contrastive learning The calculation formula is: ,in This is the weight matrix for the second layer. This is the bias vector for the second layer. The output is the low-dimensional latent feature. Used for subsequent InfoNCE contrast loss calculation. In one implementation, Set to 512, Set to 256, Set to 128; in another implementation, Set to 256, Set to 128. Set to 64. During the pre-training phase, the projector head and Mamba encoder are trained jointly. The gradient of the contrastive loss is backpropagated to the encoder through the projector head to update the parameters of both. After pre-training, the projector head learns parameters that map features to a specific contrastive learning space. This space is optimized for the contrastive loss and is not directly applicable to the downstream action classification task. Therefore, in the downstream stage, the non-linear projector head is removed, and only the Mamba encoder is retained for feature extraction. The skeletal feature sequence output by the encoder is directly input into the cross-attention fusion module.

[0034] Existing splicing or bidirectional attention fusion interactions are insufficient.

[0035] Based on this, the method of using infrared skeleton feature sequences as queries and visible light skeleton feature sequences as keys and values, performing feature fusion through a cross-attention mechanism, and generating an enhanced infrared feature sequence after normalization, weighted aggregation, and residual connection, includes the following steps: linearly transforming the infrared skeleton feature sequence into a query matrix using a first learnable weight matrix; linearly transforming the visible light skeleton feature sequence into a key matrix using a second learnable weight matrix; and linearly transforming the visible light skeleton feature sequence into a value matrix using a third learnable weight matrix; calculating the matrix product of the transpose of the query matrix and the key matrix, dividing the matrix product by a scaling factor, and inputting it into a Softmax function for normalization to obtain an attention weight matrix; multiplying the attention weight matrix with the value matrix to obtain a cross-attention output feature sequence; and performing a residual connection between the cross-attention output feature sequence and the original infrared skeleton feature sequence to generate an enhanced infrared feature sequence; the cross-attention mechanism is a unidirectional cross-attention mechanism, retrieving visible light skeleton feature sequences only using the infrared skeleton feature sequence as a query; the scaling factor is the square root of the feature dimension of the key matrix.

[0036] The infrared skeleton feature sequence Shape Visible light skeletal feature sequence The shape is also The fusion module will Linear transformation to query matrix ,Will Linear transformations into bond matrices respectively Sum matrix The calculation formula is: , , ,in This is the first learnable weight matrix. This is the second learnable weight matrix. This is the third learnable weight matrix. For the key and query feature dimensions, The feature dimension for the value. Query matrix. Each row represents a feature of the infrared mode at a certain time step, serving as a query vector and a key matrix. Each row represents a feature of the visible light mode at a certain time step, serving as a key vector. The attention weight matrix is ​​calculated by the dot product similarity between the query and the key, using the following formula: ,in To query the matrix product of the key matrix and the transpose of the key matrix, with the shape of... Each element represents the similarity between a certain infrared time step and a certain visible light time step; As a scaling factor, the dot product value is divided by the square root of the key dimension to prevent the dot product value from becoming too large as the dimension increases, causing the Softmax function to enter the gradient saturation region. The Softmax function normalizes each row so that the sum of the attention weights of each infrared time step to all visible light time steps is 1. The calculation formula is as follows: Attention weight matrix AND-value matrix Multiplication yields cross-attention output , shape is The output of each infrared time step is a weighted sum of the value vectors from all visible light time steps. The weights are determined by the attention matrix. This means that the infrared features dynamically retrieve corresponding spatial location information from the visible light sequence based on their structural context, aggregating the rich spatial location information from the visible light into the infrared features. (Cross-attention output) The dimension is reduced from [the previous value] by a linear transformation. Map back Then, compared with the original infrared skeleton feature sequence Perform residual connections to obtain enhanced infrared feature sequences. The residual connection ensures the complete preservation of the original infrared structural information, and the fused features supplement the infrared structural information with visible light spatial location information. The fusion is a unidirectional cross-attention method, using only infrared features as a query to retrieve visible light features, without performing a reverse visible light query to retrieve infrared features. This is because infrared modes are insensitive to changes in illumination and their structural information is more stable in the dim environment of the well. Using infrared as a guide avoids interference from low-quality visible light features in the fusion result. In one embodiment, and All are set to 64, and the number of attention heads is set to 8. The cross-attention is calculated independently for each head, and the results are then concatenated.

[0037] The existing global pooling method loses key timing discrimination information.

[0038] Based on this, the step of splicing a learnable classification label at the beginning of the enhanced infrared feature sequence and inputting the feature sequence after splicing the classification label into the Transformer encoding layer for feature interaction includes the following steps: initializing a learnable classification label with the same feature dimension as the enhanced infrared feature sequence; updating the parameters of the learnable classification label through backpropagation during model training; splicing the learnable classification label at the beginning of the time dimension of the enhanced infrared feature sequence to form an input sequence containing the classification label and the enhanced infrared feature sequence; inputting the input sequence into the Transformer encoding layer, which consists of a multi-head self-attention sub-layer, a feedforward neural network sub-layer, layer normalization, and residual connections; the classification label interacts with the enhanced infrared features at all time steps in the input sequence through the multi-head self-attention sub-layer, and outputs a feature sequence containing the interacted classification label.

[0039] The learnable classification label is a shape of The trainable vectors are denoted as Its feature dimensions With enhanced infrared signature sequence They have the same feature dimensions. The initial values ​​are randomly initialized using a normal distribution. During model training, the gradient of the classification loss is backpropagated through the Transformer encoding layer to... Continuously updated The parameter values ​​enable it to learn the ability to aggregate global action discrimination information. It is spliced ​​into the temporal header of the enhanced infrared feature sequence to form the input sequence. , shape is The semicolon indicates concatenation along the time dimension. The first time step of the concatenated sequence is the classification label, and subsequent steps are... Each time step is used to enhance infrared features. Input sequence Input Transformer coding layer, Transformer coding layer consists of The system consists of stacked identical coding layers. Each coding layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. Each sub-layer employs a pre-normalization structure, meaning the sub-layer input is first normalized before the sub-layer operation, and the result is concatenated with the sub-layer input residual. The multi-head self-attention sub-layer maps the input features into three matrices: query, key, and value, and divides them along the feature dimension. Each head is an independent head, and each head computes self-attention. ,Will The outputs of each head are concatenated and then linearly transformed to obtain a multi-head self-attention output. Self-attention allows each time step in the sequence to interact with all other time steps. The feedforward neural network sublayer consists of two fully connected layers and one GELU activation function layer. The first layer reduces the dimension from... Extended to After GELU activation, the second layer will change the dimension from Compress back The calculation formula is: In a multi-head self-attention sublayer, the part located at the beginning of the sequence... Attention weights are calculated using the query and all time steps as keys, aggregating information from each time step in the enhanced infrared feature sequence. Above, at the same time It is also considered as a key and value by other time steps, after After the layer coding layer, The global spatiotemporal context information of the entire sequence has been aggregated, enabling it to characterize the overall discriminative features of the action. The output shape of the Transformer encoding layer remains... The feature sequence has an interactive classification label at the beginning and interactive enhanced infrared features at subsequent positions. In one implementation, Set to 2, Set to 8, Set it to 512, and set the intermediate dimension of the feedforward network to 2048.

[0040] Existing average pooling classification cannot be adapted to pre-trained representations.

[0041] Based on this, the process of extracting the post-interaction classification labels as input to the classification head, outputting the action category probability distribution, and using the cross-entropy loss function for supervised fine-tuning includes the following steps: extracting the post-interaction classification labels located at the beginning of the time dimension from the feature sequence output by the Transformer encoding layer; inputting the post-interaction classification labels as input to the classification head composed of fully connected layers; the fully connected layers map the feature dimension of the classification labels to the action category number dimension; and outputting the probability distribution of each action category after normalization by the Softmax function; calculating the cross-entropy loss based on the labeled action category labels and the predicted action category probability distribution; and performing supervised fine-tuning of the dual-stream temporal encoder, cross-attention fusion module, Transformer encoding layer, and classification head through backpropagation and gradient update.

[0042] The feature sequence output by the Transformer encoding layer has the following shape: The first time step of the sequence corresponds to the learnable classification label concatenated at input. The processor extracts the feature vector at this position as the global action representation, denoted as... Classification tags The input is a classification head consisting of fully connected layers, which divide the feature dimensions from... Linear transformation to the number of action categories , thus obtaining the logits vector ,in This is the class head weight matrix. This is the bias vector. The logits vector, after being normalized using the Softmax function, outputs the probability distribution for each action category, calculated using the following formula: ,in For the first The predicted probabilities for each action category are given, with the sum of all probabilities equal to 1. During training, the processor calculates the cross-entropy loss based on the labeled ground truth action category labels and the predicted probability distribution. The ground truth labels are represented using one-hot encoding. Only the true class position is 1, and the rest are 0. The cross-entropy loss is calculated using the following formula: When the real category is the first When classifying, the loss is simplified to Predicted probability The closer the value is to 1, the smaller the loss value. The processor calculates this using the backpropagation algorithm. The gradients of all network parameters, including the parameters of the two-stream Mamba temporal encoder and the weight matrix of the cross-attention fusion module. , , All parameters of the Transformer encoding layer and the classification head are updated end-to-end using gradient descent. During the fine-tuning phase, a small amount of labeled underground coal mine action data is used for training. Because the Mamba encoder has already acquired general action representation capabilities through self-supervised contrastive pre-training, fine-tuning only requires adapting the general representation to the specific underground action classification task; therefore, a small number of labeled samples are sufficient for model convergence. In one implementation, the number of action categories... Set to 10, including standing, walking, bending over, squatting, climbing, carrying, inspecting, operating equipment, resting, and abnormal falls; fine-tune the learning rate to [value missing]. The training rounds were set to 50, and the batch size was set to 16. During the inference phase, the processor input the bimodal skeleton sequence to be recognized into the fine-tuned network and output the category with the highest probability as the final recognition result.

[0043] Directly outputting existing low-confidence results is prone to misjudgment.

[0044] Based on this, a low-confidence recognition result correction step is also included: using the supervised fine-tuning model to infer the bimodal skeleton sequence to be recognized, outputting the probability distribution and corresponding confidence of each action category; when the confidence of the recognition result is lower than a preset threshold, the selective scanning mechanism of the Mamba encoder is activated, and the skeleton sequence of multiple consecutive frames is dynamically weighted according to the action context information. By analyzing the motion trajectory changes of key joints on the time axis, the motion trend of the occluded part is inferred; using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, the feature is re-fused through the cross-attention mechanism, and the cross-attention output feature sequence is residually fused with the original infrared skeleton feature sequence to recalculate the fused feature sequence; the classification label is extracted from the recalculated fused feature sequence, and the corrected action category probability distribution is input into the classification head again. If the corrected confidence increases to above the preset threshold, the correction result is adopted; if the corrected confidence does not reach the preset threshold, it is marked as pending manual review.

[0045] The supervised fine-tuning of the model performs forward inference on the bimodal skeletal sequence to be identified, and the network output shape is... dimensional action category probability distribution The processor takes the maximum value in the probability distribution as the confidence level of the recognition result. Simultaneously, the category index corresponding to the maximum value is taken as the preliminary identification result. The processor then calculates the confidence level. With preset threshold In comparison, when When the initial identification results are reliable, they are output directly; when... If the recognition result is unreliable, a correction process is triggered. This correction process activates the selective scanning mechanism of the Mamba encoder, where the state-space model unit dynamically generates step size parameters based on the input sequence. Input matrix and output matrix Selective scanning mechanism adjusts The value of is dynamically weighted across multiple consecutive frames of skeletal sequences. When the skeletal points in a particular frame are complete and their coordinates are stable, Taking a larger value increases the contribution of that frame to the hidden state; when a frame contains occlusion or noise, The smaller value is chosen to reduce the contribution of that frame. The processor analyzes the coordinate change trajectory of key joints on the time axis. Key joints include the head, neck, shoulder, elbow, wrist, hip, knee, and ankle. This is done by calculating the displacement vector of the same joint between adjacent frames. Leveraging the temporal continuity of actions, the missing joint coordinates in the occluded frame are extrapolated linearly or fitted using displacement vectors from several preceding and following frames to infer the motion trend and coordinate position of the occluded part in that frame, thus completing the skeleton sequence. The completed visible light and infrared skeleton sequences are then re-inputted into the Mamba encoder to extract features. The processor uses the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, respectively, and recalculates the fused features through a cross-attention mechanism. The cross-attention output is then residually fused with the original infrared skeleton feature sequence to obtain a recalculated fused feature sequence. The processor appends a classification label to the beginning of the recalculated fused feature sequence, extracts the classification label after interaction with the Transformer encoding layer, and inputs it into the classification head to output the corrected probability distribution and confidence level. If the corrected confidence level is greater than or equal to a preset threshold... If the corrected recognition result is adopted, then the corrected confidence level will be adopted; if the corrected confidence level is still less than 1, then the corrected recognition result will be adopted. If the sample is not found to be ready for manual review, the automatic identification result will not be output. In one implementation, a preset threshold is used. Set to 0.7, and set the number of reference frames before and after dynamic weighting to 5 frames.

[0046] When existing bimodal videos are not aligned, positive sample pairs do not correspond.

[0047] Based on this, the acquisition of visible light and infrared dual-modal video data synchronously collected in underground coal mines includes the following steps: synchronously acquiring visible light video streams and infrared thermal imaging video streams through dual-spectrum cameras deployed in the underground coal mine working area, aligning the visible light video streams and infrared thermal imaging video streams on the time axis; performing frame decoding on the acquired visible light video streams and infrared thermal imaging video streams, judging the image quality frame by frame, removing invalid frames with missing images or environmental noise exceeding the noise threshold, and retaining valid frames to form a visible light image sequence and an infrared image sequence.

[0048] The dual-spectrum camera integrates a visible light sensor and an infrared thermal imaging sensor, both housed in the same housing. Synchronous acquisition is achieved via a hardware trigger signal connected to the exposure pins of both sensors. When the trigger signal arrives, both sensors begin exposure simultaneously, ensuring that each frame of the visible light image is aligned with the corresponding infrared thermal imaging image on the timeline, with a one-to-one frame sequence. The dual-spectrum camera is deployed on a fixed bracket in the underground coal mine working area, with the shooting direction aimed at the worker's work area. The visible light sensor acquires reflected light images in the visible light band with a resolution of 1920×1080 and a frame rate of 25 frames per second. The infrared thermal imaging sensor acquires thermal radiation images in the long-wave infrared band with a resolution of 640×512 and a frame rate consistent with the visible light sensor at 25 frames per second. The field of view of both sensors is calibrated to ensure that the shooting ranges largely overlap, and the position of the same human body roughly corresponds in both modal images. The acquired visible light video stream and infrared thermal imaging video stream are transmitted to a processor via wired transmission. The processor performs frame decoding on both video streams, restoring the compressed video data into a frame-by-frame pixel matrix. The processor assesses image quality frame by frame. This assessment includes image loss detection and environmental noise detection. Image loss detection determines the percentage of regions with zero or near-zero pixel values ​​by statistically analyzing the proportion of these regions; when the percentage exceeds a threshold, the image is considered missing. Environmental noise detection calculates the signal-to-noise ratio (SNR) of the image. The SNR calculation formula is... ,in For signal power, The noise power is used to determine excessive environmental noise when the signal-to-noise ratio (SNR) is below the noise threshold. For frames with missing frames or environmental noise exceeding the noise threshold, the processor marks them as invalid and discards them to prevent invalid frames from entering subsequent processing, increasing computational load, and introducing interference. After discarding invalid frames, the processor arranges the remaining valid frames in chronological order to form a visible light image sequence and an infrared image sequence. Both image sequences have the same number of frames and correspond one-to-one with their timestamps. In one implementation, the missing frame rate threshold is set to 0.3, the noise threshold is set to 20 dB, and the acquisition frame rate of the dual-spectrum camera can be adjusted to 15 frames per second or 30 frames per second depending on the operational scenario.

[0049] To verify the effectiveness of the above method, ablation experiments were conducted on a motion dataset collected underground in a coal mine. The dataset included 10 motion categories: standing, walking, bending, squatting, climbing, carrying, maintenance, operating equipment, resting, and abnormal falls. 200 bimodal video sequences were collected for each motion category, with 80% used for training and 20% for testing. Only 10% of the labeled samples in the training set were used for supervised fine-tuning, while the remaining 90% of the unlabeled samples were used for self-supervised comparative pre-training. The contribution of each technical feature to the recognition accuracy was verified by gradually adding modules. The ablation experiment results are shown in the table below.

[0050] Table 1 As shown in the table above, Experimental Group 1, using infrared single-modal input and a single Mamba encoder, achieved a Top-1 accuracy of 0.795, precision of 0.805, recall of 0.776, and F1 score of 0.790. Experimental Group 2, using visible light single-modal input and a single Mamba encoder, achieved a Top-1 accuracy of 0.735, precision of 0.742, recall of 0.715, and F1 score of 0.728. All indicators of the infrared single-modal input were higher than those of the visible light single-modal input, with a Top-1 accuracy that was 0.060 higher. This indicates that in the low-light environment of underground coal mines, the infrared modality is more stable in extracting skeletal key points, while the visible light modality suffers from a decrease in feature quality due to dim lighting and dust interference. Experiment 3 employed dual-modal input (infrared and visible light), dual Mamba encoders, and feature splicing fusion. The Top-1 accuracy improved to 0.832, precision to 0.842, recall to 0.815, and F1 score to 0.828. Compared to Experiment 2 with single-modal visible light input, the Top-1 accuracy improved by 0.097, validating the complementary effect of dual-modal input and dual-stream encoders. Experiment 4 replaced splicing fusion with infrared-guided cross-attention fusion, further improving the Top-1 accuracy to 0.868, precision to 0.878, recall to 0.852, and F1 score to 0.865. Compared to Experiment 3 with splicing fusion, the Top-1 accuracy improved by 0.036, validating that the cross-attention mechanism can dynamically retrieve visible light spatial information based on the infrared structural context, and its fusion effect is superior to simple feature splicing. Experimental group 5, building upon experimental group 4, incorporated multimodal contrastive learning pre-training. The Top-1 accuracy reached 0.895, precision 0.903, recall 0.886, and F1 score 0.894. Compared to experimental group 4 without pre-training, the Top-1 accuracy improved by 0.027, validating the role of self-supervised contrastive pre-training in learning robust action representations under small sample conditions. From experimental group 1 to experimental group 5, all four metrics—precision, recall, F1 score, and Top-1 accuracy—showed a gradual upward trend, indicating that bimodal input, dual Mamba encoders, cross-attention fusion, and contrastive learning pre-training are all necessary technical features for improving recognition performance.

[0051] Under the same dataset and the same small sample settings, the method of this invention is compared with mainstream methods based on different temporal modeling architectures, including VA-LSTM and AGC-LSTM methods based on recurrent neural networks, VA-CNN methods based on convolutional neural networks, HD-GCN methods, SARGCN methods and BlockGCN methods based on graph convolutional networks, ST-TR methods, Hyperformer methods and LTPEformer methods based on Transformer. All methods use the same bimodal input and the same training data partitioning. The comparison results are shown in the table below.

[0052] Table 2 As shown in the table above, the Top-1 accuracy of the VA-LSTM method based on recurrent neural networks is 0.742, and that of the AGC-LSTM method is 0.795. Recurrent neural networks have difficulty modeling long-term temporal dependencies due to the vanishing gradient problem, resulting in the lowest accuracy among the two methods. The Top-1 accuracy of the VA-CNN method based on convolutional neural networks is 0.812. One-dimensional convolution extracts local temporal features through a fixed convolutional kernel, but its receptive field is limited, making it unable to capture global action dependencies. The Top-1 accuracy of the HD-GCN method based on graph convolutional networks is 0.858, that of the SARGCN method is 0.825, and that of the BlockGCN method is 0.862. Graph convolutional networks use the graph structure of skeletal joints to model spatial relationships, but they rely on predefined graph connections, and the graph structure fails when skeletal points are occluded or drift. The ST-... The TR method achieved a Top-1 accuracy of 0.834, the Hyperformer method 0.865, and the LTPEformer method 0.873. The Transformer modeled global temporal dependencies through a self-attention mechanism, achieving higher accuracy than recurrent neural networks and convolutional neural networks, but its computational complexity increased quadratically with sequence length. The MCL-Mamba method of this invention achieved a Top-1 accuracy of 0.895, the highest among all compared methods. It improved by 0.022 compared to the best Transformer-class method LTPEformer and by 0.033 compared to the best graph convolutional method BlockGCN. This verifies the performance advantage of the Mamba-based selective scanning mechanism combined with multimodal contrastive learning pre-training and infrared-guided cross-attention fusion in action recognition under small sample conditions.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multimodal contrastive learning-guided method for skeletal motion recognition in coal mines, comprising acquiring visible light and infrared dual-modal video data synchronously collected in a coal mine, and using a dual-stream OpenPose network to perform inference on the visible light and infrared images respectively to obtain visible light skeleton sequences and infrared skeleton sequences, characterized in that, It also includes the following steps: S1: The visible light skeleton sequence and the infrared skeleton sequence are used to form two heterogeneous views of the same action instance. The visible light skeleton sequence and the infrared skeleton sequence are modeled in a long time series by a Mamba-based dual-stream temporal encoder to extract the visible light skeleton feature sequence and the infrared skeleton feature sequence. S2: The visible light skeleton feature sequence and the infrared skeleton feature sequence are mapped to the low-dimensional latent feature space by the non-linear projection head to obtain the visible light low-dimensional latent features and the infrared low-dimensional latent features. The multimodal InfoNCE contrast loss function is constructed. The visible light low-dimensional latent features and the infrared low-dimensional latent features of the same action instance are positive sample pairs, and the other low-dimensional latent features in the batch are negative samples. The bidirectional contrast loss in the visible light to infrared direction and the infrared to visible light direction is calculated. The dual-stream temporal encoder is pre-trained by minimizing the total contrast loss. After the pre-training is completed, the dual-stream temporal encoder is retained and the non-linear projection head is removed. S3: Load the pre-trained dual-stream temporal encoder and extract visible light skeleton feature sequences and infrared skeleton feature sequences respectively; S4: Using the infrared skeleton feature sequence as the query and the visible light skeleton feature sequence as the key and value, feature fusion is performed through a cross-attention mechanism. After normalization and weighted aggregation, residual connections are superimposed to generate an enhanced infrared feature sequence. S5: Learnable classification labels are spliced ​​into the head of the enhanced infrared feature sequence. The feature sequence after splicing the classification labels is input into the Transformer encoding layer for feature interaction. The classification labels after interaction are extracted and input into the classification head. The action category probability distribution is output, and the cross-entropy loss function is used for supervised fine-tuning.

2. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The method of using a dual-stream OpenPose network to infer visible light and infrared images to obtain visible light and infrared skeleton sequences includes the following steps: Visible light and infrared video streams are processed using a weighted, independent visible light OpenPose network and an infrared OpenPose network, respectively. Both networks use ResNet as a deep convolutional backbone to extract multi-scale feature maps, outputting a two-dimensional confidence map and a partial affinity domain. Line integrals are calculated based on the partial affinity domain to measure the correlation strength between candidate keypoints. Discrete keypoints are assembled into a human skeleton structure using a greedy parsing algorithm and bipartite graph matching, outputting the visible light and infrared skeleton sequences.

3. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The Mamba-based dual-stream temporal encoder includes a visible light Mamba encoder and an infrared Mamba encoder. Both the visible light and infrared Mamba encoders are composed of N cascaded Mamba blocks. Each Mamba block includes a layer normalization unit, a first linear transformation layer, a one-dimensional convolutional layer, a first activation function layer, a state space model unit, a second linear transformation layer, a second activation function layer, and a third linear transformation layer. After being normalized by the layer normalization unit, the input sequence is divided into two branches. The first branch is processed sequentially by the first linear transformation layer, the one-dimensional convolutional layer, and the first activation function layer before being input to the state space model unit. The state space model unit generates selective scanning parameters based on the input sequence and performs temporal modeling on the input sequence using a selective scanning algorithm. The second branch is processed by the second linear transformation layer and the second activation function layer, and then multiplied element-wise with the output of the state space model unit. The multiplication result is transformed by the third linear transformation layer and then residual-connected with the input sequence to output a feature sequence.

4. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The construction of the multimodal InfoNCE contrast loss function uses the visible light low-dimensional latent features and infrared low-dimensional latent features of the same action instance as positive sample pairs, and the remaining low-dimensional latent features in the batch as negative samples. It calculates the bidirectional contrast loss in the visible-to-infrared direction and the infrared-to-visible direction, and pre-trains a dual-stream temporal encoder by minimizing the total contrast loss. The steps include: For a training batch containing N action samples, the visible light low-dimensional latent features and infrared low-dimensional latent features of the i-th action sample are paired as positive samples, and the remaining 2(N-1) low-dimensional latent features in the batch are used as negative samples; the contrast loss in the visible-to-infrared direction is calculated, using the visible light low-dimensional latent features of the i-th action sample as the benchmark, with the numerator being the temperature-coefficient-normalized exponent of the similarity between the visible light low-dimensional latent features and the corresponding infrared low-dimensional latent features of the i-th action sample. The denominator is the sum of the temperature-normalized exponents of the similarity between the visible low-dimensional latent features of the i-th action sample and all infrared low-dimensional latent features in the batch; the contrast loss in the infrared-to-visible direction is calculated using the infrared low-dimensional latent features of the i-th action sample as the benchmark, the numerator is the temperature-normalized exponent of the similarity between the infrared low-dimensional latent features of the i-th action sample and the corresponding visible low-dimensional latent features, and the denominator is the sum of the temperature-normalized exponents of the similarity between the infrared low-dimensional latent features of the i-th action sample and all visible low-dimensional latent features in the batch; the total contrast loss is the average of the contrast loss in the visible-to-infrared direction and the contrast loss in the infrared-to-visible direction, and the total contrast loss is minimized through backpropagation and gradient update, and a pre-trained dual-stream temporal encoder is used; the temperature coefficient is a hyperparameter used to adjust the smoothness of the feature distribution.

5. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The nonlinear projection head includes a visible light projection head and an infrared projection head. The weights of the visible light projection head and the infrared projection head are independent of each other. Both the visible light projection head and the infrared projection head are composed of a first fully connected layer, an activation function layer, and a second fully connected layer. The first fully connected layer maps the skeletal feature sequence to intermediate dimension features. The activation function layer performs a nonlinear transformation on the intermediate dimension features. The second fully connected layer maps the nonlinearly transformed intermediate dimension features to a low-dimensional latent feature space and outputs low-dimensional latent features.

6. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The process involves using infrared skeleton feature sequences as queries and visible light skeleton feature sequences as keys and values, fusing features through a cross-attention mechanism, normalizing and weighting the results, and then performing residual joins to generate an enhanced infrared feature sequence. The steps include: linearly transforming the infrared skeleton feature sequence into a query matrix using a first learnable weight matrix; linearly transforming the visible light skeleton feature sequence into a key matrix using a second learnable weight matrix; and linearly transforming the visible light skeleton feature sequence into a value matrix using a third learnable weight matrix; calculating the matrix product of the transpose of the query matrix and the key matrix, dividing the product by a scaling factor, and then normalizing it using a Softmax function to obtain an attention weight matrix; multiplying the attention weight matrix by the value matrix to obtain a cross-attention output feature sequence; and performing a residual join between the cross-attention output feature sequence and the original infrared skeleton feature sequence to generate an enhanced infrared feature sequence. The cross-attention mechanism is unidirectional, retrieving visible light skeleton feature sequences only using the infrared skeleton feature sequence as the query; the scaling factor is the square root of the feature dimension of the key matrix.

7. The multimodal contrastive learning-guided method for recognizing skeletal movements in underground coal mines according to claim 1, characterized in that, The process of splicing a learnable classification label at the beginning of the enhanced infrared feature sequence and inputting the feature sequence with the spliced ​​classification label into the Transformer encoding layer for feature interaction includes the following steps: initializing a learnable classification label with the same feature dimension as the enhanced infrared feature sequence; updating the parameters of the learnable classification label through backpropagation during model training; splicing the learnable classification label at the beginning of the time dimension of the enhanced infrared feature sequence to form an input sequence containing the classification label and the enhanced infrared feature sequence; inputting the input sequence into the Transformer encoding layer, which consists of a multi-head self-attention sub-layer, a feedforward neural network sub-layer, layer normalization, and residual connections; the classification label interacts with the enhanced infrared features at all time steps in the input sequence through the multi-head self-attention sub-layer, and outputting a feature sequence containing the interacted classification label.

8. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The process of extracting the post-interaction classification labels and inputting them into the classification head, outputting the action category probability distribution, and using the cross-entropy loss function for supervised fine-tuning includes the following steps: extracting the post-interaction classification labels located at the beginning of the temporal dimension from the feature sequence output by the Transformer encoding layer; inputting the post-interaction classification labels into the classification head composed of fully connected layers; mapping the feature dimension of the classification labels to the action category number dimension; and outputting the probability distribution of each action category after normalization by the Softmax function; calculating the cross-entropy loss based on the labeled action category labels and the predicted action category probability distribution; and performing supervised fine-tuning of the dual-stream temporal encoder, cross-attention fusion module, Transformer encoding layer, and classification head through backpropagation and gradient update.

9. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The process also includes a low-confidence recognition result correction step: using a supervised fine-tuned model to infer the bimodal skeletal sequence to be recognized, outputting the probability distribution and corresponding confidence of each action category; when the confidence of the recognition result is lower than a preset threshold, the selective scanning mechanism of the Mamba encoder is activated, dynamically weighting the skeletal sequence of multiple consecutive frames according to the action context information, and inferring the movement trend of the occluded part by analyzing the movement trajectory changes of key joints on the time axis; using the infrared skeletal feature sequence as the query and the visible light skeletal feature sequence as the key and value, the feature is re-fused through a cross-attention mechanism, and the cross-attention output feature sequence is residually fused with the original infrared skeletal feature sequence to recalculate the fused feature sequence; the classification label is extracted from the recalculated fused feature sequence, and the corrected action category probability distribution is input into the classification head again; if the corrected confidence increases to above the preset threshold, the correction result is adopted; if the corrected confidence does not reach the preset threshold, it is marked as awaiting manual review.

10. The multimodal contrastive learning-guided method for skeletal motion recognition in coal mines according to claim 1, characterized in that, The acquisition of visible light and infrared dual-modal video data synchronously collected in underground coal mines includes the following steps: synchronously acquiring visible light video streams and infrared thermal imaging video streams using dual-spectrum cameras deployed in the underground coal mine operating area, aligning the visible light video streams and infrared thermal imaging video streams on the time axis; performing frame decoding on the acquired visible light video streams and infrared thermal imaging video streams, judging the image quality frame by frame, removing invalid frames with missing images or environmental noise exceeding the noise threshold, and retaining valid frames to form a visible light image sequence and an infrared image sequence.

Images