A dam personnel behavior monitoring and early warning method, system, device and medium

Abstract

The application provides a dam personnel behavior monitoring and early warning method, system, device and medium, the method comprises the following steps: S1, obtain the visible light image and low resolution infrared image of dam;S2, using super-resolution reconstruction method to reconstruct low resolution infrared image into high resolution infrared image;S3, the high resolution infrared image and visible light image are cooperated and input to human body posture recognition network, obtain the current monitored personnel's joint node feature vector;S4, the joint node feature vector is input to personnel behavior monitoring network, and whether the behavior of the current monitored personnel belongs to abnormal behavior state is judged according to judging threshold value.The application improves the precision of personnel behavior monitoring by infrared image super-resolution reconstruction and infrared and visible light image cooperative personnel posture estimation, can more accurately capture the subtle action and posture change of personnel.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy facility monitoring technology, specifically relating to a method, system, device, and medium for monitoring and early warning of personnel behavior at dams. Background Technology

[0002] Dams are core facilities in water conservancy projects, therefore, monitoring the behavior of personnel entering the dam's safety zone is a crucial aspect of ensuring the stable operation of the project. Traditional dam personnel behavior monitoring technologies suffer from the following core problems: Existing monitoring systems primarily rely on single-modal data from visible light cameras, resulting in low image resolution and loss of detail in long-distance monitoring scenarios. In complex environments such as nighttime or foggy conditions, traditional visual systems are easily affected by changes in lighting, leading to the failure to recognize key behavioral features (such as the wearing status of safety ropes and climbing actions). Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, system, device and medium for monitoring and early warning of personnel behavior at dams, which can accurately identify personnel behavior at dams in complex environments.

[0004] This invention is implemented as follows: a method for monitoring and early warning of personnel behavior at dams, comprising the following steps:

[0005] S1. Acquire visible light and low-resolution infrared images of the dam;

[0006] S2. Use super-resolution reconstruction methods to reconstruct low-resolution infrared images into high-resolution infrared images;

[0007] S3. Input the high-resolution infrared image and the visible light image together into the human posture recognition network to obtain the joint feature vector of the currently monitored person.

[0008] S4. Input the key feature vector into the personnel behavior monitoring network, and determine whether the behavior of the currently monitored personnel belongs to an abnormal behavior state based on the judgment threshold.

[0009] Furthermore, S2 includes the following steps:

[0010] S201. Perform convolution processing on the low-resolution infrared image to obtain shallow features;

[0011] S202. Using shallow features as input, deep features are obtained through a deep feature extraction algorithm.

[0012] S203. After performing residual connection between shallow and deep features, reconstruction processing is carried out to obtain a high-resolution infrared image.

[0013] Furthermore, S202 includes the following steps:

[0014] S20201. Perform feature compression, residual connection and refinement on shallow features to obtain processed features;

[0015] S20202. Adaptive average pooling is applied to the processed features to obtain adaptive average pooling features.

[0016] S20203. Adaptive convolution processing is performed on the adaptive average pooling features to obtain the local dependencies between channels;

[0017] S20204, Perform local dependency analysis. sigmoid The activation process is performed, and the output values ​​are mapped to the [0,1] interval to obtain the channel attention weights;

[0018] S20205. Perform efficient channel attention activation processing on the channel attention weights and shallow features to obtain activated features;

[0019] S20206. Perform multi-head self-attention calculation on the activation features to obtain the multi-head self-attention value;

[0020] S20207. Perform line normalization on the multi-head self-attention value to obtain the line normalization result;

[0021] S20208. Based on the line normalization result, shallow features, and activation features, intermediate features are calculated.

[0022] S20209. Perform multi-layer perception and layer normalization on the intermediate features to obtain the layer normalization result;

[0023] S20210. Calculate the output features based on the layer normalization results and intermediate features;

[0024] S20211. Perform multiple residual multi-head self-attention processing on the output features to obtain deep features.

[0025] Furthermore, S3 includes the following steps:

[0026] S301. Acquire visible light images and high-resolution infrared images;

[0027] S302. Perform preliminary fusion processing on the high-resolution infrared image and the visible light image to obtain the fused image;

[0028] S303. The ViTPose encoder is used to extract features from the fused image to obtain a multi-scale fused feature map.

[0029] S304. An adaptive frame weighting method is adopted to dynamically assign weights to the multi-scale fusion feature map based on the correlation of frames.

[0030] S305. Weight the multi-scale fused feature map using weights to obtain a weighted feature map;

[0031] S306. Using the feature map of the center frame of the feature map as the query vector and the feature map of the context frame as the key vector and value vector, calculate the attention weights between the positions in the center frame and the positions in the context frame.

[0032] S307. Update all feature maps according to attention weights and generate human pose key node feature vectors through the ViTPose decoder.

[0033] Furthermore, S302 includes the following steps:

[0034] S30201. Perform multi-scale decomposition on high-resolution infrared images and visible light images respectively to obtain low-frequency components and high-frequency components at different scales.

[0035] S30202. The low-frequency components are fused using a weighted average method to obtain the low-frequency component fusion result;

[0036] S30203. The energy of high-frequency components in each region is calculated using a region-based energy selection fusion method, and the region with higher energy is taken as the high-frequency classification fusion result.

[0037] S30204. Reconstruct the image by combining the low-frequency component fusion result and the high-frequency classification fusion result to obtain the fused image.

[0038] Furthermore, S303 includes the following steps:

[0039] S30301. Segment the fused image into primitives, use the ViTPose encoder to extract features at depth, and obtain local and global spatial information.

[0040] S30302. For primitives, feature maps are extracted from different layers in the feature extraction network, and each feature map is subjected to adaptive pooling at multiple scales to generate adaptive pooling features.

[0041] S30303: Convolution is used to reduce the embedding dimension to one-quarter of the original. Then, the height and width dimensions of the feature are restored through the difference layer. Finally, the original feature map is connected with the upsampled pooled feature map to obtain a multi-scale feature map.

[0042] S30304. Perform convolutional layer processing on multi-scale feature maps to unify feature dimensions, and then perform layer normalization and... Activation processing yields activated multi-scale feature maps;

[0043] S30305. Connect the activated multi-scale feature maps of different layers along the channel dimension to obtain features, and then calculate the spatial position dependency between consecutive frames through a multi-head attention algorithm.

[0044] S30306. Residual connections are made between features and dependencies, and a unified multi-scale representation is obtained through average pooling and linear transformation. Finally, a multi-scale fused feature map is obtained through a projection layer.

[0045] Furthermore, S4 includes the following steps:

[0046] S401. Input the joint feature vector and the normal joint feature vector into the human normal behavior model in sequence to obtain the prediction result and the normal result;

[0047] S402. Calculate the log-likelihood probability of the predicted results and the normal results to obtain the probability values;

[0048] S403. If the probability value is less than the set threshold, the personnel behavior is considered abnormal; otherwise, it is considered normal.

[0049] This invention also provides a dam personnel behavior monitoring and early warning system, comprising:

[0050] The image acquisition module is used to acquire visible light and low-resolution infrared images of the dam.

[0051] The image reconstruction module is used to reconstruct low-resolution infrared images into high-resolution infrared images using super-resolution reconstruction methods.

[0052] The feature recognition module is used to input high-resolution infrared images and visible light images into the human posture recognition network to obtain the joint feature vectors of the currently monitored person.

[0053] The behavior judgment module is used to input the key feature vector into the personnel behavior monitoring network and determine whether the behavior of the currently monitored personnel belongs to an abnormal behavior state based on the judgment threshold.

[0054] The present invention also provides a monitoring and early warning device for personnel behavior at dams, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method.

[0055] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, characterized in that, when the instructions are executed by a processor, the processor performs the above-described method.

[0056] The beneficial effects of this invention are:

[0057] 1. This invention significantly improves the accuracy of personnel behavior monitoring by using infrared image super-resolution reconstruction and combined infrared and visible light imagery for personnel posture estimation, enabling more accurate capture of subtle movements and posture changes. Simultaneously, infrared images maintain good imaging quality even at night and in adverse weather conditions. Combined with visible light images, this achieves all-weather personnel behavior monitoring, ensuring effective surveillance of personnel on the dam in any environment.

[0058] 2. This invention, through the identification of abnormal human behavior, can promptly detect abnormal behavior and issue early warnings, effectively preventing potential safety risks, ensuring the safe operation of the dam, realizing automated and intelligent management, reducing manual intervention, improving the efficiency of dam safety management, and reducing management costs.

[0059] 3. This invention can accumulate a large amount of monitoring data, providing data support and decision-making basis for subsequent dam safety management, which helps to optimize management strategies and improve the scientificity and effectiveness of dam safety management. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method in this invention;

[0061] Figure 2 This is a flowchart of S2 in this invention;

[0062] Figure 3 This is a flowchart of S202 in the present invention;

[0063] Figure 4 This is a flowchart of S3 in this invention;

[0064] Figure 5 This is a flowchart of S302 in the present invention;

[0065] Figure 6 This is a flowchart of S303 in this invention;

[0066] Figure 7 This is a flowchart of S4 in this invention. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0068] Example 1: As Figure 1 As shown in the figure, this embodiment provides a method for monitoring and early warning of personnel behavior at dams, including the following steps:

[0069] S1. Obtain a visible light image of the dam. I vis and low-resolution infrared imagesI LR , I LR ∈ R Cin×H×W .in, i Represents an infrared image. C in Indicates the number of input channels. H Indicates the image height. W Indicates the image width.

[0070] S2. Use super-resolution reconstruction methods to reconstruct low-resolution infrared images. I LR Reconstructed into high-resolution infrared images I SR , I SR ∈ R Cout×H×W , C out Indicates the number of output channels. Specifically, for example... Figure 2 As shown, S2 includes the following steps:

[0071] S201. Extract low-resolution infrared images using a 3×3 convolutional layer. I LR shallow features F 0 , I 0 ∈ R Cin×H×W The formula is: ;

[0072] in, Conv 3×3 (·) indicates a 3×3 convolution.

[0073] S202, Shallow features F 0 As input, deep features are obtained through a deep feature extraction algorithm. F’ final Deep feature extraction algorithms utilize i A residual multi-head attention group and a 3×3 convolutional layer H Conv (·) to process. Specifically, such as Figure 3 As shown, S202 includes the following steps:

[0074] S20201, For shallow features F 0 First, two convolutional layers are used. H Conve1 (·) HConve2 (·) and an activation layer H Acti (·) Perform compressed feature processing, and then combine with shallow features F 0 Residual join processing is performed, and finally, refinement is carried out to remove redundant information and obtain the processed features. F Z The specific formula is as follows:

[0075] F Z = H Conve2 ( H Acti ( H Conve1 ( F 0 ))) + F 0 .

[0076] S20202, Regarding the processing features F Z First, it is processed through an adaptive average pooling layer. H avg (·) is processed to obtain adaptive average pooling features. F avg Adaptive average pooling can automatically adjust the pooling region based on the characteristics of the features, thereby reducing the spatial dimension of the feature map while retaining more valuable information in the channels. The specific formula is as follows:

[0077] F avg = H avg ( F Z ).

[0078] S20203, Using Adaptive 1D Convolutional Layers H k Adap_Conv (·) For features after adaptive average pooling F avg Perform one-dimensional convolution processing to adaptively capture local dependencies between channels. F Conv The specific formula is as follows:

[0079] F Conv = H k Adap_Conv ( F avg ).

[0080] Among them, the adaptive 1D convolutional layerH k Adap_Conv The size of the convolution kernel in (·) k It is based on shallow features F 0 Number of input channels C in The adaptive determination is based on the following formula: ,in, γ and b This represents the pre-defined mapping function parameters in this invention. γ and b Set them to 2 and 1 respectively. This indicates that the integer part is rounded down to the nearest odd number, ensuring that the kernel size is odd to guarantee the symmetry of the convolution process.

[0081] S20204, Use sigmoid Activation function H Sig (·) Handling local dependencies F Conv The output values ​​are then mapped to the [0,1] interval to obtain the channel attention weights. W channel The specific formula is as follows:

[0082] W channel = H Sig ( F Conv ).

[0083] S20205, Adjust channel attention weights W channel shallow features F 0 Perform efficient channel attention activation processing, specifically, channel attention weights. W channel shallow features F 0 Multiply and combine with shallow features F 0 Residual connections yield activation features. F’ 0 The specific formula is as follows:

[0084] F’ 0 = F 0 · W channel + F 0 .

[0085] S20206, Activation FeaturesF’ 0 Perform multi-head self-attention calculation to obtain multi-head self-attention values. F MSA Multi-head self-attention mechanisms capture richer information by focusing on different parts of the input sequence in parallel across different representation subspaces. The specific formula is as follows:

[0086] F MSA = MSA ( F’ 0 );

[0087] in, MSA (·) indicates multi-head self-attention processing. F MSA This represents the multi-head self-attention value.

[0088] S20207, Focus on Bullish Self-Attention Value F MSA Perform line normalization to obtain the line normalization result. F LN Linear normalization makes the feature distribution more stable and reduces the gradient vanishing problem. The specific formula is as follows:

[0089] F LN = LN ( F MSA );

[0090] in, LN (·) indicates line normalization.

[0091] S20208, Based on the line normalization results F LN Shallow features F 0 and activation features F’ 0 Calculate intermediate features F mid The specific formula is as follows:

[0092] F mid = F LN + F 0 + βF’ 0；

[0093] in, β This represents a learnable parameter used to control the weights of the fusion process.

[0094] S20209. Perform multi-layer perceptron and layer normalization processing on the intermediate features to obtain the layer normalization result. LN ( MLP ( F mid Multilayer sensing utilizes multiple fully connected layers and activation functions to perform nonlinear transformations on features, enhancing the model's expressive power. Multilayer sensing processes... MLP (·), then the features after multi-layer perceptron processing are MLP ( F mid (Again, regarding the features) MLP ( F mid Perform layer normalization to obtain the layer normalization result. LN ( MLP ( F mid )).

[0095] S20210, Normalize the layer results LN ( MLP ( F mid and intermediate features F mid Residual connections are used to obtain output features. F final Residual connections can help alleviate the vanishing gradient problem, making the model easier to train. The specific formula is as follows:

[0096] F final = LN ( MLP ( F mid )) + F mid .

[0097] S20211. Perform multiple residual multi-head self-attention processes on the output features to obtain deep features. F’ final .

[0098] S203, Shallow features F 0 with deep features F’ final After performing residual connection, reconstruction processing is performed to obtain a high-resolution infrared image. I SR The specific formula is as follows:

[0099] I SR = H Rec ( F0 + F’ final ) + F 0 ;

[0100] in, H Rec (·) indicates image reconstruction processing.

[0101] S3. The high-resolution infrared image and the visible light image are collaboratively input into the human posture recognition network to obtain the key feature vectors of the currently monitored person. Specifically, such as... Figure 4 As shown, S3 includes the following steps:

[0102] S301. Acquire visible light image I vis and high-resolution infrared images I SR .

[0103] S302, View visible light image I vis and high-resolution infrared images I SR Preliminary fusion processing is performed to obtain the fused image. I fused Specifically, such as Figure 5 As shown, S302 includes the following steps:

[0104] S30201. Perform multi-scale decomposition on the high-resolution infrared image and the visible light image respectively to obtain low-frequency components and high-frequency components at different scales. L n ir and H n ir These represent the infrared images of the first and second generations, respectively. n The low-frequency and high-frequency components of the layer, L n vis and H n vis These represent the visible light images. The low-frequency and high-frequency components of the layer.

[0105] S30202. For low-frequency components, a weighted average method is used to fuse the low-frequency components to obtain the low-frequency component fusion result. L n fused The specific formula is as follows:

[0106] ;

[0107] in,α n Indicates the first n The fusion weights of the low-frequency components of the layer are learnable parameters, initially set to 1.

[0108] S30203. For high-frequency components, a region-based energy-based selective fusion method is used to calculate the energy of the high-frequency components in each region. E n ir and energy E n vis Regions with higher energy are selected as high-frequency classification fusion results. H n fused The specific formula is as follows:

[0109] .

[0110] S30204, Fusing the low-frequency components L n fused Fusion results with high-frequency classification H n fused Image reconstruction is performed to obtain the fused image. I fused .

[0111] S303, adopts ViTPose The encoder processes the fused image. I fused Feature extraction is performed to obtain a multi-scale fused feature map. F t,output Specifically, such as Figure 6 As shown, S303 includes the following steps:

[0112] S30301, The merged image I fused Divide into primitives and use ViTPose The encoder extracts features at depth to obtain local and global spatial information.

[0113] S30302. For primitives, extract feature maps from different layers in the feature extraction network (in this example, layers 9, 21, and the last layer are selected). F t and for each feature map F t Adaptive pooling is performed at multiple scales (1×1, 2×2, 3×3, and 6×6) to generate adaptive pooling features; the specific formula is as follows:

[0114] ;

[0115] in, F t Indicates the first t Feature map of frame image AvgPool (·) indicates average pooling. Conv 1×1 (·) indicates a 1×1 convolution. BN (·) indicates layer normalization processing. ReLU (·) represents the activation function.

[0116] S30303: Convolutional processing is used to reduce the embedding dimension to one-quarter of the original. Then, the height and width dimensions of the feature are restored through a difference layer. Finally, the original feature map is concatenated with the upsampled pooled feature map to obtain a multi-scale feature map. The specific formula is as follows:

[0117] ;

[0118] Wherein, [·] indicates that multiple feature maps are concatenated along the channel dimension. F (l) t Indicates the number of features extracted from the feature extraction network. l The first layer extracted t Feature map of the frame P (p*) This indicates different pooling scales.

[0119] S30304, Multi-scale feature maps Perform unified feature dimension processing on convolutional layers, and then perform layer normalization and... ReLU Activation processing yields activated multi-scale feature maps. The specific formula is as follows:

[0120] .

[0121] S30305, Multi-scale feature maps after activation of different layers Features are obtained by connecting along the channel dimension F t,cat Then, the spatial position dependencies between consecutive frames are calculated using a multi-head attention algorithm. Attn ( F t,cat The specific formula is as follows:

[0122] ;

[0123] in, W q Indicates features F t,cat Mapped to a query vectorW k Indicates features F t,cat Mapped to key vectors W v Indicates features F t,cat Mapped to a value vector express Features obtained by connecting along the channel dimension T This represents the matrix transpose operation. d k Indicates the scaling factor. softmax (·) represents the activation function.

[0124] S30306, Features F t,cat and dependency Attn ( F t,cat The residual connections are performed, and then processed by average pooling and linear transformation. Finally, a multi-scale fused feature map is obtained through a projection layer. F t,output The specific formula is as follows:

[0125] ;

[0126] ;

[0127] ;

[0128] .

[0129] in, Norm (·) indicates normalization. This represents the characteristics after normalization. AvgPool (·) indicates average pooling. F t,fused This represents the features after average pooling. Linear (·) indicates a linear transformation. Reshape (·) indicates reshaping. B Indicates the batch size.

[0130] S304. An adaptive frame weighting method is adopted to fuse feature maps at multiple scales based on the correlation of frames. F t,output Dynamically assign weights. This step calculates the quality score for each feature map using the learned quality estimation function, and then... softmax Function normalization yields weights w t , softmax The formula for the function is:

[0131] ;

[0132] The formula for the quality estimation function is:

[0133] ;

[0134] in, Conv 3×3 (·) indicates a 3×3 convolution process. ReLU (·) indicates activation function processing. AvgPool (·) indicates average pooling. Flatten (·) indicates flattening. Linear (·) denotes a linear transformation; where, S This indicates the quality score of the current frame. S t Indicates the first t Frame image quality score.

[0135] S305, Utilizing Weights w t Multi-scale fusion feature maps F t,output We perform weighting to obtain a weighted feature map. F t,w , F t,w = w t · F t,output This step allows the model to focus more on key information frames, improving the accuracy of pose estimation in complex situations.

[0136] S306, using feature maps F t,w Feature map of the center frame F tc For the query vector, the feature map of the context frame consists of a key vector and a value vector. Before calculating cross-attention, the context frame features are first processed. F contest A self-attention layer is applied to calculate the attention weights between the positions in the center frame and the positions in the context frames. α i,j The specific formula is as follows:

[0137] ;

[0138] in, F tc,i Indicates the first i Feature map of frame image q Represents the query vector. k Represents the key vector. v Represents a value vector. dk This represents the scaling factor.

[0139] S307, through Update all feature maps and through ViTPose The decoder generates keypoint prediction results, which are then used as feature vectors for key nodes in human pose. X t , . ViTPose The decoder, consisting of two deconvolutional blocks and one convolutional layer, is responsible for converting the feature maps generated by the feature extraction network into keypoint heatmaps representing the estimated locations of human joints.

[0140] S4. Transfer the feature vector of the joint points X t The data is input into the personnel behavior monitoring network, and a judgment threshold is used to determine whether the behavior of the monitored person constitutes an abnormal behavior state. Specifically, for example... Figure 7 As shown, S4 includes the following steps:

[0141] S401, Transfer the joint feature vector X t The predicted and normal results are obtained by sequentially inputting the feature vectors of normal joint points into the normal human behavior model. The normal human behavior model is constructed using a Gaussian mixture model, which consists of M Gaussian distributions with the following probability density function:

[0142] ;

[0143] in, ω m Indicates the first m The weights of a Gaussian distribution No. m A Gaussian distribution, x Represents the feature vector of the key point. μ m Let ∑ represent the mean vector. m Representing the covariance matrix, the model parameters are estimated using the expectation-maximization algorithm. .

[0144] S402. Calculate the log-likelihood probability of the predicted and normal results to obtain the probability values. The specific formula is as follows:

[0145] .

[0146] S403, Set threshold θ ,when LL ( X t )< θIf the behavior is abnormal, then it is considered normal; otherwise, it is considered normal.

[0147] Example 2: Based on the same inventive concept, this example also provides a dam personnel behavior monitoring and early warning system, including:

[0148] The image acquisition module is used to acquire visible light and low-resolution infrared images of the dam.

[0149] The image reconstruction module is used to reconstruct low-resolution infrared images into high-resolution infrared images using super-resolution reconstruction methods.

[0150] The feature recognition module is used to input high-resolution infrared images and visible light images into the human posture recognition network to obtain the joint feature vectors of the currently monitored person.

[0151] The behavior judgment module is used to input the key feature vector into the personnel behavior monitoring network and determine whether the behavior of the currently monitored personnel belongs to an abnormal behavior state based on the judgment threshold.

[0152] Example 3: Based on the same inventive concept, this example also provides a dam personnel behavior monitoring and early warning device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the method of Example 1.

[0153] Example 4: Based on the same inventive concept, this example also provides a computer-readable storage medium storing executable instructions thereon, characterized in that, when the instructions are executed by a processor, the processor performs the method of Example 1.

[0154] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring and early warning of personnel behavior at dams, characterized in that, Includes the following steps: S1. Acquire visible light and low-resolution infrared images of the dam; S2. Reconstruct the low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; S2 includes the following steps: S201. Perform convolution processing on the low-resolution infrared image to obtain shallow features; S202. Using shallow features as input, obtain deep features through a deep feature extraction algorithm; S202 includes the following steps: S20201. Perform feature compression, residual connection and refinement on shallow features to obtain processed features; S20202. Adaptive average pooling is applied to the processed features to obtain adaptive average pooling features. S20203. Adaptive convolution processing is performed on the adaptive average pooling features to obtain the local dependencies between channels; S20204. Perform sigmoid activation on the local dependencies and map the output values ​​to the [0,1] interval to obtain the channel attention weights; S20205. Perform efficient channel attention activation processing on the channel attention weights and shallow features to obtain activated features; S20206. Perform multi-head self-attention calculation on the activation features to obtain the multi-head self-attention value; S20207. Perform line normalization on the multi-head self-attention value to obtain the line normalization result; S20208. Based on the line normalization result, shallow features, and activation features, intermediate features are calculated. S20209. Perform multi-layer perception and layer normalization on the intermediate features to obtain the layer normalization result; S20210. Calculate the output features based on the layer normalization results and intermediate features; S20211. Perform multiple residual multi-head self-attention processes on the output features to obtain deep features; S203. After performing residual connection between shallow and deep features, reconstruction processing is carried out to obtain a high-resolution infrared image. S3. Perform preliminary fusion processing on the high-resolution infrared image and the visible light image to obtain the fused image, and input it into the human posture recognition network to obtain the joint feature vector of the currently monitored person. S4. Input the key feature vector into the personnel behavior monitoring network, and determine whether the behavior of the currently monitored personnel belongs to an abnormal behavior state based on the judgment threshold.

2. The method for monitoring and early warning of personnel behavior at a dam according to claim 1, characterized in that, S3 includes the following steps: S301. Acquire visible light images and high-resolution infrared images; S302. Perform preliminary fusion processing on the high-resolution infrared image and the visible light image to obtain the fused image; S303. The ViTPose encoder is used to extract features from the fused image to obtain a multi-scale fused feature map. S304. An adaptive frame weighting method is adopted to dynamically assign weights to the multi-scale fusion feature map based on the correlation of frames. S305. Weight the multi-scale fused feature map using weights to obtain a weighted feature map; S306. Using the feature map of the center frame of the feature map as the query vector and the feature map of the context frame as the key vector and value vector, calculate the attention weights between the positions in the center frame and the positions in the context frame. S307. Update all feature maps according to attention weights and generate human pose key node feature vectors through the ViTPose decoder.

3. The method for monitoring and early warning of personnel behavior at a dam according to claim 2, characterized in that, S302 includes the following steps: S30201. Perform multi-scale decomposition on high-resolution infrared images and visible light images respectively to obtain low-frequency components and high-frequency components at different scales. S30202. The low-frequency components are fused using a weighted average method to obtain the low-frequency component fusion result; S30203. The energy of high-frequency components in each region is calculated using a region-based energy selection fusion method, and the region with higher energy is taken as the high-frequency classification fusion result. S30204. Reconstruct the image by combining the low-frequency component fusion result and the high-frequency classification fusion result to obtain the fused image.

4. The method for monitoring and early warning of personnel behavior at a dam according to claim 2, characterized in that, S303 includes the following steps: S30301. Segment the fused image into primitives, use the ViTPose encoder to extract features at depth, and obtain local and global spatial information. S30302. For primitives, feature maps are extracted from different layers in the feature extraction network, and each feature map is subjected to adaptive pooling at multiple scales to generate adaptive pooling features. S30303: Convolution is used to reduce the embedding dimension to one-quarter of the original. Then, the height and width dimensions of the feature are restored through the difference layer. Finally, the original feature map is connected with the upsampled pooled feature map to obtain a multi-scale feature map. S30304. Perform convolutional layer processing on multi-scale feature maps to unify feature dimensions, and then perform layer normalization and... ReLU Activation processing yields activated multi-scale feature maps; S30305. Connect the activated multi-scale feature maps of different layers along the channel dimension to obtain features, and then calculate the spatial position dependency between consecutive frames through a multi-head attention algorithm. S30306. Residual connections are made between features and dependencies, and a unified multi-scale representation is obtained through average pooling and linear transformation. Finally, a multi-scale fused feature map is obtained through a projection layer.

5. The method for monitoring and early warning of personnel behavior at a dam according to claim 1, characterized in that, S4 includes the following steps: S401. Input the joint feature vector and the normal joint feature vector into the human normal behavior model in sequence to obtain the prediction result and the normal result; S402. Calculate the log-likelihood probability of the predicted results and the normal results to obtain the probability values; S403. If the probability value is less than the set threshold, the personnel behavior is considered abnormal; otherwise, it is considered normal.

6. A dam personnel behavior monitoring and early warning system, applied to the dam personnel behavior monitoring and early warning method described in claim 1, characterized in that, include: The image acquisition module is used to acquire visible light and low-resolution infrared images of the dam. The image reconstruction module is used to reconstruct low-resolution infrared images into high-resolution infrared images using super-resolution reconstruction methods. The feature recognition module is used to input high-resolution infrared images and visible light images into the human posture recognition network to obtain the joint feature vectors of the currently monitored person. The behavior judgment module is used to input the key feature vector into the personnel behavior monitoring network and determine whether the behavior of the currently monitored personnel belongs to an abnormal behavior state based on the judgment threshold.

7. A monitoring and early warning device for personnel behavior at a dam, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to perform the method according to any one of claims 1 to 5.

Images