Dam personnel behavior monitoring and early warning method, system and device and medium

Abstract

The invention provides a dam personnel behavior monitoring and early warning method, system and device and a medium. The method comprises the following steps: S1, obtaining a visible light image and a low-resolution infrared image of a dam; s2, reconstructing the low-resolution infrared image into a high-resolution infrared image by using a super-resolution reconstruction method; s3, cooperatively inputting the high-resolution infrared image and the visible light image into a human body posture recognition network to obtain a joint point feature vector of the current monitored person; and S4, inputting the joint point feature vector into a personnel behavior monitoring network, and judging whether the behavior of the current monitored personnel belongs to an abnormal behavior state or not according to a judgment threshold value. According to the method, through infrared image super-resolution reconstruction and infrared and visible light image collaborative personnel attitude estimation, the personnel behavior monitoring precision is remarkably improved, and subtle actions and attitude changes of personnel can be captured more accurately.

Description

Technical Field

[0001] The invention belongs to the technical field of water conservancy facility monitoring, and in particular relates to a dam personnel behavior monitoring and early warning method, system, device and medium. Background Art

[0002] The dam is the core facility of the water conservancy project, so the safety monitoring of the behavior of people entering the safe area of ​​the dam is a key link to ensure the stable operation of the project. The traditional dam personnel behavior monitoring technology has the following core problems: the existing monitoring system mainly relies on the single-mode data of visible light cameras, and has problems such as low image resolution and loss of details in long-distance monitoring scenarios. In complex environments such as nighttime and haze, the traditional visual system is easily affected by the interference of lighting changes, resulting in the failure of recognition of key behavioral features (such as safety rope wearing status and climbing movements). Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a dam personnel behavior monitoring and early warning method, system, device and medium, which can realize accurate identification of dam personnel behavior in a complex environment.

[0004] The present invention is implemented as follows: a dam personnel behavior monitoring and early warning method comprises the following steps: S1, obtain visible light images and low-resolution infrared images of the dam; S2, reconstructing the low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; S3, inputting the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint point feature vector of the currently monitored person; S4. Input the joint point feature vector into the personnel behavior monitoring network, and judge whether the behavior of the currently monitored person belongs to an abnormal behavior state according to the judgment threshold.

[0005] Further, S2 includes the following steps: S201, performing convolution processing on the low-resolution infrared image to obtain shallow features; S202, using shallow features as input, and obtaining deep features through a deep feature extraction algorithm; S203, performing residual connection on the shallow features and the deep features and then reconstructing them to obtain a high-resolution infrared image.

[0006] Further, S202 includes the following steps: S20201, compressing features, performing residual connection and refining processing on shallow features to obtain processed features; S20202, performing adaptive average pooling processing on the processed features to obtain adaptive average pooling features; S20203, performing adaptive convolution processing on the adaptive average pooling features to obtain local dependencies between channels; S20204. Perform local dependencies sigmoid Activate the process and map the output value to the interval [0,1] to obtain the channel attention weight; S20205. Perform efficient channel attention activation processing on the channel attention weight and the shallow feature to obtain the activated feature; S20206. Perform multi-head self-attention calculation on the activated features to obtain a multi-head self-attention value; S20207. Perform line normalization processing on the multi-head self-attention values ​​to obtain a line normalization result; S20208. Calculate the intermediate features according to the line normalization result, the shallow features and the activation features; S20209, performing multi-layer perception and layer normalization processing on the intermediate features to obtain a layer normalization result; S20210, calculating output features according to the layer normalization result and the intermediate features; S20211. Perform multiple residual multi-head self-attention processes on the output features to obtain deep features.

[0007] Furthermore, S3 includes the following steps: S301, acquiring a visible light image and a high-resolution infrared image; S302, performing preliminary fusion processing on the high-resolution infrared image and the visible light image to obtain a fused image; S303, using the ViTPose encoder to extract features from the fused image to obtain a multi-scale fusion feature map; S304, using an adaptive frame weighting method to dynamically assign weights to the multi-scale fusion feature map according to the relevance of the frames; S305, weighting the multi-scale fusion feature map using weights to obtain a weighted feature map; S306, using the feature map of the center frame of the feature map as the query vector, and the feature map of the context frame as the key vector and the value vector, calculating the attention weights of the position in the center frame and the position in the context frame; S307, update all feature maps according to the attention weights, and generate key node feature vectors of human posture through the ViTPose decoder.

[0008] Further, S302 includes the following steps: S30201, performing multi-scale decomposition on the high-resolution infrared image and the visible light image respectively to obtain low-frequency components and high-frequency components of different scales; S30202, using a weighted average method to fuse the low-frequency components to obtain a low-frequency component fusion result; S30203, using a regional energy-based selection fusion method to calculate the energy of the high-frequency component in each region, and taking the region with larger energy as the high-frequency classification fusion result; S30204. Reconstruct the low-frequency component fusion result and the high-frequency classification fusion result to obtain a fused image.

[0009] Further, S303 includes the following steps: S30301, segment the fused image into primitives, use the ViTPose encoder to deeply extract features, and obtain local and global spatial information; S30302. For the primitive, extract feature maps from different layers in the feature extraction network, and perform adaptive pooling processing on each feature map at multiple scales to generate adaptive pooling features; S30303, using convolution processing to reduce the embedding dimension to one-fourth of the original, then restoring the height and width dimensions of the feature through a difference layer, and finally connecting the original feature map with the upsampled pooled feature map to obtain a multi-scale feature map; S30304, perform convolution layer unified feature dimension processing on the multi-scale feature map, and then perform layer normalization and Activation processing to obtain the activated multi-scale feature map; S30305, connecting the activated multi-scale feature maps of different layers along the channel dimension to obtain features, and then calculating the spatial position dependency between consecutive frames through a multi-head attention algorithm; S30306. Perform residual connection on features and dependencies, and obtain a unified multi-scale representation through average pooling and linear transformation. Finally, obtain a multi-scale fusion feature map through the projection layer.

[0010] Further, S4 includes the following steps: S401, inputting the joint point feature vector and the normal joint point feature vector into the normal human behavior model in sequence to obtain a prediction result and a normal result; S402, performing log-likelihood probability calculation on the predicted result and the normal result to obtain a probability value; S403. If the probability value is less than the set threshold, the personnel behavior is abnormal behavior, otherwise it is normal behavior.

[0011] The present invention also provides a dam personnel behavior monitoring and early warning system, comprising: An image acquisition module, used to acquire visible light images and low-resolution infrared images of the dam; An image reconstruction module, used for reconstructing a low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; The feature recognition module is used to input the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint feature vector of the currently monitored person; The behavior judgment module is used to input the joint point feature vector into the personnel behavior monitoring network and judge whether the behavior of the currently monitored person is in an abnormal behavior state according to the judgment threshold.

[0012] The present invention also provides a dam personnel behavior monitoring and early warning device, comprising a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0013] The present invention also provides a computer-readable storage medium on which executable instructions are stored, characterized in that when the instructions are executed by a processor, the processor executes the above method.

[0014] The beneficial effects brought by the present invention are: 1. The present invention significantly improves the accuracy of personnel behavior monitoring through infrared image super-resolution reconstruction and personnel posture estimation in coordination with infrared and visible light images, and can more accurately capture the subtle movements and posture changes of personnel. At the same time, infrared images can still maintain good imaging effects at night and in bad weather conditions. Combined with visible light images, all-weather personnel behavior monitoring is achieved, ensuring that personnel on the dam can be effectively monitored in any environment.

[0015] 2. The present invention can detect abnormal behavior in time and issue early warnings through the identification of abnormal behavior of personnel, effectively prevent potential safety risks, ensure the safe operation of the dam, realize automated and intelligent management, reduce manual intervention, improve the efficiency of dam safety management, and reduce management costs.

[0016] 3. The present invention can accumulate a large amount of monitoring data, provide data support and decision-making basis for subsequent dam safety management, help optimize management strategies, and improve the scientificity and effectiveness of dam safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method in the present invention; Figure 2 is a flow chart of S2 in the present invention; Figure 3 This is a flow chart of S202 in the present invention; Figure 4 is a flow chart of S3 in the present invention; Figure 5 This is a flow chart of S302 in the present invention; Figure 6 This is a flow chart of S303 in the present invention; Figure 7 This is a flow chart of S4 in the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0019] Embodiment 1: Figure 1 As shown, this embodiment provides a dam personnel behavior monitoring and early warning method, including the following steps: S1. Obtaining visible light images of the dam I vis and low-resolution infrared images I LR , I LR ∈ R Cin×H×W .in, i represents an infrared image, C in Indicates the number of input channels, H Indicates the image height, W Indicates the image width.

[0020] S2. Use super-resolution reconstruction method to reconstruct low-resolution infrared images I LR Reconstruction into high-resolution infrared images I SR , I SR ∈ R Cout×H×W , C out Indicates the number of output channels. Specifically, Figure 2 As shown, S2 includes the following steps: S201, use a 3×3 convolutional layer to extract low-resolution infrared images I LR Shallow features F 0 , I 0 ∈ R Cin×H×W , the formula is: ; in, Conv 3×3 (·) indicates 3×3 convolution processing.

[0021] S202, shallow features F0 As input, deep features are obtained through deep feature extraction algorithm F’ final . Deep feature extraction algorithm uses i A residual multi-head attention group and a 3×3 convolutional layer H Conv (·) for processing. Specifically, Figure 3 As shown, S202 includes the following steps: S20201. For shallow features F 0 , first use two convolutional layers H Conve1 (·), H Conve2 (·) and an activation layer H Acti (·) Compress the features and then combine them with the shallow features F 0 Perform residual connection processing, and finally refine it to remove redundant information and obtain processing features F Z , the specific formula is as follows: F Z = H Conve2 ( H Acti ( H Conve1 ( F 0 ))) + F 0 .

[0022] S20202. For processing features F Z , first processed by the adaptive average pooling layer H avg (·) is processed to obtain the adaptive average pooling feature F avg Adaptive average pooling can automatically adjust the pooling area according to the characteristics of the feature to reduce the spatial dimension of the feature map while retaining more valuable information in the channel. The specific formula is as follows: F avg = H avg ( F Z ).

[0023] S20203. Use adaptive 1D convolutional layers H k Adap_Conv(·) For features after adaptive average pooling F avg Perform one-dimensional convolution to adaptively capture local dependencies between channels F Conv , the specific formula is as follows: F Conv = H k Adap_Conv ( F avg ).

[0024] Among them, adapt to the 1D convolution layer H k Adap_Conv (·) The size of the convolution kernel k Based on shallow features F 0 Number of input channels C in Adaptively determined, the calculation formula is: ,in, γ and b Represents a preset mapping function parameter. γ and b Set to 2 and 1 respectively, It means rounding down to the nearest odd number to ensure that the convolution kernel size is an odd number to ensure the symmetry of the convolution process.

[0025] S20204、Use sigmoid Activation Function H Sig (·) Handling local dependencies F Conv , and map the output value to the [0,1] interval to obtain the channel attention weight W channel , the specific formula is as follows: W channel = H Sig ( F Conv ).

[0026] S20205. Channel attention weight W channel With shallow features F 0 Perform efficient channel attention activation processing, specifically, channel attention weight W channel With shallow features F 0 Multiply and add to the shallow features F0 Residual connection to get activation features F’ 0 The specific formula is as follows: F’ 0 = F 0 · W channel + F 0 .

[0027] S20206, activate features F’ 0 Perform multi-head self-attention calculation to obtain multi-head self-attention value F MSA The multi-head self-attention mechanism captures richer information by focusing on different parts of the input sequence in parallel in different representation subspaces. The specific formula is as follows: F MSA = MSA ( F’ 0 ); in, MSA (·) indicates multi-head self-attention processing, F MSA Represents the multi-head self-attention value.

[0028] S20207, for multi-head self-attention value F MSA Perform line normalization processing to obtain line normalization results F LN ; Line normalization makes the feature distribution more stable and reduces the problem of gradient disappearance. The specific formula is as follows: F LN = LN ( F MSA ); in, LN (·) indicates line normalization processing.

[0029] S20208, Normalize the results based on the line F LN , shallow features F 0 and Activate Features F’ 0 Calculate the intermediate features F mid , the specific formula is as follows: F mid = FLN + F 0 + βF’ 0； in, β Represents a learnable parameter that controls the weight of the fusion.

[0030] S20209. Perform multi-layer perception and layer normalization on the intermediate features to obtain the layer normalization result. LN ( MLP ( F mid )). Multi-layer perception uses multiple fully connected layers and activation function processing to perform nonlinear transformations on features and enhance the model's expressiveness. MLP (·), then the feature after multi-layer perception processing is MLP ( F mid ). Again, MLP ( F mid ) to perform layer normalization processing and obtain the layer normalization result LN ( MLP ( F mid )).

[0031] S20210, normalize the layer LN ( MLP ( F mid )) and intermediate features F mid Residual connection to obtain output features F final ; Residual connections can help alleviate the gradient vanishing problem and make the model easier to train. The specific formula is as follows: F final = LN ( MLP ( F mid )) + F mid .

[0032] S20211. Perform multiple residual multi-head self-attention processing on the output features to obtain deep features F’ final .

[0033] S203, shallow features F 0 With deep features F’ final After residual connection, reconstruction processing is performed to obtain a high-resolution infrared imageI SR , the specific formula is as follows: I SR = H Rec ( F 0 + F’ final ) + F 0 ; in, H Rec (·) indicates image reconstruction processing.

[0034] S3, input the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint feature vector of the currently monitored person. Figure 4 As shown, S3 includes the following steps: S301, obtaining visible light image I vis and high-resolution infrared images I SR .

[0035] S302: visible light image I vis and high-resolution infrared images I SR Perform preliminary fusion processing to obtain the fused image I fused . Specifically, Figure 5 As shown, S302 includes the following steps: S30201. Perform multi-scale decomposition on the high-resolution infrared image and the visible light image respectively to obtain low-frequency components and high-frequency components of different scales. L n ir and H n ir Represents the infrared image n The low-frequency and high-frequency components of the layer, L n vis and H n vis Represents the visible light image The low-frequency and high-frequency components of the layer.

[0036] S30202: For low-frequency components, a weighted average method is used to fuse the low-frequency components to obtain a low-frequency component fusion result. L n fused; The specific formula is as follows: ; in, α n Indicates n The fusion weight of the low-frequency component of the layer is a learnable parameter and is initially set to 1.

[0037] S30203. For high-frequency components, a selection fusion method based on regional energy is used to calculate the energy of high-frequency components in each region. E n ir and energy E n vis , select the area with larger energy as the high-frequency classification fusion result H n fused ; The specific formula is as follows: .

[0038] S30204, low frequency component fusion result L n fused And high frequency classification fusion results H n fused Perform image reconstruction to obtain the fused image I fused .

[0039] S303, use ViTPose The encoder pairs the fused image I fused Perform feature extraction to obtain a multi-scale fusion feature map F t,output . Specifically, Figure 6 As shown, S303 includes the following steps: S30301, the fused image I fused Split into primitives, use ViTPose The encoder deeply extracts features and obtains local and global spatial information.

[0040] S30302. For primitives, extract feature maps from different layers in the feature extraction network (in this case, the 9th, 21st, and last layers) F t , and for each feature map F t Adaptive pooling is performed on multiple scales (1×1, 2×2, 3×3, and 6×6) to generate adaptive pooling features; the specific formula is as follows: ; in, F t Indicates t Feature map of frame image, AvgPool (·) represents average pooling processing, Conv 1×1 (·) indicates 1×1 convolution processing, BN (·) indicates layer normalization processing, ReLU (·) denotes the activation function.

[0041] S30303, use convolution processing to reduce the embedding dimension to one-fourth of the original, then restore the height and width dimensions of the feature through the difference layer, and finally connect the original feature map with the upsampled pooled feature map to obtain a multi-scale feature map ; The specific formula is as follows: ; Among them, [·] means concatenating multiple feature maps in the channel dimension. F (l) t Represents the feature extraction network l The first layer of extraction t Feature map of the frame, P (p*) Indicates different pooling scales.

[0042] S30304. Multi-scale feature map Perform convolutional layer to unify feature dimensions, and then perform layer normalization and ReLU Activation processing to obtain the activated multi-scale feature map ; The specific formula is as follows: .

[0043] S30305, the multi-scale feature maps after activation of different layers Connect along the channel dimension to get features F t,cat , and then the multi-head attention algorithm is used to calculate the spatial position dependency between consecutive frames Attn ( F t,cat ); the specific formula is as follows: ; in, W q Indicates that the feature F t,cat Mapped into a query vector, W k Indicates that the feature F t,cat Mapped to a key vector,W v Indicates that the feature F t,cat Mapped to a vector of values, express The features obtained by connecting along the channel dimension, T represents the matrix transpose operation, d k represents the scaling factor, softmax (·) denotes the activation function.

[0044] S30306, the characteristics F t,cat and dependencies Attn ( F t,cat ) is used for residual connection, and processed by average pooling and linear transformation, and finally a multi-scale fusion feature map is obtained through the projection layer. F t,output The specific formula is as follows: ; ; ; .

[0045] in, Norm (·) indicates normalization, represents the normalized features, AvgPool (·) represents average pooling, F t,fused represents the feature after average pooling, Linear (·) indicates linear transformation processing, Reshape (·) indicates reshaping, B Indicates the batch size.

[0046] S304, using an adaptive frame weighting method to generate a multi-scale fusion feature map according to the correlation of the frames F t,output Dynamically assign weights. This step uses the learned quality estimation function to calculate the quality score of each feature map, and then softmax Function normalization to get weight w t , softmax The function formula is: ; The formula of the quality estimation function is: ; in, Conv 3×3 (·) indicates 3×3 convolution processing, ReLU (·) indicates activation function processing, AvgPool (·) represents average pooling processing, Flatten (·) indicates flattening. Linear (·) represents linear transformation processing; where S Indicates the quality score of the current frame. S t Indicates t Frame image quality score.

[0047] S305, using weight w t Multi-scale fusion feature map F t,output Weighted to get the weighted feature map F t,w , F t,w = w t · F t,output This step makes the model pay more attention to key information frames and improves the accuracy of posture estimation in complex situations.

[0048] S306, feature map F t,w Feature map of the center frame F tc is the query vector, and the feature map of the context frame is the key vector and the value vector. Before calculating the cross attention, the context frame feature F contest Apply a self-attention layer to calculate the attention weights of the position in the center frame and the position in the context frame α i,j The specific formula is as follows: ; in, F tc,i Indicates i Feature map of frame image, q represents the query vector, k represents the key vector, v represents a value vector, d k Represents the scaling factor.

[0049] S307, through Update all feature maps and through ViTPose The decoder generates key point prediction results and uses the key point prediction results as the key node feature vector of the human body posture X t , . ViTPoseThe decoder, consisting of two deconvolution blocks and one convolutional layer, is responsible for converting the feature maps generated by the feature extraction network into keypoint heatmaps representing the estimated locations of human joints.

[0050] S4. The joint point feature vector X t The data is input into the personnel behavior monitoring network, and the judgment threshold is used to determine whether the behavior of the currently monitored person is abnormal. Figure 7 As shown, S4 includes the following steps: S401, joint point feature vector X t The normal joint feature vectors are input into the normal human behavior model in turn to obtain the predicted results and normal results. The normal human behavior model is constructed by a Gaussian mixture model, which consists of M Gaussian distributions, and its probability density function is: ; in, ω m Indicates m The weights of a Gaussian distribution, No. m Gaussian distribution, x represents the feature vector of the joint point, μ m represents the mean vector, ∑ m Represents the covariance matrix, and estimates the model parameters through the expectation maximization algorithm .

[0051] S402: Perform log-likelihood probability calculation on the predicted result and the normal result to obtain a probability value. The specific formula is as follows: .

[0052] S403: Setting threshold θ ,when LL ( X t )< θ , then the personnel behavior is abnormal, otherwise it is normal.

[0053] Embodiment 2: Based on the same inventive concept, this embodiment also provides a dam personnel behavior monitoring and early warning system, including: An image acquisition module, used to acquire visible light images and low-resolution infrared images of the dam; An image reconstruction module, used for reconstructing a low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; The feature recognition module is used to input the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint feature vector of the currently monitored person; The behavior judgment module is used to input the joint point feature vector into the personnel behavior monitoring network and judge whether the behavior of the currently monitored person is in an abnormal behavior state according to the judgment threshold.

[0054] Embodiment 3: Based on the same inventive concept, this embodiment also provides a dam personnel behavior monitoring and early warning device, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method of embodiment 1.

[0055] Embodiment 4: Based on the same inventive concept, this embodiment further provides a computer-readable storage medium on which executable instructions are stored, characterized in that when the instructions are executed by a processor, the processor executes the method of embodiment 1.

[0056] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dam personnel behavior monitoring and early warning method, characterized in that: The steps include: S1, obtain visible light images and low-resolution infrared images of the dam; S2, reconstructing the low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; S3, inputting the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint point feature vector of the currently monitored person; S4. Input the joint point feature vector into the personnel behavior monitoring network, and judge whether the behavior of the currently monitored person belongs to an abnormal behavior state according to the judgment threshold.

2. A dam personnel behavior monitoring and early warning method according to claim 1, characterized in that: S2 includes the following steps: S201, performing convolution processing on the low-resolution infrared image to obtain shallow features; S202, using shallow features as input, and obtaining deep features through a deep feature extraction algorithm; S203, performing residual connection on the shallow features and the deep features and then reconstructing them to obtain a high-resolution infrared image.

3. A dam personnel behavior monitoring and early warning method according to claim 2, characterized in that: S202 includes the following steps: S20201, compressing features, performing residual connection and refining processing on shallow features to obtain processed features; S20202, performing adaptive average pooling processing on the processed features to obtain adaptive average pooling features; S20203, performing adaptive convolution processing on the adaptive average pooling features to obtain local dependencies between channels; S20204. Perform local dependencies sigmoid Activate the process and map the output value to the interval [0,1] to obtain the channel attention weight; S20205. Perform efficient channel attention activation processing on the channel attention weight and the shallow feature to obtain the activated feature; S20206. Perform multi-head self-attention calculation on the activated features to obtain a multi-head self-attention value; S20207. Perform line normalization processing on the multi-head self-attention values ​​to obtain a line normalization result; S20208. Calculate the intermediate features according to the line normalization result, the shallow features and the activation features; S20209, performing multi-layer perception and layer normalization processing on the intermediate features to obtain a layer normalization result; S20210, calculating output features according to the layer normalization result and the intermediate features; S20211. Perform multiple residual multi-head self-attention processes on the output features to obtain deep features.

4. A dam personnel behavior monitoring and early warning method according to claim 1, characterized in that: S3 includes the following steps: S301, acquiring a visible light image and a high-resolution infrared image; S302, performing preliminary fusion processing on the high-resolution infrared image and the visible light image to obtain a fused image; S303, using the ViTPose encoder to extract features from the fused image to obtain a multi-scale fusion feature map; S304, using an adaptive frame weighting method to dynamically assign weights to the multi-scale fusion feature map according to the relevance of the frames; S305, weighting the multi-scale fusion feature map using weights to obtain a weighted feature map; S306, using the feature map of the center frame of the feature map as the query vector, and the feature map of the context frame as the key vector and the value vector, calculating the attention weights of the position in the center frame and the position in the context frame; S307, update all feature maps according to the attention weights, and generate key node feature vectors of human posture through the ViTPose decoder.

5. A dam personnel behavior monitoring and early warning method according to claim 4, characterized in that: S302 includes the following steps: S30201, performing multi-scale decomposition on the high-resolution infrared image and the visible light image respectively to obtain low-frequency components and high-frequency components of different scales; S30202, using a weighted average method to fuse the low-frequency components to obtain a low-frequency component fusion result; S30203, using a regional energy-based selection fusion method to calculate the energy of the high-frequency component in each region, and taking the region with larger energy as the high-frequency classification fusion result; S30204. Reconstruct the low-frequency component fusion result and the high-frequency classification fusion result to obtain a fused image.

6. A dam personnel behavior monitoring and early warning method according to claim 4, characterized in that: S303 includes the following steps: S30301, segment the fused image into primitives, use the ViTPose encoder to deeply extract features, and obtain local and global spatial information; S30302. For the primitive, extract feature maps from different layers in the feature extraction network, and perform adaptive pooling processing on each feature map at multiple scales to generate adaptive pooling features; S30303, using convolution processing to reduce the embedding dimension to one-fourth of the original, then restoring the height and width dimensions of the feature through a difference layer, and finally connecting the original feature map with the upsampled pooled feature map to obtain a multi-scale feature map; S30304, perform convolution layer unified feature dimension processing on the multi-scale feature map, and then perform layer normalization and Activation processing to obtain the activated multi-scale feature map; S30305, connecting the activated multi-scale feature maps of different layers along the channel dimension to obtain features, and then calculating the spatial position dependency between consecutive frames through a multi-head attention algorithm; S30306. Perform residual connection on features and dependencies, and obtain a unified multi-scale representation through average pooling and linear transformation. Finally, obtain a multi-scale fusion feature map through the projection layer.

7. A dam personnel behavior monitoring and early warning method according to claim 1, characterized in that: S4 includes the following steps: S401, inputting the joint point feature vector and the normal joint point feature vector into the normal human behavior model in sequence to obtain a prediction result and a normal result; S402, performing log-likelihood probability calculation on the predicted result and the normal result to obtain a probability value; S403. If the probability value is less than the set threshold, the personnel behavior is abnormal behavior, otherwise it is normal behavior.

8. A dam personnel behavior monitoring and early warning system, characterized in that: include: An image acquisition module, used to acquire visible light images and low-resolution infrared images of the dam; An image reconstruction module, used for reconstructing a low-resolution infrared image into a high-resolution infrared image using a super-resolution reconstruction method; The feature recognition module is used to input the high-resolution infrared image and the visible light image into the human posture recognition network to obtain the joint feature vector of the currently monitored person; The behavior judgment module is used to input the joint point feature vector into the personnel behavior monitoring network and judge whether the behavior of the currently monitored person is in an abnormal behavior state according to the judgment threshold.

9. A dam personnel behavior monitoring and early warning device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is caused to perform a method according to any one of claims 1 to 7.

Images