Multi-modal tunnel construction human body pose detection and estimation model construction method based on R2Fusion

Abstract

The invention relates to a multi-modal tunnel construction human body pose detection and estimation model construction method based on R2Fusion, and belongs to the field of tunnel safety monitoring. Comprising the following steps: acquiring data of an RGB image, a depth image and an infrared image, and realizing multi-modal image fusion by combining alignment processing with an adaptive weight network; human body joint points are detected through HRNet, human body non-rigid deformation is captured through WPNet, and a corresponding feature map is generated; an R2Fusion model is constructed by fusing AGCA and DLCA, key points are described, and human body pose detection is achieved; iMU inertial measurement data are collected, data obtained by the IMU and visual data are fused, attitude estimation and correction are further performed by constructing a key point minimum cost function and solving a rotation matrix and a translation matrix, and real-time monitoring and feedback are ensured. The method improves the positioning precision of the human body in a complex tunnel environment, and enables the tunnel safety inspection to be safer and more reliable.

Description

Technical Field

[0001] The invention relates to a method for constructing a multi-modal tunnel construction human body posture detection and estimation model based on R2Fusion, and belongs to the technical field of tunnel safety monitoring. Background Art

[0002] With the continuous advancement of urbanization and the improvement of transportation infrastructure, tunnels have become an indispensable and important part of the modern transportation system. During the construction of tunnels, due to the complex structure and closed environment, the safety of workers is always the primary concern. However, during the long-term construction and operation process, the environmental factors such as light in the tunnel pose severe challenges to human posture detection. Traditional human posture detection relies on manual monitoring and static monitoring equipment, and has problems such as low detection efficiency, insufficient spatial coverage, and large subjective judgment errors. Especially in the tunnel environment with weak light and narrow space, the difficulty of manual monitoring increases significantly, and there is a high risk.

[0003] To solve the above problems, the present invention proposes a method for constructing a multi-modal human posture detection and estimation model for tunnel construction based on R2Fusion. Summary of the invention

[0004] The present invention provides a method for constructing a multi-modal human posture detection and estimation model for tunnel construction based on R2Fusion, so as to solve the problems of insufficient efficiency and accuracy of human posture detection in tunnel construction, improve the positioning accuracy of the human body in the complex environment of tunnel construction, make the tunnel construction process safer and more reliable, and meet the actual application scenarios.

[0005] The technical solution of the present invention is: a method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion, and the specific steps of the method include:

[0006] Step 1: Use a multi-camera system to collect RGB images, depth images, and infrared images, and realize multimodal image fusion through alignment processing and combining with an adaptive weight network;

[0007] Step 2: HRNet detects human joints, WPNet captures non-rigid deformation of the human body, and generates corresponding feature maps;

[0008] Step 3, by fusing the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, the R2Fusion model is constructed to describe the key points and realize the detection of human body posture;

[0009] Step 4: Collect IMU inertial measurement data, integrate the data obtained by IMU with visual data, construct the minimum cost function of key points and obtain the rotation matrix R and translation matrix t, further perform posture estimation and correction, and ensure real-time monitoring and feedback.

[0010] As a further solution of the present invention, Step 1 includes:

[0011] Each camera group in the multi-camera system consists of the following three parts: RGB camera, depth camera and infrared thermal imaging camera. Since the depth camera has a built-in 6-axis IMU sensor, it can simultaneously collect inertial measurement data to help the subsequent process for attitude estimation and correction. Assuming that there are a total of n camera groups, represented by (k 1 ,k 2 ,…k n ), the image capture range of these n camera groups covers the location of tunnel construction. For image preprocessing, it is specifically divided into the following three parts:

[0012] First, for the RGB image, convert it to the YCbCr color space:

[0013] Y=0.299·R+0.587·G+0.114·B;

[0014] Cb=128-0.168736·R-0.331264·G+0.5·B;

[0015] Cr=128+0.5·R-0.418688·G-0.081312·B;

[0016] Where R, G, B are the values ​​of the red, green, and blue channels of the input RGB image, Y, Cb, Cr are the brightness and chromaticity values, respectively, and the center values ​​of Cb and Cr are set to 128;

[0017] Then, the depth image is normalized as follows:

[0018]

[0019] Where D is the pixel value of the original depth image, D min and D max are the minimum and maximum values ​​of the depth image, D norm is the normalized depth image pixel value;

[0020] Finally, the infrared image is histogram equalized; r is defined as the gray level of the image to be enhanced, and r is a continuous function. Assume that r is normalized to the interval [0, 1], and stipulate that r = 1 represents black and r = 0 represents white. Consider a discretization formula and allow the pixel value to be in [0, L-1], where L is the gray level. The gray level of the image after equalization s, there is a transformation T such that s = T (r), where T needs to meet the following two conditions:

[0021] (1) T(r) is a single-valued and monotonically increasing function in the interval [0, L-1];

[0022] (2) r is in the interval [0, L-1], T(r) is in the interval [0, L-1];

[0023] Assume that the normalized gray level in the interval [0, 1] is a continuous quantity, and let P r (r) represents the probability density function of the gray level in an image, where the subscript is used to distinguish the probability density function of the input image and the output image; assuming that the following transformation is performed on the input gray level t, the processed gray level s is obtained, the formula is:

[0024]

[0025] Among them, w is an integral imaginary variable, and the probability density function of the output gray level is uniform, that is:

[0026]

[0027] When the gray value is a discrete value, let P r (r j ), j = 0, 1, 2, ... L-1 represents a histogram associated with the gray level of a given image. For discrete gray levels, the summation method is adopted, and its balanced transformation is:

[0028]

[0029] Where k = 0, 1, 2, ..., L-1, s k is the gray value of the processed image, and its corresponding image input value is r k ;

[0030] Through the above preprocessing, the aligned images of each modality are obtained. Let Y represent the Y channel brightness image, D represent the depth image, and IR represent the infrared image. The image of each modality is represented as a two-dimensional matrix, in which each pixel position corresponds to a feature vector. The three input matrices are represented as follows:

[0031] Y={Y ij}, D={D ij}, IR={IR ij};

[0032] Where (i, j) represents the pixel position in the image;

[0033] Use a simple three-layer perceptron MLP as the weight generation network to achieve dynamic weighted fusion of adaptive learning weights; first input the multimodal feature M:

[0034] M=(Y ij ,D ij ,IR ij )

[0035] Then, the interaction features are extracted through the hidden layer, specifically:

[0036] h=RELU(W h ·[Y ij ,D ij ,IR ij ]+b h )

[0037] Finally, the Softmax function is used to normalize the output to ensure that the sum of the weights is 1, and we have:

[0038]

[0039] Among them, Z Y , Z D , Z IR is the output of the weight generation network, and the weight α is obtained after the Softmax operation ij , β ij , γ ij ;

[0040] The entire fused image is represented as:

[0041] F = {α ij ·Y ij +β ij ·D ij +γ ij ·IR ij} ij .

[0042] As a further solution of the present invention, the Step 2, detecting human joints by HRNet, comprises:

[0043] The HRNet is a high-resolution preserving network based on multi-scale feature fusion, which can extract and fuse features at different scales to achieve accurate detection of human joints; HRNet consists of an initial convolution module, a multi-resolution feature extraction module, a feature fusion module and an output layer;

[0044] In the initial convolution module, the fused image obtained by inputting Step 1 is successively processed through two convolution layers for initial convolution. The convolution kernel size of each layer is 3×3. This step reduces the image size to one-fourth of the original size and the number of channels becomes 64. It is then further processed through the RELU activation function. In the multi-resolution feature extraction module, it is divided into four stages. The first stage uses residual blocks to extract features, which includes three convolution layers and the number of output channels is increased to 128. The second stage adds a new low-resolution branch based on the first stage, with a resolution of one-eighth of the initial input image, and continues to extract features on the high-resolution branch. The third and fourth stages further add more resolution branches to enhance the feature expression capability. In the feature fusion module, high-resolution features are downsampled by convolution with a step size of 2, while low-resolution features are adjusted by upsampling. Finally, features of different resolutions are added to realize information exchange between high and low resolution features.

[0045] As a further solution of the present invention, in Step 2, WPNet captures the non-rigid deformation of the human body and generates the corresponding feature map including:

[0046] The core idea of ​​WRNet is to use the thin plate spline algorithm TPS to perform non-rigid transformation on the local area so that the feature descriptor can adapt to complex deformation scenes. The implementation process of WRNet is as follows:

[0047] First, define the input feature map as The detected key point coordinates are Where H, W are the height and width of the original input image, d is the number of channels, N is the number of key points, (x i ,y i ) represents the position of the i-th key point;

[0048] Then, from the feature map X feature For each key point position (x i ,y i ), extract the features of the local window size k×k, and set the local features to be Apply a convolution operation to these features:

[0049] F i =σ(W conv *X i +b conv )

[0050] in, is the convolution kernel, b conv is the bias term, σ is the RELU activation function, and the output feature is

[0051] The convolution feature Fi Straighten to vector Input into the multi-layer perceptron MLP to learn the parameters of TPS:

[0052] θ i =f MLP (f i )

[0053] in, Contains all the parameters of TPS, n is the number of control points;

[0054] The specific calculation process of MLP is expressed as:

[0055] θ i =W 2 ·(σ(W 1 ·f i +b 1 ))+b 2

[0056] Among them, W 1 , W 2 are the weight matrices of the fully connected layer, b 1 , b 2 is the bias term;

[0057] Next, define n control points To describe the deformation of the image area, given the key point (x i ,y i ) around a fixed polar grid The TPS transformation is applied in the following way:

[0058]

[0059] Among them, A i q represents affine transformation, including rotation, scaling and translation, ρ(r)=r 2 log(r) is a radial basis function used to implement non-rigid transformation. is the weight of the control point;

[0060] The transformed grid position p is sampled using a bilinear interpolation function to obtain the transformed image patch P from the input image. i :

[0061] P i =f bilinear (I,p)

[0062] Among them, f bilinear represents the bilinear interpolation function, I is the input image, and p is the position after TPS transformation;

[0063] Finally, the deformed image block Use convolutional neural network to extract feature descriptors. Assuming there are L convolutional layers, the calculation of each layer is expressed as:

[0064] F i (l+1) =σ(W l *F i (l) +b l )

[0065] Among them, F i (l) represents the feature map of the first layer, W l and b l are the convolution kernel and bias term of the first layer respectively, and σ is the RELU activation function.

[0066] As a further solution of the present invention, the Step 3 includes constructing an adaptive global context enhancement module AGCA. The specific steps of constructing the adaptive global context enhancement module AGCA are as follows:

[0067] First, input the feature map F obtained by HRNet H And the feature map F obtained by WRNet W , respectively, perform adaptive average pooling to pool them into a fixed-size 64×64×128 feature map, and then pass the feature fusion layer to generate the fused feature map F AGCA , to balance immutability and uniqueness;

[0068] Secondly, F AGCA Flattened to a 16×16 2D block B:

[0069]

[0070] The vectorized 2D blocks are mapped to a potential 128-dimensional embedding space, using trainable linear projection for feature compression and information extraction:

[0071] y=W·x+b

[0072] Among them, W is a 256×128 weight matrix, b is a bias term, and y is the output after linear projection;

[0073] Then, in order to make the local descriptor obtain the global spatial position information, learn the specific position information embedding, introduce the spatial position encoding PE, and calculate it as follows:

[0074]

[0075] Among them, pos represents the position index, i represents the dimension index, and dmodel is the dimension of the model. The generated PE will be added to the input features to enhance the model's perception of the order of elements in the sequence;

[0076] Finally, the features obtained by the above operation Input into the Transformer layer, which includes layer normalization Layer Norm, multi-head self-attention mechanism MSA and multi-layer perceptron MLP:

[0077] Perform layer normalization, we have:

[0078]

[0079] Among them, μ is the mean of the input feature, σ 2 is the variance, ε is a small constant to prevent division by zero, and x i ′ is the normalized feature. In layer normalization, the normalized feature is scaled and offset by linear transformation: i ′=γx i +β, where γ and β are learnable parameters that allow the model to rescale and offset the normalized features;

[0080] The multi-head self-attention mechanism is the core of the Transformer layer. It first maps the input sequence into queries, keys, and values ​​through three linear transformations:

[0081] Q = xW Q ; K = xW K ; V = xW V ;

[0082] Among them, W Q , W K , W V is a trainable weight matrix;

[0083] The attention weight is calculated by the following formula:

[0084]

[0085] Among them, d k is the dimension of the key, and the softmax function is used to normalize the attention weights into a probability distribution; multiple heads are calculated to generate multiple queries, keys, and values ​​through different weight matrices for the input sequence:

[0086]

[0087] Among them, i represents the i-th head;

[0088] Concatenate the outputs of all attention heads together:

[0089] Z = Concat(h 1 ,h 2 ,h 3 ,…,h h )W O

[0090] Among them, W O is the weight matrix used to transform the concatenated features;

[0091] The multilayer perceptron consists of two linear transformations and a nonlinear activation function. The process is summarized as passing the output to the next layer and repeating the steps until the output layer is reached. The overall structure of the MLP is expressed as:

[0092] y=W (L) ·f(W (L-1) ·f(…f(W (1) ·x+b (1) )…)+b (L-1) )+b(L)

[0093] Among them, W (L) is the weight matrix of the Lth layer, b (L) is the bias term of the Lth layer, and y is the final output of the MLP.

[0094] As a further solution of the present invention, the Step 3 includes constructing a diversified local context enhancement module DLCA. The specific steps of constructing the diversified local context enhancement module DLCA are as follows:

[0095] First, the feature maps of different layers obtained by HRNet in Step 2 are recorded as The feature maps of different layers obtained by WRNet are recorded as These feature maps from different layers are unified to the same size through interpolation operations, and they are concatenated into H / 4×W / 4×384 through channels:

[0096]

[0097] Then, the two types of concatenated feature maps are first subjected to 1×1 convolution dimensionality reduction, and then feature fusion is performed to generate the fused feature map F DLCA ;

[0098] Secondly, the feature map obtained by the adaptive global context enhancement module is fused and then 1×1 convolution is performed again to fuse local and global information. At the same time, dilated convolution is performed on some feature maps to extract multi-scale local context information, and finally the feature map F is obtained. fusion ;

[0099] Finally, the obtained feature descriptor is L2 normalized:

[0100]

[0101] in, It is a normalized 128-dimensional feature descriptor;

[0102] The R2Fusion model fuses the feature maps obtained by HRNet and WRNet, and effectively extracts the static and dynamic features of the human body through the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, and returns relevant information to realize the detection of human posture.

[0103] As a further solution of the present invention, Step 4 includes:

[0104] In Step 1, the depth camera has a built-in 6-axis IMU sensor and collects inertial measurement data. The IMU sensor consists of an accelerometer and a gyroscope. The accelerometer measures linear acceleration, which is recorded as:

[0105] A(t)=[A x (t),A y (t),A z (t)] T

[0106] The gyroscope measures angular velocity, recorded as:

[0107] G(t)=[G x (t),G y (t),G z (t)] T

[0108] Use the Kalman filter to estimate the state and measurement noise, setting the state equation as:

[0109] x t =Fx t-1 +Bu t

[0110] Among them, x t represents the predicted state vector at time t, x t-1 represents the predicted state vector at time t-1, F is the state transfer matrix, B is the control input matrix, u t is the control input vector;

[0111] The observation equation is set as:

[0112] z t =Hx t +v t

[0113] Among them, z trepresents the observation vector at time t, H is the observation matrix, which describes the mapping relationship from state space to observation space, v t Observation noise, usually assumed to be Gaussian white noise with zero mean and covariance R;

[0114] The Kalman filter consists of two steps: prediction and update; for the prediction step, the covariance prediction is expressed as:

[0115] C t =FC t-1 F T +Q

[0116] Among them, P t represents the prediction error covariance matrix at time t, i.e., the uncertainty of the prediction, and Q represents the process noise covariance matrix, i.e., the uncertainty in the model;

[0117] For the update step, the Kalman gain is calculated:

[0118] K t =C t H T (HC t H T +R) -1

[0119] To make a status update:

[0120] x t =x t-1 +K t (z t -Hx t-1 )

[0121] Among them, z t -Hx t-1 represents the difference between the actual observation and the predicted observation;

[0122] Perform a covariance update:

[0123] C t =(IK t H)C t-1

[0124] Among them, I is the identity matrix, which ensures the correctness of the covariance update;

[0125] Through the Kalman filter, the noisy measurement data are processed and help describe and update the estimated accuracy of the system state;

[0126] In order to achieve more accurate key point position estimation, velocity is first introduced. For velocity update, there are:

[0127] V t =Vt-1 +A t Δt

[0128] Among them, V t represents the velocity information, Δt represents the sampling time interval of IMU data;

[0129] Update location:

[0130]

[0131] Among them, P t Indicates the current location information;

[0132] Then, according to the angular velocity of the gyroscope G(t) = [G x (t),G y (t),G z (t)] T , rotation quaternion Q gyro (Δt) is calculated as:

[0133]

[0134] Among them, ||G(t)|| is the modulus of the angular velocity, indicating the rotation speed;

[0135] The update is expressed as:

[0136]

[0137] By fusing the above position information with attitude information, i.e. quaternion, the key points of IMU estimation are:

[0138] φ=[P t Q t ] T

[0139] Finally, the posture estimated by the IMU is corrected with the key points in the visual data, and the visual key points are set to D. The dimension is reduced to the same dimension as Φ through linear mapping, and then the minimum cost function equation is constructed as:

[0140]

[0141] Among them, R is the rotation matrix, t is the translation matrix, λ is the regularization parameter, which ensures the orthogonality of the rotation matrix. The gradient descent obtains the rotation matrix R and the translation matrix t at the minimum cost, and updates the key point position:

[0142] Φ′=R·φ+t; According to the real-time updated key point position φ, real-time monitoring and correction of the human body posture is achieved.

[0143] The beneficial effect of the present invention is that the present invention provides a more robust and efficient means of human posture detection by fusing data inputs of different modes, which is specially designed for complex detection needs in tunnel construction environments and can better cope with posture recognition challenges in extreme environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 is a flow chart of the present invention;

[0145] Figure 2 This is a multi-source data fusion diagram. DETAILED DESCRIPTION

[0146] Embodiment 1:

[0147] In order to solve the problem of insufficient efficiency and accuracy of human posture detection in tunnel construction, the present invention provides a multimodal human posture detection and estimation model construction method based on R2Fusion, aiming to improve the detection intelligence level in the construction environment and its recognition ability in a complex tunnel environment.

[0148] This method is applicable to but not limited to a variety of closed and light-restricted work scenarios including tunnel construction. It aims to replace traditional manual monitoring, accurately identify abnormal postures of personnel that may occur during the construction process, and ensure the comprehensiveness and efficiency of detection. This application can not only greatly improve the detection efficiency, but also reduce the high risks in manual inspections and avoid long-term exposure in harsh environments. Traditional manual posture monitoring is difficult to accurately identify in complex and narrow tunnel environments when faced with insufficient light and dynamic obstacles, resulting in low detection efficiency and hidden dangers of monitoring blind spots. Therefore, the present invention improves the posture detection accuracy and response speed through the innovative deep fusion model R2Fusion, thereby realizing efficient and safe human posture monitoring tasks in tunnel construction.

[0149] like Figure 1-Figure 2 As shown, a method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion, the specific steps of the method include:

[0150] Step 1: Use a multi-camera system to collect RGB images, depth images, and infrared images, and realize multimodal image fusion through alignment processing and combining with an adaptive weight network;

[0151] The Step 1 includes:

[0152] In order to collect image information more comprehensively, a multi-camera system is used. Each camera group in the multi-camera system consists of the following three parts: RGB camera, depth camera and infrared thermal imaging camera; since the depth camera has a built-in 6-axis IMU sensor, it can simultaneously collect inertial measurement data to help the subsequent process for attitude estimation and correction. Assuming that there are a total of n camera groups, represented by (k 1 ,k 2 ,…k n ), the image capture range of these n camera groups covers the location of tunnel construction. For image preprocessing, it is specifically divided into the following three parts:

[0153] First, for RGB images, brightness information can better reflect the structural changes of the scene than chrominance information, especially in low-light environments such as tunnel construction, where brightness information can more clearly reflect the shape and outline of objects. Therefore, convert it to the YCbCr color space:

[0154] Y=0.299·R+0.587·G+0.114·B;

[0155] Cb=128-0.168736·R-0.331264·G+0.5·B;

[0156] Cr=128+0.5·R-0.418688·G-0.081312·B;

[0157] Where R, G, B are the values ​​of the red, green, and blue channels of the input RGB image, Y, Cb, Cr are the brightness and chromaticity values, respectively, and the center values ​​of Cb and Cr are set to 128;

[0158] Then, the depth image is normalized as follows:

[0159]

[0160] Where D is the pixel value of the original depth image, D min and D max are the minimum and maximum values ​​of the depth image, D norm is the normalized depth image pixel value;

[0161] Finally, the infrared image is histogram equalized to enhance the contrast of the image and make the human body contour more obvious; r is defined as the grayscale of the image to be enhanced, and r is a continuous function. Assume that r is normalized to the interval [0, 1], and stipulate that r = 1 represents black and r = 0 represents white. Consider a discretization formula and allow the pixel value to be in [0, L-1], where L is the grayscale (the possible grayscale value range of an image is [0, 255], then its grayscale is 256). The grayscale s of the equalized image has a transformation T such that s = T(r), where T needs to satisfy the following two conditions:

[0162] (1) T(r) is a single-valued and monotonically increasing function in the interval [0, L-1];

[0163] (2) r is in the interval [0, L-1], T(r) is in the interval [0, L-1];

[0164] Assume that the normalized gray level in the interval [0, 1] is a continuous quantity, and let P r (r) represents the probability density function of the gray level in an image, where the subscript is used to distinguish the probability density function of the input image and the output image; assuming that the following transformation is performed on the input gray level t, the processed gray level s is obtained, the formula is:

[0165]

[0166] Among them, w is an integral imaginary variable, and the probability density function of the output gray level is uniform, that is:

[0167]

[0168] When the gray value is a discrete value, let P r (r j ), j = 0, 1, 2, ... L-1 represents a histogram associated with the gray level of a given image. For discrete gray levels, the summation method is adopted, and its balanced transformation is:

[0169]

[0170] Where k = 0, 1, 2, ..., L-1, s k is the gray value of the processed image, and its corresponding image input value is r k ;

[0171] Through the above preprocessing, the aligned images of each modality are obtained. Let Y represent the Y channel brightness image, D represent the depth image, and IR represent the infrared image. The image of each modality is represented as a two-dimensional matrix, in which each pixel position corresponds to a feature vector. The three input matrices are represented as follows:

[0172] Y={Y ij}, D={D ij}, IR={IR ij};

[0173] Where (i, j) represents the pixel position in the image;

[0174] Use a simple three-layer perceptron MLP as the weight generation network to achieve dynamic weighted fusion of adaptive learning weights; first input the multimodal feature M:

[0175] M=(Y ij ,D ij ,IR ij )

[0176] Then, the interaction features are extracted through the hidden layer, specifically:

[0177] h=RELU(W h ·[Y ij ,D ij ,IR ij ]+b h )

[0178] Finally, the Softmax function is used to normalize the output to ensure that the sum of the weights is 1, and we have:

[0179]

[0180] Among them, Z Y , Z D , Z IR is the output of the weight generation network, and the weight α is obtained after the Softmax operation ij , β ij , γ ij ;

[0181] The entire fused image is represented as:

[0182] F = {α ij ·Y ij +β ij ·D ij +γ ij ·IR ij} ij .

[0183] Step 2: HRNet detects human joints, WPNet captures non-rigid deformation of the human body, and generates corresponding feature maps;

[0184] As a further solution of the present invention, the Step 2, detecting human joints by HRNet, comprises:

[0185] The HRNet is a high-resolution preserving network based on multi-scale feature fusion, which can extract and fuse features at different scales to achieve accurate detection of human joints; HRNet consists of an initial convolution module, a multi-resolution feature extraction module, a feature fusion module and an output layer;

[0186] In the initial convolution module, the fused image obtained by inputting Step 1 is successively processed through two convolution layers for initial convolution. The convolution kernel size of each layer is 3×3. This step reduces the image size to one-fourth of the original size and the number of channels becomes 64. Then, the features are further processed through the RELU activation function. In the multi-resolution feature extraction module, it is divided into four stages. The first stage uses residual blocks to extract features, which includes three convolution layers and the number of output channels is increased to 128. The second stage adds a new low-resolution branch based on the first stage, with a resolution of one-eighth of the initial input image, and continues to extract features on the high-resolution branch. The third and fourth stages further add more resolution branches to enhance the feature expression capability. In the feature fusion module, high-resolution features are downsampled by convolution with a step size of 2, while low-resolution features are adjusted by upsampling (bilinear interpolation). Finally, features of different resolutions are added to realize information exchange between high- and low-resolution features.

[0187] As a further solution of the present invention, in Step 2, WPNet captures the non-rigid deformation of the human body and generates the corresponding feature map including:

[0188] The core idea of ​​WRNet is to use the thin plate spline algorithm TPS to perform non-rigid transformation on the local area so that the feature descriptor can adapt to complex deformation scenes. The implementation process of WRNet is as follows:

[0189] First, define the input feature map as The detected key point coordinates are Where H, W are the height and width of the original input image, d is the number of channels, N is the number of key points, (x i ,y i ) represents the position of the i-th key point;

[0190] Then, from the feature map X feature For each key point position (x i ,y i ), extract the features of the local window size k×k, and set the local features to be Apply a convolution operation to these features:

[0191] F i =σ(W conv *X i +b conv )

[0192] in, is the convolution kernel, b conv is the bias term, σ is the RELU activation function, and the output feature is

[0193] The convolution feature F i Straighten to vector Input into the multi-layer perceptron MLP to learn the parameters of TPS:

[0194] θ i =f MLP (f i )

[0195] in, Contains all the parameters of TPS, n is the number of control points;

[0196] The specific calculation process of MLP is expressed as:

[0197] θ i =W 2 ·(σ(W 1 ·f i +b 1 ))+b 2

[0198] Among them, W 1 , W 2 are the weight matrices of the fully connected layer, b 1 , b 2 is the bias term;

[0199] Next, define n control points To describe the deformation of the image area, given the key point (x i ,y i ) around a fixed polar grid The TPS transformation is applied in the following way:

[0200]

[0201] Among them, A i q represents affine transformation, including rotation, scaling and translation, ρ(r)=r 2 log(r) is a radial basis function used to implement non-rigid transformation. is the weight of the control point;

[0202] The transformed grid position p is sampled using a bilinear interpolation function to obtain the transformed image patch P from the input image. i :

[0203] P i =f bilinear (I,p)

[0204] Among them, f bilinear represents the bilinear interpolation function, I is the input image, and p is the position after TPS transformation;

[0205] Finally, the deformed image block Use convolutional neural network to extract feature descriptors. Assuming there are L convolutional layers, the calculation of each layer is expressed as:

[0206] F i (l+1) =σW l *F i (l) +b l )

[0207] Among them, F i (l) represents the feature map of the first layer, W l and b l are the convolution kernel and bias term of the first layer respectively, and σ is the RELU activation function.

[0208] Step 3, by fusing the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, the R2Fusion model is constructed to describe the key points and realize the detection of human body posture;

[0209] The R2Fusion model fuses the feature maps obtained by HRNet and WRNet to generate a descriptor for human posture detection. As can be seen from Step 2, HRNet obtains the human joint features, that is, the features that do not deform, and WRNet obtains the features of the human body's non-rigid deformation, that is, the features that deform. By fusing the deformable and invariant feature information, the human posture can be accurately detected. At the same time, since most local feature description methods based on convolutional neural networks only focus on local information and cannot capture the global or surrounding spatial context, the R2Fusion model solves the local limitation problem by combining the adaptive global context enhancement module (AGCA) and the diversified local context enhancement module (DLCA).

[0210] As a further solution of the present invention, the Step 3 includes constructing an adaptive global context enhancement module AGCA. The specific steps of constructing the adaptive global context enhancement module AGCA are as follows:

[0211] First, input the feature map F obtained by HRNet H And the feature map F obtained by WRNet W, respectively, perform adaptive average pooling to pool them into a fixed-size 64×64×128 feature map, and then pass the feature fusion layer to generate the fused feature map F AGCA , to balance immutability and uniqueness;

[0212] Secondly, F AGCA Flattened to a 16×16 2D block B:

[0213]

[0214] The vectorized 2D blocks are mapped to a potential 128-dimensional embedding space, using trainable linear projection for feature compression and information extraction:

[0215] y=W·x+b

[0216] Among them, W is a 256×128 weight matrix, b is a bias term, and y is the output after linear projection;

[0217] Then, in order to make the local descriptor obtain the global spatial position information, learn the specific position information embedding, introduce the spatial position encoding PE, and calculate it as follows:

[0218]

[0219] Among them, pos represents the position index, i represents the dimension index, and d model is the dimension of the model. The generated PE will be added to the input features to enhance the model's perception of the order of elements in the sequence;

[0220] Finally, the features obtained by the above operation Input into the Transformer layer, which includes layer normalization Layer Norm, multi-head self-attention mechanism MSA and multi-layer perceptron MLP:

[0221] Perform layer normalization, we have:

[0222]

[0223] Among them, μ is the mean of the input feature, σ 2 is the variance, ε is a small constant to prevent division by zero, and x i ′ is the normalized feature. In layer normalization, the normalized feature is scaled and offset by linear transformation: i ′=γx i +β, where γ and β are learnable parameters that allow the model to rescale and offset the normalized features;

[0224] The multi-head self-attention mechanism is the core of the Transformer layer. It first maps the input sequence into queries, keys, and values ​​through three linear transformations:

[0225] Q = xW Q ; K = xW K ; V = xW V ;

[0226] Among them, W Q , W K , W V is a trainable weight matrix;

[0227] The attention weight is calculated by the following formula:

[0228]

[0229] Among them, d k is the dimension of the key, and the softmax function is used to normalize the attention weights into a probability distribution; multiple heads are calculated to generate multiple queries, keys, and values ​​through different weight matrices for the input sequence:

[0230]

[0231] Among them, i represents the i-th head;

[0232] Concatenate the outputs of all attention heads together:

[0233] Z = Concat(h 1 ,h 2 ,h 3 ,…,h h )W O

[0234] Among them, W O is the weight matrix used to transform the concatenated features;

[0235] The multilayer perceptron consists of two linear transformations and a nonlinear activation function. The process is summarized as passing the output to the next layer and repeating the steps until the output layer is reached. The overall structure of the MLP is expressed as:

[0236] y=W (L) ·f(W (L-1) ·f(…f(W (1) ·x+b (1) )…)+b (L-1) )+b(L)

[0237] Among them, W (L) is the weight matrix of the Lth layer, b (L) is the bias term of the Lth layer, and y is the final output of the MLP.

[0238] As a further solution of the present invention, the Step 3 includes constructing a diversified local context enhancement module DLCA. The specific steps of constructing the diversified local context enhancement module DLCA are as follows:

[0239] First, the feature maps of different layers obtained by HRNet in Step 2 are recorded as The feature maps of different layers obtained by WRNet are recorded as These feature maps from different layers are unified to the same size through interpolation operations, and they are concatenated into H / 4×W / 4×384 through channels:

[0240]

[0241] Then, the two types of concatenated feature maps are first subjected to 1×1 convolution dimensionality reduction, and then feature fusion is performed to generate the fused feature map F DLCA ;

[0242] Secondly, the feature map obtained by the adaptive global context enhancement module is fused and then 1×1 convolution is performed again to fuse local and global information. At the same time, dilated convolution is performed on some feature maps to extract multi-scale local context information, and finally the feature map F is obtained. fusion ;

[0243] Finally, the obtained feature descriptor is L2 normalized:

[0244]

[0245] in, It is a normalized 128-dimensional feature descriptor;

[0246] The R2Fusion model fuses the feature maps obtained by HRNet and WRNet, and effectively extracts the static and dynamic features of the human body through the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, and returns relevant information to realize the detection of human posture.

[0247] Step 4: Collect IMU inertial measurement data, integrate the data obtained by IMU with visual data, construct the minimum cost function of key points and obtain the rotation matrix R and translation matrix t, further perform posture estimation and correction, and ensure real-time monitoring and feedback.

[0248] As a further solution of the present invention, Step 4 includes:

[0249] In Step 1, the depth camera has a built-in 6-axis IMU sensor and collects inertial measurement data. The IMU sensor consists of an accelerometer and a gyroscope. The accelerometer measures linear acceleration, which is recorded as:

[0250] A(t)=[A x (t),A y (t),A z (t)] T

[0251] The gyroscope measures angular velocity, recorded as:

[0252] G(t)=[G x (t),G y (t),G z (t)] T

[0253] Use the Kalman filter to estimate the state and measurement noise, setting the state equation as:

[0254] x t =Fx t-1 +Bu t

[0255] Among them, x t represents the predicted state vector at time t, x t-1 represents the predicted state vector at time t-1, F is the state transfer matrix, B is the control input matrix, u t is the control input vector;

[0256] The observation equation is set as:

[0257] z t =Hx t +v t

[0258] Among them, z t represents the observation vector at time t, H is the observation matrix, which describes the mapping relationship from state space to observation space, v t Observation noise, usually assumed to be Gaussian white noise with zero mean and covariance R;

[0259] The Kalman filter consists of two steps: prediction and update; for the prediction step, the covariance prediction is expressed as:

[0260] C t =FC t-1 F T +Q

[0261] Among them, P t represents the prediction error covariance matrix at time t, i.e., the uncertainty of the prediction, and Q represents the process noise covariance matrix, i.e., the uncertainty in the model;

[0262] For the update step, the Kalman gain is calculated:

[0263] K t =C t H T (HC t H T +R) -1

[0264] To make a status update:

[0265] x t =x t-1 +K t (z t -Hx t-1 )

[0266] Among them, z t -Hx t-1 represents the difference between the actual observation and the predicted observation;

[0267] Perform a covariance update:

[0268] C t =(IK t H)C t-1

[0269] Among them, I is the identity matrix, which ensures the correctness of the covariance update;

[0270] Through the Kalman filter, the noisy measurement data are processed and help describe and update the estimated accuracy of the system state;

[0271] In order to achieve more accurate key point position estimation, velocity is first introduced. For velocity update, there are:

[0272] V t =V t-1 +A t Δt

[0273] Among them, V t represents the velocity information, Δt represents the sampling time interval of IMU data;

[0274] Update location:

[0275]

[0276] Among them, P t Indicates the current location information;

[0277] Then, according to the angular velocity of the gyroscope G(t) = [G x (t),G y (t),G z (t)] T , rotation quaternion Q gyro (Δt) is calculated as:

[0278]

[0279] Among them, ||G(t)|| is the modulus of the angular velocity, indicating the rotation speed;

[0280] The update is expressed as:

[0281]

[0282] By fusing the above position information with attitude information, i.e. quaternion, the key points of IMU estimation are:

[0283] Φ=[P t Q t ] T

[0284] Finally, the posture estimated by the IMU is corrected with the key points in the visual data, and the visual key points are set to D. The dimension is reduced to the same dimension as φ through linear mapping, and then the minimum cost function equation is constructed as:

[0285]

[0286] Among them, R is the rotation matrix, t is the translation matrix, λ is the regularization parameter, which ensures the orthogonality of the rotation matrix. The gradient descent obtains the rotation matrix R and the translation matrix t at the minimum cost, and updates the key point position:

[0287] Φ′=R·Φ+t;

[0288] According to the real-time updated key point position Φ, real-time monitoring and correction of human body posture is achieved.

[0289] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multi-modal tunnel construction human posture detection and estimation model construction method based on R2Fusion, characterized by: The specific steps of the method include: Step 1: Use a multi-camera system to collect RGB images, depth images, and infrared images, and realize multimodal image fusion through alignment processing and combining with an adaptive weight network; Step 2: HRNet detects human joints, WPNet captures non-rigid deformation of the human body, and generates corresponding feature maps; Step 3, by fusing the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, the R2Fusion model is constructed to describe the key points and realize the detection of human body posture; Step 4: Collect IMU inertial measurement data, integrate the data obtained by IMU with visual data, construct the minimum cost function of key points and obtain the rotation matrix R and translation matrix t, further perform posture estimation and correction, and ensure real-time monitoring and feedback.

2. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: The Step 1 includes: Each camera group in the multi-camera system consists of the following three parts: RGB camera, depth camera and infrared thermal imaging camera. Since the depth camera has a built-in 6-axis IMU sensor, it can simultaneously collect inertial measurement data to help the subsequent process for attitude estimation and correction. Assume that there are a total of n camera groups, represented by (k1, k2, … k n ), the image capture range of these n camera groups covers the location of tunnel construction. For image preprocessing, it is specifically divided into the following three parts: First, for the RGB image, convert it to the YCbCr color space: Y=0.299·R+0.587·G+0.114·B; Cb=128-0.168736·R-0.331264·G+0.5·B; Cr=128+0.5·R-0.418688·G-0.081312·B; Where R, G, B are the values ​​of the red, green, and blue channels of the input RGB image, Y, Cb, Cr are the brightness and chromaticity values, respectively, and the center values ​​of Cb and Cr are set to 128; Then, the depth image is normalized as follows: Where D is the pixel value of the original depth image, D min and D max are the minimum and maximum values ​​of the depth image, D norm is the normalized depth image pixel value; Finally, the infrared image is histogram equalized; r is defined as the gray level of the image to be enhanced, and r is a continuous function. Assume that r is normalized to the interval [0, 1], and stipulate that r = 1 represents black and r = 0 represents white. Consider a discretization formula and allow the pixel value to be in [0, L-1], where L is the gray level. The gray level of the image after equalization s, there is a transformation T such that s = T (r), where T needs to meet the following two conditions: (1) T(r) is a single-valued and monotonically increasing function in the interval [0, L-1]; (2) r is in the interval [0, L-1], T(r) is in the interval [0, L-1]; Assume that the normalized gray level in the interval [0, 1] is a continuous quantity, and let P r (r) represents the probability density function of the gray level in an image, where the subscript is used to distinguish the probability density function of the input image and the output image; assuming that the following transformation is performed on the input gray level t, the processed gray level s is obtained, the formula is: Among them, w is an integral imaginary variable, and the probability density function of the output gray level is uniform, that is: When the gray value is a discrete value, let P r (r j ), j = 0, 1, 2, ... L-1 represents a histogram associated with the gray level of a given image. For discrete gray levels, the summation method is adopted, and its balanced transformation is: Where k = 0, 1, 2, ..., L-1, s k is the gray value of the processed image, and its corresponding image input value is r k ; Through the above preprocessing, the aligned images of each modality are obtained. Let Y represent the Y channel brightness image, D represent the depth image, and IR represent the infrared image. The image of each modality is represented as a two-dimensional matrix, in which each pixel position corresponds to a feature vector. The three input matrices are represented as follows: And={And ij },D={D ij },GO={GO ij }; Where (i, j) represents the pixel position in the image; Use a simple three-layer perceptron MLP as the weight generation network to achieve dynamic weighted fusion of adaptive learning weights; first input the multimodal feature M: M=(Y ij ,D ij ,GO ij ) Then, the interaction features are extracted through the hidden layer, specifically: h=RELU(W h ·[Y ij ,D ij ,IR ij ]+b h ) Finally, the Softmax function is used to normalize the output to ensure that the sum of the weights is 1, and we have: Among them, Z Y , Z D , Z IR is the output of the weight generation network, and the weight α is obtained after the Softmax operation ij , β ij , γ ij ; The entire fused image is represented as: F={α ij ·AND ij +β ij ·D ij +γ ij ·GO ij } ij 。 3. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: The Step 2, detecting human joints through HRNet, includes: The HRNet is a high-resolution preserving network based on multi-scale feature fusion, which can extract and fuse features at different scales to achieve accurate detection of human joints; HRNet consists of an initial convolution module, a multi-resolution feature extraction module, a feature fusion module and an output layer; In the initial convolution module, the fused image obtained by inputting Step 1 is successively processed through two convolution layers for initial convolution. The convolution kernel size of each layer is 3×3. This step reduces the image size to one-fourth of the original size and the number of channels becomes 64. It is then further processed through the RELU activation function. In the multi-resolution feature extraction module, it is divided into four stages. The first stage uses residual blocks to extract features, which includes three convolution layers and the number of output channels is increased to 128. The second stage adds a new low-resolution branch based on the first stage, with a resolution of one-eighth of the initial input image, and continues to extract features on the high-resolution branch. The third and fourth stages further add more resolution branches to enhance the feature expression capability. In the feature fusion module, high-resolution features are downsampled by convolution with a step size of 2, while low-resolution features are adjusted by upsampling. Finally, features of different resolutions are added to realize information exchange between high and low resolution features.

4. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: In Step 2, WPNet captures the non-rigid deformation of the human body and generates corresponding feature maps including: The core idea of ​​WRNet is to use the thin plate spline algorithm TPS to perform non-rigid transformation on the local area so that the feature descriptor can adapt to complex deformation scenes. The implementation process of WRNet is as follows: First, define the input feature map as The detected key point coordinates are Where H, W are the height and width of the original input image, d is the number of channels, N is the number of key points, (x i ,y i ) represents the position of the i-th key point; Then, from the feature map X feature For each key point position (x i ,y i ), extract the features of the local window size k×k, and set the local features to be Apply a convolution operation to these features: F i =σ(W conv *X i +b conv ) in, is the convolution kernel, b conv is the bias term, σ is the RELU activation function, and the output feature is The convolution feature F i Straighten to vector Input into the multi-layer perceptron MLP to learn the parameters of TPS: i i =f MLP (f i ) in, Contains all the parameters of TPS, n is the number of control points; The specific calculation process of MLP is expressed as: i i =W2·(σ(W1·f i +b1))+b2 Among them, W1 and W2 are the weight matrices of the fully connected layers, and b1 and b2 are bias terms; Next, define n control points To describe the deformation of the image area, given the key point (x i ,y i ) around a fixed polar grid The TPS transformation is applied in the following way: Among them, A i q represents affine transformation, including rotation, scaling and translation, ρ(r)=r 2 log(r) is a radial basis function used to implement non-rigid transformation. is the weight of the control point; The transformed grid position p is sampled using a bilinear interpolation function to obtain the transformed image patch P from the input image. i : P i =f bilinear (I,p) Among them, f bilinear represents the bilinear interpolation function, I is the input image, and p is the position after TPS transformation; Finally, the deformed image block Use convolutional neural network to extract feature descriptors. Assuming there are L convolutional layers, the calculation of each layer is expressed as: F i (l+1) =σ(W l *F i (l) +b l ) Among them, F i (l) represents the feature map of the first layer, W l and b l are the convolution kernel and bias term of the first layer respectively, and σ is the RELU activation function.

5. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: The Step 3 includes constructing an adaptive global context enhancement module AGCA. The specific steps of constructing the adaptive global context enhancement module AGCA are as follows: First, input the feature map F obtained by HRNet H And the feature map F obtained by WRNet W , respectively, perform adaptive average pooling to pool them into a fixed-size 64×64×128 feature map, and then pass the feature fusion layer to generate the fused feature map F AGCA , to balance immutability and uniqueness; Secondly, F AGCA Flattened to a 16×16 2D block B: The vectorized 2D blocks are mapped to a potential 128-dimensional embedding space, using trainable linear projection for feature compression and information extraction: y=W·x+b Among them, W is a 256×128 weight matrix, b is a bias term, and y is the output after linear projection; Then, in order to make the local descriptor obtain the global spatial position information, learn the specific position information embedding, introduce the spatial position encoding PE, and calculate it as follows: Among them, pos represents the position index, i represents the dimension index, and d model is the dimension of the model. The generated PE will be added to the input features to enhance the model's perception of the order of elements in the sequence; Finally, the features obtained by the above operation Input into the Transformer layer, which includes layer normalization Layer Norm, multi-head self-attention mechanism MSA and multi-layer perceptron MLP: Perform layer normalization, we have: Among them, μ is the mean of the input feature, σ 2 is the variance, ε is a small constant to prevent division by zero, and x i ′ is the normalized feature. In layer normalization, the normalized feature is scaled and offset by linear transformation: i ′=γx i +β, where γ and β are learnable parameters that allow the model to rescale and offset the normalized features; The multi-head self-attention mechanism is the core of the Transformer layer. It first maps the input sequence into queries, keys, and values ​​through three linear transformations: Q=xW Q ;K=xW K ;V=xW V ; Among them, W Q , W K , W V is a trainable weight matrix; The attention weight is calculated by the following formula: Among them, d k is the dimension of the key, and the softmax function is used to normalize the attention weights into a probability distribution; Compute multiple heads, passing the input sequence through different weight matrices to generate multiple queries, keys, and values: h i =A(QW Qi ,KW Ki ,VW Vi ) Among them, i represents the i-th head; Concatenate the outputs of all attention heads together: Z=Concat(h1,h2,h3,…,h h )W O Among them, W O is the weight matrix used to transform the concatenated features; The multilayer perceptron consists of two linear transformations and a nonlinear activation function. The process is summarized as passing the output to the next layer and repeating the steps until the output layer is reached. The overall structure of the MLP is expressed as: y=W (L) ·f(W (L-1) ·f(…f(W (1) ·x+b (1) )…)+b (L-1) )+b(L) Among them, W (L) is the weight matrix of the Lth layer, b (L) is the bias term of the Lth layer, and y is the final output of the MLP.

6. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: The Step 3 includes constructing a diversified local context enhancement module DLCA. The specific steps of constructing the diversified local context enhancement module DLCA are as follows: First, the feature maps of different layers obtained by HRNet in Step 2 are recorded as The feature maps of different layers obtained by WRNet are recorded as These feature maps from different layers are unified to the same size through interpolation operations, and they are concatenated into H / 4×W / 4×384 through channels: Then, the two types of concatenated feature maps are first subjected to 1×1 convolution dimensionality reduction, and then feature fusion is performed to generate the fused feature map F DLCA ; Secondly, the feature map obtained by the adaptive global context enhancement module is fused and then 1×1 convolution is performed again to fuse local and global information. At the same time, dilated convolution is performed on some feature maps to extract multi-scale local context information, and finally the feature map F is obtained. fusion ; Finally, the obtained feature descriptor is L2 normalized: in, It is a normalized 128-dimensional feature descriptor; The R2Fusion model fuses the feature maps obtained by HRNet and WRNet, and effectively extracts the static and dynamic features of the human body through the adaptive global context enhancement module AGCA and the diversified local context enhancement module DLCA, and returns relevant information to realize the detection of human posture.

7. The method for constructing a multi-modal tunnel construction human posture detection and estimation model based on R2Fusion according to claim 1, characterized in that: The Step 4 includes: In Step 1, the depth camera has a built-in 6-axis IMU sensor and collects inertial measurement data. The IMU sensor consists of an accelerometer and a gyroscope. The accelerometer measures linear acceleration, which is recorded as: A(t)=[A x (t),A y (t),A z (t)] T The gyroscope measures angular velocity, recorded as: G(t)=[G x (t),G y (t),G z (t)] T Use the Kalman filter to estimate the state and measurement noise, setting the state equation as: x t =Fx t-1 +Bu t Among them, x t represents the predicted state vector at time t, x t-1 represents the predicted state vector at time t-1, F is the state transfer matrix, B is the control input matrix, u t is the control input vector; The observation equation is set as: z t =Hx t +v t Among them, z t represents the observation vector at time t, H is the observation matrix, which describes the mapping relationship from state space to observation space, v t Observation noise, usually assumed to be Gaussian white noise with zero mean and covariance R; The Kalman filter consists of two steps: prediction and update; for the prediction step, the covariance prediction is expressed as: C t =FC t-1 F T +Q Among them, P t represents the prediction error covariance matrix at time t, i.e., the uncertainty of the prediction, and Q represents the process noise covariance matrix, i.e., the uncertainty in the model; For the update step, the Kalman gain is calculated: K t =C t H T (HC t H T +R) -1 To make a status update: x t =x t-1 +K t (z t -Hx t-1 ) Among them, z t -Hx t-1 represents the difference between the actual observation and the predicted observation; Perform a covariance update: C t =(I-K t H)C t-1 Among them, I is the identity matrix, which ensures the correctness of the covariance update; Through the Kalman filter, the noisy measurement data are processed and help describe and update the estimated accuracy of the system state; In order to achieve more accurate key point position estimation, velocity is first introduced. For velocity update, there are: V t =V t-1 +A t Δt Among them, V t represents the velocity information, Δt represents the sampling time interval of IMU data; Update location: Among them, P t Indicates the current location information; Then, according to the angular velocity of the gyroscope G(t) = [G x (t),G y (t),G z (t)] T , rotation quaternion Q gyro (Δt) is calculated as: Among them, ||G(t)|| is the modulus of the angular velocity, indicating the rotation speed; The update is expressed as: By fusing the above position information with attitude information, i.e. quaternion, the key points of IMU estimation are: φ=[P t Q t ] T Finally, the posture estimated by the IMU is corrected with the key points in the visual data, and the visual key points are set to D. The dimension is reduced to the same dimension as Φ through linear mapping, and then the minimum cost function equation is constructed as: Among them, R is the rotation matrix, t is the translation matrix, λ is the regularization parameter, which ensures the orthogonality of the rotation matrix. The gradient descent obtains the rotation matrix R and the translation matrix t at the minimum cost, and updates the key point position: φ′=R·Φ+t, and the real-time monitoring and correction of the human body posture is realized according to the real-time updated key point position Φ.

Images