Unmanned aerial vehicle tunnel safety inspection pose recognition optimization method based on DDANet

Abstract

The invention relates to an unmanned aerial vehicle tunnel safety inspection pose recognition optimization method based on DDANet, and belongs to the technical field of tunnel safety monitoring. The method comprises the following steps of: introducing multi-level hole convolution on the basis of the traditional DenseNet-201, and constructing an initial network architecture; constructing an attention mechanism model (CBAM), sequentially connecting a channel attention mechanism and a space attention mechanism in series, and then accessing the channel attention mechanism and the space attention mechanism into a network to finally obtain a new network architecture DDANet; the method comprises the following steps: acquiring a video stream of unmanned aerial vehicle tunnel inspection through a monocular camera, and transmitting the video stream data to a new network architecture DDANet network for feature extraction to obtain corresponding feature points; and establishing an inter-frame matching tunnel feature point least square model according to the obtained feature points, and obtaining values of an unmanned aerial vehicle pose rotation matrix and a translation matrix by adopting a singular value decomposition method. According to the invention, the recognition precision and response speed of the unmanned aerial vehicle in the tunnel inspection process are improved.

Description

Technical Field

[0001] The invention relates to a DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method, and belongs to the technical field of tunnel safety monitoring. Background Art

[0002] With the acceleration of urbanization and the continuous development of transportation infrastructure, tunnels have become an indispensable part of the modern transportation system. Tunnels have complex structures and closed environments. After long-term use, they are prone to cracks, water seepage, and structural aging. Regular tunnel safety inspections are essential to ensure traffic safety. Traditional tunnel inspection methods mainly rely on manual inspections, which have problems such as low inspection efficiency, difficulty in covering the entire area, and subjective judgment errors. Especially in tunnels with insufficient light and narrow space, the difficulty and risk of manual inspections are further increased. At the same time, with the development of drone technology, automated tunnel inspections based on drones have gradually become an efficient and safe inspection method. The drone inspection system can not only cover complex areas in the tunnel, but also significantly improve the inspection efficiency. However, the harsh environment in the tunnel, insufficient light, large noise interference, and limited image quality have brought great challenges to the positioning and posture recognition of drones. In the inspection process, it is crucial to accurately identify the posture of the drone, which is directly related to the accuracy and safety of the inspection. Although existing posture recognition algorithms, such as those based on SLAM algorithms, can solve the posture estimation problem of drones to a certain extent, they still have shortcomings such as limited feature extraction capabilities and low computational efficiency in extreme environments such as tunnels.

[0003] In order to overcome the above problems, the present invention proposes a method for constructing an optimization model for posture recognition of UAV tunnel safety inspection based on DDANet, which provides a more robust and efficient solution for posture recognition of UAV in complex environments. Summary of the invention

[0004] The present invention provides a posture recognition optimization method for UAV tunnel safety inspection based on DDANet, which is used to solve the problem of low posture recognition efficiency and accuracy of UAV during tunnel safety inspection, aiming to improve the intelligence level of UAV inspection and its recognition ability in complex tunnel environment.

[0005] The technical solution of the present invention is: a posture recognition optimization method for unmanned aerial vehicle tunnel safety inspection based on DDANet, the method comprising the following steps:

[0006] Step 1: Based on the traditional DenseNet-201, multi-level hole convolution is introduced to build the initial network architecture;

[0007] Step 2: Build the attention mechanism model CBAM. First, connect the channel attention mechanism and the spatial attention mechanism in series, then connect them to the network, and finally get the new network architecture DDANet.

[0008] Step 3: Collect the video stream of the UAV tunnel inspection through a monocular camera, and transmit the video stream data to the new network architecture DDANet network for feature extraction to obtain the corresponding feature points;

[0009] Step 4: Based on the obtained feature points, a least squares model for inter-frame matching of tunnel feature points is established, and the singular value decomposition method is used to obtain the values ​​of the UAV posture rotation matrix and translation matrix.

[0010] As a further solution of the present invention, the Step 1 includes:

[0011] In the traditional DenseNet-201 network architecture, dilated convolution is only used in the second and third layers of dense blocks DenseBlock. When introducing multi-level dilated convolution, the HDC design criteria are followed and the convolution expansion rate sequence is set to [3,5].

[0012] The specific process is as follows: In the second layer DenseBlock, a dilation rate r0=3 is used to replace the original convolution. The actual convolution kernel size after adding the dilation rate is:

[0013] K1=K0+(K0-1)(r0-1);

[0014] Among them, K0 is the original convolution kernel size, K1 is the convolution kernel size after the hole, and r0 is the first convolution expansion rate;

[0015] The dilated convolution achieves the expansion of the receptive field by adding empty zeros, so the actual parameter quantity remains unchanged, and the parameter quantity of the same layer is the same. The specific parameter quantity is:

[0016] W = M*K*K*N;

[0017] Among them, M is the number of input channels, N is the number of output channels, and K is the actual convolution kernel size;

[0018] In the third layer, DenseBlock uses a dilation rate r1=5 to replace the original convolution. The actual convolution kernel size after adding the dilation is:

[0019] K2 = K0 + (K0 - 1) (r1 - 1);

[0020] Among them, K0 is the original convolution kernel size, K2 is the convolution kernel size after the hole in this layer, and r1 is the second convolution expansion rate;

[0021] After introducing the dilated convolution, the feature map is expressed as:

[0022]

[0023] in, is the convolution weight matrix, I r (x l ) is the convolution representation after introducing the dilation rate r of the dilated convolution, F d-1 It is the feature map representation of the previous layer, and n is the number of network layers.

[0024] As a further solution of the present invention, Step 2 includes:

[0025] The attention mechanism model CBAM is introduced into the DenseNet-201 network of the initial network architecture; the attention mechanism model CBAM includes a channel attention mechanism module and a spatial attention mechanism module;

[0026] The channel attention mechanism module and the spatial attention mechanism module are sequentially connected in series and then connected to each layer of dense blocks to obtain a new network architecture DDANet. The final feature map of the new network architecture DDANet is expressed as:

[0027]

[0028] in, and Represent the channel attention and spatial attention weights of the lth layer respectively, F d It is the feature map representation after the introduction of the dilated convolution, that is, a three-dimensional feature map is extracted from the main network, and n is the number of network layers.

[0029] As a further solution of the present invention, the channel attention mechanism module includes:

[0030] First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0031] Secondly, after global maximum pooling and global average pooling, two 1×1×C feature maps are obtained, denoted as and Through this step, the maximum response value and the average response value in each channel of the feature map are captured, that is, the balance between the most significant feature and the overall information in each channel;

[0032] Then, the two pooling feature maps and They are fed into a two-layer neural network MLP. The number of neurons in the first layer of the network is C / r, r is the reduction rate, and the activation function is Relu. The number of neurons in the second layer is C, no activation function is used, and a 1×1×C feature map is output to restore the original size of the channel dimension. The two-layer neural network is shared, which means that they share the same weights but input different pooling results.

[0033] Again, after MLP processing, and Perform element-by-element addition to obtain a 1×1×C feature map that combines the maximum pooling and average pooling features, and apply the Sigmoid function to activate the added feature map to compress the weighted value of each channel to between 0 and 1, thereby generating the final channel attention feature map M c ;

[0034] Finally, the final channel attention feature map M is generated c With the original input feature map F d Perform element-by-element multiplication to enhance the feature response of important channels and suppress the features of unimportant channels. According to the above process, the expression of the final output feature map is:

[0035]

[0036] Among them, W0 and W1 are shared weights, and It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

[0037] As a further solution of the present invention, the spatial attention mechanism module includes:

[0038] First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map; secondly, after global maximum pooling and global average pooling, two H×W×1 feature maps are generated respectively. and These two feature maps represent the maximum response and average response of each spatial position on all channels, respectively, and can provide a balance between local and global information;

[0039] Then, these two spatial feature maps and Through the splicing operation of the channel dimension, a H×W×2 feature map is generated to ensure all-round consideration of the spatial position;

[0040] Again, a convolution operation is performed on the concatenated feature map, using a 7×7 convolution kernel. The convolution operation reduces the number of channels of the feature map from 2 to 1, and a H×W×1 feature map F is obtained. s , where the value of each position represents the importance of the position in the entire feature map;

[0041] Then the feature map F s Normalized by Sigmoid activation function, its value is compressed to the range of 0 to 1 to generate the final spatial attention feature map M s ;

[0042] Finally, the final generated spatial attention feature map M s The feature map F after dimensionality reduction of the input s Perform element-by-element multiplication to generate the final enhanced feature map. The expression of the final output enhanced feature map is:

[0043]

[0044] Among them, f 7*7 is the size of the convolution kernel, It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

[0045] As a further solution of the present invention, Step 3 includes:

[0046] When the number of feature points in a frame image n1≥800, the pose estimation is initialized from this frame, and this frame is called the initial frame; let the pixel coordinates corresponding to the feature points of the initial frame of the model be:

[0047]

[0048] At the same time The corresponding tunnel feature point coordinates in the world coordinate system are:

[0049]

[0050] but and The coordinate transformation relationship is expressed as:

[0051]

[0052] Among them, (1u i,1 v i ) represents the number of rows and columns of the i-th feature point pixel in the first frame, Represents the pixel coordinates of the i-th feature point in the first frame, Represents the world coordinates of the i-th feature point in the first frame, 1x i 1y i 1z i respectively

[0053] , , are the horizontal and vertical coordinates, K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation matrix, T CW is the transformation matrix from the world coordinate system to the camera coordinate system;

[0054] Assuming that the initial frame camera coordinate system and the world coordinate system coincide, we can set T CW is the identity matrix; for and The coordinate transformation relationship formula of the feature point is missing the depth information of the tunnel scene. The depth data stream of the monocular camera is used to calculate the coordinates of each pixel. Restore the corresponding tunnel world coordinates The true depth z i ;

[0055] Suppose the pixel coordinates of the feature points extracted by the network model in the second frame after the initial frame are:

[0056]

[0057] Similarly, the coordinates of the corresponding tunnel landmark points in the world coordinates are:

[0058]

[0059] The fast approximate nearest neighbor method is used to perform feature matching on the initial frame and the second frame. Assume that the feature point sets that match well between the two frames are:

[0060]

[0061] Applying the above process to the entire video stream, the total set of feature points that match well between two frames is obtained as follows:

[0062] P=(P1,P2,…,P τ );

[0063] Among them, k≤min(n,m), τ is the total number of video frames.

[0064] As a further solution of the present invention, Step 4 includes:

[0065] The feature point sets P1 and P2 constitute the ICP problem. The singular value decomposition method is used to solve the transformation matrix from the initial frame to the second frame, that is, to solve the rotation matrix R and the translation matrix t. The robust ICP algorithm is used to calculate the error e of the i-th pair of matching feature points. i It is expressed as:

[0066]

[0067] in, is the world coordinate of the i-th feature point in the first frame, is the world coordinate of the i-th feature point in the second frame, R is the rotation matrix, and t is the translation matrix;

[0068] Construct a robust loss function:

[0069]

[0070] Among them, δ is a threshold. When the error is small, the square error is used, and when the error is large, the linear error is used to reduce the impact of outliers;

[0071] After introducing the robust loss function, the objective function of ICP becomes:

[0072]

[0073] The weighted least squares method is used to optimize the objective function, and the error of each pair of matching points is assigned a weight ω i , used to adjust the influence of each point:

[0074]

[0075] Applying the weights to the error term, the objective function is updated as:

[0076]

[0077] According to the optimization theory, the rotation matrix R and translation matrix t that minimize the sum of squared errors are the closest to the actual posture of the tunnel drone. According to the solution idea of ​​singular value decomposition, it is necessary to use the centroid coordinate pair E of the tunnel feature point set P1 and P2 min (R, t) is simplified; therefore, the centroids of the tunnel feature point sets P1 and P2 are first defined:

[0078]

[0079] Then the centroid coordinates are expressed as:

[0080]

[0081] Substituting into the ICP objective function, we can get:

[0082]

[0083] To solve for R, define the matrix M:

[0084]

[0085] Performing singular value decomposition on the point set after centroid alignment, we can get:

[0086] M=VUQ T ;

[0087] Among them, U is a diagonal matrix composed of singular values, V and Q are orthogonal matrices;

[0088] When M is full rank, R is:

[0089] R=VQ T ;

[0090] After obtaining the rotation matrix R, use the following formula to solve the translation matrix t:

[0091] t=p2-Rp1;

[0092] According to the rotation matrix R and translation matrix t obtained above, the position and posture recognition of the UAV during the tunnel safety inspection process is realized.

[0093] The beneficial effects of the present invention are as follows: the present invention improves the recognition accuracy and response speed of the UAV during the tunnel inspection process by innovating the deep learning network mechanism DDANet, thereby realizing efficient and safe tunnel inspection tasks;

[0094] The present invention solves the problem of low efficiency and accuracy in posture recognition of UAVs during tunnel safety inspections, and aims to improve the intelligence level of UAV inspections and their recognition capabilities in complex tunnel environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 for the invention flow chart;

[0096] Figure 2 This is the DDANet network architecture diagram;

[0097] Figure 3 This is a topological diagram of the drone tunnel safety inspection scenario, which specifically shows the changes in the rotation matrix and translation matrix when the drone is inspecting in the tunnel. DETAILED DESCRIPTION

[0098] Example 1: In order to solve the problem of low efficiency and accuracy of posture recognition of UAVs during tunnel safety inspections, the present invention provides a UAV tunnel safety inspection posture recognition optimization method based on DDANet, aiming to improve the intelligence level of UAV inspections and its recognition capabilities in complex tunnel environments.

[0099] Specific applicable environments include but are not limited to tunnel structure safety inspection scenarios. The drone is designed to replace manual inspections to perform safety inspection tasks in tunnels. It can accurately identify possible structural problems in tunnels and ensure the comprehensiveness and efficiency of inspections. This application can not only greatly improve the efficiency of inspection operations, but also reduce the high risks faced by manual inspections, and avoid long-term inspections in harsh environments. In complex and narrow tunnel environments, traditional drone inspections are unable to accurately perform posture recognition when faced with insufficient light and changing obstacles, resulting in low inspection efficiency and the risk of missed inspections. Therefore, the present invention improves the recognition accuracy and response speed of drones during tunnel inspections through the innovative deep learning network structure DDANet, thereby achieving efficient and safe tunnel inspection tasks.

[0100] like Figure 1-Figure 3 As shown, a posture recognition optimization method for unmanned aerial vehicle tunnel safety inspection based on DDANet comprises the following steps:

[0101] Step 1: Based on the traditional DenseNet-201, multi-level hole convolution is introduced to build the initial network architecture;

[0102] The traditional DenseNet-201 network consists of four DenseBlock layers (dense blocks) and three TransitionLayer layers (transition layers). Each DenseBlock layer is connected through the Transition Layer layer; the input of each layer of DenseNet-201 not only includes the feature map outputs of all previous layers in the same DenseBlock, but also the connection of these feature maps will become part of the output. Therefore, we can get:

[0103]

[0104] in, is a nonlinear transformation function, including a series of batch normalization, ReLU activation function and 3×3 convolution operations; l is the number of layers; x l is the current layer output, x l-1 is the current layer input, x l-n is the input of the previous layer. After each convolution operation, k feature maps are generated, which means that k channels or k convolution kernels are used. The specific calculation of the number of channels is as follows:

[0105] C l =C l-1 +k(l-1);

[0106] Among them, k represents the growth rate of the network, l is the number of layers, and C l-1 is the number of channels of the input layer feature map, C lis the number of channels of the output layer feature map.

[0107] Although k is set very small, the dense connection structure of DenseNet still causes the input of the subsequent layers to be very large. Therefore, in order to solve the problem of excessive input, a 1×1 convolution layer is introduced before each 3×3 convolution layer, called BottleNeck. The role of the 1×1 convolution layer is to reduce the number of features, reduce the amount of calculation, and improve the calculation efficiency. After the output, a BN-ReLU transformation is performed again, and then through the 3×3 convolution layer, it is restored to the final k output feature maps for the next convolution.

[0108] Atrous convolution, also known as dilated convolution or expanded convolution, is simply the process of adding some spaces (zeros) between the convolution kernel elements to expand the convolution kernel. A major feature of atrous convolution is that it can increase the receptive field while maintaining the spatial resolution of the feature map. Assuming that a variable r is used to measure the dilation coefficient of atrous convolution, the relationship between the actual convolution kernel size after adding the hole and the original convolution kernel size is:

[0109] K l =K l-1 +(K l-1 )(r-1);

[0110] Among them, K is the convolution kernel size and r is the convolution expansion rate.

[0111] The relationship between the actual receptive field and the original receptive field is:

[0112]

[0113] Among them, γ l is the receptive field size corresponding to the current feature map, γ l-1 is the receptive field size corresponding to the feature map of the previous layer, K l is the convolution kernel size of the current convolution layer, Represents the stride product of the previous convolutional layer.

[0114] As a further solution of the present invention, the Step 1 includes:

[0115] In the traditional DenseNet-201 network architecture, there are four layers of DenseBlock. Considering that replacing the original convolution with dilated convolution may increase the amount of calculation, dilated convolution is only used in the second and third layers of dense blocks DenseBlock; when introducing multi-level dilated convolution, the HDC design criteria are followed and the convolution expansion rate sequence is set to [3,5]; the expansion rate sequence is set in this way because: 1) a larger receptive field can be obtained in a shallower network layer; 2) multi-level convolution expansion rates can be used for multi-scale feature extraction.

[0116] The specific process is as follows: In the second layer DenseBlock, a dilation rate r0=3 is used to replace the original convolution. The actual convolution kernel size after adding the dilation rate is:

[0117] K1=K0+(K0-1)(r0-1);

[0118] Among them, K0 is the original convolution kernel size, K1 is the convolution kernel size after the hole, and r0 is the first convolution expansion rate;

[0119] The dilated convolution achieves the expansion of the receptive field by adding empty zeros, so the actual parameter quantity remains unchanged, and the parameter quantity of the same layer is the same. The specific parameter quantity is:

[0120] W = M*K*K*N;

[0121] Among them, M is the number of input channels, N is the number of output channels, and K is the actual convolution kernel size;

[0122] In the third layer, DenseBlock uses a dilation rate r1=5 to replace the original convolution. The actual convolution kernel size after adding the dilation is:

[0123] K2 = K0 + (K0 - 1) (r1 - 1);

[0124] Among them, K0 is the original convolution kernel size, K2 is the convolution kernel size after the hole in this layer, and r1 is the second convolution expansion rate;

[0125] After introducing the dilated convolution, the feature map is expressed as:

[0126]

[0127] in, is the convolution weight matrix, I r (x l ) is the convolution representation after introducing the dilation rate r of the dilated convolution, F d-1 It is the feature map representation of the previous layer, and n is the number of network layers.

[0128] Step 2: Build the attention mechanism model CBAM. First, connect the channel attention mechanism and the spatial attention mechanism in series, then connect them to the network, and finally get the new network architecture DDANet.

[0129] As a further solution of the present invention, Step 2 includes:

[0130] In order to better obtain the key information of the image and enhance the network's ability to capture long-range dependencies, the attention mechanism model CBAM is introduced into the initial network architecture DenseNet-201 network; the attention mechanism model CBAM includes a channel attention mechanism module and a spatial attention mechanism module;

[0131] The channel attention mechanism module and the spatial attention mechanism module are sequentially connected in series and then connected to each layer of dense blocks to obtain a new network architecture DDANet. The final feature map of the new network architecture DDANet is expressed as:

[0132]

[0133] in, and Represent the channel attention and spatial attention weights of the lth layer respectively, F d It is the feature map representation after the introduction of the dilated convolution, that is, a three-dimensional feature map is extracted from the main network, and n is the number of network layers.

[0134] As a further solution of the present invention, the channel attention mechanism module includes:

[0135] First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0136] Secondly, after global maximum pooling and global average pooling, two 1×1×C feature maps are obtained, denoted as and Through this step, the maximum response value and the average response value in each channel of the feature map are captured, that is, the balance between the most significant feature and the overall information in each channel;

[0137] Then, the two pooling feature maps and They are fed into a two-layer neural network MLP. The number of neurons in the first layer of the network is C / r, r is the reduction rate, and the activation function is Relu. The number of neurons in the second layer is C, no activation function is used, and a 1×1×C feature map is output to restore the original size of the channel dimension. The two-layer neural network is shared, which means that they share the same weights but input different pooling results.

[0138] Again, after MLP processing, and Perform element-by-element addition to obtain a 1×1×C feature map that combines the maximum pooling and average pooling features, and apply the Sigmoid function to activate the added feature map to compress the weighted value of each channel to between 0 and 1, thereby generating the final channel attention feature map M c ;

[0139] Finally, the final channel attention feature map M is generated c With the original input feature map F d Perform element-by-element multiplication to enhance the feature response of important channels and suppress the features of unimportant channels. According to the above process, the expression of the final output feature map is:

[0140]

[0141] Among them, W0 and W1 are shared weights, and It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

[0142] As a further solution of the present invention, the spatial attention mechanism module includes:

[0143] First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map; secondly, after global maximum pooling and global average pooling, two H×W×1 feature maps are generated respectively. and These two feature maps represent the maximum response and average response of each spatial position on all channels, respectively, and can provide a balance between local and global information;

[0144] Then, these two spatial feature maps and Through the splicing operation of the channel dimension, a H×W×2 feature map is generated to ensure all-round consideration of the spatial position;

[0145] Again, a convolution operation is performed on the concatenated feature map, using a 7×7 convolution kernel. The convolution operation reduces the number of channels of the feature map from 2 to 1, and a H×W×1 feature map F is obtained. s , where the value of each position represents the importance of the position in the entire feature map;

[0146] Then the feature map F sNormalized by Sigmoid activation function, its value is compressed to the range of 0 to 1 to generate the final spatial attention feature map M s ;

[0147] Finally, the final generated spatial attention feature map M s The feature map F after dimensionality reduction of the input s Perform element-by-element multiplication to generate the final enhanced feature map. The expression of the final output enhanced feature map is:

[0148]

[0149] Among them, f 7*7 is the size of the convolution kernel, It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

[0150] Step 3: Collect the video stream of the UAV tunnel inspection through a monocular camera, and transmit the video stream data to the new network architecture DDANet network for feature extraction to obtain the corresponding feature points;

[0151] As a further solution of the present invention, Step 3 includes:

[0152] When the number of feature points in a frame image n1≥800, the pose estimation is initialized from this frame, and this frame is called the initial frame; let the pixel coordinates corresponding to the feature points of the initial frame of the model be:

[0153]

[0154] At the same time The corresponding tunnel feature point coordinates in the world coordinate system are:

[0155]

[0156] but and The coordinate transformation relationship is expressed as:

[0157]

[0158] Among them, (1u i,1 v i ) represents the number of rows and columns of the i-th feature point pixel in the first frame, Represents the pixel coordinates of the i-th feature point in the first frame, Represents the world coordinates of the i-th feature point in the first frame, 1x i 1y i 1z irespectively

[0159] , , are the horizontal and vertical coordinates, K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation matrix, T CW is the transformation matrix from the world coordinate system to the camera coordinate system;

[0160] Assuming that the initial frame camera coordinate system and the world coordinate system coincide, we can set T CW is the identity matrix; for and The coordinate transformation relationship formula of the feature point is missing the depth information of the tunnel scene. The depth data stream of the monocular camera is used to calculate the coordinates of each pixel. Restore the corresponding tunnel world coordinates The true depth z i ;

[0161] Suppose the pixel coordinates of the feature points extracted by the network model in the second frame after the initial frame are:

[0162]

[0163] Similarly, the coordinates of the corresponding tunnel landmark points in the world coordinates are:

[0164]

[0165] The fast approximate nearest neighbor method is used to perform feature matching on the initial frame and the second frame. Assume that the feature point sets that match well between the two frames are:

[0166]

[0167] Applying the above process to the entire video stream, the total set of feature points that match well between two frames is obtained as follows:

[0168] P=(P1,P2,…,P τ );

[0169] Among them, k≤min(n,m), τ is the total number of video frames.

[0170] Step 4. Based on the obtained feature points, a least squares model for inter-frame matching of tunnel feature points is established, and the singular value decomposition method is used to obtain the values ​​of the UAV posture rotation matrix and translation matrix, thereby improving the positioning accuracy in complex environments.

[0171] As a further solution of the present invention, Step 4 includes:

[0172] The feature point sets P1 and P2 constitute the ICP problem. The singular value decomposition method is used to solve the transformation matrix from the initial frame to the second frame, that is, to solve the rotation matrix R and the translation matrix t. Due to the influence of illumination and material sensor properties, there is an error between the conversion relationship between the initial frame and the second frame. Using the robust ICP algorithm, the error e of the i-th pair of matching feature points is i It is expressed as:

[0173]

[0174] in, is the world coordinate of the i-th feature point in the first frame, is the world coordinate of the i-th feature point in the second frame, R is the rotation matrix, and t is the translation matrix;

[0175] Construct a robust loss function:

[0176]

[0177] Among them, δ is a threshold. When the error is small, the square error is used, and when the error is large, the linear error is used to reduce the impact of outliers;

[0178] After introducing the robust loss function, the objective function of ICP becomes:

[0179]

[0180] The weighted least squares method is used to optimize the objective function, and the error of each pair of matching points is assigned a weight ω i , used to adjust the influence of each point:

[0181]

[0182] Applying the weights to the error term, the objective function is updated as:

[0183]

[0184] According to the optimization theory, the rotation matrix R and translation matrix t that minimize the sum of squared errors are the closest to the actual posture of the tunnel drone. According to the solution idea of ​​singular value decomposition, it is necessary to use the centroid coordinate pair E of the tunnel feature point set P1 and P2 min (R, t) is simplified; therefore, the centroids of the tunnel feature point sets P1 and P2 are first defined:

[0185]

[0186] Then the centroid coordinates are expressed as:

[0187]

[0188] Substituting into the ICP objective function, we can get:

[0189]

[0190] To solve for R, define the matrix M:

[0191]

[0192] Performing singular value decomposition on the point set after centroid alignment, we can get:

[0193] M=VUQ T ;

[0194] Among them, U is a diagonal matrix composed of singular values, V and Q are orthogonal matrices;

[0195] When M is full rank, R is:

[0196] R=VQ T ;

[0197] After obtaining the rotation matrix R, use the following formula to solve the translation matrix t:

[0198] t=p2-Rp1;

[0199] According to the rotation matrix R and translation matrix t obtained above, the position and posture recognition of the UAV during the tunnel safety inspection process is realized.

[0200] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. The optimization method of posture recognition for UAV tunnel safety inspection based on DDANet is characterized by: The method comprises the following steps: Step 1: Based on the traditional DenseNet-201, multi-level hole convolution is introduced to build the initial network architecture; Step 2: Build the attention mechanism model CBAM. First, connect the channel attention mechanism and the spatial attention mechanism in series, then connect them to the network, and finally get the new network architecture DDANet. Step 3: Collect the video stream of the UAV tunnel inspection through a monocular camera, and transmit the video stream data to the new network architecture DDANet network for feature extraction to obtain the corresponding feature points; Step 4: Based on the obtained feature points, a least squares model for inter-frame matching of tunnel feature points is established, and the singular value decomposition method is used to obtain the values ​​of the UAV posture rotation matrix and translation matrix.

2. According to the DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method of claim 1, it is characterized in that: The Step 1 includes: In the traditional DenseNet-201 network architecture, dilated convolution is only used in the second and third layers of dense blocks DenseBlock. When introducing multi-level dilated convolution, the HDC design criteria are followed and the convolution expansion rate sequence is set to [3,5]. The specific process is as follows: In the second layer DenseBlock, a dilation rate r0=3 is used to replace the original convolution. The actual convolution kernel size after adding the dilation rate is: K1=K0+(K0-1)(r0-1); Among them, K0 is the original convolution kernel size, K1 is the convolution kernel size after the hole, and r0 is the first convolution expansion rate; The dilated convolution achieves the expansion of the receptive field by adding empty zeros, so the actual parameter quantity remains unchanged, and the parameter quantity of the same layer is the same. The specific parameter quantity is: W = M*K*K*N; Among them, M is the number of input channels, N is the number of output channels, and K is the actual convolution kernel size; In the third layer, DenseBlock uses a dilation rate r1=5 to replace the original convolution. The actual convolution kernel size after adding the dilation is: K2 = K0 + (K0 - 1) (r1 - 1); Among them, K0 is the original convolution kernel size, K2 is the convolution kernel size after the hole in this layer, and r1 is the second convolution expansion rate; After introducing the dilated convolution, the feature map is expressed as: in, is the convolution weight matrix, I r (x l ) is the convolution representation after introducing the dilation rate r of the dilated convolution, F d-1 It is the feature map representation of the previous layer, and n is the number of network layers.

3. The DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method according to claim 1 is characterized in that: The Step 2 includes: The attention mechanism model CBAM is introduced into the DenseNet-201 network of the initial network architecture; the attention mechanism model CBAM includes a channel attention mechanism module and a spatial attention mechanism module; The channel attention mechanism module and the spatial attention mechanism module are sequentially connected in series and then connected to each layer of dense blocks to obtain a new network architecture DDANet. The final feature map of the new network architecture DDANet is expressed as: in, and Represent the channel attention and spatial attention weights of the lth layer respectively, F d It is the feature map representation after the introduction of the dilated convolution, that is, a three-dimensional feature map is extracted from the main network, and n is the number of network layers.

4. According to claim 3, the DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method is characterized in that: The channel attention mechanism module includes: First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map; Secondly, after global maximum pooling and global average pooling, two 1×1×C feature maps are obtained, denoted as and Through this step, the maximum response value and the average response value in each channel of the feature map are captured, that is, the balance between the most significant feature and the overall information in each channel; Then, the two pooling feature maps and They are fed into a two-layer neural network MLP. The number of neurons in the first layer of the network is C / r, r is the reduction rate, and the activation function is Relu. The number of neurons in the second layer is C, no activation function is used, and a 1×1×C feature map is output to restore the original size of the channel dimension. The two-layer neural network is shared, which means that they share the same weights but input different pooling results. Again, after MLP processing, and Perform element-by-element addition to obtain a 1×1×C feature map that combines the maximum pooling and average pooling features, and apply the Sigmoid function to activate the added feature map to compress the weighted value of each channel to between 0 and 1, thereby generating the final channel attention feature map M c ; Finally, the final channel attention feature map M is generated c With the original input feature map F d Perform element-by-element multiplication to enhance the feature response of important channels and suppress the features of unimportant channels. According to the above process, the expression of the final output feature map is: Among them, W0 and W1 are shared weights, and It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

5. The DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method according to claim 3 is characterized in that: The spatial attention mechanism module includes: First, a three-dimensional feature map F is extracted from the main network d , the shape is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map; secondly, after global maximum pooling and global average pooling, two H×W×1 feature maps are generated respectively. and These two feature maps represent the maximum response and average response of each spatial position on all channels, respectively, and can provide a balance between local and global information; Then, these two spatial feature maps and Through the splicing operation of the channel dimension, a H×W×2 feature map is generated to ensure all-round consideration of the spatial position; Again, a convolution operation is performed on the concatenated feature map, using a 7×7 convolution kernel. The convolution operation reduces the number of channels of the feature map from 2 to 1, and a H×W×1 feature map F is obtained. s , where the value of each position represents the importance of the position in the entire feature map; Then the feature map F s Normalized by Sigmoid activation function, its value is compressed to the range of 0 to 1 to generate the final spatial attention feature map M s ; Finally, the final generated spatial attention feature map M s The feature map F after dimensionality reduction of the input s Perform element-by-element multiplication to generate the final enhanced feature map. The expression of the final output enhanced feature map is: Among them, f 7*7 is the size of the convolution kernel, It is the feature map after the global average pooling operation. It is the feature map after the global maximum pooling operation, and σ represents the Sigmoid activation function.

6. The DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method according to claim 1 is characterized in that: The Step 3 includes: When the number of feature points in a frame image n1≥800, the pose estimation is initialized from this frame, and this frame is called the initial frame; let the pixel coordinates corresponding to the feature points of the initial frame of the model be: At the same time The corresponding tunnel feature point coordinates in the world coordinate system are: but and The coordinate transformation relationship is expressed as: Among them, (1u i,1 v i ) represents the number of rows and columns of the i-th feature point pixel in the first frame, Represents the pixel coordinates of the i-th feature point in the first frame, Represents the world coordinates of the i-th feature point in the first frame, 1x i1 y i1 z i Respectively, , are the horizontal and vertical coordinates, K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation matrix, T CW is the transformation matrix from the world coordinate system to the camera coordinate system; Assuming that the initial frame camera coordinate system and the world coordinate system coincide, we can set T CW is the identity matrix; for and The coordinate transformation relationship formula of the feature point is missing the depth information of the tunnel scene. The depth data stream of the monocular camera is used to calculate the coordinates of each pixel. Restore the corresponding tunnel world coordinates The true depth z i ; Suppose the pixel coordinates of the feature points extracted by the network model in the second frame after the initial frame are: Similarly, the coordinates of the corresponding tunnel landmark points in the world coordinates are: The fast approximate nearest neighbor method is used to perform feature matching on the initial frame and the second frame. Assume that the feature point sets that match well between the two frames are: Applying the above process to the entire video stream, the total set of feature points that match well between two frames is obtained as follows: P=(P1,P2,…,P τ ); Among them, k≤min(n,m), τ is the total number of video frames.

7. The DDANet-based unmanned aerial vehicle tunnel safety inspection posture recognition optimization method according to claim 1 is characterized in that: The Step 4 includes: The feature point sets P1 and P2 constitute the ICP problem. The singular value decomposition method is used to solve the transformation matrix from the initial frame to the second frame, that is, to solve the rotation matrix R and the translation matrix t. The robust ICP algorithm is used to calculate the error e of the i-th pair of matching feature points. i It is expressed as: in, is the world coordinate of the i-th feature point in the first frame, is the world coordinate of the i-th feature point in the second frame, R is the rotation matrix, and t is the translation matrix; Construct a robust loss function: Among them, δ is a threshold. When the error is small, the square error is used, and when the error is large, the linear error is used to reduce the impact of outliers; After introducing the robust loss function, the objective function of ICP becomes: The weighted least squares method is used to optimize the objective function, and the error of each pair of matching points is assigned a weight ω i , used to adjust the influence of each point: Applying the weights to the error term, the objective function is updated as: According to the optimization theory, the rotation matrix R and translation matrix t that minimize the sum of squared errors are the closest to the actual posture of the tunnel drone. According to the solution idea of ​​singular value decomposition, it is necessary to use the centroid coordinate pair E of the tunnel feature point set P1 and P2 min (R, t) is simplified; therefore, the centroids of the tunnel feature point sets P1 and P2 are first defined: Then the centroid coordinates are expressed as: Substituting into the ICP objective function, we can get: To solve for R, define the matrix M: Performing singular value decomposition on the point set after centroid alignment, we can get: M=VUQ T ; Among them, U is a diagonal matrix composed of singular values, V and Q are orthogonal matrices; When M is full rank, R is: R=VQ T ; After obtaining the rotation matrix R, use the following formula to solve the translation matrix t: t=p2-Rp1; According to the rotation matrix R and translation matrix t obtained above, the position and posture recognition of the UAV during the tunnel safety inspection process is realized.

Images