VR / AR operation-oriented ground pollution airflow identification and tracking method

Abstract

The invention discloses a VR / AR operation-oriented ground pollution airflow identification and tracking method. The method comprises the following steps of: constructing a pollution airflow identification model; collecting an airflow picture, inputting the airflow picture into the pollution airflow identification model, and identifying and tracking the pollution airflow; the airflow recognition model comprises a backbone network, a multi-scale pooling stripe convolution module MPSCM, a relation attention network and a context feature fusion network; the backbone network, the multi-scale pooling stripe convolution module MPSCM and the context feature fusion network are connected in sequence; the relational attention network comprises a plurality of relational attention modules RAM, and the context feature fusion network comprises a plurality of context feature fusion modules CFFM; the relation attention module RAM is used for fusing output information of adjacent layers in the backbone network and outputting a feature map to the corresponding context feature fusion module CFFM for fusion, and the context feature fusion network outputs a first feature map.

Description

Technical Field

[0001] The present invention belongs to the field of image data processing, and in particular relates to a method for a drone for VR / AR operation to identify and track ground polluted airflow. Background of the Invention

[0002] In the current low-altitude economy, an important task is to identify and track pollutants. The existing mature solutions are mainly for the identification of water and solid pollutants, but there is relatively little research on gaseous pollutants. The difficulty is mainly that gases will drift with the wind. Generally speaking, people driving cars cannot keep up with the airflow. Therefore, in the low-altitude economy, the identification and tracking of polluted airflow has become a problem that needs to be tackled.

[0003] Nowadays, many units have tried to carry collectors of different wavelengths on drones to collect polluted gases and form pictures for inspection in key areas to track and measure polluted airflows. However, traditional, general semantic segmentation networks such as UNet do not have obvious feature extraction and fusion effects on polluted airflow segmentation in complex atmospheric and ground backgrounds, and may have problems such as missed detection, false detection, inadequate segmentation details, inaccurate segmentation positions, and poor generalization performance.

[0004] Therefore, people need a method that can effectively detect polluted gas in complex atmosphere and ground backgrounds, improve the control of details and spatial position, and determine the accurate target segmentation boundary for the polluted gas images collected by collectors of different wavelengths on drone platforms. Then display them on the drone operating platform, such as VR / AR glasses and display screens. Summary of the invention

[0005] Based on the existing UAV technology, artificial intelligence, and virtual reality technology, the present invention proposes a method for identifying and tracking ground polluted airflow for VR / AR operations. The method can effectively detect the distribution area of ​​polluted gases in complex atmosphere and ground backgrounds based on polluted airflow images collected by collectors of different wavelengths on the UAV platform, improve the control of details and spatial position, and determine the accurate target segmentation boundary, which is finally displayed to the VR / AR helmet.

[0006] The method for identifying and tracking ground polluted airflow for VR / AR operation of the present invention specifically comprises: constructing a polluted airflow identification model; collecting airflow pictures, inputting the airflow pictures into the polluted airflow identification model, and identifying and tracking the polluted airflow;

[0007] The airflow recognition model includes a backbone network, a multi-scale pooling strip convolution module MPSCM, a relational attention network and a context feature fusion network.

[0008] The backbone network, the multi-scale pooling strip convolution module MPSCM and the context feature fusion network are connected in sequence;

[0009] The relational attention network includes a plurality of relational attention modules RAM, and the contextual feature fusion network includes a plurality of contextual feature fusion modules CFFM;

[0010] The number of relational attention module RAMs is the same as the number of contextual feature fusion modules CFFM, and they correspond one to one; the relational attention module RAM is used to fuse the output information of adjacent layers in the backbone network, and output the feature map to the corresponding contextual feature fusion module CFFM for fusion, and output the first feature map.

[0011] Furthermore, the backbone network is ResNet50, including Layer1, Layer2, Layer3, Layer4 and Layer5 connected in sequence; there are four relational attention modules RAM, namely RAM1, RAM2, RAM3 and RAM4; RAM1 is used to fuse the feature maps output by Layer1 and Layer2; RAM2 is used to fuse the feature maps output by Layer2 and Layer3; RAM3 is used to fuse the feature maps output by Layer3 and Layer4; RAM4 is used to fuse the feature maps output by Layer4 and Layer5.

[0012] Furthermore, there are four context feature fusion modules CFFM, namely CFFM1, CFFM2, CFFM3 and CFFM4 connected in sequence; the outputs of RAM1, RAM2, RAM3 and RAM4 correspond to an input feature map of CFFM1, CFFM2, CFFM3 and CFFM4 respectively;

[0013] The output of RAM4 is used as a low-level feature map, and the output of the multi-scale pooling strip convolution module MPSCM is used as a high-level feature map and is input into CFFM4 for feature fusion;

[0014] The output of RAM3 is used as the low-level feature map, and the output of CFFM4 is used as the high-level feature map, which are input into CFFM3 for feature fusion;

[0015] The output of RAM2 is used as a low-level feature map, and the output of CFFM3 is used as a high-level feature map and is input into CFFM2 for feature fusion;

[0016] The output of RAM1 is used as a low-level feature map, and the output of CFFM2 is used as a high-level feature map, which are input into CFFM1 for feature fusion, and CFFM1 outputs the first feature map.

[0017] Furthermore, each relational attention module RAM includes a self-attention module and a cross-attention module;

[0018] The shallow feature map F1 is input into the self-attention module for feature extraction to obtain the feature map F1 ″ :

[0019] The adjacent deep high-level feature map F2 is upsampled and input into the cross attention module together with the shallow feature map F1 to capture the correlation information between different levels and the dependency relationship within the same level, and obtain the feature map F″2;

[0020] Finally, the feature map F″1 output by the self-attention module and the feature map F″2 output by the cross-attention module are added to obtain the output F of the relational attention module RAM.

[0021] For the relational attention module RAM1, the output of Layer1 is the shallow feature map F1, and the output of Layer1 is the deep high-level feature map F2 adjacent to Layer1; the shallow and deep layers mentioned here are relative; similarly, for the relational attention module RAM2, the output of Layer2 is the shallow feature map F1, and the output of Layer3 is the deep high-level feature map F2 adjacent to Layer2.

[0022] Furthermore, the shallow feature map F1 is input into the self-attention module for feature extraction, specifically: the shallow feature map F1 undergoes three 3×3 two-dimensional convolutions and reshape operations to obtain three features of the query vector Q1, the key vector K1 and the value vector V1 respectively, and then the query vector Q1 and the key vector K1 undergo a matrix multiplication operation to obtain a feature map, which is then normalized by a softmax operation and then matrix multiplication is performed with the value vector V1 to obtain the feature map F′1. After a 1×1 convolution and reshape operation, the feature map F′1 is restored to the same size as the shallow feature map F1, and finally, after a residual connection, the output feature map F″1 of the self-attention module is obtained.

[0023] Furthermore, the adjacent deep high-level feature map F2 is upsampled and input into the cross-attention module together with the shallow feature map F1 to capture the correlation information between different levels and the dependency relationship within the same level. Specifically, the high-level feature map F2 is input into the cross-attention module and undergoes two 3×3 convolution and reshape operations respectively to obtain the key vector K2 and the value vector V2. The primary feature map F1 is input into the cross-attention module and undergoes a 3×3 convolution and reshape operation to obtain the query vector Q2. Then the query vector Q2 and the key vector K2 undergo a matrix multiplication operation, and then a Softmax operation and then matrix multiplication with the value vector V2 to obtain the feature map F′2. After a 1×1 convolution and reshape operation, it is restored to the same size as the high-level feature map F2. Finally, after a residual connection, the output feature map F″2 of the cross-attention module is obtained.

[0024] For RAM1 in this application, the output of Layer1 in the backbone network is used as the shallow feature map F1, and the output of Layer2 is used as the deep high-level feature map F2, which are input into RAM1 for fusion;

[0025] Similarly, for RAM2, the output of Layer2 in the backbone network is used as the shallow feature map F1, and the output of Layer3 is used as the deep high-level feature map F2, which are input into RAM3 for fusion; and the same applies to RAM3 and RAM4.

[0026] Furthermore, the context feature fusion module CFFM includes a detail branch and a stream alignment branch; the input of the context feature fusion module CFFM includes a low-level feature map F D and high-level feature map F F ;

[0027] Among them, the low-level feature map F D is the output feature map of the corresponding relational attention module RAM;

[0028] High-level feature map F F It is the output feature map of the previous module of the context feature fusion module CFFM. For CFFM4, the previous module is the multi-scale pooling strip convolution module MPSCM; for CFFM3, the previous module is CFFM3; for CFFM2 and CFFM1, the same applies;

[0029] For the flow alignment branch: the low-level feature map F D Through two 3×3 convolutions, we get the feature map F′ D1 and F′ D2 ; High-level feature map F F Through a 3×3 convolution, we get the feature map F′ F1; Then, the feature map F F and F′ F1 Perform bilinear upsampling to make its resolution the same as the low-level feature map F′ D1 Equal, we get F′ F and F″ F1 ; The feature map F″ F1 and F′ D2 Splice them together and pass the convolution kernel size 3×3 to obtain a semantic flow field; F′ F Align (warp) with the semantic flow field to obtain the aligned feature map;

[0030] For the detail branch: the high-level feature map F″ after upsampling F1 Activated by Sigmoid, each pixel value in the feature map is mapped to the range of (0, 1) and then convolved with the low-level feature map F′ after 3×3 convolution. D1 The obtained feature map is then concatenated with the aligned feature map, and then subjected to a 3×3 convolution to output the first feature map.

[0031] Furthermore, the multi-scale pooling strip convolution module MPSCM includes four branches and one residual connection;

[0032] The four branches include a first strip average pooling branch, a second strip average pooling branch, a global average pooling branch and a 1×1 convolution branch;

[0033] After the input features undergo strip average pooling and 1×1 convolution in the first strip average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the first strip average pooling branch;

[0034] After the input features undergo strip average pooling and 1×1 convolution in the second strip average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the second strip average pooling branch;

[0035] After the input features undergo global average pooling and 1×1 convolution in the global average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the global average pooling branch;

[0036] The output feature maps of the first strip average pooling branch, the second strip average pooling branch, the global average pooling branch and the 1×1 convolution branch are merged, and then connected through 1×1 convolution and residual to obtain the feature map output by the multi-scale pooling strip convolution module MPSCM.

[0037] Furthermore, the convolution kernel sizes of the strip average pooling branches are 1×N, N×1, and N×N, respectively; N=3 in the first strip average pooling branch, and N=5 in the second strip average pooling branch.

[0038] Furthermore, the polluted airflow identification model of the present invention further includes a boundary enhancement module BEM;

[0039] The feature map output by the context feature fusion network is input into the boundary enhancement module BEM for boundary enhancement;

[0040] In the boundary enhancement module BEM, the feature map undergoes two 3×3 convolutions and a residual structure, and then passes through the SegHead structure to obtain the final feature map; among them, a Dropout layer is added to the SegHead to discard neurons with a probability of 0.1.

[0041] Beneficial effects: In the method for ground polluted airflow identification and tracking for VR / AR operation proposed in the present invention, ResNet50 is used as the backbone network, which can efficiently capture complex image features and has good generalization ability; the present invention also designs and uses a multi-scale pooling strip convolution module (MPSCM) to help the model extract polluted airflow features of different sizes; a relational attention module RAM is designed to enhance feature fusion and global context understanding through cross and self-attention mechanisms to improve segmentation accuracy; a context feature fusion module CFFM is designed to fuse high-level and low-level features to retain spatial details and semantic information; the addition of a boundary enhancement module BEM accelerates convergence, strengthens boundary information, and improves prediction quality. Finally, the predicted segmentation results are displayed to the operator through a VR / AR helmet to facilitate further operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall network structure diagram of the present invention;

[0043] Figure 2 It is a structural diagram of the residual network in the present invention;

[0044] Figure 3 It is a structural diagram of the multi-scale pooling strip convolution module MPSCM of the present invention;

[0045] Figure 4 is a structural diagram of the relational attention module RAM of the present invention;

[0046] Figure 5 It is a structural diagram of the context feature fusion module CFFM of the present invention;

[0047] Figure 6 It is a structural diagram of the boundary enhancement module BEM of the present invention. DETAILED DESCRIPTION

[0048] The following is a specific embodiment of the present invention Figures 1 to 6 , and Table 1 to specifically describe the method proposed in the present invention.

[0049] The present invention proposes a method for identifying and tracking ground polluted airflow based on an unmanned aerial vehicle, comprising: first, building a polluted airflow identification model; second, collecting airflow pictures, inputting the airflow pictures into the polluted airflow identification model, identifying and tracking the polluted airflow; and finally displaying the predicted segmentation results to the operator through a VR / AR helmet to facilitate further operations.

[0050] The overall structure of the polluted airflow identification model is as follows: Figure 1 As shown. It includes the backbone network, multi-scale pooling strip convolution module MPSCM, relation attention module RAM, context feature fusion module CFFM and boundary enhancement module BEM;

[0051] The backbone network, multi-scale pooling strip convolution module MPSCM, context feature fusion module CFFM and boundary enhancement module BEM are connected in sequence; the number of relation attention module RAMs is the same as the number of context feature fusion modules CFFM, and they correspond one to one; the relation attention module RAM is used to fuse the output information of adjacent layers in the backbone network, and output the feature map to the corresponding context feature fusion module CFFM for fusion.

[0052] The overall introduction of the polluted air flow identification model is as follows:

[0053] The accuracy of polluted airflow feature extraction directly affects the accuracy of semantic category assignment for each pixel or pixel area in the final image. The present invention adopts ResNet50 as the backbone network (for specific content, please refer to https: / / blog.csdn.net / weixin_42419898 / article / details / 107187494). ResNet50 can efficiently capture complex image features and has good generalization ability. ResNet50 uses the same residual block structure to share network parameters, reduce the complexity of the network, and make training more efficient. In order to further improve the multi-scale information extraction capability, the present invention also designs and uses a multi-scale pooling strip convolution module (MPSCM) to help the model extract polluted airflow features of different sizes.

[0054] In the decoding stage, the existing general segmentation network cannot effectively restore the lost details. For this reason, the present invention designs a relational attention module RAM, which enhances feature fusion and global context understanding through cross and self-attention mechanisms to improve segmentation accuracy. The cross attention channel can enhance the communication between the two channels, so that the two adjacent main network layers can guide each other to perform feature mining, thereby promoting image feature fusion. Secondly, the self-attention channel can capture the relationship between different positions in the image on a global scale, thereby providing richer global context information. By calculating global correlations, the model can obtain the semantic structure and content of the entire image, rather than just being limited to local areas.

[0055] At the same time, the context feature fusion module CFFM fuses high-level and low-level features to retain spatial details and semantic information. The boundary enhancement module BEM accelerates convergence, strengthens boundary information, and improves prediction quality. Table 1 lists the specific parameter settings of the model. The overall design aims to comprehensively optimize the segmentation effect of polluted airflow.

[0056] 1. Backbone network

[0057] ResNet50 is a relatively deep neural network model (for details, please refer to https: / / blog.csdn.net / weixin_42419898 / article / details / 107187494). ResNet50 contains 50 layers of depth and has a strong representation ability to capture rich semantic information in images. Deep networks are usually able to learn more abstract and complex feature representations, thereby improving the performance of the model. In addition, ResNet50 introduces residual connections to make the network easier to train, accelerate the convergence of the model, and reduce the risk of overfitting. The expression of residual connection is as follows:

[0058] y=x+Conv2[σ(Conv1(x))]

[0059] Among them, Conv represents convolution, σ represents the nonlinear function ReLU, x represents input, and y represents output. The detailed structure is as follows Figure 2 As shown, Figure 2 The Weight layer in it means the weight layer.

[0060] As shown in Table 1, ResNet50 in the present invention includes layer1, layer2, layer3, layer4 and layer5 connected in sequence.

[0061] Table 1 Number of layers, parameters, inputs and outputs of the model backbone network

[0062]

[0063] 2. Multi-scale pooling strip convolution module (MPSCM)

[0064] For complex natural scenes and changeable target objects in semantic segmentation, traditional deep neural networks can only extract limited local areas through convolution operations, but cannot capture the contextual content of different areas, and the global information is not enough to be reflected. For this reason, relevant scholars proposed PSPNet, and the PSPNet model proposed a pyramid pooling module. The pyramid pooling module uses pooling operations to connect global features and local features, aggregates contextual information of different regions, and thus improves the performance of model segmentation. However, due to the larger square kernel, the traditional pooling operation extracts too much information from irrelevant areas, which will interfere with the final prediction of the model and reduce the accuracy of segmentation. For this reason, the present invention proposes a multi-scale pooling strip convolution module (MPSCM) here.

[0065] The multi-scale pooling strip convolution module MPSCM uses strip pooling to replace the square kernel pooling operation in the pyramid pooling module, which can reduce the impact of redundant information when capturing representative features, and more effectively distinguish small-scale polluted airflow and background areas, thereby reducing missed detection and false detection, and improving the final segmentation results. Adding strip convolutions of different scales can better enhance the features of the backbone contour of the polluted airflow and prevent feature loss. In addition, in order to further remove redundant information in the small feature map, a residual structure is used here to add the features output by the convolution kernels of different scales to obtain more fine-grained features.

[0066] like Figure 3 As shown in the figure, there are four branches and one residual connection in the multi-scale pooling strip convolution module MPSCM, which consists of a series of pooling layers and convolution layers for feature extraction at different scales.

[0067] The four branches include two strip average pooling branches, one global average pooling branch and one 1×1 convolution branch. The convolution kernel sizes of the strip average pooling branches are 1×N, N×1 and N×N (N=3, 5), respectively. After the input features are subjected to parallel strip pooling operations, they are added to enhance the weights of the central cross parts to obtain more refined features. After the strip average pooling operations and the global average pooling operations, the present invention uses the ReLU activation function, the BatchNorm layer and the 1×1 convolution to reduce the parameters of the model, and then restores the acquired feature map to the input size, and then connects it in parallel with the feature map after the 1×1 convolution. Finally, the present invention merges features of different scales, and then reduces the number of channels through 1×1 convolution and residual connection to obtain the refined feature map required for the next stage. The formula for the pooling process of the four branches is as follows:

[0068] y i =Avg1×N (x)+Avg N×1 (x)+Avg N×N (x), N = 3, 5

[0069] y i+1 =Up(Conv 1×1 (y i ))+Conv 1×1 (x)

[0070] Among them, x represents the input feature map, y i+1 Represents the feature map output by the multi-scale pooling strip convolution module MPSCM. Avg N×1 , Avg N×1 , Avg N×N Represents the average pooling layer of different convolution kernels, and N represents the size of the convolution kernel. 1×1 represents a two-dimensional convolution with a kernel size of 1, and Up represents an upsampling operation. The overall formula is as follows:

[0071]

[0072] F out =Conv 1×1 (y c )+Conv 1×1 (x)

[0073] Among them, x n Represents the input feature map of different branches, y c is the result of splicing each branch, F out Represents the feature map output by the multi-scale pooling strip convolution module MPSCM, n represents different branches from top to bottom, Global pool represents the global average pooling layer, C(·) represents the strip average pooling operation above, and {· represents the concatenation operation.

[0074] 3. Relational Attention Module (RAM)

[0075] Skip connections can improve the model's understanding of the target and segmentation accuracy by combining deep semantic information in the decoder with shallow detail information in the encoder. Shallow feature maps have high spatial resolution and can capture subtle structures and texture information in the image, but have weaker expression capabilities for global semantic information, while adjacent feature maps have richer and more advanced semantic information. By fusing these two types of information together, the accuracy of semantic information can be enhanced while retaining details, and the accuracy of spatial information can be improved, allowing the decoder to better utilize features. To this end, the present invention proposes a relational attention module RAM. The relational attention module RAM is mainly composed of two parts in parallel, including a self-attention module and a cross-attention module.

[0076] The self-attention mechanism establishes the relationship between positions in the shallow feature map, effectively captures the global dependency of the image, and improves the flexibility of the model in handling long-distance dependencies. Cross-attention promotes the interaction between different feature maps, enhances information transfer, and enhances the information transfer and communication between features. In the semantic segmentation task, features at different levels often contain semantic information of different scales and levels. By introducing the cross-attention mechanism, the model can better utilize the correlation between these features, thereby improving the accuracy and robustness of semantic segmentation.

[0077] The input of the self-attention module is the shallow low-level features, and the input of the cross-attention module is the adjacent deep high-level features. Through this parallel structure, the model can simultaneously capture the association information between different levels and the dependencies within the same level. Finally, the outputs of the self-attention module and the cross-attention module are added together to further integrate the feature information extracted by the cross-attention and self-attention, which can enhance the accuracy of semantic information while retaining details and save sufficient spatial information.

[0078] The input features of the self-attention module are Represents a shallow, low-level feature map, where H, W, and C represent height, width, and channel, respectively. After three 3×3 two-dimensional convolutions and reshape operations, the query vector (Q1), key vector (K1), and value vector (V1) are obtained, with a size of R C′×HW Then the query vector Q1 and the key vector K1 undergo a matrix multiplication operation to obtain a matrix of size R HW×HW The feature map is then normalized by a softmax operation and matrix multiplication is performed with the value vector V1 to obtain a feature map of size R C′×HW After a 1×1 convolution and reshape operation, the feature map is restored to R C×H×W Finally, through a residual connection, the mapping relationship between input and output is learned. The specific operation is as follows:

[0079] Q1=Re(Conv 3×3 (F1)

[0080] K1=Re(ConH 3×3 (F1)

[0081] V1=Re(Conv 3×3 (F1)

[0082] F′1=V1·σ(Tranpose(K1)·Q1)

[0083] F″1=Re(ConH 1×1(F′1))+F1

[0084] Among them, F1 represents the input, F″1 represents the output of the self-attention module, Re represents the reshape operation, Tranpose represents the transposition operation, and σ represents the Softmax activation function.

[0085] In the criss-cross attention module, the input features are and Represents deep, semantic feature maps. After two 3×3 convolutions and reshape operations, the key vector (K2) and value vector (V2) are obtained, and the query vector (Q2) is the feature After a 3×3 convolution and reshape operation, the size is R C ′×HW Then the query vector Q2 and the key vector K2 undergo a matrix multiplication operation to obtain a matrix of size R HW×HW The feature map of is obtained by performing a matrix multiplication operation with the value vector V2 through a Softmax operation to obtain a size of R C′×HW Then, after a 1×1 convolution and reshape operation, the feature map is restored to R C×H×W Size. Finally, after a residual connection, the features are learned and The mapping relationship between them. The specific operations are as follows:

[0086] Q2=Re(Conv 3×3 (F1)

[0087] K2=Re(Conv 3×3 (F2)

[0088] V2=Re(ConH 3×3 (F2)

[0089] F′2=V2·σ(Tranpose(K2)·Q2)

[0090] F″2=Re(ConH 1×1 (F′2))+F1+F2

[0091] Among them, F1 and F2 represent shallow low-level features and deep high-level features, F″2 represents the output of the cross-attention module, Re represents the reshape operation, Tranpose represents the transposition operation, and σ represents the Softmax activation function.

[0092] Finally, the two inputs F″1 and F″2 are concatenated and the final output feature map F is obtained through 1×1 convolution. The overall structure is as follows Figure 4 shown.

[0093] 4. Contextual Feature Fusion Module (CFFM)

[0094] In order to fuse low-level semantic information and high-level detail information, while retaining spatial information and details, the global perception ability of deep features can be utilized to improve the accuracy of model segmentation. The present invention proposes a context feature fusion module (CFFM) here, which includes a detail branch and a stream alignment branch. The detail branch is responsible for spatial details, fusing high-level semantic information with spatial position information in the original image to better restore the details and structure of the image. The stream alignment branch is responsible for aligning features at different levels, refining semantic features, further reducing position errors in the upsampling process, and reducing the probability of false detection and missed detection. The specific structure is as follows Figure 5 shown.

[0095] The flow alignment module aligns the feature maps of two adjacent levels by predicting the flow field (for relevant knowledge on predicting the flow field, please refer to the literature: Li, X., et al. Semantic Jlow Jor Jast and accurate scene parsing. in Computer Vision-ECCV2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I16.2020. Springer). The size of the input low-level feature map and high-level feature map is and First, the low-level feature map FD is passed through two 3×3 convolutions to obtain the feature map F′ D1 and F′ D2 . High-level feature map F F Through a 3×3 convolution, we get the feature map F′ F1 Then the feature map F F and F′ F1 Perform bilinear upsampling to make its resolution the same as the low-level feature map F′ D1 Equal, we get F′ F and F″ F1 In the process of predicting the boundary, the feature map F″ F1 and F′ D2 Spliced ​​together and convolved with a convolution kernel size of 3×3, a semantic flowfield is obtained with a size of Then F′ F Align (warp) with the semantic flow field to obtain the aligned feature map, the size is The formula is as follows:

[0096] Flow=Conv 3×3 [Conv 3×3 (F D ); Up(Conv 3×3 (F F ))]

[0097] F′1=Warp(Flow,Up(F F ))

[0098] Where Flow represents the semantic flow field, F D , F F They represent the low-level feature maps and high-level feature maps of the input, Up represents upsampling, Warp represents alignment operation, and Conv 3×3 represents a 3×3 convolution, and [·] represents a concatenation operation.

[0099] The detail branch is the high-level feature map F″ after upsampling F1 Activated by a Sigmoid, each pixel value in the feature map is mapped to the range of (0, 1) and then convolved with the low-level feature map F′ after 3×3 convolution. D1 Multiplying together can better maintain resolution while paying more attention to important features. The formula is as follows:

[0100] F=σ(Up(Conv 3×3 (F F )))·Conv 3×3 (F D )

[0101] Where σ represents the Sigmoid activation function, Conv 3×3 represents 3×3 convolution, F represents the input feature map, F D , F F They represent the low-level feature map and high-level feature map of the input respectively, and Up represents upsampling.

[0102] Finally, the features of the detail branch and the flow alignment branch are concatenated and passed through a 3×3 convolution to output the final feature map.

[0103] 5. Boundary Enhancement Module (BEM)

[0104] In the traditional plume boundary processing process, post-processing techniques such as conditional random fields (CRF) are required, and end-to-end training cannot be completed. To this end, the present invention designs a new boundary enhancement module (BEM) here, and the specific structure is as follows Figure 6As shown. After two 3×3 convolutions and a residual structure, the model can improve the expressiveness of the input features. Secondly, a Dropout layer is added to SegHead to discard neurons with a probability of 0.1, which can enhance the robustness of the model, improve the generalization ability of the model, improve the segmentation accuracy to a certain extent, obtain a prediction map with good segmentation effect, and prevent the network from overfitting. The formula is as follows:

[0105] y′=Conv 3×3 (Conv 3×3 (x))+x

[0106] y″=Up(Conv 1×1 (Drop(Conv 3×3 (y′))))

[0107] Among them, Conv 3×3 Represents 3×3 convolution, Conv 1×1 represents 1×1 convolution, Up represents upsampling, Drop represents Dropout layer, the drop rate is 0.1, and x and y″ are input and output respectively.

[0108] In this way, the inventors can obtain a method for identifying and tracking ground polluted airflow for VR / AR operation by implementing the present invention. It can effectively detect polluted gas in complex atmosphere and ground backgrounds, improve the control of details and spatial position, and determine the accurate target segmentation boundary. Finally, the predicted segmentation results are displayed to the operator through the VR / AR helmet to facilitate further operation.

Claims

1. A method for identifying and tracking ground polluted airflow, characterized in that: Construct a polluted airflow identification model; Collecting airflow pictures, inputting the airflow pictures into the polluted airflow identification model, and identifying and tracking the polluted airflow; The airflow recognition model includes a backbone network, a multi-scale pooling strip convolution module MPSCM, a relational attention network and a context feature fusion network; The backbone network, the multi-scale pooling strip convolution module MPSCM and the context feature fusion network are connected in sequence; The relational attention network includes a plurality of relational attention modules RAM, and the contextual feature fusion network includes a plurality of contextual feature fusion modules CFFM; The number of RAMs in the relation attention module is the same as that in the context feature fusion module CFFM, and they correspond one to one; The relational attention module RAM is used to fuse the output information of adjacent layers in the backbone network, and output the feature map to the corresponding context feature fusion module CFFM for fusion, and the context feature fusion network outputs the first feature map.

2. The method for identifying and tracking ground polluted airflow according to claim 1, characterized in that: The backbone network is ResNet50, including Layer1, Layer2, Layer3, Layer4 and Layer5 connected in sequence; The relationship attention module RAM has four, namely RAM1, RAM2, RAM3 and RAM4; RAM1 is used to fuse the feature maps output by Layer1 and Layer2; RAM2 is used to fuse the feature maps output by Layer2 and Layer3; RAM3 is used to fuse the feature maps output by Layer3 and Layer4; RAM4 is used to fuse the feature maps output by Layer4 and Layer5.

3. The method for identifying and tracking ground polluted airflow according to claim 2, characterized in that: There are four context feature fusion modules CFFM, namely CFFM1, CFFM2, CFFM3 and CFFM4 connected in sequence; The outputs of RAM1, RAM2, RAM3 and RAM4 correspond to an input feature map of CFFM1, CFFM2, CFFM3 and CFFM4 respectively.

4. The method for identifying and tracking ground polluted airflow according to claim 2, characterized in that: Each relational attention module RAM includes a self-attention module and a cross-attention module; The shallow feature map F1 is input into the self-attention module for feature extraction to obtain the feature map F1″: The adjacent deep high-level feature map F2 is upsampled and input into the cross attention module together with the shallow feature map F1 to capture the correlation information between different layers and the dependency relationship within the same layer, and obtain the feature map F2″; Finally, the feature map F1″ output by the self-attention module and the feature map F2″ output by the cross-attention module are added to obtain the output F of the relational attention module RAM.

5. The method for identifying and tracking ground polluted airflow according to claim 4, characterized in that: The shallow feature map F1 is input into the self-attention module for feature extraction. Specifically, the shallow feature map F1 undergoes three 3×3 two-dimensional convolutions and reshape operations to obtain three features, namely, the query vector Q1, the key vector K1, and the value vector V1. Then, the query vector Q1 and the key vector K1 undergo a matrix multiplication operation to obtain a feature map, which is then normalized by a softmax operation and then matrix multiplied with the value vector V1 to obtain the feature map F1. ′ , feature map F1 ′ After a 1×1 convolution and reshape operation, it is restored to the same size as the shallow feature map F1, and finally through a residual connection to obtain the output feature map F1″ of the self-attention module.

6. The method for identifying and tracking ground polluted airflow according to claim 4, characterized in that: The adjacent deep high-level feature map F2 is upsampled and input into the cross attention module together with the shallow feature map F1 to capture the correlation information between different levels and the dependency relationship within the same level, specifically: The high-level feature map F2 is input to the cross-attention module and undergoes two 3×3 convolutions and reshape operations to obtain the key vector K2 and the value vector V2. The primary feature map F1 is input to the cross-attention module and undergoes a 3×3 convolution and reshape operation to obtain the query vector Q2. Then the query vector Q2 and the key vector K2 undergo a matrix multiplication operation, and then a Softmax operation and matrix multiplication with the value vector V2 to obtain the feature map F2. ′ , after a 1×1 convolution and reshape operation, it is restored to the same size as the high-level feature map F2, and finally through a residual connection to obtain the output feature map F2″ of the cross attention module.

7. The method for identifying and tracking ground polluted airflow according to claim 3, characterized in that: The context feature fusion module CFFM includes a detail branch and a stream alignment branch; The input of the context feature fusion module CFFM includes the low-level feature map F D and high-level feature map F F ; Among them, the low-level feature map F D is the output feature map of the corresponding relational attention module RAM; the high-level feature map F F It is the feature map output by the previous module of the context feature fusion module CFFM; For the flow alignment branch: the low-level feature map F D Through two 3×3 convolutions, we get the feature map F ′ D1 and F ′ D2 ; High-level feature map F F Through a 3×3 convolution, we get the feature map F ′ F1 ; Then, the feature map F F and F ′ F1 Perform bilinear upsampling to make its resolution the same as the low-level feature map F ′ D1 Equal, get F ′ F and F ′ F ′ 1; The feature map F ′ F ′ 1 and F ′ D2 Splice them together and pass the convolution kernel size 3×3 to obtain a semantic flow field; ′ F Align (warp) with the semantic flow field to obtain the aligned feature map; For the detail branch: the upsampled high-level feature map F ′ F ′ 1 is activated by Sigmoid, and each pixel value in the feature map is mapped to the range of (0,1), and then combined with the low-level feature map F after 3×3 convolution. ′ D1 The obtained feature map is then concatenated with the aligned feature map, and then subjected to a 3×3 convolution to output the first feature map.

8. The method for identifying and tracking ground polluted airflow according to claim 3, characterized in that: The multi-scale pooling strip convolution module MPSCM includes four branches and one residual connection; The four branches include a first strip average pooling branch, a second strip average pooling branch, a global average pooling branch and a 1×1 convolution branch; After the input features undergo strip average pooling and 1×1 convolution in the first strip average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the first strip average pooling branch; After the input features undergo strip average pooling and 1×1 convolution in the second strip average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the second strip average pooling branch; After the input features undergo global average pooling and 1×1 convolution in the global average pooling branch, they are upsampled to restore to the input feature size, and then connected in parallel with the feature map obtained by the 1×1 convolution branch to obtain the output feature map of the global average pooling branch; The output feature maps of the first strip average pooling branch, the second strip average pooling branch, the global average pooling branch and the 1×1 convolution branch are merged, and then connected through 1×1 convolution and residual to obtain the feature map output by the multi-scale pooling strip convolution module MPSCM.

9. The method for identifying and tracking ground polluted airflow according to claim 8, characterized in that: The convolution kernel sizes of the strip average pooling branches are 1×N, N×1, and N×N respectively; N=3 in the first strip average pooling branch and N=5 in the second strip average pooling branch.

10. The method for identifying and tracking ground polluted airflow according to claim 8, characterized in that: It also includes Boundary Enhancement Module (BEM); The feature map output by the context feature fusion network is then input into the boundary enhancement module BEM for boundary enhancement; After the feature map is input into the boundary enhancement module BEM, it passes through two 3×3 convolutions and a residual structure, and then passes through the SegHead structure to obtain the final feature map.

Images