Gas identification method based on multi-source information fusion and environmental perception

Abstract

The invention discloses a gas recognition method based on multi-source information fusion and environmental perception, and the method comprises the steps: constructing a deep feature learning framework of multi-source fusion through combining the spatial response features, time sequence features and external environmental information of gas; the method comprises the following specific steps: preprocessing collected gas data, and respectively extracting features of an image mode, a sequence mode and an environment mode; fusing the image features and the sequence features through a cross attention fusion module, and capturing the space-time correlation of the data; a cross-modal attention compensation module is introduced, so that main-modal gas data adaptively gathers key information in an auxiliary-modal environment, and effective compensation of environmental factors on gas recognition performance is realized; and finally, gas category prediction is performed through a classification decision head. According to the method, the problems that the detection is easily interfered by environmental factors, the stability is poor or the qualification is inaccurate due to the fact that modeling depends on single modal data in the existing gas identification technology are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of gas identification, and particularly relates to a gas identification method based on multi-source information fusion and environment perception. BACKGROUND

[0002] Embodied intelligence is to embed artificial intelligence technology into physical entities such as robots, so that they have the ability to perceive, learn and dynamically interact with the environment. Among various sensing technologies, the electronic nose is a sensor system that simulates human olfactory function, which can capture the response characteristics of gas molecules through a sensor array, and combine pattern recognition algorithms to classify and analyze gas samples. It has been widely used in many key scenarios in intelligent sensing systems, such as food safety detection, environmental pollutant monitoring, etc.

[0003] Early gas identification methods usually extract steady-state and transient responses from gas response curves as features, and then use machine learning algorithms such as K-nearest neighbor (KNN) or support vector machine (SVM) for modeling and classification. These methods require manual feature extraction and are relatively cumbersome. Current mainstream gas identification methods are mostly based on deep learning technology. Some research converts gas sensor response curves into grayscale images and uses convolutional neural networks (CNN) to extract spatial features. Some methods use long short-term memory networks (LSTM) to capture dynamic change features of gas sequence data. These methods can automatically extract deep features from gas data, thereby improving the accuracy of gas identification. However, they still have certain limitations. Existing models only start from the image or sequence perspective, only focus on single modal features and ignore the coupling information between the two types of data, which limits the feature expression ability of the model. At the same time, most models do not consider environmental factors, and sensors are easily disturbed by external conditions such as temperature and humidity. Lack of environmental compensation mechanism will lead to insufficient stability of the model. SUMMARY

[0004] The application provides a gas identification method based on multi-source information fusion and environment perception to solve the problems of the prior art.

[0005] To achieve the above purpose, the application provides a gas identification method based on multi-source information fusion and environment perception, comprising the following steps:

[0006] Obtain multi-modal data and preprocess to obtain image data, sequence data and environment data;

[0007] Extract feature representations of the image data, sequence data and environment data through a multi-modal feature extraction encoder to obtain spatial features, time sequence features and environment features;

[0008] Fuse the spatial features and time sequence features through a cross-attention fusion module to generate spatio-temporal fusion features of the gas.

[0009] The spatio-temporal fusion feature is adaptively fused with the environment feature through a cross-modal attention compensation module to obtain a compensated gas feature;

[0010] According to the compensated gas feature, a gas category is predicted.

[0011] Optionally, the preprocessing of the multi-modal data comprises:

[0012] The original image data is smoothed using a Savitzky-Golay filtering method, the smoothed data is down-sampled, and is reshaped, the reshaped image is filled into a preset grid to form multi-channel image data;

[0013] The original sequence data is subjected to sensor noise elimination using a Savitzky-Golay method, and the sampling frequency is reduced to obtain sequence data of sensor measurement values.

[0014] Optionally, the obtaining of the spatial feature, the time sequence feature and the environment feature comprises:

[0015] The multi-modal feature extraction encoder comprises an image encoder, a sequence encoder and an environment encoder;

[0016] The image encoder extracts the spatial feature of the image data through a convolutional neural network;

[0017] The sequence encoder extracts the time sequence feature of the sequence data through a Transformer encoder;

[0018] The environment encoder extracts the environment feature of the environment data through a multi-layer perception.

[0019] Optionally, the extraction of the spatial feature of the image data comprises:

[0020] The image data is input into a basic convolutional block for preliminary feature extraction to obtain a basic feature;

[0021] The preliminary feature is subjected to hierarchical spatial feature extraction through a residual block to obtain the spatial feature, wherein each residual block comprises two stacked convolutional sub-modules.

[0022] Optionally, the extraction of the time sequence feature of the sequence data comprises:

[0023] The sequence data is mapped to a high-dimensional feature space through an embedding layer and position encoding is added;

[0024] Information between time steps is globally modeled through a multi-head self-attention mechanism to obtain sequence features with time sequence context association.

[0025] Optionally, the spatio-temporal fusion feature of the gas includes:

[0026] The spatial feature is projected into a query vector, the temporal feature is projected into a key vector and a value vector, an attention score is calculated through an attention mechanism, and outputs of all attention heads are spliced along a channel dimension to obtain a fused attention representation;

[0027] The fused attention feature is connected with the original spatial feature through a residual connection to obtain a final spatial fusion feature;

[0028] The temporal feature is mapped into a query vector, the spatial feature is mapped into a key vector and a value vector, a complementary attention feature is generated through an attention mechanism, and the complementary attention feature and the original temporal feature are connected and fused through a residual connection to obtain a temporal fusion feature;

[0029] The temporal fusion feature is linearly mapped to the same dimension as the spatial fusion feature, and the mapped features are finally fused to obtain the spatio-temporal fusion feature of the gas.

[0030] Optionally, the compensated gas feature includes:

[0031] The spatio-temporal fusion feature is projected into a query vector, and the environmental feature is projected into a key vector and a value vector;

[0032] An attention score of the query vector and the key vector is calculated through a cross-modal attention mechanism, and is applied to the value vector to generate a weighted environmental feature;

[0033] The weighted environmental features of the attention heads are spliced to obtain a compensation feature;

[0034] The compensation feature and the spatio-temporal fusion feature are spliced to obtain the compensated gas feature.

[0035] Optionally, the gas category prediction includes: inputting the compensated gas feature into a fully connected layer to output a prediction result of the gas category.

[0036] The application also provides a computer device including a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to realize the steps of the method.

[0037] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to realize the steps of the method.

[0038] Compared with the prior art, the application has the following advantages and technical effects:

[0039] The present application aims at the problems of detection susceptible to environmental factors, poor stability or inaccurate qualitative caused by the existing gas recognition technology relying on single modal data modeling, and proposes a gas recognition method based on multi-source information fusion and environmental perception. The method constructs a multi-source fusion deep feature learning framework by combining the spatial response characteristics, time series characteristics and external environmental information of the gas, significantly improving the accuracy and stability of gas recognition. Specifically, the present application extracts the features of image modal, sequence modal and environment modal through image encoder, sequence encoder and environment encoder respectively, and then uses the cross attention fusion module to fuse the image features and sequence features, capturing the spatio-temporal correlation of the data. At the same time, the cross-modal attention compensation module is introduced to make the main modal gas data adaptively aggregate the key information in the auxiliary modal environment, effectively compensating the interference of environmental factors on the performance of gas recognition. Through multi-source information fusion and environmental perception technology, the present application also enhances the spatio-temporal joint expression ability of the model to gas features, improves the generalization ability and adaptability of the gas recognition system, and can better adapt to different application scenarios and environmental conditions, providing an efficient and reliable technical path and solution for gas recognition tasks in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their

[0041] Figure 1 It is a whole architecture diagram of the multi-modal feature fusion network of the embodiment of the present application;

[0042] Figure 2 It is a multi-source data preprocessing result diagram of the embodiment of the present application, wherein (1) is an image data preprocessing result diagram, (2) is a sequence data preprocessing result diagram, and (3) is an environment data preprocessing result diagram;

[0043] Figure 3 It is a multi-modal feature extraction encoder structure schematic diagram of the embodiment of the present application, wherein (1) is a structure diagram of an image encoder, (2) is a structure diagram of a sequence encoder, and (3) is a structure diagram of an environment encoder;

[0044] Figure 4 It is a cross attention feature fusion module architecture diagram of the embodiment of the present application;

[0045] Figure 5 It is a cross-modal attention compensation module architecture diagram of the embodiment of the present application;

[0046] Figure 6Figures of t-SNE feature visualization results of different input modalities combinations of embodiments of the present application, wherein (1) is a sequence modality feature visualization figure, (2) is an image modality feature visualization figure, (3) is a sequence and image fusion feature visualization figure, and (4) is a multi-modal feature visualization figure of sequence, image and environment fusion.

[0047] Figure 7 A confusion matrix figure of the multi-modal feature fusion network of embodiments of the present application in a gas recognition task. DETAILED DESCRIPTION

[0048] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0049] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0050] Embodiment one

[0051] A gas recognition method based on multi-source information fusion and environment perception is provided in the present embodiment, comprising the following steps:

[0052] Obtain multi-modal data and pre-process to obtain image data, sequence data and environment data;

[0053] Extract feature representations of the image data, sequence data and environment data respectively through a multi-modal feature extraction encoder to obtain spatial features, time series features and environment features;

[0054] Fuse the spatial features and time series features through a cross-attention fusion module to generate spatio-temporal fusion features of the gas;

[0055] Adaptively fuse the spatio-temporal fusion features and the environment features through a cross-modal attention compensation module to obtain compensated gas features;

[0056] According to the compensated gas features, perform gas category prediction.

[0057] In this embodiment, multi-modal data is obtained: the experimental data set comes from the UCI machine learning library, which collects 18000 time series gas instances in a wind tunnel environment, including ten high-priority chemical gases. The sensor array of the gas detection platform is composed of nine identical modules, each module has eight different metal oxide sensors, a total of 72 gas sensors. Each gas sample is recorded for 260 seconds at a sampling frequency of 100 Hz. During the experiment, environmental factors such as exhaust fan wind speed (0.1 m / s, 0.21 m / s, 0.34 m / s), sampling position (L1-L6), and environmental temperature and humidity are set. These information is evenly distributed in all gas samples, providing data support for subsequent multi-modal feature modeling and implementation of environmental compensation strategy.

[0058] Further, the pre-processing of the multi-modal data includes:

[0059] The original image data is smoothed using the Savitzky-Golay filtering method, the smoothed data is down-sampled, and is reshaped, and the reshaped image is filled into a preset grid to form multi-channel image data.

[0060] The original sequence data is processed using the Savitzky-Golay method to eliminate sensor noise and reduce the sampling frequency, and the sequence data of the sensor measurement values is obtained.

[0061] In this embodiment, specifically:

[0062] In order to obtain multi-modal input features, the original data is processed through a series of preprocessing steps. For image data, the original data is smoothed using the Savitzky-Golay filtering method, and then the processed data is down-sampled to 256 dimensions, and the 256-dimensional data of each sensor is reshaped into a (16, 16) image. There are 8 different sensors in a module, which produces 8 (16, 16) images, and then these 8 images are filled into a 3x3 grid, and the 9th grid is filled with 0. 9 modules form 9 channels of data, each channel is a (48, 48) image. For sequence data, the Savitzky-Golay method is used to eliminate sensor noise, and for each sensor, the sampling frequency is reduced to 1 Hz, and the down-sampling matrix of the original data becomes 260x72. This data can be regarded as 72 sensor measurements with 260 time steps. For environmental data, the exhaust fan wind speed, sampling position, temperature and humidity information is extracted from the sample, and these information is arranged into a 1x4 one-dimensional environmental feature. The processing results are shown in Figure 2 (1) is the image data preprocessing result image, (2) is the sequence data preprocessing result image, and (3) is the environmental data preprocessing result image.

[0063] In this embodiment, the overall architecture of the multi-modal feature fusion network is as shown in Figure 1 The network is composed of three key parts: (1) a multi-modal backbone network for spatial feature, temporal feature and environmental feature extraction (2) a neck network for feature fusion and environmental compensation (3) a classification decision head for gas identification. First, ResNet is used for deep feature extraction of image representation of gas data, while Transformer encoder is used to model the temporal representation. MLP structure is used to process environmental data to obtain global environmental information. Then, cross-attention fusion module is used for fusion of gas spatio-temporal features, which fully represents gas data from the spatio-temporal comprehensive perspective. Next, cross-modal attention compensation module is introduced to enable environmental information to effectively compensate gas features. Finally, the fused high-dimensional features are input into the fully connected layer for gas category prediction. The structure of the cross-modal attention compensation module is as shown in Figure 5 .

[0064] Further, obtaining spatial features, temporal features and environmental features includes:

[0065] The multi-modal feature extraction encoder includes an image encoder, a sequence encoder and an environment encoder;

[0066] The image encoder extracts spatial features of image data through a convolutional neural network;

[0067] The sequence encoder extracts temporal features of sequence data through a Transformer encoder;

[0068] The environment encoder extracts environmental features of environmental data through a multi-layer perceptron.

[0069] Wherein, the spatial features of the image data include:

[0070] The image data is input into a basic convolutional block for preliminary feature extraction to obtain basic features;

[0071] The preliminary features are extracted by a residual block to obtain spatial features, wherein each residual block includes two stacked convolutional sub-modules.

[0072] Wherein, the temporal features of the sequence data include:

[0073] The sequence data is mapped to a high-dimensional feature space through an embedding layer and position encoding is added;

[0074] The information between time steps is globally modeled through a multi-head self-attention mechanism to obtain sequence features with temporal context association.

[0075] Specifically, in order to extract semantic features from the pre-processed multi-modal data, a specific modal encoder is designed to extract preliminary features of the image modal, sequence modal and environment modal respectively. The structural diagram of the encoder is shown in Figure 3 ; wherein (1) is the structural diagram of the image encoder, (2) is the structural diagram of the sequence encoder, and (3) is the structural diagram of the environment encoder. Let I, S and E represent the input data of the three modalities respectively, and the output feature vectors of the encoder are denoted as I g , S g and E g respectively. In the encoder, Conv represents convolution operation, BN represents batch normalization, and ReLU represents activation function.

[0076] For the image data of the gas , spatial features are extracted by convolution operation. The image input size is 9x48x48, which is first subjected to a basic convolution block for preliminary feature extraction, and the convolution block is composed of a 3x3 convolution layer, a BatchNorm layer and a ReLU activation function. The calculation process is as follows:

[0077] X c = ReLU(BN(Conv1(I))) (1)

[0078] The feature X c is input into two residual blocks in turn, and each residual block is composed of two stacked convolution sub-modules. Each convolution sub-module can be represented as:

[0079] C i,j (x) = ReLU(BN i,j (Conv i,j (x c ))) (2)

[0080] wherein C i,j represents the jth convolution sub-module in the ith residual block, i∈{1,2} represents the number of residual blocks, and j∈{1,2} represents the jth convolution operation in the block. In the ith residual block, the input feature z i-1 is subjected to feature transformation of two sub-modules in turn, and is added to the input through residual connection to obtain the output feature z i , and this process is represented as:

[0081] z i = C i,2 (C i,1 (z i-1 ))+z i-1 (3)

[0082] Let z0=x cThe input of the first residual block is the feature extracted by the base convolution. Finally, the output z2 of the second residual block is the final spatial feature I of the image encoder g .

[0083] For the sequence data of gas The 260x72 sequence matrix is input into the Transformer encoder. First, it is mapped to a high-dimensional feature space through the embedding layer and position encoding is added to enhance the model's perception of sequence order. Then, the multi-head self-attention mechanism is used to model the information between time steps globally, and finally the sequence feature S with temporal context association is output g Since the Transformer itself does not have position information of sequence elements, position encoding based on different frequencies of sine and cosine functions is used to explicitly model the position of each time step. The encoding method is as follows:

[0084]

[0085] where pos∈[0,T-1] and k∈[0,...,(D / 2)]. T represents the time step length, and D represents the number of sensors.

[0086] For environmental data Nonlinear mapping is performed through two layers of fully connected neural networks (MLP). The entire encoding process can be represented as:

[0087] E1=ReLU(BN(FC1(E))) (6)

[0088] E g =BN(FC2(Dropout(E1))) (7)

[0089] where FC represents fully connected mapping, and Dropout is a regularization mechanism. The final obtained environmental modal feature is denoted as E g .

[0090] Further, the spatio-temporal fusion feature of gas includes:

[0091] The spatial feature is projected into the query vector, and the temporal feature is projected into the key vector and value vector. The attention score is calculated through the attention mechanism, and the outputs of all attention heads are concatenated along the channel dimension to obtain the fused attention representation;

[0092] The fused attention feature and the original spatial feature are connected through the residual connection to obtain the final spatial fusion feature;

[0093] The time sequence feature is mapped to a query vector, the space feature is mapped to a key vector and a value vector, a complementary attention feature is generated through an attention mechanism, the complementary attention feature and the original time sequence feature are connected and fused through a residual connection to obtain a time sequence fusion feature;

[0094] The time sequence fusion feature is linearly mapped to the same dimension as the space fusion feature, and the mapped features are finally fused to obtain a space-time fusion feature of the gas.

[0095] In this embodiment, in order to realize efficient fusion and perception between the two representations of gas data, a cross-attention feature fusion module is designed based on an attention mechanism, and the structure thereof is as shown in Figure 4 The module introduces a bidirectional attention interaction process, respectively takes the space feature and the time sequence feature as the dominant, performs two rounds of complementary feature guidance, thereby realizes information guidance and deep matching between different data, and finally forms a more discriminative gas space-time fusion feature F g .

[0096] The space feature I g extracted by the image encoder is projected into a query vector Q I , and the time sequence feature S g extracted by the sequence encoder is projected into a key vector K S and a value vector V S , and the projection process is as follows:

[0097]

[0098] wherein and are learnable linear projection matrices. Then, an attention mechanism is used to calculate an attention score, and the formula is as follows:

[0099]

[0100] wherein represents the output of the i-th attention head, d k is the dimension of the key vector. Different attention scores reflect the modeling ability of the model in multiple subspaces for the heterogeneous association and semantic alignment between the image and the sequence. The outputs of all attention heads are spliced along the channel dimension to obtain a fused attention representation:

[0101]

[0102] wherein h represents the number of attention heads. Finally, the fused attention feature F1 and the original space feature I g are connected through a residual connection to form a final space fusion feature F , and the formula is as follows:

[0103]

[0104] The above is a one-way attention interaction process. In order to realize the feature-guided bidirectional complement, the time sequence feature is further taken as the leading factor to guide the spatial feature to complete deep fusion. The time sequence feature S g is mapped into a query vector Q S , the spatial feature I g is mapped into a key vector K I and a value vector V I , and the attention feature F2 is obtained by using the above method. The F2 and the original sequence feature S g are connected and fused by using a residual connection, and the time sequence fusion feature F is obtained. g .

[0105]

[0106] Further, the compensated gas feature includes:

[0107] The time-space fusion feature is projected into a query vector, and the environment feature is projected into a key vector and a value vector.

[0108] The attention score of the query vector and the key vector is calculated by the cross-modal attention mechanism, and is applied to the value vector to generate a weighted environment feature.

[0109] The weighted environment features of each attention head are spliced to obtain a compensation feature.

[0110] The compensation feature and the time-space fusion feature are spliced to obtain the compensated gas feature.

[0111] In the embodiment, the core idea of the cross-modal attention is to use the data (Query) of one modality to focus on the data (Key-Value) of another modality, and to adaptively fuse the features through the attention score. In the module, the cross-modal attention calculation process is constructed, so that the gas main modality actively perceives the key semantic information in the environment auxiliary modality, thereby enhancing the adaptive ability of the model to the environment change. Under the multi-head attention mechanism, the gas time-space fusion feature and the environment feature are respectively mapped into the query vector, the key and the value vector corresponding to the jth attention head, and the mapping process is as follows:

[0112]

[0113] wherein, Qj represents the query vector of the gas feature under the jth attention head, and are the key and value vectors of the environmental features, respectively. and is a learnable linear projection matrix, and j represents the number of attention heads. The cross-modal attention score matrix is used to represent the feature correlation between the two modalities, and the cross-modal attention score of the jth head can be defined as:

[0114]

[0115] wherein, represents the compensated feature output by the jth attention head, and the final compensated feature Y is obtained by concatenating the output results of all attention heads in the channel dimension. g .

[0116]

[0117] Further, the gas category prediction includes: inputting the compensated gas feature into a fully connected layer to output the prediction result of the gas category.

[0118] The present embodiment also performs the following verification:

[0119] To verify the effectiveness of the proposed model, all experiments are performed on an NVIDIA RTX 3090 GPU, and the network training and evaluation are realized by using the PyTorch deep learning framework. In the backbone network, the image branch uses a Resnet model containing two residual blocks to extract hierarchical spatial features of the gas image, and the sequence branch uses a Transformer encoder, and the number of attention heads is set to 8. In the cross and cross-modal attention module, 8-head attention mechanism is also used to enhance the information interaction between modalities. For the environmental information processing module, a multi-layer perceptron (MLP) structure is used, and Dropout regularization is introduced to suppress overfitting, and the dropout rate is set to 0.2. During the training process, the model uses the cross-entropy loss function for optimization, and the batch size of training and testing is set to 32. The optimizer is AdamW, the total number of training rounds is 100 rounds, the initial learning rate is set to 0.01, and the cosine annealing strategy is used to gradually decay the learning rate, and the learning rate is reduced to 0.003 at the end of training.

[0120] Compared with the traditional single-modal gas sensing method, the fusion and compensation modules introduced in this paper aim to improve the representation ability of the model under complex gas distribution and environmental disturbance background. To systematically evaluate the independent and joint contributions of the two modules, multiple sub-models are constructed, and ablation experiments are carried out under four different input combinations and fusion paths, and the experimental results are shown in Table 1 (the optimal performance is indicated in bold).

[0121] Table 1

[0122]

[0123] In the G-1 group, three types of data were used as single input for training, respectively, to evaluate the respective data representation capabilities. Without any module involved, both image and sequence features showed high discriminative ability, with 96.13% and 94.20%, respectively, while the environment information was 58.20%, much lower than other data types, which indicated that relying solely on environmental data for gas classification was not good, and this information was suitable for participating in multi-modal fusion as auxiliary information. Further, in the G-2 group, the image and sequence features of the gas were fused by introducing the cross-attention fusion module, and the accuracy after fusion was improved to 97.18%, which was improved by 1.05% and 2.98% compared with any single representation, which verified the effectiveness of the module in the same source and heterogeneous feature extraction. In the subsequent G-3 group, the environment information was introduced through the cross-modal attention compensation module to realize cross-modal attention fusion, and the “image + environment” and “sequence + environment” configurations reached an accuracy of 97.85% and 96.56%, respectively, which was 1.72% and 2.36% higher than the respective single modal performance, indicating that the compensation module could effectively adjust the interference of environmental factors on sensor perception and help to enhance the model's attention to key gas features. In the final G-4 group, both the fusion and compensation modules were used, and the model fused multiple modal data of image, sequence and environment, and realized efficient information integration across representations and cross-modal through a two-level fusion mechanism. The complete model achieved the highest accuracy of 99.07% and the lowest standard deviation of ±0.48, which represented stronger stability and generalization ability. In order to more intuitively feel the influence of different modal inputs on the performance of the model, the t-SNE visualization tool was used to visualize the features of the image and sequence groups of G-1, the G-2 group, and the G-4 group, and the results are shown in Figure 6

[0124] ​To comprehensively verify the effectiveness of the proposed multi-source information fusion and environment perception method in the gas identification task, the classification performance is compared with multiple classical and advanced models under the unified dataset division, and the comparison results are shown in Table 2 (the optimal performance is indicated in bold). The results show that the learning strategies of different gas features have their own advantages, and the proposed multi-modal feature fusion network achieves the highest classification accuracy of 99.07% and F1 score of 99.25%, which is significantly better than the current mainstream models. Among them, compared with the better GasNet and SimResNet9, the accuracy is improved by 4.32% and 3.84% respectively, and the standard deviation of the method is the lowest, only ± 0.48, which shows that the proposed multi-source feature fusion strategy can effectively integrate the spatio-temporal relationship of gas image and sequence features, and the environment compensation mechanism helps the model to learn the correlation between gas and external environment. Figure 7 The confusion matrix of the multi-modal feature fusion network in the gas identification task is shown in Table 3. The model achieves 100% accuracy on 7 gas categories, and the probability that the predicted class is equal to the real class is greater than 99.20%.

[0125] Table 2

[0126]

[0127] The embodiment also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the method.

[0128] The embodiment also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the method.

[0129] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical range disclosed in the present application can be easily thought by those skilled in the art, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A gas identification method based on multi-source information fusion and environmental perception, characterized in that, Includes the following steps: Multimodal data is acquired and preprocessed to obtain image data, sequence data, and environmental data; The multimodal feature extraction encoder extracts feature representations from the image data, sequence data, and environmental data respectively, to obtain spatial features, temporal features, and environmental features; The spatial and temporal features are fused by the cross-attention fusion module to generate spatiotemporal fusion features of gas. The spatiotemporal fusion features and environmental features are adaptively fused by the cross-modal attention compensation module to obtain the compensated gas features; Based on the compensated gas characteristics, gas category prediction is performed.

2. The method according to claim 1, characterized in that, Preprocessing of multimodal data includes: The Savitzky-Golay filtering method is used to smooth the original image data. The smoothed data is then downsampled and reshaped. The reshaped image is then filled into a preset grid to form multi-channel image data. The Savitzky-Golay method is used to eliminate sensor noise from the original sequence data and reduce the sampling frequency to obtain sequence data of sensor measurements.

3. The method according to claim 1, characterized in that, The obtained spatial features, temporal features, and environmental features include: The multimodal feature extraction encoder includes an image encoder, a sequence encoder, and an environment encoder; The image encoder extracts spatial features from the image data using a convolutional neural network; The sequence encoder extracts the temporal features of the sequence data using a Transformer encoder; The environmental encoder extracts environmental features from the environmental data using a multilayer perceptron.

4. The method according to claim 1, characterized in that, Extracting the spatial features of the image data includes: Image data is input into basic convolutional blocks for preliminary feature extraction to obtain basic features; The spatial features are obtained by performing hierarchical spatial feature extraction on the preliminary features through residual blocks, wherein each residual block includes two stacked convolutional sub-modules.

5. The method according to claim 1, characterized in that, Extracting the temporal features of the sequence data includes: Sequence data is mapped to a high-dimensional feature space through an embedding layer, and positional encoding is added; By using a multi-head self-attention mechanism to globally model the information between each time step, sequence features with temporal contextual association are obtained.

6. The method according to claim 1, characterized in that, The spatiotemporal fusion characteristics of the generated gas include: Spatial features are projected onto the query vector, temporal features are projected onto the key vector and value vector, attention scores are calculated through an attention mechanism, and the outputs of all attention heads are concatenated along the channel dimension to obtain the fused attention representation; The fused attention features are connected to the original spatial features through residuals to obtain the final fused spatial features; Temporal features are mapped to query vectors, spatial features are mapped to key vectors and value vectors, complementary attention features are generated through an attention mechanism, and residual connections are used to connect and fuse the complementary attention features and the original temporal features to obtain temporal fusion features. The temporal fusion features are linearly mapped to the same dimension as the spatial fusion features, and the mapped features are then fused to obtain the spatiotemporal fusion features of the gas.

7. The method according to claim 1, characterized in that, The compensated gas characteristics include: The spatiotemporal fusion features are projected into query vectors, and the environmental features are projected into key vectors and value vectors; The attention scores of the query vector and the key vector are calculated using a cross-modal attention mechanism and applied to the value vector to generate weighted environmental features. The weighted environmental features of each attention point are spliced ​​together to obtain the compensated features; The compensated gas feature is obtained by splicing the compensated feature with the spatiotemporal fusion feature.

8. The method according to claim 1, characterized in that, The gas category prediction process includes: inputting the compensated gas features into a fully connected layer and outputting the gas category prediction result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-8.

Images