Gas detection method, system and terminal based on multi-modal data

Abstract

The invention discloses a gas detection method and system based on multi-modal data, and a terminal. The method comprises the following steps: collecting VOCs gas data and corresponding VOCs text data; performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values from the standardized VOCs gas data; text features are extracted according to the VOCs text data, multi-modal data fusion is carried out according to the gas feature values and the text features, and fusion feature vectors are generated; and constructing a gas recognition model according to the fusion feature vector, inputting the to-be-detected gas data into the gas recognition model, and outputting a detection result of the to-be-detected gas data. On the basis of multi-modal data fusion and deep learning technologies, by integrating sensor data and applying a data processing algorithm, high-precision recognition of a complex gas mixture is achieved, gas information easy to understand can be provided in real time, and higher practicability and application value are provided.

Description

Technical Field

[0001] The present invention relates to the field of gas identification technology, and in particular to a gas detection method, system, terminal and computer-readable storage medium based on multimodal data. Background Art

[0002] Gas detection is mainly used in the fields of toxic and harmful gas monitoring, food quality and freshness inspection, disease breath detection, etc. Volatile organic compounds (VOCs) are a class of organic compounds with a boiling point between 50°C and 250°C and are volatile at room temperature. Due to their high volatility and the toxicity of some gases, they pose potential hazards to the ecological environment and human health. Their monitoring and control issues have always been highly valued.

[0003] VOCs mainly come from industrial emissions (such as printing, coating, surface treatment, etc.), solvent use (such as paint, detergent, printing ink, etc.), chemical production (such as plastics, rubber, fertilizers, etc.), daily necessities (such as furniture, carpets, detergents, etc.), as well as automobile exhaust, building materials, etc. In addition to the wide range of sources, the composition of VOCs is also complex and diverse, among which the common ones are alcohols: methanol, ethanol, propanol; aldehydes: formaldehyde, acetaldehyde; ketones: acetone; hydrocarbons: methane, ethane, ethylene, propylene, benzene, toluene, etc.

[0004] VOCs come from various sources and are rich in variety. Their concentration and types in the environment fluctuate with time and location, which makes the detection of VOCs extremely complex. In real life, since the concentration of VOCs in the environment is usually low, the detection method needs to be highly sensitive. Currently, the VOCs detectors on the market can only measure some gases and can only measure a few fixed gases. Since VOCs are a mixture of multiple gases, the detector can only measure the total concentration of the mixed gas.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a gas detection method, system, terminal and computer-readable storage medium based on multimodal data, aiming to solve the technical problem that the gas detection method in the prior art is difficult to detect the VOCs gas of each component present in the mixed gas.

[0007] To achieve the above object, the present invention provides a gas detection method based on multimodal data, and the gas detection method based on multimodal data comprises the following steps:

[0008] Collect VOCs gas data and corresponding VOCs text data;

[0009] Performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data;

[0010] Extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature values ​​and the text features to generate a fused feature vector;

[0011] A gas identification model is constructed according to the fused feature vector, the gas data to be detected is input into the gas identification model, and the detection result of the gas data to be detected is output.

[0012] Optionally, in the gas detection method based on multimodal data, the step of collecting VOCs gas data and corresponding VOCs text data comprises:

[0013] Acquiring VOCs gas data in a target area collected by a gas sensor array, wherein the gas sensor array includes a metal oxide semiconductor sensor, an electrochemical sensor, and an optical sensor;

[0014] A gas tag of the VOCs gas data is extracted, and the gas tag is input into a search engine to collect VOCs text data corresponding to the VOCs gas data, wherein the VOCs text data includes chemical properties, physical properties and characteristic descriptions.

[0015] Optionally, in the gas detection method based on multimodal data, the steps of performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data include:

[0016] The VOCs gas data is denoised, the denoised VOCs gas data is normalized, and the standard deviation of the VOCs gas data is calculated:

[0017]

[0018] in, is the average value, x i is the i-th VOCs gas data, σ is the standard deviation, and n is the total number of VOCs gas data;

[0019] According to the Raida criterion, the standard deviation is VOCs gas data outside the range are screened out and standardized VOCs gas data are output;

[0020] Gas characteristic values ​​are extracted according to the standardized VOCs gas data, and the gas characteristic values ​​include gas concentration change rate, time series characteristics and fluctuation amplitude characteristic values.

[0021] Optionally, in the gas detection method based on multimodal data, the step of extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature value and the text feature to generate a fused feature vector comprises:

[0022] Calculate the word frequency and inverse document frequency of the VOCs text data:

[0023]

[0024] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D);

[0025] Among them, TF(t,d) is the word frequency, n t,d is the number of times word t appears in VOCs text data d, k is the total number of words in VOCs text data, n k,d is the number of times the kth word appears in VOCs text data d, IDF(t,D) is the inverse document frequency, N is the total number of VOCs text data, D is the set of VOCs text data, |{d∈D:t∈d} represents the number of documents of word t, and TF-IDF(t,d,D) represents the importance of word t;

[0026] Select words with importance higher than a preset threshold as text features of VOCs;

[0027] The text features are converted into word vectors, the gas feature values ​​and the word vectors are fused to generate fused feature vectors, and the fused feature vectors are uniformly represented in a feature space.

[0028] Optionally, in the gas detection method based on multimodal data, the step of constructing a gas recognition model according to the fused feature vector comprises:

[0029] Build a deep learning model based on attention mechanism;

[0030] The fused feature vector is input into the deep learning model for training to obtain a gas recognition model.

[0031] Optionally, in the gas detection method based on multimodal data, the step of inputting the fused feature vector into the deep learning model for training to obtain a gas recognition model comprises:

[0032] Extracting time series data features according to the fused feature vector;

[0033] The deep learning model is trained according to the time series data features, and multiple recognition models are obtained through different model parameters;

[0034] The plurality of identification models are verified, and the best identification model is selected as the gas identification model.

[0035] In addition, to achieve the above-mentioned purpose, the present invention further provides a gas detection system based on multimodal data, wherein the gas detection system based on multimodal data comprises:

[0036] Data acquisition module, used to collect VOCs gas data and corresponding VOCs text data;

[0037] A data processing module, used for performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data;

[0038] A feature fusion module, used for extracting text features according to the VOCs text data, performing multimodal data fusion according to the gas feature value and the text feature, and generating a fused feature vector;

[0039] The gas identification module is used to construct a gas identification model according to the fused feature vector, input the gas data to be detected into the gas identification model, and output the detection result of the gas data to be detected.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a gas detection program based on multimodal data stored in the memory and executable on the processor, wherein the gas detection program based on multimodal data, when executed by the processor, implements the steps of the gas detection method based on multimodal data as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a gas detection program based on multimodal data, and when the gas detection program based on multimodal data is executed by a processor, the steps of the gas detection method based on multimodal data as described above are implemented.

[0042] In the present invention, VOCs gas data and corresponding VOCs text data are collected; data preprocessing is performed on the VOCs gas data, standardized VOCs gas data is output, and gas characteristic values ​​are extracted from the standardized VOCs gas data; text features are extracted according to the VOCs text data, and multimodal data fusion is performed according to the gas characteristic values ​​and the text features to generate a fused feature vector; a gas recognition model is constructed according to the fused feature vector, the gas data to be detected is input into the gas recognition model, and the detection result of the gas data to be detected is output. The present invention is based on multimodal data fusion and deep learning technology, which improves the detection accuracy of VOCs gas, and realizes high-precision recognition of complex gas mixtures by integrating sensor data and applying data processing algorithms, and can provide easy-to-understand gas information in real time, providing higher practicality and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a preferred embodiment of a gas detection method based on multimodal data of the present invention;

[0044] Figure 2 It is a schematic diagram of the principle of a preferred embodiment of a gas detection system based on multimodal data of the present invention;

[0045] Figure 3 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The gas detection method based on multimodal data described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the gas detection method based on multimodal data includes the following steps:

[0048] Step S10: Collect VOCs gas data and corresponding VOCs text data.

[0049] Specifically, in the process of multimodal data collection, the key first step is to select appropriate sensors to collect VOCs gas data. VOCs (volatile organic compounds) refer to organic compounds that can evaporate into gas under certain conditions, and usually have potential impacts on the environment and human health. The present invention mainly selects VOCs gas data and corresponding VOCs text data as components of multimodal data to detect VOCs gas. The collected VOCs gas data needs to be collected and stored in real time or periodically for subsequent data processing and analysis.

[0050] Furthermore, the step of collecting VOCs gas data and corresponding VOCs text data includes:

[0051] The VOCs gas data in the target area collected by the gas sensor array is obtained, and the gas sensor array includes a metal oxide semiconductor sensor, an electrochemical sensor and an optical sensor.

[0052] A gas tag of the VOCs gas data is extracted, and the gas tag is input into a search engine to collect VOCs text data corresponding to the VOCs gas data, wherein the VOCs text data includes chemical properties, physical properties and characteristic descriptions.

[0053] Specifically, in order to cover the detection needs of various VOCs, various types of sensors can be selected. Selecting an appropriate sensor array to monitor and collect VOCs gas in the target area is the key first step. The selection of sensors is determined according to the type of VOCs to be detected and the environmental conditions of the target area. As an example, the sensor types in the present invention may include metal oxide semiconductor sensors (MOS sensors, such as SnO2 sensors), which are used to detect general volatile organic compounds, such as formaldehyde, benzene, etc.; electrochemical sensors, which are suitable for detecting specific chemical components, such as methane, ethane, etc.; optical sensors, which can be used to detect certain specific VOCs, such as aromatic hydrocarbon compounds. The sensors are reasonably arranged in the target area to ensure that the VOCs gas data at different locations can be captured. In addition to collecting from the target area (polluted area, industrial emission source, etc.), VOCs gas data can also be obtained by injecting pre-collected VOCs gas, so that the VOCs data obtained are more accurate and convenient for processing and analysis.

[0054] Specifically, the data collection frequency and method can set an appropriate data collection frequency, such as once every minute or every hour, and use the data acquisition system to collect VOCs gas data output by the sensor in real time. For the collected VOCs gas data, each VOCs needs to be identified and classified so that it can be subsequently associated with the corresponding text data. The gas label can include unique identifiers such as chemical name, CAS number (Chemical Abstracts Service number, a digital identification number for chemical substances), and corresponding concentration information. The gas label is entered into a search engine, and literature databases in the fields of chemistry and environmental science are accessed to obtain VOCs text data corresponding to the collected VOCs gas data, and further obtain detailed chemical properties, physical properties and characteristic description information about these compounds, laying the foundation for subsequent data preprocessing, feature extraction and multimodal data fusion to build an efficient gas detection and analysis system.

[0055] Step S20: preprocess the VOCs gas data, output standardized VOCs gas data, and extract gas characteristic values ​​from the standardized VOCs gas data.

[0056] Specifically, for the raw VOCs gas data collected from the sensor, a sensor curve is drawn according to each VOCs gas data sample, the data distribution scale is analyzed and data preprocessing is performed to output standardized VOCs gas data, and gas characteristic values ​​are extracted from the standardized VOCs gas data. This process helps to improve data quality, with the aim of standardizing sensor data, unifying the data format, preparing for subsequent multimodal data fusion, and providing more accurate input for deep learning models.

[0057] Furthermore, the steps of performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data include:

[0058] The VOCs gas data is denoised, the denoised VOCs gas data is normalized, and the standard deviation of the VOCs gas data is calculated:

[0059]

[0060] in, is the average value, x i is the i-th VOCs gas data, σ is the standard deviation, and n is the total number of VOCs gas data.

[0061] According to the Raida criterion, the standard deviation is VOCs gas data outside the range are screened out and standardized VOCs gas data are output.

[0062] Gas characteristic values ​​are extracted according to the standardized VOCs gas data, and the gas characteristic values ​​include gas concentration change rate, time series characteristics and fluctuation amplitude characteristic values.

[0063] Specifically, in the data preprocessing stage, denoising is aimed at eliminating random noise and outliers in the VOCs gas data collected by the sensor to ensure the accuracy and stability of subsequent analysis. Different filtering techniques can be used, such as moving average filtering, median filtering or Gaussian filtering. As an example, for VOCs gas data in a continuous time series, moving average filtering technology can be used to calculate the average value within a certain window size around each time point to smooth the data and reduce the impact of noise. Normalization is to unify VOCs gas data of different scales into a standard data range for subsequent statistical analysis and feature extraction. As an example, VOCs gas data can be scaled to a specific range, such as [0,1] or [-1,1], and the data can be converted to a standard normal distribution with a mean of 0 and a standard deviation of 1. If the output range of a VOCs gas sensor is between 0 and 100, it can be converted to a value between 0 and 1 through linear normalization. When the data distribution is uneven or statistical inference is required, standardization can ensure the balance of the data. For example, adjusting the data to a standard normal distribution is conducive to subsequent time series analysis and model training. The standard deviation of VOCs gas data is calculated to measure the degree of data variation and use the Raida criterion to exclude outliers. In general, the range of the mean plus or minus 3 times the standard deviation is used as the boundary of normal data. Data points outside this range can be regarded as outliers and processed to further improve data quality and analysis accuracy. For example, if the mean of a VOCs gas data is 10 and the standard deviation is 2, according to the Raida criterion, the range of normal data should be (10-3×2,10+3×2)=(4,16). Data points outside this range can be considered abnormal and need further verification and processing or directly screened out as outliers to obtain standardized VOCs gas data.

[0064] Specifically, after the data preprocessing is completed, meaningful gas characteristic values are extracted from the standardized VOCs gas data for subsequent gas identification and analysis. As an example, the extracted gas characteristic values include the gas concentration change rate, time series characteristics, and fluctuation amplitude characteristic values. Among them, the gas concentration change rate is used to analyze the change rate of the VOCs gas concentration over time, reflecting the dynamic change of the pollutant concentration in the target area, which can help identify potential pollution sources or emission events; the time series characteristics are used to extract the statistical characteristics of time series data, such as mean, variance, maximum value, and minimum value, etc., revealing the distribution law and periodicity of the data. Extracting statistical characteristics such as the daily average value and weekly change amplitude of the VOCs gas data can be used to establish time-related models and predictions; the fluctuation amplitude characteristic value is used to calculate the fluctuation amplitude of the VOCs gas data over time, reflecting the fluctuation of the VOCs gas concentration in the environment. Calculate the concentration fluctuation of a certain VOC under different seasons or different weather conditions to evaluate the degree of its impact on environmental changes. By performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting meaningful gas characteristic values, it provides effective support for environmental monitoring and pollution control.

[0065] Step S30: Extract text features according to the VOCs text data, and perform multimodal data fusion based on the gas characteristic values and the text features to generate a fusion feature vector.

[0066] Specifically, before extracting text features, it is first necessary to preprocess the VOCs text data. For example, remove special symbols, stop words (such as "of", "is", etc.) and non-critical information in the text to reduce the impact of noise on feature extraction. Then split the text into words or phrases to form a corpus, and convert the words into their basic forms to reduce the impact of word form changes on feature extraction to ensure the consistency and usability of the data. As an example, the text description extracted from the VOCs text data is "Ethanol is a common organic solvent with a strong alcohol smell". After text preprocessing, the segmented text "[ethanol, common, organic solvent, strong, alcohol, smell]" can be obtained. Further extract the keywords to get the key text "[ethanol, organic solvent, alcohol, smell]", and convert the key text into a quantifiable and analyzable text feature, that is, a feature vector, so as to fuse with the gas characteristic values to obtain a fusion feature vector.

[0067] Further, the step of extracting text features according to the VOCs text data and performing multimodal data fusion based on the gas characteristic values and the text features to generate a fusion feature vector includes:

[0068] Calculate the word frequency and inverse document frequency of the VOCs text data:

[0069]

[0070] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D);

[0071] Among them, TF(t,d) is the word frequency, n t,d is the number of times word t appears in VOCs text data d, k is the total number of words in VOCs text data, n k,d is the number of times the kth word appears in the VOCs text data d, IDF(t,D) is the inverse document frequency, N is the total number of VOCs text data, D is the set of VOCs text data, |{d∈D:t∈d} represents the number of documents of word t, and TF-IDF(t,d,D) represents the importance of word t.

[0072] Select words with importance higher than a preset threshold as text features of VOCs, convert the text features into word vectors, fuse the gas feature values ​​and the word vectors to generate a fused feature vector, and uniformly represent the fused feature vector in a feature space.

[0073] Specifically, TF (Term Frequency) is the term frequency, which measures the frequency of a word appearing in the current document. The more times a word appears in a text, the more important it is in the text; IDF (Inverse Document Frequency) is the inverse document frequency, which is used to measure the general importance of a word. The lower the frequency of a word appearing in multiple documents, the more important it is in the text; TF-IDF (Term Frequency-Inverse Document Frequency) is the product of TF and IDF, which combines term frequency and inverse document frequency to measure the importance of a word in a document collection. According to the calculated TF-IDF value, words with an importance higher than the preset threshold can be selected as text features of VOCs. These words have significant information and representativeness when describing the characteristics of VOCs. The TF-IDF algorithm believes that if a word has a high term frequency in a document and a low document frequency in the entire document collection, then this word is likely to be the keyword of this document. The selected text features (words above a preset threshold) are converted into word vectors, usually using word embedding techniques such as Word2Vec, GloVe, etc., to map each word to a vector representation in a high-dimensional space. For example, the converted text features are ethanol: [0.234, -0.123, 0.456, ...], organic solvents: [0.567, 0.234, -0.789, ...], alcohol: [0.345, -0.567, 0.678, ...], odor: [0.123, 0.345, -0.456, ...], and the previously extracted gas feature values ​​(such as concentration change rate, time series features, fluctuation amplitude features, etc.) are fused with the obtained word vector to generate a fused feature vector. The fusion method can be to connect the two together. For example, the current VOCs sample data is ethanol: [0.234, -0.123, 0.456, ...], and the corresponding text data (such as the original text: wine aroma) is normalized (such as the mean is 0, the variance is 1) to obtain the word vector [0.1, 0.1, 0.1, ..., 0.1]. The fused feature vector obtained by fusing the two is [0.234, -0.123, 0.456, ..., 0.1, 0.1, 0.1, ..., 0.1]. Finally, the fused feature vector is represented in a unified feature space to ensure the consistency and comparability of the data for subsequent data analysis, machine learning model training or other data processing tasks.

[0074] Step S40: construct a gas identification model according to the fused feature vector, input the gas data to be detected into the gas identification model, and output the detection result of the gas data to be detected.

[0075] Specifically, before building a gas recognition model, it is necessary to first determine the machine learning or deep learning method to be used. Models based on deep learning, such as convolutional neural networks (CNN) or recurrent neural networks (RNN), can be used. When processing multimodal data, the attention mechanism and feature fusion technology used are used to extract and integrate feature information from different data sources (such as text and gas sensor data). More complex fusion technologies such as attention mechanisms or multi-layer perceptrons (MLP) can also be used to combine information from different sources.

[0076] Furthermore, the step of inputting the fused feature vector into the deep learning model for training to obtain a gas recognition model includes:

[0077] Build a deep learning model based on attention mechanism.

[0078] The fused feature vector is input into the deep learning model for training to obtain a gas recognition model.

[0079] The plurality of identification models are verified, and the best identification model is selected as the gas identification model.

[0080] Specifically, the attention mechanism can be used to strengthen the emphasis on different features, such as the degree of attention paid to the rate of change of gas concentration and the description of chemical properties, because it can effectively integrate text features and gas features, and improve the robustness and accuracy of the model. As an example, the present invention uses the Transformer model as a basis and combines the self-attention mechanism to process the fused feature vector. By adding a layer suitable for processing gas features on the basis of the Transformer model, a model suitable for gas recognition can be constructed. The fused feature vector and the extracted time series data features are used as inputs of the deep learning model. The model is trained multiple times to explore different model parameter and architecture combinations to obtain the best gas recognition performance. The training process aims to optimize the model parameters, and different recognition models are obtained through multiple parameter and architecture combinations. The gas recognition model with the best performance is screened and can learn and capture effective patterns from the input multimodal data for subsequent gas detection tasks. The generalization ability of the model is verified using cross-validation technology, for example, by dividing the data set into multiple training sets and validation sets, and training the model on different training sets to evaluate the performance of the model.

[0081] Furthermore, the step of inputting the gas data to be detected into the gas identification model and outputting the detection result of the gas data to be detected includes:

[0082] The gas data to be detected is acquired in real time, and the gas data to be detected is input into the gas identification model, and the gas identification model determines the gas type and gas concentration to generate a gas identification result.

[0083] If the gas identification result is VOCs gas, an alarm prompt is given and text features corresponding to the VOCs gas are output.

[0084] Specifically, the gas data in the environment are regularly collected through the gas sensor array. These data include the concentration values ​​of different types of VOCs gases. The gas sensors can also be customized to select the corresponding sampling frequency. For example, for gas sensors with faster response speeds, a higher sampling frequency can be selected to capture rapid changes in gas concentration and ensure the real-time and accuracy of the data.

[0085] For example, by deploying multiple sensors, including metal oxide semiconductor sensors and electrochemical sensors, they are able to measure the concentration data of VOCs such as benzene and toluene in the target area in real time, and transmit the data to the central processing unit for further processing and analysis.

[0086] Furthermore, the gas data to be detected is input into the gas identification model. The gas data to be detected is input into the pre-trained gas identification model, and the model can determine the gas type and concentration level in the input data through the above-mentioned learning and analysis process.

[0087] As an example, the real-time data of the gas to be detected is preprocessed similar to the training data to ensure the consistency of the data format and features, and the preprocessed data is input into the gas recognition model. The model uses feature extraction and pattern recognition technology to judge the type (such as benzene, toluene, etc.) and concentration of the gas. When receiving real-time data from the sensor, for example, if the concentration of benzene exceeds the preset threshold, the data will be sent to the pre-trained deep learning model. The model can quickly identify the gas type and concentration level and obtain the gas recognition result.

[0088] Furthermore, through the fusion of multimodal data, the gas identification results obtained include not only the gas type and gas concentration, but also the text features of the gas. When the gas identification model determines that the gas to be detected is VOCs, it will trigger an alarm prompt and output the corresponding text features of the VOCs gas. Relevant personnel can understand the characteristic information of the VOCs gas more intuitively, which facilitates the quick adoption of treatment measures.

[0089] Specifically, if the gas identification model confirms that the gas to be detected is VOCs, an alarm signal will be generated to notify relevant personnel or automatically trigger emergency measures, and text features related to the detected VOCs gas will be output. These features may include chemical properties, physical properties and other descriptive information, which can be output through a display or voice broadcast to help relevant personnel further analyze and deal with the problem.

[0090] For example, when the gas identification model determines that the detected gas is benzene, the system will immediately send an alarm to notify relevant personnel, and output benzene's chemical structure description, toxicity information and emergency treatment recommendations so that necessary safety measures and treatments can be taken. It can be effectively applied to various environments and application scenarios that require real-time monitoring and response to ensure the timeliness and reliability of gas detection.

[0091] In another embodiment, in addition to triggering an alarm by detecting VOCs gas, a microphone array can be used to collect sound signals in the environment. After similar data processing, the sound signal is converted into text information, and keywords are extracted therefrom. The keywords are converted into word vectors using the conversion method described above, and input into the recognition model for comparison and analysis to see whether the gas described in the text information is a harmful gas. For example, the text information in the environment collected by the microphone array is "gas with a rotten egg smell". At this time, the feature information of ['rotten egg smell', 'gas'] is obtained by extracting keywords, and after conversion into word vectors, it is input into the model for comparison or prediction. If the corresponding gas is compared or predicted to be VOCs gas, an alarm prompt is also triggered, and the information of the VOCs gas is transmitted to relevant personnel.

[0092] Furthermore, if Figure 2 As shown, based on the above-mentioned gas detection method based on multimodal data, the present invention also provides a gas detection system based on multimodal data, wherein the gas detection system based on multimodal data includes:

[0093] A data collection module 51 is used to collect VOCs gas data and corresponding VOCs text data;

[0094] A data processing module 52 is used to perform data preprocessing on the VOCs gas data, output standardized VOCs gas data, and extract gas characteristic values ​​from the standardized VOCs gas data;

[0095] A feature fusion module 53 is used to extract text features according to the VOCs text data, perform multimodal data fusion according to the gas feature value and the text feature, and generate a fused feature vector;

[0096] The gas identification module 54 is used to construct a gas identification model according to the fused feature vector, input the gas data to be detected into the gas identification model, and output the detection result of the gas data to be detected.

[0097] Furthermore, if Figure 3 As shown, based on the above-mentioned gas detection method and system based on multimodal data, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 3 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0098] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a gas detection program 40 based on multimodal data is stored on the memory 20, and the gas detection program 40 based on multimodal data can be executed by the processor 10, thereby realizing the gas detection method based on multimodal data in the present application.

[0099] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the gas detection method based on multimodal data.

[0100] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0101] In one embodiment, when the processor 10 executes the gas detection program 40 based on multimodal data in the memory 20, the following steps are implemented:

[0102] Collect VOCs gas data and corresponding VOCs text data;

[0103] Performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data;

[0104] Extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature values ​​and the text features to generate a fused feature vector;

[0105] A gas identification model is constructed according to the fused feature vector, the gas data to be detected is input into the gas identification model, and the detection result of the gas data to be detected is output.

[0106] The step of collecting VOCs gas data and corresponding VOCs text data includes:

[0107] Acquiring VOCs gas data in a target area collected by a gas sensor array, wherein the gas sensor array includes a metal oxide semiconductor sensor, an electrochemical sensor, and an optical sensor;

[0108] A gas tag of the VOCs gas data is extracted, and the gas tag is input into a search engine to collect VOCs text data corresponding to the VOCs gas data, wherein the VOCs text data includes chemical properties, physical properties and characteristic descriptions.

[0109] The steps of preprocessing the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data include:

[0110] The VOCs gas data is denoised, the denoised VOCs gas data is normalized, and the standard deviation of the VOCs gas data is calculated:

[0111]

[0112] in, is the average value, x i is the i-th VOCs gas data, σ is the standard deviation, and n is the total number of VOCs gas data;

[0113] According to the Raida criterion, the standard deviation is VOCs gas data outside the range are screened out and standardized VOCs gas data are output;

[0114] Gas characteristic values ​​are extracted according to the standardized VOCs gas data, and the gas characteristic values ​​include gas concentration change rate, time series characteristics and fluctuation amplitude characteristic values.

[0115] The step of extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature values ​​and the text features to generate a fused feature vector includes:

[0116] Calculate the word frequency and inverse document frequency of the VOCs text data:

[0117]

[0118] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D);

[0119] Among them, TF(t,d) is the word frequency, n t,d is the number of times word t appears in VOCs text data d, k is the total number of words in VOCs text data, n k,d is the number of times the kth word appears in VOCs text data d, IDF(t,D) is the inverse document frequency, N is the total number of VOCs text data, D is the set of VOCs text data, |{d∈D:t∈d} represents the number of documents of word t, and TF-IDF(t,d,D) represents the importance of word t;

[0120] Select words with importance higher than a preset threshold as text features of VOCs;

[0121] The text features are converted into word vectors, the gas feature values ​​and the word vectors are fused to generate fused feature vectors, and the fused feature vectors are uniformly represented in a feature space.

[0122] The step of inputting the fused feature vector into the deep learning model for training to obtain a gas recognition model includes:

[0123] Extracting time series data features according to the fused feature vector;

[0124] The deep learning model is trained according to the time series data features, and multiple recognition models are obtained through different model parameters;

[0125] The plurality of identification models are verified, and the best identification model is selected as the gas identification model.

[0126] The step of inputting the gas data to be detected into the gas identification model and outputting the detection result of the gas data to be detected comprises:

[0127] Acquire the gas data to be detected in real time, input the gas data to be detected into the gas identification model, the gas identification model determines the gas type and gas concentration, and generates a gas identification result;

[0128] If the gas identification result is VOCs gas, an alarm prompt is given and text features corresponding to the VOCs gas are output.

[0129] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a gas detection program based on multimodal data, and when the gas detection program based on multimodal data is executed by a processor, the steps of the gas detection method based on multimodal data as described above are implemented.

[0130] In summary, the present invention provides a gas detection method, system, terminal and computer-readable storage medium based on multimodal data, the method comprising: collecting VOCs gas data and corresponding VOCs text data; performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data; extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas characteristic values ​​and the text features to generate a fused feature vector; constructing a gas recognition model according to the fused feature vector, inputting the gas data to be detected into the gas recognition model, and outputting the detection result of the gas data to be detected. The present invention is based on multimodal data fusion and deep learning technology, which improves the detection accuracy of VOCs gas, realizes high-precision recognition of complex gas mixtures by integrating sensor data and applying data processing algorithms, and can provide easy-to-understand gas information in real time, providing higher practicality and application value.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0132] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0133] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A gas detection method based on multimodal data, characterized in that: The gas detection method based on multimodal data includes: Collect VOCs gas data and corresponding VOCs text data; Performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data; Extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature values ​​and the text features to generate a fused feature vector; A gas identification model is constructed according to the fused feature vector, the gas data to be detected is input into the gas identification model, and the detection result of the gas data to be detected is output.

2. The method according to claim 1, characterized in that The step of collecting VOCs gas data and corresponding VOCs text data includes: Acquiring VOCs gas data in a target area collected by a gas sensor array, wherein the gas sensor array includes a metal oxide semiconductor sensor, an electrochemical sensor, and an optical sensor; A gas tag of the VOCs gas data is extracted, and the gas tag is input into a search engine to collect VOCs text data corresponding to the VOCs gas data, wherein the VOCs text data includes chemical properties, physical properties and characteristic descriptions.

3. The method according to claim 1, characterized in that The steps of performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data include: The VOCs gas data is denoised, the denoised VOCs gas data is normalized, and the standard deviation of the VOCs gas data is calculated: in, is the average value, x i is the i-th VOCs gas data, σ is the standard deviation, and n is the total number of VOCs gas data; According to the Raida criterion, the standard deviation is VOCs gas data outside the range are screened out and standardized VOCs gas data are output; Gas characteristic values ​​are extracted according to the standardized VOCs gas data, and the gas characteristic values ​​include gas concentration change rate, time series characteristics and fluctuation amplitude characteristic values.

4. The method according to claim 1, characterized in that The step of extracting text features according to the VOCs text data, and performing multimodal data fusion according to the gas feature values ​​and the text features to generate a fused feature vector comprises: Calculate the word frequency and inverse document frequency of the VOCs text data: TF-IDF(t,d,D)=TF(t,d)×IDF(t,D); Among them, TF(t,d) is the word frequency, n t,d is the number of times word t appears in VOCs text data d, k is the total number of words in VOCs text data, n k,d is the number of times the kth word appears in VOCs text data d, IDF(t,D) is the inverse document frequency, N is the total number of VOCs text data, D is the set of VOCs text data, |{d∈D:t∈d} represents the number of documents of word t, and TF-IDF(t,d,D) represents the importance of word t; Select words with importance higher than a preset threshold as text features of VOCs; The text features are converted into word vectors, the gas feature values ​​and the word vectors are fused to generate fused feature vectors, and the fused feature vectors are uniformly represented in a feature space.

5. The method according to claim 1, characterized in that The step of constructing a gas recognition model according to the fused feature vector comprises: Build a deep learning model based on attention mechanism; The fused feature vector is input into the deep learning model for training to obtain a gas recognition model.

6. The method according to claim 5, characterized in that The step of inputting the fused feature vector into the deep learning model for training to obtain a gas recognition model comprises: Extracting time series data features according to the fused feature vector; The deep learning model is trained according to the time series data features, and multiple recognition models are obtained through different model parameters; The plurality of identification models are verified, and the best identification model is selected as the gas identification model.

7. The method according to claim 1, characterized in that The step of inputting the gas data to be detected into the gas identification model and outputting the detection result of the gas data to be detected comprises: Acquire the gas data to be detected in real time, input the gas data to be detected into the gas identification model, the gas identification model determines the gas type and gas concentration, and generates a gas identification result; If the gas identification result is VOCs gas, an alarm prompt is given and text features corresponding to the VOCs gas are output.

8. A gas detection system based on multimodal data, characterized in that: The gas detection system based on multimodal data includes: Data acquisition module, used to collect VOCs gas data and corresponding VOCs text data; A data processing module, used for performing data preprocessing on the VOCs gas data, outputting standardized VOCs gas data, and extracting gas characteristic values ​​from the standardized VOCs gas data; A feature fusion module, used for extracting text features according to the VOCs text data, performing multimodal data fusion according to the gas feature value and the text feature, and generating a fused feature vector; The gas identification module is used to construct a gas identification model according to the fused feature vector, input the gas data to be detected into the gas identification model, and output the detection result of the gas data to be detected.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a gas detection program based on multimodal data stored in the memory and executable on the processor. When the gas detection program based on multimodal data is executed by the processor, the steps of the gas detection method based on multimodal data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a gas detection program based on multimodal data, and when the gas detection program based on multimodal data is executed by a processor, the steps of the gas detection method based on multimodal data as described in any one of claims 1-7 are implemented.

Images