A multi-channel infrared gas detection system and method based on big data

Abstract

The present invention discloses a multi-channel infrared gas detection system and method based on big data, belonging to the field of big data technology. The present invention includes: a multi-channel infrared sensor module: adopting an infrared sensor array, including several independent channels, each channel is responsible for detecting an infrared spectrum of a specific wavelength; a data preprocessing module: preprocessing the original data, extracting features from the preprocessed data; normalizing the extracted features, and visually displaying the preprocessed data; a pattern recognition and classification module: using a support vector machine model, training through pre-labeled training data, and establishing a gas classification model; a real-time monitoring and alarm module: continuously monitoring the gas concentration data obtained from the data preprocessing module, and triggering an alarm mechanism when the gas concentration is monitored to exceed a set threshold; a data storage and management module: storing the processed data, implementing a distributed data backup strategy, and establishing a disaster recovery mechanism.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a multi-channel infrared gas detection system and method based on big data. Background Art

[0002] Big data technology refers to a series of technologies and tools used to process, store and analyze large-scale data sets. Through distributed storage systems, data can be stored in multiple nodes to improve capacity and performance. Data lake architecture is a data storage and management model that allows the storage of data of various types and formats and supports a variety of data processing and analysis tools. Discrete Fourier transform is a mathematical tool that converts time domain signals into frequency domain signals and is often used in signal processing and spectrum analysis. In the field of big data, discrete Fourier transform is often used to extract frequency domain features to analyze and process time domain data.

[0003] In the existing technology, relying solely on traditional feature extraction methods cannot fully capture the frequency domain characteristics of spectral intensity, resulting in incomplete feature representation; the lack of unified standardization of various features may lead to inconsistency in the feature space, making it difficult to effectively train and generalize the model. In addition, the use of traditional classification algorithms, such as logistic regression or decision trees, will not be able to fully utilize the complex structure of the data and the relationship between features, resulting in relatively low performance of the classification model. In terms of real-time data processing, the lack of an efficient streaming processing engine may cause the system to be unable to respond and process real-time data streams in a timely manner, thereby affecting the accuracy and timeliness of monitoring and early warning. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-channel infrared gas detection system and method based on big data to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A multi-channel infrared gas detection system based on big data, comprising:

[0007] Multi-channel infrared sensor module: It uses an infrared sensor array, which contains several independent channels, each of which is responsible for detecting infrared spectra of a specific wavelength; each channel is equipped with an independent filter and detector component to select a specific infrared wavelength, convert optical signals into electrical signals, and amplify and process weak signals output from the detector;

[0008] Data preprocessing module: obtains raw data from the multi-channel infrared sensor module, the raw data including the spectral intensity, timestamp, environmental parameters and sensor status of each channel; performs analog-to-digital conversion, filters the raw data, removes high-frequency noise and interference signals, calibrates and corrects the data output by the sensor, and extracts features from the preprocessed data; normalizes the extracted features, performs dimensionality reduction on the feature space, detects and processes outliers in the data, and visualizes the preprocessed data;

[0009] Pattern recognition and classification module: obtain features from the data preprocessing module, use the support vector machine model, train with pre-labeled training data, and establish a gas classification model; use the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model based on the evaluation results, and perform classification prediction on the data;

[0010] Real-time monitoring and alarm module: continuously monitors the gas concentration data obtained from the data preprocessing module and maintains real-time communication with the sensor module; sets the gas concentration threshold to determine whether the alarm condition is met, and can be dynamically adjusted as needed; when the gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event;

[0011] Data storage and management module: Use HDFS to store processed data, use Apache HBase to store structured data, establish a data lake architecture, and store structured data and unstructured data in the same storage pool; for real-time data streams, use the streaming processing engine Apache Kafka for real-time data processing and storage; implement a distributed data backup strategy, store data backups in several geographical locations, establish a disaster recovery mechanism, and use encryption technology to encrypt and store data.

[0012] The data preprocessing module uses an ADC chip to convert the analog signal obtained from the multi-channel infrared sensor module into a digital form; the spectral intensity of each channel reflects the infrared signal intensity detected by each channel, and a low-pass filter is used, and the user selects the cut-off frequency to remove the high-frequency noise of the spectral intensity of each channel; a median filter is used to take the median in the sliding window to remove the outliers and impulse noise of the spectral intensity of each channel; a Gaussian filter is used to smooth the spectral intensity of each channel and reduce the noise level of the spectral intensity of each channel; the timestamp data is verified and corrected to ensure the correctness of the time sequence; for environmental parameters, a moving average filter is used to reduce the random fluctuations of the environmental parameters, and weighted filtering is performed according to the weight of the data in the environmental parameters; the sensor state reflects the operating state and fault information of the sensor, and a state detection algorithm is used to detect the abnormal state or fault of the sensor, and the abnormal data is corrected and marked; a hysteresis filter is used to smooth the state change of the sensor and reduce the impact of sudden changes in the sensor state.

[0013] The data preprocessing module uses Fourier transform to extract frequency domain features from the spectral intensity of each channel. For the value x(n) of the time domain signal of the spectral intensity of each channel at the nth sampling point, discrete Fourier transform is used to convert x(n) into the frequency domain signal X(k) of the spectral intensity of each channel. The formula is as follows:

[0014]

[0015] Where k is the frequency index, indicating the position of each frequency component in the frequency domain signal in the discrete frequency domain; N is the number of sampling points of the spectral intensity data of each channel; n is the time domain sampling point index, starting from 0 and increasing until N-1; e -i2πkn / N is the rotation factor used to convert the time domain signal into the frequency domain signal, e is the base of the natural logarithm, and i is the imaginary unit.

[0016] Based on the frequency domain signal X(k) of the spectral intensity of each channel, the spectral energy is calculated as follows:

[0017]

[0018] Where E is the spectrum energy, which represents the total energy of the spectrum intensity data of each channel in the frequency domain, reflecting the overall spectrum characteristics of the channel;

[0019] Based on X(k), the spectrum mean is calculated using the following formula:

[0020]

[0021] in, It is the spectrum mean, which indicates the mean frequency of the frequency domain signal. The spectrum mean is used to indicate the average frequency of the spectral intensity data of each channel in the frequency domain, reflecting the central position of the spectral distribution.

[0022] Based on X(k), the spectrum bandwidth is calculated as follows:

[0023]

[0024] Among them, BW is the spectrum bandwidth, which indicates the distribution range of the spectral intensity data of each channel in the frequency domain, reflecting the width and volatility of the spectral signal.

[0025] When extracting features from timestamps, count the time intervals between timestamps, including the average time interval, the maximum time interval, and the minimum time interval, to reflect the sampling frequency and stability of the data; convert the timestamps into time units accurate to seconds to extract the periodicity of the data; analyze the distribution density of the data within the time period to detect sampling deviations or anomalies in the data;

[0026] When extracting features from environmental parameters, the average value, variance, maximum value, and minimum value of the environmental parameters are statistically calculated to describe the overall distribution of the environmental parameters; the correlation between environmental parameters is analyzed, and the correlation coefficient is used to evaluate the degree of correlation between different environmental parameters; if the environmental parameters have periodic changes, the periodic features are extracted, and the periodic features include the amplitude, period, and phase of the periodic changes;

[0027] When extracting features from sensor states, count the duration of the sensor states, and calculate the average duration, maximum duration, and minimum duration of each state; analyze the transitions between sensor states, calculate the frequency or pattern of state transitions, and evaluate the stability and performance of the sensor; and encode and classify the sensor states.

[0028] The pattern recognition and classification module prepares a training data set from samples of known categories, each sample containing a feature vector and a corresponding category label; the feature vector includes spectral intensity features, timestamp features, environmental parameter features and sensor status features of each channel, and the category label indicates the type of gas to which the sample belongs; the feature vector is normalized using mean normalization; the training data set is trained using a support vector machine algorithm, and during the training process, the support vector machine model will learn an optimal hyperplane to maximize the interval of the data and correctly classify the known samples;

[0029] The loss function of the support vector machine adopts the hinge loss function, and the formula is defined as follows:

[0030]

[0031] Among them, w is the normal vector of the hyperplane, which is used to separate the data in the feature space; b is the bias term, which is used to adjust the position of the decision boundary; u i is the feature vector of sample i; y i is the true category label of sample i, indicating the type of gas to which the sample belongs; M is the number of samples; max(0,1-y i (w·u i +b)) is the hinge loss function, which is used to measure the error between the model prediction and the true label. For misclassified samples, the loss value is non-zero.

[0032] The objective function of the support vector machine consists of two parts: the regularization term and the loss function. The regularization term is used to control the complexity of the model and prevent overfitting. The objective function is expressed as:

[0033]

[0034] Among them, ||w|| 2 is the norm of w, used for regularization; C is the penalty parameter, used to balance the regularization term and the loss function term; the optimization goal of the objective function is to minimize the sum of the regularization term and the loss function term;

[0035] In order to solve the optimization problem of support vector machine, the dual problem method is adopted. Through the Lagrange multiplier method, the following dual optimization problem formula is obtained:

[0036]

[0037] Among them, α is the Lagrange multiplier vector, which represents the weight of the training sample in the decision function and reflects the importance of the support vector; u i is the eigenvector of sample i, u j is the feature vector of sample j, u i and u j are all features extracted from the data preprocessing module; i is the true category label of sample i, indicating the type of gas to which sample i belongs; y j is the true category label of sample j, indicating the type of gas to which sample j belongs.

[0038] A multi-channel infrared gas detection method based on big data comprises the following steps:

[0039] S100, using an infrared sensor array, each channel is responsible for detecting a specific wavelength of infrared spectrum, converting optical signals into electrical signals, and amplifying and processing weak signals output from the detector;

[0040] S200, acquiring raw data, wherein the raw data includes spectral intensity, timestamp, environmental parameters and sensor status of each channel, and preprocessing the raw data;

[0041] S300, extracting features from the preprocessed data, normalizing the extracted features, reducing the dimension of the feature space, detecting and processing outliers in the data, and visually displaying the preprocessed data;

[0042] S400, using a support vector machine model to train with pre-labeled training data to establish a gas classification model;

[0043] S500, using the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model according to the evaluation results, and perform classification prediction on the data;

[0044] S600, continuously monitor gas concentration data and maintain real-time communication with the sensor module; set a threshold value of gas concentration to determine whether an alarm condition is reached;

[0045] S700, when the monitored gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event;

[0046] S800, uses HDFS to store processed data, uses Apache HBase to store structured data, and establishes a data lake architecture to store structured and unstructured data in the same storage pool;

[0047] S900, for real-time data streams, uses the stream processing engine Apache Kafka for real-time data processing and storage; implements a distributed data backup strategy, establishes a disaster recovery mechanism, uses encryption technology, and implements an automated operation and maintenance mechanism.

[0048] According to step S700, the system sets three alarm levels, namely low, medium and high, and sets corresponding thresholds for different gas concentration ranges; when the monitored gas concentration exceeds a specific threshold, the system will trigger an alarm of the corresponding level; the alarm mechanism uses several methods for notification, including sound alarm, light flash alarm, email and SMS notification;

[0049] When the system triggers a low-level alarm, early warning measures will be taken to alert relevant personnel and conduct preliminary emergency processing; when the system triggers a medium-level alarm, personnel will need to be evacuated and areas closed; when the system triggers a high-level alarm, emergency measures will be taken immediately to ensure personnel safety; the system will record detailed information on each alarm event, including the alarm level, trigger time, gas concentration value, trigger reason and alarm handling personnel, and store them in the data storage and management module.

[0050] According to step S800, using HDFS to store the processed data is a common big data storage solution. HDFS is a distributed file system specially designed for storing and processing large-scale data sets. Its main features include high reliability, high throughput, fault tolerance and scalability. In this system, data is divided into multiple blocks and distributedly stored on different nodes of the cluster, thereby achieving parallel processing and efficient storage of data. Apache HBase is an open source, distributed, column-oriented NoSQL database built on HDFS. HBase is mainly used to store structured data and provide fast random read / write access. It is suitable for scenarios that require real-time access to large amounts of structured data. For example, in the present invention, it is used to store structured data in a gas detection system, such as information such as the spectral intensity, timestamp, environmental parameters and sensor status of each channel.

[0051] Building a data lake architecture means storing data of different types and formats in the same storage pool, including structured data, semi-structured data, and unstructured data. This architecture makes data more flexible and facilitates unified management and analysis, while also improving data availability and scalability. Storing structured and unstructured data in the same storage pool can provide a broader perspective and richer information for data analysis and mining, thereby better supporting data-driven decision-making and applications.

[0052] According to step S900, an Apache Kafka cluster is deployed in the data storage and management module, and the cluster includes several Kafka brokers for receiving, processing and storing real-time data streams; Kafka Topics are configured to send the raw data of each channel as messages to the corresponding Topic, and each message contains spectral intensity, timestamp, environmental parameters and sensor status; on the consumer side, a corresponding Kafka consumer application is developed to consume data from each Topic and perform real-time processing; the processed data is sent again to other Topics in Kafka according to demand;

[0053] Configure a replication mechanism in the Kafka cluster to ensure that the data of each Topic is backed up between multiple Brokers; deploy several Kafka clusters in different geographical locations to ensure distributed data backup. When a cluster fails, other clusters can continue to provide services. For each Kafka cluster, set up a monitoring system to monitor the health of the cluster in real time, and configure an automatic failover mechanism to automatically transfer traffic to the backup cluster when a failure is detected.

[0054] Use TLS / SSL protocol for encryption in communications between Kafka clusters to ensure data security during transmission; configure access control lists (ACLs) within the Kafka cluster to limit access to topics so that only authenticated users can access data; perform end-to-end encryption on data to ensure data security during storage and processing; deploy a monitoring system to monitor the operating status of the Kafka cluster in real time, including cluster throughput, latency, and disk usage; develop automated scripts and tools to implement automatic deployment, configuration, and expansion of the cluster, as well as automatic detection and recovery of faults.

[0055] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0056] The data preprocessing module of the present invention utilizes Fourier transform to extract frequency domain features, and can more accurately analyze the spectral intensity of each channel, thereby improving the accuracy and stability of gas detection.

[0057] The pattern recognition and classification module of the present invention adopts a support vector machine algorithm and combines multiple feature vectors to train and classify samples. Compared with traditional methods, it can better identify gas types and improve the performance and generalization ability of the classification model.

[0058] The data storage and management module of the present invention adopts technologies such as HDFS, Apache HBase and Apache Kafka, which can achieve efficient data storage, processing and backup, and improve the efficiency and reliability of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 This is a system structure diagram of a multi-channel infrared gas detection system based on big data of the present invention;

[0061] Figure 2 It is a schematic diagram of the steps of a multi-channel infrared gas detection method based on big data of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0064] According to one embodiment of the present invention, Figure 1 As shown in the system structure diagram of a multi-channel infrared gas detection system based on big data, a multi-channel infrared gas detection system based on big data includes:

[0065] Multi-channel infrared sensor module: It uses an infrared sensor array, which contains several independent channels, each of which is responsible for detecting infrared spectra of a specific wavelength; each channel is equipped with an independent filter and detector component to select a specific infrared wavelength, convert optical signals into electrical signals, and amplify and process weak signals output from the detector;

[0066] Data preprocessing module: obtains raw data from the multi-channel infrared sensor module, the raw data including the spectral intensity, timestamp, environmental parameters and sensor status of each channel; performs analog-to-digital conversion, filters the raw data, removes high-frequency noise and interference signals, calibrates and corrects the data output by the sensor, and extracts features from the preprocessed data; normalizes the extracted features, performs dimensionality reduction on the feature space, detects and processes outliers in the data, and visualizes the preprocessed data;

[0067] Pattern recognition and classification module: obtain features from the data preprocessing module, use the support vector machine model, train with pre-labeled training data, and establish a gas classification model; use the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model based on the evaluation results, and perform classification prediction on the data;

[0068] Real-time monitoring and alarm module: continuously monitors the gas concentration data obtained from the data preprocessing module and maintains real-time communication with the sensor module; sets the gas concentration threshold to determine whether the alarm condition is met, and can be dynamically adjusted as needed; when the gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event;

[0069] Data storage and management module: Use HDFS to store processed data, use Apache HBase to store structured data, establish a data lake architecture, and store structured data and unstructured data in the same storage pool; for real-time data streams, use the streaming processing engine Apache Kafka for real-time data processing and storage; implement a distributed data backup strategy, store data backups in several geographical locations, establish a disaster recovery mechanism, and use encryption technology to encrypt and store data.

[0070] The data preprocessing module uses an ADC chip to convert the analog signal obtained from the multi-channel infrared sensor module into a digital form; the spectral intensity of each channel reflects the infrared signal intensity detected by each channel, and a low-pass filter is used, and the user selects the cut-off frequency to remove the high-frequency noise of the spectral intensity of each channel; a median filter is used to take the median in the sliding window to remove the outliers and impulse noise of the spectral intensity of each channel; a Gaussian filter is used to smooth the spectral intensity of each channel and reduce the noise level of the spectral intensity of each channel; the timestamp data is verified and corrected to ensure the correctness of the time sequence; for environmental parameters, a moving average filter is used to reduce the random fluctuations of the environmental parameters, and weighted filtering is performed according to the weight of the data in the environmental parameters; the sensor state reflects the operating state and fault information of the sensor, and a state detection algorithm is used to detect the abnormal state or fault of the sensor, and the abnormal data is corrected and marked; a hysteresis filter is used to smooth the state change of the sensor and reduce the impact of sudden changes in the sensor state.

[0071] The data preprocessing module uses Fourier transform to extract frequency domain features from the spectral intensity of each channel. For the value x(n) of the time domain signal of the spectral intensity of each channel at the nth sampling point, discrete Fourier transform is used to convert x(n) into the frequency domain signal X(k) of the spectral intensity of each channel. The formula is as follows:

[0072]

[0073] Where k is the frequency index, indicating the position of each frequency component in the frequency domain signal in the discrete frequency domain; N is the number of sampling points of the spectral intensity data of each channel; n is the time domain sampling point index, starting from 0 and increasing until N-1; e -i2πkn / N is the rotation factor used to convert the time domain signal into the frequency domain signal, e is the base of the natural logarithm, and i is the imaginary unit.

[0074] Based on the frequency domain signal X(k) of the spectral intensity of each channel, the spectral energy is calculated as follows:

[0075]

[0076] Where E is the spectrum energy, which represents the total energy of the spectrum intensity data of each channel in the frequency domain, reflecting the overall spectrum characteristics of the channel;

[0077] Based on X(k), the spectrum mean is calculated using the following formula:

[0078]

[0079] in, It is the spectrum mean, which indicates the mean frequency of the frequency domain signal. The spectrum mean is used to indicate the average frequency of the spectral intensity data of each channel in the frequency domain, reflecting the central position of the spectral distribution.

[0080] Based on X(k), the spectrum bandwidth is calculated as follows:

[0081]

[0082] Among them, BW is the spectrum bandwidth, which indicates the distribution range of the spectral intensity data of each channel in the frequency domain, reflecting the width and volatility of the spectral signal.

[0083] When extracting features from timestamps, count the time intervals between timestamps, including the average time interval, the maximum time interval, and the minimum time interval, to reflect the sampling frequency and stability of the data; convert the timestamps into time units accurate to seconds to extract the periodicity of the data; analyze the distribution density of the data within the time period to detect sampling deviations or anomalies in the data;

[0084] When extracting features from environmental parameters, the average value, variance, maximum value, and minimum value of the environmental parameters are statistically calculated to describe the overall distribution of the environmental parameters; the correlation between environmental parameters is analyzed, and the correlation coefficient is used to evaluate the degree of correlation between different environmental parameters; if the environmental parameters have periodic changes, the periodic features are extracted, and the periodic features include the amplitude, period, and phase of the periodic changes;

[0085] When extracting features from sensor states, count the duration of the sensor states, and calculate the average duration, maximum duration, and minimum duration of each state; analyze the transitions between sensor states, calculate the frequency or pattern of state transitions, and evaluate the stability and performance of the sensor; and encode and classify the sensor states.

[0086] The pattern recognition and classification module prepares a training data set from samples of known categories, each sample containing a feature vector and a corresponding category label; the feature vector includes spectral intensity features, timestamp features, environmental parameter features and sensor status features of each channel, and the category label indicates the type of gas to which the sample belongs; the feature vector is normalized using mean normalization; the training data set is trained using a support vector machine algorithm, and during the training process, the support vector machine model will learn an optimal hyperplane to maximize the interval of the data and correctly classify the known samples;

[0087] The loss function of the support vector machine adopts the hinge loss function, and the formula is defined as follows:

[0088]

[0089] Among them, w is the normal vector of the hyperplane, which is used to separate the data in the feature space; b is the bias term, which is used to adjust the position of the decision boundary; u i is the feature vector of sample i; y i is the true category label of sample i, indicating the type of gas to which the sample belongs; M is the number of samples; max(0,1-y i (w·u i +b)) is the hinge loss function, which is used to measure the error between the model prediction and the true label. For misclassified samples, the loss value is non-zero.

[0090] The objective function of the support vector machine consists of two parts: the regularization term and the loss function. The regularization term is used to control the complexity of the model and prevent overfitting. The objective function is expressed as:

[0091]

[0092] Among them, ||w|| 2 is the norm of w, used for regularization; C is the penalty parameter, used to balance the regularization term and the loss function term; the optimization goal of the objective function is to minimize the sum of the regularization term and the loss function term;

[0093] In order to solve the optimization problem of support vector machine, the dual problem method is adopted. Through the Lagrange multiplier method, the following dual optimization problem formula is obtained:

[0094]

[0095] Among them, α is the Lagrange multiplier vector, which represents the weight of the training sample in the decision function and reflects the importance of the support vector; u i is the eigenvector of sample i, u j is the feature vector of sample j, u i and u jare all features extracted from the data preprocessing module; i is the true category label of sample i, indicating the type of gas to which sample i belongs; y j is the true category label of sample j, indicating the type of gas to which sample j belongs.

[0096] According to another embodiment of the present invention, Figure 2 As shown in a schematic diagram of the steps of a multi-channel infrared gas detection method based on big data, a multi-channel infrared gas detection method based on big data includes the following steps:

[0097] S100, using an infrared sensor array, each channel is responsible for detecting a specific wavelength of infrared spectrum, converting optical signals into electrical signals, and amplifying and processing weak signals output from the detector;

[0098] S200, acquiring raw data, wherein the raw data includes spectral intensity, timestamp, environmental parameters and sensor status of each channel, and preprocessing the raw data;

[0099] S300, extracting features from the preprocessed data, normalizing the extracted features, reducing the dimension of the feature space, detecting and processing outliers in the data, and visually displaying the preprocessed data;

[0100] S400, using a support vector machine model to train with pre-labeled training data to establish a gas classification model;

[0101] S500, using the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model according to the evaluation results, and perform classification prediction on the data;

[0102] S600, continuously monitor gas concentration data and maintain real-time communication with the sensor module; set a threshold value of gas concentration to determine whether an alarm condition is reached;

[0103] S700, when the monitored gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event;

[0104] S800, uses HDFS to store processed data, uses Apache HBase to store structured data, and establishes a data lake architecture to store structured and unstructured data in the same storage pool;

[0105] S900, for real-time data streams, uses the stream processing engine Apache Kafka for real-time data processing and storage; implements a distributed data backup strategy, establishes a disaster recovery mechanism, uses encryption technology, and implements an automated operation and maintenance mechanism.

[0106] According to step S100, we have three channels of infrared sensors that detect infrared spectra with wavelengths of 800nm, 1000nm and 1200nm respectively. The input light signal intensity data of these three channels are randomly generated to represent the light intensity at these wavelengths. The infrared sensor converts the input light signal into an electrical signal. We can perform some conversions on the input light signal, such as adding noise, and simulate the response curve of the sensor to convert the light signal into an electrical signal. When the signal output from the sensor is weak, we can simulate the amplification of these signals, such as multiplying them by an amplification factor, and perform some signal processing, such as filtering, denoising, etc.

[0107] According to step S200, the obtained pre-processed partial data is as follows: timestamp: 2024-03-2010:00, channel 1 spectral intensity: 0.6, channel 2 spectral intensity: 0.8, channel 3 spectral intensity: 0.7, environmental parameter 1: 25°C, environmental parameter 2: 50%RH, sensor status: normal;

[0108] Timestamp: 2024-03-2010:01, Channel 1 spectral intensity: 0.55, Channel 2 spectral intensity: 0.78, Channel 3 spectral intensity: 0.68, Environmental parameter 1: 24℃, Environmental parameter 2: 49%RH, Sensor status: Normal;

[0109] Timestamp: 2024-03-2010:02, Channel 1 spectral intensity: 0.58, Channel 2 spectral intensity: 0.76, Channel 3 spectral intensity: 0.72, Environmental parameter 1: 25℃, Environmental parameter 2: 50%RH, Sensor status: Normal.

[0110] According to step S300, feature extraction is performed: Spectral intensity average feature: calculate the average value of the spectral intensity of each channel as one of the features; environmental parameters: directly used as features; sensor state: converted into binary features, normal state is 1, abnormal state is 0. Normalization processing: normalize each feature, and use the Min-Max standardization method to scale the feature value to the range of [0, 1]. Dimensionality reduction processing: If the dimension of the feature space is high, use the principal component analysis PCA technology to reduce the dimension to retain most of the information while reducing the number of features. Outlier detection and processing: use an outlier detection algorithm, such as a statistical method or a distance-based method, to identify and process outliers, correct them to appropriate values ​​or delete outliers. Visualization display: visualize the preprocessed data, and use scatter plots, box plots, etc. to display feature distribution and abnormal conditions.

[0111] According to step S400, a set of training data is prepared, each sample contains a feature vector and a corresponding category label, and then these data are used to train the SVM model. Next, we will use these training data to train the SVM model. In the simulation experiment, we use the existing machine learning library scikit-learn to implement this step. The training process will find an optimal hyperplane based on the SVM algorithm to maximize the interval of the data and correctly classify the known samples.

[0112] According to step S500, we will use the trained SVM model to predict these validation data and evaluate the performance and generalization ability of the model, using the accuracy index to evaluate the performance of the model.

[0113] According to step S700, we have a gas concentration sensor that can collect data once a second. We set three thresholds to trigger different levels of alarms: low, medium, and high. When the gas concentration is 0.4, no alarm is triggered and it is recorded as a low-level alarm. When the gas concentration is 0.6, a medium-level alarm is triggered and it is recorded as a medium-level alarm. When the gas concentration is 0.8, a high-level alarm is triggered and it is recorded as a high-level alarm.

[0114] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0115] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-channel infrared gas detection system based on big data, characterized by: include: Multi-channel infrared sensor module: It uses an infrared sensor array, which contains several independent channels, each of which is responsible for detecting infrared spectra of a specific wavelength; each channel is equipped with an independent filter and detector component to select a specific infrared wavelength, convert optical signals into electrical signals, and amplify and process weak signals output from the detector; Data preprocessing module: obtains raw data from the multi-channel infrared sensor module, the raw data including the spectral intensity, timestamp, environmental parameters and sensor status of each channel; performs analog-to-digital conversion, filters the raw data, removes high-frequency noise and interference signals, calibrates and corrects the data output by the sensor, and extracts features from the preprocessed data; Normalize the extracted features, reduce the dimension of the feature space, detect and process outliers in the data, and visualize the preprocessed data; The data preprocessing module uses Fourier transform to extract frequency domain features from the spectral intensity of each channel. For the value x(n) of the time domain signal of the spectral intensity of each channel at the nth sampling point, discrete Fourier transform is used to convert x(n) into the frequency domain signal X(k) of the spectral intensity of each channel. The formula is as follows: Where k is the frequency index, indicating the position of each frequency component in the frequency domain signal in the discrete frequency domain; N is the number of sampling points of the spectral intensity data of each channel; n is the time domain sampling point index, starting from 0 and increasing until N-1; e -i2πkn / N is the rotation factor used to convert the time domain signal into the frequency domain signal, e is the base of the natural logarithm, and i is the imaginary unit; Based on the frequency domain signal X(k) of the spectral intensity of each channel, the spectral energy is calculated as follows: Where E is the spectrum energy, which represents the total energy of the spectrum intensity data of each channel in the frequency domain, reflecting the overall spectrum characteristics of the channel; Based on X(k), the spectrum mean is calculated using the following formula: in, It is the spectrum mean, which indicates the mean frequency of the frequency domain signal. The spectrum mean is used to indicate the average frequency of the spectral intensity data of each channel in the frequency domain, reflecting the central position of the spectral distribution. Based on X(k), the spectrum bandwidth is calculated as follows: Among them, BW is the spectrum bandwidth, which indicates the distribution range of the spectral intensity data of each channel in the frequency domain, reflecting the width and volatility of the spectral signal; When extracting features from timestamps, count the time intervals between timestamps, including the average time interval, the maximum time interval, and the minimum time interval, to reflect the sampling frequency and stability of the data; convert the timestamps into time units accurate to seconds to extract the periodicity of the data; analyze the distribution density of the data within the time period to detect sampling deviations or anomalies in the data; When extracting features from environmental parameters, the average value, variance, maximum value, and minimum value of the environmental parameters are statistically calculated to describe the overall distribution of the environmental parameters; the correlation between environmental parameters is analyzed, and the correlation coefficient is used to evaluate the degree of correlation between different environmental parameters; if the environmental parameters have periodic changes, the periodic features are extracted, and the periodic features include the amplitude, period, and phase of the periodic changes; When extracting features from sensor states, count the duration of sensor states, calculate the average duration, maximum duration, and minimum duration of each state; analyze the transitions between sensor states, calculate the frequency or pattern of state transitions, and evaluate the stability and performance of the sensor; encode and classify sensor states; Pattern recognition and classification module: obtain features from the data preprocessing module, use the support vector machine model, train with pre-labeled training data, and establish a gas classification model; use the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model based on the evaluation results, and perform classification prediction on the data; The pattern recognition and classification module prepares a training data set from samples of known categories, each sample containing a feature vector and a corresponding category label; the feature vector includes spectral intensity features, timestamp features, environmental parameter features and sensor status features of each channel, and the category label indicates the type of gas to which the sample belongs; the feature vector is normalized using mean normalization; the training data set is trained using a support vector machine algorithm, and during the training process, the support vector machine model will learn an optimal hyperplane to maximize the interval of the data and correctly classify the known samples; The loss function of the support vector machine adopts the hinge loss function, and the formula is defined as follows: Among them, w is the normal vector of the hyperplane, which is used to separate the data in the feature space; b is the bias term, which is used to adjust the position of the decision boundary; u i is the feature vector of sample i; y i is the true category label of sample i, indicating the type of gas to which the sample belongs; M is the number of samples; max(0,1-y i (w·u i +b)) is the hinge loss function, which is used to measure the error between the model prediction and the true label. For misclassified samples, the loss value is non-zero; Real-time monitoring and alarm module: continuously monitors the gas concentration data obtained from the data preprocessing module and maintains real-time communication with the sensor module; sets the gas concentration threshold to determine whether the alarm condition is met, and can be dynamically adjusted as needed; when the gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event; Data storage and management module: Use HDFS to store processed data, use Apache HBase to store structured data, establish a data lake architecture, and store structured data and unstructured data in the same storage pool; for real-time data streams, use the streaming processing engine Apache Kafka for real-time data processing and storage; implement a distributed data backup strategy, store data backups in several geographical locations, establish a disaster recovery mechanism, and use encryption technology to encrypt and store data.

2. The multi-channel infrared gas detection system based on big data according to claim 1 is characterized in that: The data preprocessing module uses an ADC chip to convert the analog signal obtained from the multi-channel infrared sensor module into a digital form; the spectral intensity of each channel reflects the infrared signal intensity detected by each channel, and a low-pass filter is used, and the user selects the cut-off frequency to remove the high-frequency noise of the spectral intensity of each channel; Use median filtering to take the median in the sliding window to remove outliers and impulse noise in the spectral intensity of each channel; Use Gaussian filtering to smooth the spectral intensity of each channel and reduce the noise level of the spectral intensity of each channel; verify and correct the timestamp data to ensure the correctness of the time sequence; for environmental parameters, use moving average filtering to reduce the random fluctuations of environmental parameters, and perform weighted filtering according to the weight of the data in the environmental parameters; the sensor state reflects the operating state and fault information of the sensor, and uses the state detection algorithm to detect the abnormal state or fault of the sensor, and correct and mark the abnormal data; Use hysteresis filtering to smooth the state changes of the sensor and reduce the impact of sudden changes in the sensor state.

3. The multi-channel infrared gas detection system based on big data according to claim 1 is characterized in that: The objective function of the support vector machine consists of two parts: the regularization term and the loss function. The regularization term is used to control the complexity of the model and prevent overfitting. The objective function is expressed as: Among them, ||w|| 2 is the norm of w, used for regularization; C is the penalty parameter, used to balance the regularization term and the loss function term; the optimization goal of the objective function is to minimize the sum of the regularization term and the loss function term; In order to solve the optimization problem of support vector machine, the dual problem method is adopted. Through the Lagrange multiplier method, the following dual optimization problem formula is obtained: Among them, α is the Lagrange multiplier vector, which represents the weight of the training sample in the decision function and reflects the importance of the support vector; u i is the eigenvector of sample i, u j is the feature vector of sample j, u i and u j are all features extracted from the data preprocessing module; i is the true category label of sample i, indicating the type of gas to which sample i belongs; y j is the true category label of sample j, indicating the type of gas to which sample j belongs.

4. A multi-channel infrared gas detection method based on big data, applied to a multi-channel infrared gas detection system based on big data as claimed in any one of claims 1 to 3, characterized in that: The steps include: S100, using an infrared sensor array, each channel is responsible for detecting a specific wavelength of infrared spectrum, converting optical signals into electrical signals, and amplifying and processing weak signals output from the detector; S200, acquiring raw data, wherein the raw data includes spectral intensity, timestamp, environmental parameters and sensor status of each channel, and preprocessing the raw data; S300, extracting features from the preprocessed data, normalizing the extracted features, reducing the dimension of the feature space, detecting and processing outliers in the data, and visually displaying the preprocessed data; S400, using a support vector machine model to train with pre-labeled training data to establish a gas classification model; S500, using the validation data set to evaluate the trained gas classification model, evaluate the classification performance and generalization ability of the model, optimize the model according to the evaluation results, and perform classification prediction on the data; S600, continuously monitor gas concentration data and maintain real-time communication with the sensor module; set a threshold value of gas concentration to determine whether an alarm condition is reached; S700, when the monitored gas concentration exceeds the set threshold, the alarm mechanism is triggered, and different alarm levels are set according to different gas concentrations; the system records detailed information of each alarm event; S800, uses HDFS to store processed data, uses Apache HBase to store structured data, and establishes a data lake architecture to store structured and unstructured data in the same storage pool; S900, for real-time data streams, uses the stream processing engine Apache Kafka for real-time data processing and storage; implements a distributed data backup strategy, establishes a disaster recovery mechanism, uses encryption technology, and implements an automated operation and maintenance mechanism.

5. The multi-channel infrared gas detection method based on big data according to claim 4 is characterized in that: According to step S700, the system sets three alarm levels, namely low, medium and high, and sets corresponding thresholds for different gas concentration ranges; when the monitored gas concentration exceeds a specific threshold, the system will trigger an alarm of the corresponding level; the alarm mechanism uses several methods for notification, including sound alarm, light flash alarm, email and SMS notification; The system will record detailed information of each alarm event, including alarm level, trigger time, gas concentration value, trigger reason and alarm handling personnel, and store them in the data storage and management module.

6. The multi-channel infrared gas detection method based on big data according to claim 4 is characterized in that: According to step S900, an Apache Kafka cluster is deployed in the data storage and management module, and the cluster includes several Kafka brokers for receiving, processing and storing real-time data streams; Kafka Topic is configured, and the raw data of each channel is sent to the corresponding Topic as a message, and each message contains spectral intensity, timestamp, environmental parameters and sensor status; on the consumer side, a corresponding Kafka consumer application is developed to consume data from each Topic and perform real-time processing; the processed data is sent to other Topics in Kafka again according to demand; Configure a replication mechanism in the Kafka cluster to ensure that the data of each Topic is backed up between multiple Brokers. Deploy several Kafka clusters in different geographical locations to ensure distributed backup of data. When a cluster fails, other clusters can continue to provide services. For each Kafka cluster, set up a monitoring system to monitor the health of the cluster in real time, and configure an automatic failover mechanism to automatically transfer traffic to a backup cluster when a failure is detected; Use TLS / SSL protocol for encryption in the communication between Kafka clusters to ensure the security of data during transmission; configure access control list ACL within the Kafka cluster to limit access to topics, so that only authenticated users can access data; encrypt data end-to-end to ensure the security of data during storage and processing; Deploy a monitoring system to monitor the operating status of the Kafka cluster in real time, including cluster throughput, latency, and disk usage; Develop automated scripts and tools to achieve automatic deployment, configuration, and expansion of clusters, as well as automatic detection and recovery of failures.

Images