Atmospheric pollution control big data analysis method and system based on FaaS function service architecture

Abstract

The invention discloses an air pollution control big data analysis method and system based on a FaaS function service architecture. The analysis method comprises the following steps: identifying and confirming data sources of pollution factor molecules, pollution assessment indexes and air quality indexes; based on different data sources, compiling corresponding FaaS functions to perform data extraction on the data sources; establishing connection with the HDFS through an HDFS client provided by the Hadoop ecological system, preprocessing the extracted data, uploading the preprocessed data to a specified position in the HDFS through an application gateway for distributed storage, and displaying the preprocessed data through a visual tool; and based on the newly acquired data source, triggering the FaaS function to execute a data analysis process. According to the method, the FaaS function is adopted, the server configuration, maintenance or expansibility problem does not need to be considered, the period from development to deployment is greatly shortened, the data analysis efficiency is improved, meanwhile, waste or performance bottleneck caused by fixed configuration of hardware resources in a traditional mode is avoided, and prediction of the environmental protection prediction big data analysis model is more accurate.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and more specifically to a method and system for analyzing big data for air pollution prevention and control based on a FaaS function service architecture. Background Art

[0002] Due to the current atmospheric pollution factors [including conventional sulfur dioxide (SO2), nitrogen dioxide (NO2), carbon monoxide (CO), ozone (O3), particulate matter (particle size less than or equal to 2.5um), and particulate matter (particle size less than or equal to 10um)], data computing services are usually manually installed on each physical server node by operation and maintenance personnel; developers use programming languages ​​or scripting languages ​​(such as Scala, Java, Shell, etc.) to implement ETL data processing; and developers compile the code after the function development is completed and package it into a jar or war type executable file and upload it to the server installed by the operation and maintenance personnel, and then use the spark command or shell script to execute the application server war or jar type executable program.

[0003] However, as more and more data on atmospheric pollution factors are generated, the operating system performance is heavily relied upon. In addition, the migration of massive data, the expansion of the scale of computing service clusters, and the complexity of operation and maintenance are becoming increasingly complex. Memory overflow and insufficient disk space are often found, resulting in abnormalities in running programs. This will force developers to apply for server resources again or alleviate the problem of memory shortage by deleting some historical data, logs and other files. The end result is that some avoidable work is done, which reduces work efficiency. Summary of the invention

[0004] In this embodiment, a method, system, electronic device and storage medium for analyzing big data for air pollution prevention and control based on the FaaS function service architecture are provided to achieve the goal of not having to consider server configuration, maintenance or scalability issues, thereby greatly accelerating the cycle from development to deployment, making iterations faster, being able to respond quickly to market changes, and providing analysis suggestions based on pollution factor data through AI models.

[0005] In a first aspect, an embodiment of the present invention provides an air pollution prevention and control big data analysis method based on a FaaS function service architecture, the analysis method comprising: Identify and confirm the data sources of pollution factor molecules, pollution assessment indicators, and air quality index; based on different data sources, write corresponding FaaS functions to extract data from the data sources; Establish a connection with HDFS through the HDFS client provided by the Hadoop ecosystem, upload the extracted data to the designated location in HDFS through the application gateway after preprocessing for distributed storage, and display it through visualization tools; Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and predictive analysis is performed through the pre-built environmental prediction big data analysis model.

[0006] In an optional embodiment, the data sources of pollution factor molecules, pollution assessment indicators, and air quality index are identified and confirmed, and based on different data sources, corresponding FaaS functions are written to extract data from the data sources, including: Obtain various pollution factor parameters through the client, and generate assessment indicators, real-time air quality index report data, daily air quality index data, and air quality index category color data based on various pollution factor parameters; Indicators and data generated based on various pollution factor parameters are used as data sources; Based on different data sources, write corresponding FaaS functions to extract data.

[0007] In an optional embodiment, a connection is established with HDFS through an HDFS client provided by the Hadoop ecosystem, and the extracted data is pre-processed and uploaded to a designated location in HDFS through an application gateway for distributed storage, and displayed through a visualization tool, including: Use the HDFS client provided by the Hadoop ecosystem to establish a connection with HDFS, and upload the processed data to the specified directory in HDFS for storage; the data storage format is Parquet; Use the command line tools provided by HDFS to verify whether the data is successfully stored and check the file path, size, and permission information; Through data visualization BI tools, various data stored in HDFS are processed and visualized.

[0008] In an optional embodiment, the extracted data is pre-processed and then uploaded through the application gateway, including: Check whether the extracted data has missing values, outliers or duplicate data, and correct the data; The data is partitioned and indexed based on time and location dimensions, converted into Parquet format, and written to the specified directory in HDFS for storage through application gateway transmission.

[0009] In an optional embodiment, based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and the prediction analysis is performed through the pre-built environmental prediction big data analysis model, including: Obtain new data sources, activate corresponding FaaS instances, and trigger FaaS functions to perform predefined processing. The data after predefined processing is stored in HDFS; Based on the new data source, the pre-built environmental protection prediction big data analysis model performs predictive analysis on it.

[0010] In an optional embodiment, the construction of the environmental protection prediction big data analysis model includes: Collect environmental data, including but not limited to air quality index, water pH value, concentration of various pollutants, heavy metal content in soil, and distribution of organic pollutants; Clean and preprocess environment-related data; Divide the preprocessed data into training set, test set and validation set; Use the data in the training set to train the neural network model, and use the validation set to verify and adjust the model training process; The model is optimized through sparse classification cross entropy loss function, and the optimized and trained neural network model is used as an environmental protection prediction big data analysis model.

[0011] Compared with the prior art, the air pollution prevention and control big data analysis method based on the FaaS function service architecture of the present invention has the following beneficial effects: By adopting FaaS functions, developers can focus entirely on writing business logic, that is, focusing on the development of functional function code without having to consider server configuration, maintenance or scalability issues, which greatly accelerates the cycle from development to deployment, making iteration faster and able to respond quickly to market changes; FaaS service providers are responsible for the security, monitoring, backup and recovery of the underlying infrastructure and other operation and maintenance tasks, which means that developers no longer need to worry about operating system management, software updates or server failures, and can focus more on optimizing business logic and improving user experience.

[0012] The application of FaaS in the field of environmental prediction big data analysis has significant advantages and far-reaching impact. It provides an efficient, flexible and scalable computing platform for environmental big data analysis by abstracting computing resources into callable functions. When the environmental prediction big data analysis model needs to process massive data or execute complex algorithms, FaaS can respond quickly and dynamically schedule sufficient computing resources, such as CPU, memory and storage, to ensure that the model can complete large-scale data processing and deep learning tasks in the shortest time. This instant response and elastic scaling capabilities greatly improve the efficiency of data analysis, while avoiding the waste or performance bottlenecks caused by fixed configuration of hardware resources in the traditional model.

[0013] FaaS services can monitor and respond to changes in data sources in real time through event-driven methods, whether such changes are due to updates in real-time data collected by environmental monitoring stations or to the flow of information from multiple data sources such as social media and satellite remote sensing. Once the data changes, FaaS can immediately trigger the corresponding data processing function to start working, ensuring that the environmental prediction big data analysis model can capture environmental change trends in real time and quickly generate accurate prediction results. This near-delay-free real-time processing capability is of immeasurable value in ensuring environmental safety, preventing environmental disasters, and formulating effective environmental policies.

[0014] In a second aspect, an embodiment of the present invention provides an air pollution prevention and control big data analysis system based on a FaaS function service architecture, which is used to implement the air pollution prevention and control big data analysis method based on a FaaS function service architecture described in the first aspect, including: a client, which is used to receive data sources from different sources, wherein the data source includes pollution factor data; The data extraction system is used to extract specific types of pollution factor information from the data obtained from the client, and then package and compress the pollution factor information and upload it to HDFS storage through the application gateway; HDFS storage, used for centralized management and storage of processed data pushed by the data extraction system; Data analysis and processing system, including multiple specialized data processing modules, for in-depth processing and analysis of data in HDFS; The AI ​​algorithm engine, based on the pre-built environmental prediction big data analysis model, generates analysis results according to the newly arrived pollution factor data.

[0015] In an optional embodiment, the data analysis and processing system includes a pollution factor analysis data processing module, a pollution assessment index analysis data processing module, an air quality index real-time report data processing module, an air quality index daily report data processing module, and an air quality index category color data processing module, which is used to generate pollution assessment indicators, air quality index real-time report, air quality index daily report, and air quality index category color data based on pollution factor data processing.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the big data analysis method for air pollution prevention and control based on the FaaS function service architecture as described in the first aspect are implemented.

[0018] Compared with the prior art, the beneficial effects of the air pollution prevention and control big data analysis system, electronic device and storage medium based on the FaaS function service architecture of the present invention are the same as the air pollution prevention and control big data analysis method based on the FaaS function service architecture described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a flow chart of a method for analyzing big data for air pollution prevention and control based on the FaaS function service architecture in an embodiment of the present invention; Figure 2 This is a structural block diagram of an air pollution prevention and control big data analysis system based on the FaaS function service architecture in an embodiment of the present invention; Figure 3 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] First, the prior art and related concepts involved in the embodiments of the present invention are described: Hadoop ecosystem: Hadoop ecosystem is a collection of open source software and tools formed around the Hadoop big data processing platform, designed to support large-scale data processing, storage, management, analysis and visualization applications. Hadoop's core components mainly include three parts: HDFS, MapReduce and YARN. These components together form the basis of the Hadoop distributed computing framework.

[0024] Among them, HDFS is a key component of Hadoop, which provides high-throughput data access services for applications and can run on low-cost hardware. HDFS is designed to handle very large data sets and supports streaming data access. It does this by splitting files into multiple blocks and distributing them on different nodes in the cluster. In addition, HDFS is also highly fault-tolerant because it saves data copies on multiple nodes to prevent data loss caused by hardware failure.

[0025] In an embodiment of the present invention, a method for analyzing big data on air pollution prevention and control based on a FaaS function service architecture is provided. Figure 1is a flowchart of the method for analyzing air pollution prevention and control big data based on the FaaS function service architecture of the present invention. Figure 1 As shown, the process includes the following steps: S100, identifying and confirming the data sources of pollution factor molecules, pollution assessment indicators, and air quality index; based on different data sources, writing corresponding FaaS functions to extract data from the data sources; It should be noted that there are six types of pollution factors: sulfur dioxide (SO2), nitrogen dioxide (NO2), carbon monoxide (CO), ozone (O3), particulate matter (particle size less than or equal to 2.5um), and particulate matter (particle size less than or equal to 10um). The molecular data source of the pollution factor is the pollution factor data obtained through the client (such as an air monitoring station), and then analyzed and processed according to the dimensions of provincial control and townships (including AQI and IAQI parameters. The calculation standard adopts the "Ambient Air Quality Standard" (GB3095-2012) and the Ambient Air Quality Index (AQI) technical regulations to generate pollution assessment indicators and air quality index.

[0026] Specifically, identify and confirm the data sources of pollution factor molecules, pollution assessment indicators, and air quality indexes. Based on different data sources, write corresponding FaaS functions to extract data from the data sources, including: Obtain various pollution factor parameters through the client, and generate assessment indicators, real-time air quality index report data, daily air quality index data, and air quality index category color data based on various pollution factor parameters; In this embodiment, the pollution assessment indicators include target values, cumulative values, compliance rates and other parameters of sulfur dioxide (SO2), nitrogen dioxide (NO2), carbon monoxide (CO), PM2.5, PM10, and ozone (O3).

[0027] Real-time air quality index data refers to the real-time air quality index data generated based on pollution factor parameter data and pollution assessment indicators, such as the 24-hour average value of sulfur dioxide (SO2), the 24-hour average value of nitrogen dioxide (NO2), the 24-hour average value of carbon monoxide (CO), the 24-hour average value of particulate matter (particle size less than or equal to 2.5um), the 24-hour average value of particulate matter (particle size less than or equal to 10um), the maximum 1-hour average value of ozone (O3), the maximum 8-hour sliding average value of ozone (O3), air quality index AQI, etc.

[0028] Daily air quality index data refers to daily air quality index data generated based on pollution factor parameter data and pollution assessment indicators, such as the 1-hour average value of sulfur dioxide (SO2), the 1-hour average value of nitrogen dioxide (NO2), the 1-hour average value of carbon monoxide (CO), the 1-hour average value of particulate matter (particle size less than or equal to 2.5um), the 24-hour sliding average value of particulate matter (particle size less than or equal to 2.5um), the 1-hour average value of particulate matter (particle size less than or equal to 10um), the 24-hour sliding average value of particulate matter (particle size less than or equal to 10um), the 1-hour average value of ozone (O3), the 8-hour sliding average value of ozone (O3), air quality index AQI, etc.

[0029] It should be noted that the calculation method of the air quality index (IAQI) is as follows: ; In the formula, IAQI P Refers to the air quality index of pollutant item P; C P Refers to the mass concentration value of pollutant item P; BP Hi Refers to the air quality sub-index of the corresponding area and the corresponding pollutant item concentration index and C P The higher value of similar pollutant concentration limit; BP Lo Refers to the air quality sub-index of the corresponding area and the corresponding pollutant item concentration index and C P The lower value of similar pollutant concentration limit; IAQI Hi Refers to the air quality sub-index of the corresponding area and the corresponding pollutant item concentration index and BP Hi Corresponding air quality index; IAQI Lo Refers to the air quality sub-index of the corresponding area and the corresponding pollutant item concentration index and BP Lo The corresponding air quality index.

[0030] The Air Quality Index (AQI) is calculated as follows: AQI=max{IAQI1,IAQI2,IAQI3,...,IAQIn}; In the formula, AQI is the maximum value of the air quality index (IAQI) of each pollutant. It should be noted that when AQI is greater than 50, the corresponding pollutant is the primary pollutant, and the pollutant with IAQI greater than 100 is an excessive pollutant. According to the Technical Provisions on Ambient Air Quality Index (AQI) (Trial) (HJ 633-2012), the air pollution index is divided into six levels: 0-50, 51-100, 101-150, 151-200, 201-300 and greater than 300, corresponding to the six levels of air quality. The larger the index, the higher the level, indicating that the pollution is more serious and the impact on human health is more obvious.

[0031] Air quality index category color data refers to air quality index category color data generated based on various pollution factor parameter data; illustratively, the index category colors include green, yellow, orange, red, purple, and maroon, which are calculated based on the air quality index, and different colors represent the six levels of the air quality index.

[0032] The indicators and data generated based on various pollution factor parameters are used as data sources; that is, the above-mentioned pollution factor parameters, indicators and data are used as data sources.

[0033] Based on different data sources, write corresponding FaaS functions to extract data.

[0034] For example, according to the different characteristics of data sources, such as database query, API interface call or log file parsing, corresponding FaaS functions are developed to automatically extract the required data.

[0035] S200, establishing a connection with HDFS through an HDFS client provided by the Hadoop ecosystem, uploading the extracted data to a designated location in HDFS through an application gateway after preprocessing for distributed storage, and displaying it through a visualization tool; First of all, it should be noted that HDFS (Hadoop Distributed File System), as a distributed file system, can efficiently store and manage large-scale data sets.

[0036] Specifically, a connection is established with HDFS through the HDFS client provided by the Hadoop ecosystem, and the extracted data is pre-processed and uploaded to the designated location in HDFS through the application gateway for distributed storage, and displayed through visualization tools, including: Establish a connection with HDFS through the HDFS client provided by the Hadoop ecosystem, and upload the processed data to the specified directory in HDFS for storage; the data storage format uses the Parquet format, and the Parquet file format is used to optimize the read and write performance.

[0037] The HDFS client is part of the Hadoop ecosystem and provides a set of APIs (application programming interfaces) that allow developers to write code to communicate with HDFS. To interact with HDFS, you need to create a connection to the HDFS cluster. This involves configuring the correct host name, port number, and other necessary parameters so that the client can find and connect to the HDFS NameNode. The NameNode is one of the core components in the HDFS architecture and is responsible for managing the file system's namespace and controlling client access to files. During the data extraction phase, the FaaS function can call the HDFS client API to upload preprocessed data packets to HDFS; during the data analysis phase, the required data sets can be read from HDFS for further processing.

[0038] Use the command line tools provided by HDFS to verify whether the data is successfully stored and check the file path, size, and permission information; Through data visualization BI tools, various data stored in HDFS are processed and visualized.

[0039] Through data visualization BI tools, the pollution factor analysis, pollution assessment index analysis, real-time air quality index report, daily air quality index report, and air quality index category color data stored in HDFS are displayed in the form of charts and reports, making it easier for decision makers and researchers to understand the air quality status intuitively.

[0040] Furthermore, the extracted data is pre-processed and uploaded through the application gateway, including: Check whether the extracted data has missing values, outliers or duplicate data, and correct the data; The data is partitioned and indexed based on time and location dimensions, converted into Parquet format, and written to the specified directory in HDFS for storage through application gateway transmission.

[0041] Specifically, the extracted data is preprocessed before being pushed to the application gateway. In this embodiment, the preprocessing includes checking whether there are missing values, abnormal values ​​or duplicate data to ensure data integrity. Abnormal values ​​are removed or corrected, and missing data are processed by averaging and interpolation substitution methods to ensure data quality.

[0042] The data is partitioned according to time, location and other dimensions, and indexes are created. The data is converted to Parquet format and written to the specified directory in HDFS for storage through the gateway. The Gzip compression algorithm is used to compress the data to save storage space and improve data transmission efficiency. For sensitive information data, the RSA encryption algorithm is used to encrypt the data to ensure data security.

[0043] Furthermore, the components MapReduce, Hive, and Spark in the Hadoop ecosystem can be used to conduct in-depth analysis and mining of the pollution factor analysis, pollution assessment index analysis, real-time air quality index report, daily air quality index report, and air quality index category color data stored in HDFS to extract valuable information and patterns.

[0044] S300: Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and predictive analysis is performed through the pre-built environmental prediction big data analysis model.

[0045] Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and the pre-built environmental prediction big data analysis model is used to perform predictive analysis, including: Obtain new data sources, activate corresponding FaaS instances, and trigger FaaS functions to perform predefined processing. The data after predefined processing is stored in HDFS; Based on the new data source, the pre-built environmental protection prediction big data analysis model performs predictive analysis on it.

[0046] When new environmental monitoring data is generated from sensors, weather stations (clients) or other sources, these data will first be collected and transmitted to the cloud data lake or object storage service; once new data is detected, whether it is through an object storage trigger or an HTTP POST request received by the API gateway, the corresponding FaaS function instance will be activated immediately. This ensures that new data can be responded to in a timely manner even during non-working hours.

[0047] For example, the triggered FaaS function will perform a series of predefined operations, including but not limited to: Decompression: If the uploaded file is a compressed package, decompress it first; Parsing structure: read data in CSV, JSON and other formats and convert them into a form suitable for subsequent processing; Quality control: Check data integrity and remove or correct outliers and missing values.

[0048] Standardization: unify units, coding methods, etc. so that data from different sources can be compatible with each other.

[0049] The data that has been initially processed will be further verified for its correctness and consistency, and then partitioned according to characteristics such as time and geographic location. At the same time, indexes will be created for commonly used query fields to optimize future retrieval efficiency. The processed data will be uploaded to HDFS.

[0050] Furthermore, the construction of the environmental protection prediction big data analysis model includes: Collect environmental data, including but not limited to air quality index, water pH value, concentration of various pollutants, heavy metal content in soil, and distribution of organic pollutants; Clean and preprocess environment-related data; Divide the preprocessed data into training set, test set and validation set; Use the data in the training set to train the neural network model, and use the validation set to verify and adjust the model training process; The model is optimized through sparse classification cross entropy loss function, and the optimized and trained neural network model is used as an environmental protection prediction big data analysis model.

[0051] Specifically, before building the big data analysis model for environmental prediction, data collection and preprocessing are performed. This stage requires obtaining environmental data from multiple sources, including but not limited to indicators such as air quality index, water quality pH value, and various pollutant concentrations collected regularly by various environmental monitoring stations, as well as characteristic data such as heavy metal content in soil and distribution of organic pollutants. At the same time, it also involves extracting historical environmental policies, meteorological data and other factors from public databases to fully reflect all aspects of factors affecting environmental quality.

[0052] After obtaining the raw data, data cleaning and preprocessing are performed. This includes removing outliers, handling missing values, standardizing data formats and units from different sources, and filling in missing data caused by equipment failure or extreme weather conditions through interpolation, mean value substitution, etc., to ensure the integrity and reliability of the data set. In addition, statistical methods or machine learning algorithms are used to identify and remove noise components in the data to ensure the accuracy and effectiveness of input data during the modeling process. The collected and preprocessed environmental data are divided into training sets, test sets, and validation sets according to a certain ratio, and then the training set data is used to train the neural network model; the validation set is used to verify and adjust the model during the training process, help monitor the generalization ability of the model and identify overfitting or underfitting phenomena; the test set is independent of the training and validation process, and is used for the final performance evaluation after the model training is completed to ensure the performance and robustness of the model on unseen data.

[0053] During model training, the cross entropy or relative quotient loss function and the Adam optimization algorithm were adopted. When solving multi-classification problems, for example, for data with integer (i.e. sparse label) features, the cross entropy or relative quotient loss function (SparseCategoricalCrossentropy) was selected as the loss function to more accurately measure the difference between the model prediction and the actual label, further enhancing the classification performance of the model. For data such as one-hot encoded category label features, CategoricalCrossentropy is used as the loss function to avoid the problem of gradient vanishing, because when the target value is very different from the predicted value, the gradient of the cross entropy is larger than the gradient of the mean square error, which can make the predicted distribution of the model closer to the true distribution, and has good convergence and robustness when training neural network models.

[0054] By adjusting and optimizing the parameters of the model, targeted improvements are made to the problems that arise during the training process. The network structure is adjusted by increasing or decreasing the number of layers and nodes of the neural network; the training process is optimized by changing hyperparameters such as the learning rate and batch size; and regularization terms, ensemble learning and other technologies are introduced to improve the generalization ability and stability of the model.

[0055] At the same time, the optimized model is verified using the validation set. By comparing the prediction results on the validation set with the actual labels, the reliability, accuracy and robustness of the model in actual application scenarios are evaluated to ensure that the model can correctly handle various complex situations.

[0056] The entire training process of the environmental prediction big data analysis model is carried out on a high-performance workstation equipped with a 2.20 GHz Intel(R) Xeon(R)E5-2699 v4 CPU, 64GB RAM and an NVIDIA A40(48G) GPU. The trained model is exported as a deployable pkl format file to ensure that the model can be correctly read and used during the deployment process. It is integrated and called through API interfaces or Web services, so that the model can respond quickly and make accurate predictions.

[0057] The main application scenarios include environmental monitoring, pollution warning, air quality forecast, etc. These scenarios all involve a large amount of data analysis and processing, and the FaaS function service architecture air pollution prevention and control big data AI analysis system can quickly and accurately provide analysis results and provide scientific basis for decision makers.

[0058] Furthermore, by constructing multi-layer perceptron (MLP), recurrent neural network (RNN), and long short-term memory network (LSTM) neural network models, we can explore and reveal the nonlinear relationship patterns hidden between complex environmental factors, and achieve accurate prediction of pollution factors. Based on characteristic variables such as pollutant concentrations, emission sources, and meteorological conditions in historical environmental monitoring data, we build a decision tree model to predict and classify future environmental quality. By analyzing the degree of influence of characteristic variables on environmental quality, we can find out the key environmental influencing factors and provide a basis for formulating targeted environmental protection measures.

[0059] Among them, the quadratic exponential smoothing forecasting algorithm is a time series forecasting method that predicts future data points by weighted averaging historical data. This algorithm is suitable for environmental data with obvious trends and seasonal changes, such as air quality and water quality. By using the quadratic exponential smoothing forecasting algorithm, the future environmental quality can be accurately predicted based on historical environmental monitoring data. This helps to discover potential environmental problems, formulate corresponding response measures, and provide forward-looking guidance for environmental protection work. At the same time, the algorithm can also evaluate the effectiveness of environmental protection measures and provide a scientific basis for policy making.

[0060] The embodiment of the present invention also provides an air pollution prevention and control big data analysis system based on the FaaS function service architecture, which is used to implement the above method embodiments, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0061] like Figure 2 As shown, Figure 2 This is a structural block diagram of the air pollution prevention and control big data analysis system based on the FaaS function service architecture in the present invention, which includes: A client, for receiving data sources from different sources, wherein the data sources include pollution factor data; The data extraction system is used to extract specific types of pollution factor information from the data obtained from the client, and then package and compress the pollution factor information and upload it to HDFS storage through the application gateway; HDFS storage, used for centralized management and storage of processed data pushed by the data extraction system; Data analysis and processing system, including multiple specialized data processing modules, for in-depth processing and analysis of data in HDFS; The AI ​​algorithm engine, based on the pre-built environmental prediction big data analysis model, generates analysis results based on the newly arrived pollution factor data.

[0062] Furthermore, the data analysis and processing system includes a pollution factor analysis data processing module, a pollution assessment index analysis data processing module, an air quality index real-time report data processing module, an air quality index daily report data processing module, and an air quality index category color data processing module, which is used to generate pollution assessment indicators, air quality index real-time report, air quality index daily report, and air quality index category color data based on pollution factor data processing.

[0063] The pollution assessment index analysis data processing module is used by technical personnel to process the data of the pollution factor analysis data processing module, and generate assessment indicators (the assessment indicators mainly include sulfur dioxide (SO2), nitrogen dioxide (NO2), carbon monoxide (CO), PM2.5, PM10, ozone (O3) target value, cumulative value, compliance rate and other parameters) and push them to the application gateway; similarly, the pollution assessment index analysis data processing module, the air quality index real-time report data processing module, the air quality index daily report data processing module, and the air quality index category color data processing module respectively analyze and process the pollution factor data and push them to the application gateway.

[0064] The data extraction system includes pollution factor analysis data extraction module; pollution assessment index analysis data extraction module; air quality index real-time reporting data extraction module; air quality index daily reporting data extraction module; air quality index category color data extraction module. The data extraction system uses FaaS functions for data extraction.

[0065] Each extraction module in the data extraction system extracts and processes the corresponding data in the application gateway and pushes it to HDFS for storage.

[0066] Specifically, determine the data sources of the pollution factor analysis data processing module, the pollution assessment index analysis data processing module, the air quality index real-time report data processing module, the air quality index daily report data processing module, and the air quality index category color data processing module, write FaaS functions to extract data, and use SQL queries, API calls, and log parsing to extract data according to different data sources. Use the HDFS client provided by Hadoop to establish a connection with HDFS, upload the processed data to the specified directory in HDFS for storage, and select the Parquet file format for data storage. After the upload is completed, use the command line tool provided by HDFS to verify whether the data has been successfully stored, and check the file path, size, permissions, and other information.

[0067] Figure 3 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: Identify and confirm the data sources of pollution factor molecules, pollution assessment indicators, and air quality index; based on different data sources, write corresponding FaaS functions to extract data from the data sources; Establish a connection with HDFS through the HDFS client provided by the Hadoop ecosystem, upload the extracted data to the designated location in HDFS through the application gateway after preprocessing for distributed storage, and display it through visualization tools; Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and predictive analysis is performed through the pre-built environmental prediction big data analysis model.

[0068] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0069] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data analysis method for air pollution prevention and control based on FaaS function service architecture, characterized in that: The analysis method comprises: Identify and confirm the data sources of pollution factor molecules, pollution assessment indicators, and air quality index; based on different data sources, write corresponding FaaS functions to extract data from the data sources; Establish a connection with HDFS through the HDFS client provided by the Hadoop ecosystem, upload the extracted data to the designated location in HDFS through the application gateway after preprocessing for distributed storage, and display it through visualization tools; Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and predictive analysis is performed through the pre-built environmental prediction big data analysis model.

2. The method for analyzing air pollution prevention and control big data based on FaaS function service architecture according to claim 1 is characterized in that: Identify and confirm the data sources of pollution factor molecules, pollution assessment indicators, and air quality index. Based on different data sources, write corresponding FaaS functions to extract data from the data sources, including: Obtain various pollution factor parameters through the client, and generate assessment indicators, real-time air quality index report data, daily air quality index data, and air quality index category color data based on various pollution factor parameters; Indicators and data generated based on various pollution factor parameters are used as data sources; Based on different data sources, write corresponding FaaS functions to extract data.

3. The method for analyzing air pollution prevention and control big data based on FaaS function service architecture according to claim 1 is characterized in that: Establish a connection with HDFS through the HDFS client provided by the Hadoop ecosystem, upload the extracted data to the designated location in HDFS through the application gateway after preprocessing for distributed storage, and display it through visualization tools, including: Use the HDFS client provided by the Hadoop ecosystem to establish a connection with HDFS, and upload the processed data to the specified directory in HDFS for storage; the data storage format is Parquet; Use the command line tools provided by HDFS to verify whether the data is successfully stored and check the file path, size, and permission information; Through data visualization BI tools, various data stored in HDFS are processed and visualized.

4. The method for analyzing air pollution prevention and control big data based on FaaS function service architecture according to claim 3 is characterized in that: The extracted data is pre-processed and uploaded through the application gateway, including: Check whether the extracted data has missing values, outliers or duplicate data, and correct the data; The data is partitioned and indexed based on time and location dimensions, converted into Parquet format, and written to the specified directory in HDFS for storage through application gateway transmission.

5. The method for analyzing air pollution prevention and control big data based on FaaS function service architecture according to claim 1 is characterized in that: Based on the newly acquired data source, the FaaS function is triggered to execute the data analysis process, and the pre-built environmental prediction big data analysis model is used to perform predictive analysis, including: Obtain new data sources, activate corresponding FaaS instances, and trigger FaaS functions to perform predefined processing. The data after predefined processing is stored in HDFS; Based on the new data source, the pre-built environmental protection prediction big data analysis model performs predictive analysis on it.

6. The method for analyzing air pollution prevention and control big data based on FaaS function service architecture according to claim 1 is characterized in that: The construction of the environmental protection prediction big data analysis model includes: Collect environmental data, including but not limited to air quality index, water pH value, concentration of various pollutants, heavy metal content in soil, and distribution of organic pollutants; Clean and preprocess environment-related data; Divide the preprocessed data into training set, test set and validation set; Use the data in the training set to train the neural network model, and use the validation set to verify and adjust the model training process; The model is optimized through sparse classification cross entropy loss function, and the optimized and trained neural network model is used as an environmental protection prediction big data analysis model.

7. A big data analysis system for air pollution prevention and control based on FaaS function service architecture, used to implement the big data analysis method for air pollution prevention and control based on FaaS function service architecture as described in any one of claims 1 to 6, characterized in that: include: A client, for receiving data sources from different sources, wherein the data sources include pollution factor data; The data extraction system is used to extract specific types of pollution factor information from the data obtained from the client, and then package and compress the pollution factor information and upload it to HDFS storage through the application gateway; HDFS storage, used for centralized management and storage of processed data pushed by the data extraction system; Data analysis and processing system, including multiple specialized data processing modules, for in-depth processing and analysis of data in HDFS; The AI ​​algorithm engine, based on the pre-built environmental prediction big data analysis model, generates analysis results based on the newly arrived pollution factor data.

8. The air pollution prevention and control big data analysis system based on FaaS function service architecture according to claim 7 is characterized in that: The data analysis and processing system includes a pollution factor analysis data processing module, a pollution assessment index analysis data processing module, an air quality index real-time report data processing module, an air quality index daily report data processing module, and an air quality index category color data processing module, which are used to generate pollution assessment indicators, air quality index real-time report, air quality index daily report, and air quality index category color data based on pollution factor data processing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the big data analysis method for air pollution prevention and control based on the FaaS function service architecture as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the big data analysis method for air pollution prevention and control based on the FaaS function service architecture are implemented as described in any one of claims 1 to 6.

Images