Sensor alarm data processing method and system based on big data

Abstract

The invention provides a big data-based sensor alarm data processing method and system. The method comprises the following steps of: collecting alarm data related to a sensor in different data sources; cleaning according to data characteristics of the alarm data to obtain preprocessed data; filtering the preprocessed data according to a preset screening rule to obtain effective data; analyzing the potential incidence relation of the effective data through customized labels or features to obtain an analysis result; and generating an operation suggestion according to the data analysis result. According to the method, efficient processing and deep analysis of real-time sensor alarm data are achieved, noise and irrelevant information in the data can be removed through data cleaning, the data false alarm rate is remarkably reduced, the most effective information can be effectively screened out through data filtering, the analysis result is more accurate and meaningful through correlation analysis, and the method is suitable for popularization and application. The operation suggestion is more specific and targeted, and the capability of the user in the aspects of data management and decision making is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method and system for processing sensor alarm data based on big data. Background Art

[0002] In today's information technology field, traditional systems have shown certain limitations in processing alarm data, especially in reducing the false alarm rate, which is not satisfactory. Since the processing and analysis of big data rely on efficient algorithms and powerful computing capabilities, while traditional methods are slow in data processing and analysis and cannot achieve real-time response to alarms. In addition, existing systems often cannot provide specific operation suggestions after analyzing the data, making it difficult for them to effectively guide the actual work process. Therefore, when facing the analysis results, users often find it difficult to take specific and effective actions based on these results, thus affecting the overall work efficiency and decision-making quality. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a method and system for processing sensor alarm data based on big data to solve the existing technical problems.

[0004] To solve the above technical problems, the present invention provides the following technical solutions:

[0005] In a first aspect, the present invention provides a method for processing sensor alarm data based on big data, including:

[0006] Collect alarm data related to sensors from different data sources;

[0007] Clean the alarm data according to the data characteristics of the alarm data to obtain preprocessed data;

[0008] Filter the preprocessed data according to preset screening rules to obtain valid data;

[0009] Obtain analysis results by customizing tags or feature analyzing potential correlation relationships of the valid data;

[0010] Generate operation suggestions according to the data analysis results.

[0011] In one embodiment, the cleaning the alarm data according to the data characteristics of the alarm data to obtain preprocessed data includes:

[0012] Check the alarm data to obtain data characteristics, where the data characteristics include data structure, data fields, missing values, and outliers;

[0013] Regard null values or placeholders in the alarm data as missing values, add marks to the missing values, and determine whether the missing values affect the analysis. If so, fill the missing values with mean, median, mode, or predicted values;

[0014] If not, delete the relevant records or fields;

[0015] Identify outliers through statistical methods, where the statistical methods include standard deviation and box plot. If the outlier is an obvious error and cannot be corrected, delete it; if the outlier can be corrected, correct the outlier according to the context; if the outlier is reasonable, retain and record it;

[0016] Find the data that is completely or partially repeated in the alarm data. For the completely repeated data, retain one and delete one; for the partially repeated data, perform a merge process;

[0017] Find the data with field values that do not conform to the business rules in the alarm data as inconsistent data, and determine whether the inconsistent data can be corrected. If so, correct the inconsistent data according to the business rules or the context; if not, delete the inconsistent data;

[0018] Check whether the data meets the analysis requirements, and save the preprocessed data after cleaning as a file or import it into the database.

[0019] In one embodiment, filtering the preprocessed data according to the preset screening rules to obtain valid data includes:

[0020] Set a first filtering condition according to the analysis target, where the first filtering condition includes a time range, a region, and a product category;

[0021] Set a second filtering condition according to the data quality requirements, where the second filtering condition includes removing missing values and removing outliers;

[0022] Use a screening tool to screen the data that meets the conditions according to the first filtering condition and / or the second filtering condition, where the screening tool includes SQL query, programming language, or Excel tool;

[0023] Combine multiple conditions according to a logical expression for screening, save the data that meets the conditions as a new data set, and record the filtering conditions and processing results. Check the integrity and data quality of the new data set, and generate a filtering report, where the filtering report includes filtering conditions, processing results, and data quality;

[0024] Save the new data set as valid data as a file or import it into the database.

[0025] In one embodiment, it further includes:

[0026] Visually display the analysis results and operation suggestions.

[0027] In a second aspect, the present invention provides a sensor alarm data processing system based on big data, including:

[0028] Data collection module: used to collect alarm data related to sensors from different data sources;

[0029] Data cleaning module: used to clean according to the data characteristics of the alarm data to obtain preprocessed data;

[0030] Data filtering module: used to filter the preprocessed data according to preset screening rules to obtain valid data;

[0031] Data analysis module: used to obtain analysis results by customizing tags or analyzing potential correlation relationships of valid data through feature analysis;

[0032] Operation suggestion module: used to generate operation suggestions according to the data analysis results.

[0033] In one embodiment, the data cleaning module is specifically used for:

[0034] Check the alarm data to obtain data characteristics, where the data characteristics include data structure, data fields, missing values, and outliers;

[0035] Regard null values or placeholders in the alarm data as missing values, add marks to the missing values, and judge whether the missing values affect the analysis. If so, fill the missing values with the mean, median, mode, or predicted value;

[0036] If not, delete the relevant records or fields;

[0037] Identify outliers through statistical methods, where the statistical methods include standard deviation and box plot. If the outliers are obvious errors and cannot be corrected, delete them; if the outliers can be corrected, correct the outliers according to the context; if the outliers are reasonable, retain and record them;

[0038] Find completely duplicate or partially duplicate data in the alarm data. For completely duplicate data, retain one and delete one; for partially duplicate data, perform a merge process;

[0039] Find data with field values that do not conform to business rules in the alarm data as inconsistent data, and judge whether the inconsistent data can be corrected. If so, correct the inconsistent data according to business rules or the context; if not, delete the inconsistent data;

[0040] Check whether the data meets the analysis requirements, and save the cleaned preprocessed data as a file or import it into the database.

[0041] In one embodiment, the data filtering module is specifically used for:

[0042] Set a first filtering condition according to the analysis target, where the first filtering condition includes time range, region, and product category;

[0043] Set the second filtering condition according to the data quality requirements, where the second filtering condition includes removing missing values and removing outliers;

[0044] Filter the qualified data according to the first filtering condition and / or the second filtering condition using a screening tool, where the screening tool includes SQL query, programming language, or Excel tool;

[0045] Combine multiple conditions according to logical expressions for screening, save the qualified data as a new data set, record the filtering conditions and processing results, check the integrity and data quality of the new data set, and generate a filtering report, where the filtering report includes filtering conditions, processing results, and data quality;

[0046] Save the new data set as a file or import it into the database as valid data.

[0047] In one embodiment, the method further includes:

[0048] Data display module: Visualize the analysis results and operation suggestions.

[0049] In a third aspect, the present invention provides an electronic device, including: a processor and a memory;

[0050] The memory is used to store a computer program;

[0051] The processor is used to execute a method for processing sensor alarm data based on big data provided in any one of the first aspects by calling the computer program.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a program, and the program is used to implement a method for processing sensor alarm data based on big data provided in any one of the first aspects when executed by a processor.

[0053] As can be seen from the above description, the embodiments of the present invention provide a method and system for processing sensor alarm data based on big data. Compared with the existing technologies, the present invention has significantly improved the efficiency and effect of big data processing and analysis, providing users with more powerful and convenient data analysis. The present invention realizes the efficient processing and in-depth analysis of real-time sensor alarm data. Through data cleaning, noise and irrelevant information in the data can be removed, thereby significantly reducing the data false alarm rate. Through data filtering, the most effective information can be effectively screened out, and the association analysis technology further enhances the connection between data, making the analysis results more accurate and meaningful. In terms of data visualization, complex data analysis results become easy to understand, and users can clearly grasp the data dynamics at a glance. The proposed operation suggestions are more specific and targeted, greatly improving the user's ability in data management and decision-making. Brief Description of the Drawings

[0054] Figure 1 The figure shows a schematic flow diagram of a method for processing sensor alarm data based on big data provided by an embodiment of the present invention;

[0055] Figure 2 The figure shows a schematic structural diagram of a system for processing sensor alarm data based on big data provided by an embodiment of the present invention;

[0056] Figure 3 The figure shows a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed Embodiments

[0057] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without any creative efforts shall fall within the scope of protection of the present invention.

[0058] Based on the disadvantages of the prior art, an embodiment of the present invention provides a specific implementation of a method for processing sensor alarm data based on big data, as Figure 1 shown, the method includes the following steps:

[0059] S110: Collect alarm data related to sensors from different data sources.

[0060] The data sources include but are not limited to database data sources, API data sources, or file data sources, etc. For database data sources, SQL statements can be used for querying, and relevant fields of alarm data can be filtered through filtering statements. For API data sources, HTTP requests can be used to obtain the alarm data of the required sensors. File data sources include but are not limited to CSV files, JSON files, and XML files, etc., and their respective corresponding libraries can be used to obtain the alarm data of sensors.

[0061] S120: Clean the alarm data according to the data characteristics of the alarm data to obtain preprocessed data.

[0062] Specifically, the data cleaning includes the following steps:

[0063] Check the alarm data to obtain data characteristics, and the data characteristics include data structure, data fields, missing values, and outliers;

[0064] Regard null values or placeholders in the alarm data as missing values, add marks to the missing values, and determine whether the missing values affect the analysis. If so, fill the missing values with the mean, median, mode, or predicted values;

[0065] If not, delete the relevant records or fields;

[0066] Identify outliers through statistical methods, including standard deviation and box plots. If the outliers are obvious errors and cannot be corrected, delete them; if the outliers can be corrected, correct the outliers according to the context; if the outliers are reasonable, retain and record them;

[0067] Find completely duplicate or partially duplicate data in the alarm data. For completely duplicate data, retain one and delete one; for partially duplicate data, merge and process it;

[0068] Find the data with field values that do not conform to the business rules in the alarm data as inconsistent data, and determine whether the inconsistent data can be corrected. If so, correct the inconsistent data according to the business rules or context; if not, delete the inconsistent data;

[0069] Check whether the data meets the analysis requirements, and save the preprocessed data after cleaning as a file or import it into the database.

[0070] When checking the data, standardize the data format of the preprocessed data to ensure that the formats of fields such as dates, times, and currencies are consistent, and convert the preprocessed data into appropriate types, such as converting strings to numerical values.

[0071] In this step, a standardized method for full inspection and preprocessing of sensor alarm data is proposed, which can greatly improve the data quality and enhance the analysis results. The standardized data cleaning process can improve the processing efficiency and reduce the workload of manual inspection and data correction. Automated operations such as missing value processing and outlier identification can quickly complete data cleaning and save time. Detecting and solving data problems in the early stage of data processing avoids wrong decisions caused by data quality problems in subsequent analysis and decision-making processes, thus reducing the error cost. By reasonably processing missing values, it is possible to avoid analysis biases caused by data missing. Using statistical methods to identify and process outliers can prevent outliers from misleading the overall data analysis results. Deleting completely duplicate data and merging partially overlapping data can effectively avoid data redundancy and improve the analysis efficiency of the provincial bureau. By checking and correcting inconsistent data that does not conform to the business rules, the accuracy and consistency of the data are ensured, making the analysis more reliable.

[0072] S130: Filter the preprocessed data according to the preset screening rules to obtain valid data.

[0073] Specifically, data filtering includes the following steps:

[0074] Set the first filtering condition according to the analysis objective. The first filtering condition includes the time range, region, and product category;

[0075] Set the second filtering condition according to the data quality requirements. The second filtering condition includes removing missing values and removing outliers.

[0076] Use a screening tool to screen the qualified data according to the first filtering condition and / or the second filtering condition. The screening tool includes SQL query, programming language or Excel tool.

[0077] Combine multiple conditions according to logical expressions (such as AND, OR) for screening, save the qualified data as a new data set, record the filtering conditions and processing results, check the integrity and data quality of the new data set, and generate a filtering report. The filtering report includes filtering conditions, processing results and data quality.

[0078] Save the new data set as a file or import it into the database as valid data.

[0079] In this step, a filtering process for preprocessed data is provided, which guarantees data quality, provides flexible and efficient screening tools, and realizes traceability and data management. It can improve the efficiency and accuracy of data analysis, and provide more powerful data support for subsequent correlation analysis. Through the first filtering condition, the screened data can accurately meet the analysis requirements, avoid interference from irrelevant data, help focus on specific business scenarios, and thus improve the effectiveness of correlation analysis. Through the second filtering condition, invalid information can be removed, the reliability and stability of the data can be enhanced, and the ability of the analysis results to reflect the real situation can be improved, thereby enhancing the credibility of the data and enabling decision-makers to make decisions more confidently based on the analysis results. Providing diverse screening tools can meet the skill levels and operation needs of different users, and supports combining multiple conditions for screening according to logical expressions, greatly improving the flexibility and accuracy of screening. The filtering report enables the data screening process to be traceable, facilitating subsequent verification. It not only ensures that the new data set meets the analysis requirements, but also improves the data management efficiency, laying a good foundation for subsequent data analysis and use.

[0080] S140: Analyze the potential correlation relationships of the valid data through customized tags or feature analysis to obtain the analysis results.

[0081] Determine the analysis objective according to business requirements, customize tags for valid data according to the analysis objective, such as tags for alarm type, severity, occurrence event, occurrence location, related devices or services, etc. Determine the features (numerical features and text features) that reflect the essential characteristics of the tags according to the customized tags. Obtain the valid data related to the determined tags and features from the database, and apply the formulated tags and features to specific valid data instances, so that each piece of valid data has clear tags and feature values for subsequent analysis. Use big data processing technologies (Hadoop and Spark, Apache Flink or Storm) to process and analyze the potential correlation relationships of the valid data to obtain the analysis results.

[0082] S150: Generate operation suggestions based on the data analysis results.

[0083] Compare the analysis results with the standard data, and use artificial intelligence technology to automatically generate more accurate and personalized operation suggestions.

[0084] S160: Visualize the analysis results and operation suggestions.

[0085] When performing visualization, different data visualization tools and methods can be used, such as D3.js or Tableau.

[0086] As can be seen from the above, compared with the existing technologies, the present invention has significantly improved the efficiency and effectiveness of big data processing and analysis, providing users with more powerful and convenient data analysis. The present invention realizes the efficient processing and in-depth analysis of real-time sensor alarm data. Through data cleaning, noise and irrelevant information in the data can be removed, thus significantly reducing the data false alarm rate. Through data filtering, the most effective information can be effectively screened out, and the correlation analysis technology further enhances the connection between data, making the analysis results more accurate and meaningful. In terms of data visualization, complex data analysis results become easy to understand, and users can clearly grasp the data dynamics at a glance. The proposed operation suggestions are more specific and targeted, greatly improving the user's ability in data management and decision-making.

[0087] Based on the same inventive concept, the embodiments of the present application further provide a sensor alarm data processing system based on big data, which can be used to implement a sensor alarm data processing method based on big data described in the above embodiments, as described in the following embodiments. Since the principle of a sensor alarm data processing system based on big data to solve problems is similar to that of a sensor alarm data processing method based on big data, the implementation of the system can refer to the method implementation, and the repeated parts will not be described again. As used below, the term "unit" or "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0088] As Figure 2 shown, the present invention provides a sensor alarm data processing system based on big data. In Figure 2 it, the system includes:

[0089] A data collection module 210: configured to collect alarm data related to sensors in different data sources;

[0090] A data cleaning module 220: configured to clean according to the data characteristics of the alarm data to obtain preprocessed data;

[0091] A data filtering module 230: configured to filter the preprocessed data according to a preset screening rule to obtain valid data;

[0092] A data analysis module 240: configured to obtain an analysis result by customizing tags or analyzing potential association relationships of the valid data;

[0093] An operation suggestion module 250: configured to generate operation suggestions according to the analysis result.

[0094] In an embodiment of the present invention, the data cleaning module 220 is specifically configured to:

[0095] Check the alarm data to obtain data characteristics, where the data characteristics include data structure, data fields, missing values, and outliers;

[0096] Regard null values or placeholders in the alarm data as missing values, add marks to the missing values, and determine whether the missing values affect the analysis. If so, fill the missing values with mean, median, mode, or predicted values;

[0097] If not, delete the relevant records or fields;

[0098] Identify outliers through statistical methods, where the statistical methods include standard deviation and box plot. If the outliers are obvious errors and cannot be corrected, delete them; if the outliers can be corrected, correct the outliers according to the context; if the outliers are reasonable, retain and record them.

[0099] Find the completely duplicate or partially duplicate data in the alarm data. For the completely duplicate data, keep one and delete the other; for the partially duplicate data, perform a merge process;

[0100] Find the data with field values not conforming to the business rules in the alarm data as inconsistent data, and determine whether the inconsistent data can be corrected. If so, correct the inconsistent data according to the business rules or context; if not, delete the inconsistent data;

[0101] Check whether the data meets the analysis requirements, and save the preprocessed data after cleaning as a file or import it into the database.

[0102] In an embodiment of the present invention, the data filtering module 230 is specifically configured to:

[0103] Set a first filtering condition according to the analysis target, and the first filtering condition includes a time range, a region, and a product category;

[0104] Set a second filtering condition according to the data quality requirements, and the second filtering condition includes removing missing values and removing outliers;

[0105] Use a screening tool to screen the data that meets the conditions according to the first filtering condition and / or the second filtering condition, and the screening tool includes SQL query, programming language, or Excel tool;

[0106] Combine multiple conditions according to logical expressions for screening, save the data that meets the conditions as a new data set, record the filtering conditions and processing results, check the integrity and data quality of the new data set, and generate a filtering report. The filtering report includes filtering conditions, processing results, and data quality;

[0107] Save the new data set as valid data as a file or import it into the database.

[0108] In an embodiment of the present invention, it further includes:

[0109] The data display module 260: visually display the analysis results and operation suggestions.

[0110] The embodiments of the present application also provide a specific implementation manner of an electronic device capable of implementing all the steps in the method in the above embodiments. Refer to Figure 3 The electronic device 300 specifically includes the following contents:

[0111] A processor 310, a memory 320, a communication unit 330, and a bus 340;

[0112] Among them, the processor 310, the memory 320, and the communication unit 330 complete mutual communication through the bus 340; the communication unit 330 is used to realize information transmission between related devices such as the server-side device and the terminal device.

[0113] The processor 310 is used to call the computer program in the memory 320. When the processor executes the computer program, all steps in a method for processing sensor alarm data based on big data in the above embodiment are realized.

[0114] Those of ordinary skill in the art should understand that the memory can be, but is not limited to, random access memory (Random Access Memory, abbreviated as RAM), read-only memory (Read Only Memory, abbreviated as ROM), programmable read-only memory (Programmable Read-Only Memory, abbreviated as PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. Among them, the memory is used to store programs, and the processor executes the programs after receiving execution instructions. Further, the software programs and modules in the above memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide a running environment for other software components.

[0115] The processor can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0116] The present application also provides a computer-readable storage medium, and the computer-readable storage medium includes a program, and the program is used to execute a method for processing sensor alarm data based on big data provided by any one of the foregoing method embodiments when executed by a processor.

[0117] Those of ordinary skill in the art should understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes, and the specific type of the medium is not limited in this application.

[0118] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for processing sensor alarm data based on big data, characterized in that Including: Collect alarm data related to sensors from different data sources; Clean the alarm data according to the data characteristics of the alarm data to obtain preprocessed data; Filter the preprocessed data according to preset screening rules to obtain valid data; Obtain the analysis result by customizing tags or analyzing the potential correlation relationship of the valid data through feature analysis; Generate operation suggestions according to the data analysis result.

2. The method for processing sensor alarm data based on big data according to claim 1, wherein, The cleaning of the alarm data according to the data characteristics of the alarm data to obtain preprocessed data includes: Check the alarm data to obtain data characteristics, and the data characteristics include data structure, data fields, missing values, and outliers; Regard the null values or placeholders in the alarm data as missing values, add marks to the missing values, and judge whether the missing values affect the analysis. If so, fill the missing values with the mean, median, mode, or predicted values; If not, delete the relevant records or fields; Identify outliers through statistical methods, and the statistical methods include standard deviation and box plot. If the outliers are obvious errors and cannot be corrected, delete them; if the outliers can be corrected, correct the outliers according to the context; if the outliers are reasonable, retain and record them; Find the completely duplicate or partially duplicate data in the alarm data. For the completely duplicate data, retain one and delete one; for the partially duplicate data, perform merging processing; Find the data with field values that do not conform to the business rules in the alarm data as inconsistent data, and judge whether the inconsistent data can be corrected. If so, correct the inconsistent data according to the business rules or context; if not, delete the inconsistent data; Check whether the data meets the analysis requirements, and save the cleaned preprocessed data as a file or import it into the database.

3. The method for processing sensor alarm data based on big data according to claim 1, wherein The filtering of the preprocessed data according to preset screening rules to obtain valid data includes: Set the first filtering condition according to the analysis target, and the first filtering condition includes time range, region, product category; Set the second filtering condition according to the data quality requirements, and the second filtering condition includes removing missing values and removing outliers; Use a screening tool to screen the data that meets the conditions according to the first filtering condition and / or the second filtering condition, and the screening tool includes SQL query, programming language, or Excel tool; Combine multiple conditions according to logical expressions for screening, save the data that meets the conditions as a new data set, record the filtering conditions and processing results, check the integrity and data quality of the new data set, and generate a filtering report. The filtering report includes filtering conditions, processing results, and data quality; Save the new data set as valid data as a file or import it into the database.

4. The method for processing sensor alarm data based on big data according to claim 1, characterized in that It also includes: Visually display the analysis result and operation suggestions.

5. A sensor alarm data processing system based on big data, characterized in that, Including: Data collection module: used to collect alarm data related to sensors from different data sources; Data cleaning module: used to clean the alarm data according to the data characteristics of the alarm data to obtain preprocessed data; Data filtering module: used to filter the preprocessed data according to preset screening rules to obtain valid data; Data analysis module: used to obtain the analysis result by customizing tags or analyzing the potential correlation relationship of the valid data through feature analysis; Operation suggestion module: used to generate operation suggestions according to the data analysis result.

6. The sensor alarm data processing system based on big data according to claim 5, characterized in that, The data cleaning module is specifically used for: Check the alarm data to obtain data features, where the data features include data structure, data fields, missing values, and outliers; Take null values or placeholders in the alarm data as missing values, add marks to the missing values, and determine whether the missing values affect the analysis. If so, fill the missing values with the mean, median, mode, or predicted values; If not, delete the relevant records or fields; Identify outliers through statistical methods, where the statistical methods include standard deviation and box plot. If the outliers are obvious errors and cannot be corrected, delete them; if the outliers can be corrected, correct the outliers according to the context; If the outliers are reasonable, retain and record them; Find the data that is completely or partially repeated in the alarm data. For the completely repeated data, retain one and delete the other; for the partially repeated data, perform a merge process; Find the data with field values that do not conform to the business rules in the alarm data as inconsistent data, and determine whether the inconsistent data can be corrected. If so, correct the inconsistent data according to the business rules or the context; if not, delete the inconsistent data; Check whether the data meets the analysis requirements, and save the preprocessed data after cleaning as a file or import it into the database.

7. The sensor alarm data processing system based on big data according to claim 5, wherein, The data filtering module is specifically used for: Set the first filtering condition according to the analysis objective, where the first filtering condition includes time range, region, and product category; Set the second filtering condition according to the data quality requirements, where the second filtering condition includes removing missing values and removing outliers; Use a screening tool to screen the data that meets the conditions according to the first filtering condition and / or the second filtering condition, where the screening tool includes SQL query, programming language, or Excel tool; Combine multiple conditions according to logical expressions for screening, save the data that meets the conditions as a new data set, record the filtering conditions and processing results, check the integrity and data quality of the new data set, and generate a filtering report, where the filtering report includes filtering conditions, processing results, and data quality; Save the new data set as valid data as a file or import it into the database.

8. The sensor alarm data processing system based on big data according to claim 5, wherein It further includes: Data display module: Visualize the analysis results and operation suggestions.

9. An electronic device, characterized in that, It includes: A processor and a memory; The memory is used to store computer programs; The processor is used to execute a method for processing sensor alarm data based on big data as described in any one of claims 1 to 4 by calling the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program, and the program is used to implement a method for processing sensor alarm data based on big data as described in any one of claims 1 to 4 when executed by the processor.

Images