Data processing system and method

Abstract

The invention discloses a data processing system and method, and relates to the field of data processing systems, and the system comprises a data storage platform which is used for building a cloud storage platform and storing data on a cloud server; the data processing module is used for collecting, classifying, cleaning, converting, aggregating and maintaining; and the data cleaning module aims to eliminate noise, abnormal values and repeated data in the data so as to ensure the quality and the accuracy of the data. According to the system, a cloud storage platform is established, data are stored on a cloud server, a data processing module collects, classifies, cleans, converts, aggregates and maintains the data, and meanwhile, the collected data are cleaned; comprising repeated data cleaning, incomplete data cleaning, data format cleaning, error data cleaning, relevance data cleaning and monitoring and solving methods for solving incomplete data and error values, so that the accuracy and consistency of data are ensured, and the effective rate of the data is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing systems, and in particular to a data processing system and method. Background Art

[0002] Data is the result of facts or observations. It is a logical summary of objective things and is the unprocessed raw material used to represent objective things. Data can be continuous values, such as sound and images, which are called analog data; it can also be discrete, such as symbols and text, which are called digital data. It not only refers to numbers in a narrow sense, but also can be a combination of text, letters, digital symbols, graphics, images, videos, audio, etc. with certain meanings. It is also an abstract representation of the attributes, quantity, position and their mutual relationships of objective things. For example, "0, 1, 2...", "cloudy, rainy, falling, temperature", "student file records, cargo transportation conditions", etc. are all data. After processing, data becomes information. In computer science, data is a general term for all symbolic media that can be input into computers and processed by computer programs. It is a general term for numbers, letters, symbols and analog quantities with certain meanings that are used to be input into electronic computers for processing. The objects stored and processed by computers are very wide, and the data representing these objects has become more and more complex. Data needs to be processed before it can be used.

[0003] The current data processing system has a poor data cleaning effect and the cleaning content is not comprehensive, which makes the data defective and the efficiency is low. Therefore, a data processing system and method are urgently needed. Summary of the invention

[0004] The purpose of the present invention is to provide a data processing system and method to solve the problems that the current data processing system has poor data cleaning effect and incomplete cleaning content, resulting in data defects, low efficiency and the data processing system cannot guarantee normal operation all the time, and requires regular fault detection and timely processing after the fault exists.

[0005] To achieve the above object, the present invention provides the following technical solution: a data processing system and method, comprising: Data storage platform: Establish a cloud storage platform to store data on cloud servers; Data processing module: including collection, classification, cleaning, conversion, aggregation and maintenance processing; Data cleaning module: The purpose of data cleaning is to eliminate noise, outliers and duplicate data in the data to ensure the quality and accuracy of the data; Fault detection unit: real-time tracking of the data processing process, and judging whether there is a fault in the processing system according to the processing flow; Fault debugging and analysis module: Analyze fault data based on the detection data provided by the fault detection unit, determine the problems in the processing method, and select system part adjustments or switch the processing system according to the fault problem; Switching of processing methods: Prepare alternative data processing systems for emergency situations in advance, and replace them with alternative systems when problems that cannot be solved by the current processing system occur; Method detection unit: After switching the system, it is necessary to perform stability detection on the system operation by running benchmark tests and stress tests to ensure the normal operation of the system.

[0006] Preferably, in the data storage platform, the cloud storage platform has the characteristics of easy expansion, high reliability and powerful computing and analysis capabilities. In the cloud storage platform, users can choose different service providers according to their own needs.

[0007] Preferably, in the data processing module, data collection: collecting raw data from various sources; data classification is the process of merging data together according to their common attributes or characteristics, in order to better organize, manage and use data; data cleaning: deleting duplicate data, processing missing values, identifying and processing outliers to ensure data accuracy and consistency, data conversion: converting data from one format or structure to another; data aggregation: merging data from multiple data sources into a consistent format, usually accompanied by data cleaning and conversion steps; data maintenance includes backing up system data, restoring the database system, generating user information tables, and authorizing the information tables, monitoring system operation status, handling system errors in a timely manner, and ensuring system data security.

[0008] Preferably, the data cleaning module includes duplicate data cleaning, incomplete data cleaning, data format cleaning, erroneous data cleaning, correlation data cleaning, and monitoring and solutions for incomplete data and erroneous values.

[0009] Preferably, in the fault detection unit, the data processing process is tracked in real time, and whether there is a fault in the processing system is determined based on the processing flow.

[0010] Preferably, in the fault debugging and analysis module, the data processing software is generally run for detection. Software failure generally refers to the incompatibility or damage of the system software, which causes the processing system to be unable to start or work normally. Virus failure is because the files in the computer system are infected with viruses, and the computer system cannot work normally after the virus attack. Failures caused by viruses can be prevented and viruses can be detected and killed by anti-virus software and anti-virus systems. For highly destructive viruses, regular inspections should be carried out to prevent the computer system from being damaged and causing irreparable losses.

[0011] Preferably, in the switching of the processing method, an alternative data processing system is prepared in advance for emergency situations. When there is a problem that cannot be solved in the currently used processing system, the controller will replace the backup system after authorization and send a fault signal to the PC end for prompting. Preferably, in the method detection unit, running a benchmark test refers to implementing a quantitative and comparable test on a certain performance indicator of a class of test objects by designing scientific test methods, test tools and test systems. Stress testing is usually used to test the performance and stability of an application or system under high load and high stress conditions.

[0012] Preferably, the data processing method comprises the following steps: Step 1: The data storage platform obtains and stores data information; Step 2: Collect, classify, clean, transform, aggregate and maintain data; Step 3: Clean the collected data to improve data efficiency: Step 4: Data processing system fault detection; Step 5: Replace the faulty data processing system.

[0013] Compared with the prior art, the present invention has the following beneficial effects: In the present invention, a cloud storage platform is established to store data on a cloud server. The data processing module collects, classifies, cleans, converts, aggregates and maintains the data. At the same time, the collected data is cleaned, including duplicate data cleaning, incomplete data cleaning, data format cleaning, erroneous data cleaning, related data cleaning, monitoring and solutions for incomplete data and erroneous values, so as to ensure the accuracy and consistency of the data and improve the efficiency of the data.

[0014] In the present invention, the data processing system is checked through software detection and virus detection, the data processing process is tracked in real time, and whether the processing system has a fault is judged and analyzed according to the processing flow. An alternative data processing system for emergency situations is prepared in advance. When there are unsolvable problems with the currently used processing system, the controller is authorized to replace the backup system and send a fault signal to the PC for prompting. By running benchmark tests and stress tests, the stability of the switched processing system is tested to ensure the normal operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 , a data processing system and method, comprising: Data storage platform: Establish a cloud storage platform to store data on cloud servers; Data processing module: including collection, classification, cleaning, conversion, aggregation and maintenance processing; Data cleaning module: The purpose of data cleaning is to eliminate noise, outliers and duplicate data in the data to ensure the quality and accuracy of the data; Fault detection unit: real-time tracking of the data processing process, and judging whether there is a fault in the processing system according to the processing flow; Fault debugging and analysis module: Analyze fault data based on the detection data provided by the fault detection unit, determine the problems in the processing method, and select system part adjustments or switch the processing system according to the fault problem; Switching of processing methods: Prepare alternative data processing systems for emergency situations in advance, and replace them with alternative systems when problems that cannot be solved by the current processing system occur; Method detection unit: After switching the system, it is necessary to perform stability detection on the system operation by running benchmark tests and stress tests to ensure the normal operation of the system.

[0018] A method for data processing comprises the following steps: Step 1: The data storage platform obtains and stores data information; Step 2: Collect, classify, clean, transform, aggregate and maintain data; Step 3: Clean the collected data to improve data efficiency: Step 4: Data processing system fault detection; Step 5: Replace the faulty data processing system.

[0019] Example 1 As a preferred embodiment of the present invention: in the data storage platform, the cloud storage platform has the characteristics of easy expansion, high reliability and powerful computing and analysis capabilities. In the cloud storage platform, users can choose different service providers according to their own needs; Cloud storage is a new concept that is extended and derived from the concept of cloud computing. Cloud computing is the development of distributed processing, parallel processing and grid computing. It is to automatically split the huge computing processing program into countless smaller sub-programs through the network, and then hand it over to a large system composed of multiple servers for calculation and analysis, and then send the processing results back to the user. Through cloud computing technology, network service providers can process tens of millions or even billions of information within seconds, achieving network services as powerful as "supercomputers". In the cloud storage platform, users can choose different service providers according to their own needs.

[0020] Example 2 As a preferred embodiment of the present invention: in the data processing module, data collection: collecting raw data from various sources; data classification is the process of merging data together according to their common attributes or characteristics, in order to better organize, manage and use data; data cleaning: deleting duplicate data, processing missing values, identifying and processing outliers to ensure data accuracy and consistency, data conversion: converting data from one format or structure to another; data aggregation: merging data from multiple data sources into a consistent format, usually accompanied by data cleaning and conversion steps; data maintenance includes backing up system data, restoring the database system, generating user information tables, and authorizing information tables, monitoring system operation status, timely handling system errors, and ensuring system data security; In the data processing module, data collection: collecting raw data from various sources; data classification is the process of grouping data together based on their common attributes or characteristics, in order to better organize, manage and use data; data cleaning: deleting duplicate data, processing missing values, identifying and processing outliers to ensure data accuracy and consistency; data conversion: converting data from one format or structure to another; data aggregation: merging data from multiple data sources into a consistent format, usually accompanied by data cleaning and conversion steps; data maintenance includes backing up system data, restoring the database system, generating user information tables, and authorizing information tables, monitoring system operation status, handling system errors in a timely manner, and ensuring system data security.

[0021] Example 3 As a preferred embodiment of the present invention: the data cleaning module includes duplicate data cleaning, incomplete data cleaning, data format cleaning, error data cleaning, correlation data cleaning, monitoring and solution of incomplete data and error values; Duplicate data cleaning: Detect and delete duplicate data through sorting algorithms and predefined duplicate identification rules, while ensuring the integrity of the original data and backing up the deleted data; Incomplete data cleaning: Develop a plan based on the missing ratio and field importance, delete unnecessary fields or fill in the missing values ​​of important fields through calculation and analysis; Data format cleaning: For information collected manually or filled in by users, clean up the format and content that does not comply with the regulations, such as the display format of time, date, and numerical values; Error data cleaning: Use statistical analysis methods to identify erroneous values ​​or outliers, such as deviation analysis, or use simple rule bases and constraints between data to detect errors; Correlation data cleaning: Check the correlation between data to ensure logical consistency and accuracy between data; Solve incomplete data: For missing values, you may need to fill them in manually or derive them from other data sources, such as using the average, maximum, minimum or probability estimate to replace the missing value; Error value detection and solution: Identify erroneous values ​​or outliers through statistical analysis methods and correct them.

[0022] Example 4 As a preferred embodiment of the present invention: in the fault detection unit, the data processing process is tracked in real time, and it is determined whether there is a fault in the processing system according to the processing flow; Check the data processing system through software detection and virus detection, track the data processing process in real time, and judge and analyze whether there is any fault in the processing system based on the processing flow.

[0023] Example 5 As a preferred embodiment of the present invention: in the fault debugging and analysis module, the data processing software is generally tested. Software faults generally refer to the incompatibility or damage of the system software, which causes the processing system to fail to start or work properly. Virus faults are caused by files in the computer system being infected with viruses, and the computer system cannot work properly after the virus attacks. Faults caused by viruses can be prevented and checked by anti-virus software and anti-virus systems. Viruses with strong destructiveness should be checked regularly to prevent the computer system from being damaged and causing irreparable losses. Software failure generally refers to the incompatibility of system software or damage that causes the processing system to be unable to start or work properly. Virus failure is because the files in the computer system are infected with viruses, and the computer system cannot work properly after the virus attack. Failures caused by viruses can be prevented and checked with anti-virus software and anti-virus systems. For more destructive viruses, regular checks should be carried out to prevent damage to the computer system and cause irreparable losses.

[0024] Example 6 As a preferred embodiment of the present invention: during the processing method switching, an alternative data processing system is prepared in advance for emergency situations. When there are unsolvable problems with the currently used processing system, the controller replaces the backup system after authorization and sends a fault signal to the PC end for prompting; Prepare an alternative data processing system for emergency situations in advance. When the current processing system has unsolvable problems, the controller will replace the backup system after authorization and send a fault signal to the PC for prompting. The alternative data processing system has the same running program as the main processing system and the processing effect is consistent.

[0025] Example 7 As a preferred embodiment of the present invention: in the method detection unit, running a benchmark test means implementing a quantitative and comparable test on a certain performance indicator of a class of test objects by designing a scientific test method, test tool and test system, and stress testing is usually used to test the performance and stability of an application or system under high load and high stress conditions; Benchmarking is to conduct quantitative and comparable testing on a certain performance indicator of a class of test objects by designing scientific testing methods, testing tools and testing systems. Stress testing is a software testing method, usually used to test the performance and stability of applications or systems under high load and high pressure. Stress testing is to test the system's load capacity, performance bottlenecks and stability by simulating a large number of users accessing the system at the same time, or performing a large amount of data processing or calculation on the system. Through stress testing, the bottlenecks and weaknesses of the system can be discovered, and then the system can be optimized to improve the system's performance and stability. Stress testing includes: Load testing: testing the system under Performance indicators under normal load to determine the system's capacity and resource utilization; concurrency testing: testing the system's performance and stability when processing multiple requests at the same time to determine the system's concurrent processing capabilities; peak testing: testing the system's performance and stability when receiving sudden peak traffic in a short period of time to determine the system's carrying capacity; delay testing: testing the system's performance and stability under long-term high load to determine the system's reliability and fault tolerance; capacity testing: testing the system's performance and stability under maximum load to determine the system's capacity and resource utilization; abnormal testing: testing the system's performance and stability under abnormal conditions to determine the system's fault tolerance and recovery capabilities.

[0026] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data processing system, characterized in that: include: Data storage platform: Establish a cloud storage platform to store data on cloud servers; Data processing module: including collection, classification, cleaning, conversion, aggregation and maintenance processing; Data cleaning module: The purpose of data cleaning is to eliminate noise, outliers and duplicate data in the data to ensure the quality and accuracy of the data; Fault detection unit: real-time tracking of the data processing process, and judging whether there is a fault in the processing system according to the processing flow; Fault debugging and analysis module: Analyze fault data based on the detection data provided by the fault detection unit, determine the problems in the processing method, and select system part adjustments or switch the processing system according to the fault problem; Switching of processing methods: Prepare alternative data processing systems for emergency situations in advance, and replace them with alternative systems when problems that cannot be solved by the current processing system occur; Method detection unit: After switching the system, it is necessary to perform stability detection on the system operation by running benchmark tests and stress tests to ensure the normal operation of the system.

2. A data processing system according to claim 1, characterized in that: In the data storage platform, the cloud storage platform has the characteristics of easy expansion, high reliability and powerful computing and analysis capabilities. In the cloud storage platform, users can choose different service providers according to their own needs.

3. A data processing system according to claim 1, characterized in that: In the data processing module, data collection: collecting raw data from various sources; data classification is the process of merging data together according to their common attributes or characteristics, in order to better organize, manage and use data; data cleaning: deleting duplicate data, processing missing values, identifying and processing outliers to ensure data accuracy and consistency, data conversion: converting data from one format or structure to another; data aggregation: merging data from multiple data sources into a consistent format, usually accompanied by data cleaning and conversion steps; data maintenance includes backing up system data, restoring the database system, generating user information tables, authorizing information tables, monitoring system operation status, handling system errors in a timely manner, and ensuring system data security.

4. A data processing system according to claim 1, characterized in that: The data cleaning module includes duplicate data cleaning, incomplete data cleaning, data format cleaning, error data cleaning, correlation data cleaning, and monitoring and solutions for incomplete data and error values.

5. A data processing system according to claim 1, characterized in that: In the fault detection unit, the data processing process is tracked in real time, and whether there is a fault in the processing system is determined based on the processing flow.

6. A data processing system according to claim 1, characterized in that: In the fault debugging and analysis module, the data processing software is generally run for detection. Software failure generally refers to the incompatibility or damage of the system software, which causes the processing system to be unable to start or work properly. Virus failure is because the files in the computer system are infected with viruses, and the computer system cannot work properly after the virus attack. Failures caused by viruses can be prevented and checked for viruses using anti-virus software and anti-virus systems. Viruses with strong destructive power should be checked regularly to prevent damage to the computer system and cause irreparable losses.

7. A data processing system according to claim 1, characterized in that: In the switching of the processing method, an alternative data processing system for emergency situations is prepared in advance. When there are unsolvable problems with the currently used processing system, the controller will replace the backup system after authorization and send a fault signal to the PC for prompting.

8. A data processing system according to claim 1, characterized in that: In the method detection unit, running a benchmark test refers to implementing a quantitative and comparable test on a certain performance indicator of a class of test objects by designing scientific test methods, test tools and test systems. Stress testing is usually used to test the performance and stability of an application or system under high load and high stress conditions.

9. A data processing method according to claim 1, characterized in that: The steps include: Step 1: The data storage platform obtains and stores data information; Step 2: Collect, classify, clean, transform, aggregate and maintain data; Step 3: Clean the collected data to improve data efficiency: Step 4: Data processing system fault detection; Step 5: Replace the faulty data processing system.

Images