A method and system for cleaning data reported by equipment

Abstract

The present invention is applicable to the field of data processing technology, and provides a method and system for cleaning data reported by equipment. The system includes: a form creation module, an interval limitation module, a value update module, an exception handling module, and an output value calculation module. The present application records the data reported by the equipment in real time and updates relevant information, so that production management decisions have more efficient and timely data support. Past systems may not be agile and accurate enough in real-time detection and analysis of data. And it has the ability to distinguish abnormalities in data reported by the equipment. If the data changes due to parameter adjustments, the system can intelligently adjust and record the start of a new evaluation cycle, rather than blindly marking abnormalities or ignoring actual changes in the production process. The complete retention of old and new data records provides perfect support for tracking the production process and analyzing historical data, and can provide a richer data analysis basis compared to the current solutions of the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for cleaning data reported by a device. Background Art

[0002] Cleaning of device-reported data refers to the process of collecting data from various industrial equipment, sensors or systems, and then improving the quality of the data through a set of processes to eliminate inconsistent, duplicate or erroneous data, thereby ensuring the accuracy and reliability of the data when used for monitoring, analysis and decision-making.

[0003] Traditional systems may have delays in data recording and analysis, resulting in untimely feedback on data changes and abnormal conditions. This may delay decision-making. It is also unable to effectively distinguish whether data changes are due to normal operating parameter changes or equipment problems, and sometimes some parameter adjustments are mistakenly marked as abnormal. Summary of the invention

[0004] The purpose of the present invention is to provide a method for cleaning data reported by a device, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0005] The present invention is implemented as follows: a method for cleaning data reported by a device, the method comprising:

[0006] Establish a data record table. When the device reports, obtain the reported data content and reporting time, and obtain the information of the reporting device, and register it in the data record table;

[0007] Set the analysis range, compare the data content reported by the device with the analysis range, and determine whether the data content reported by the device falls within the analysis range;

[0008] If the data content reported by the device falls within the analysis range, the existing time window in the data record table is obtained, and the end time of the time window is updated according to the reporting time of the device, and then the data record table is updated according to the data content reported by the device;

[0009] If the data content reported by the device does not fall within the analysis range, the data content is defined as abnormal data, and the reporting device information is read to verify the data, determine the abnormal cause of the abnormal data, and establish a new time window based on the reporting time of the abnormal data, and record the reported data in the data recording table;

[0010] Calculate the difference between the data content reported by the equipment and the data content before the equipment reported to obtain the total output value.

[0011] As a further solution of the present invention, the step of obtaining the reported data content and reporting time, and obtaining the information of the reporting device, and registering them in the data record table specifically includes:

[0012] Establish a data record table, the data record table specifically includes:

[0013] Device identifier (device_id):

[0014] The value at the beginning of the detection time window (initial_value):

[0015] The latest value (current_value):

[0016] Cumulative production of the last batch in the time window (cumulative_production):

[0017] The start timestamp of the time window (time_window_start):

[0018] The end timestamp of the time window of the current data (time_window_end):

[0019] The number of data reported within the time window (data_points):

[0020] Flag indicating whether an anomaly is detected (flag_anomaly):

[0021] When the device reports, the reported data content, reporting time and information of the reporting device are obtained, and the information of the reporting device is recorded as device_id.

[0022] As a further solution of the present invention, the setting of the analysis range interval and comparing the data content reported by the device with the analysis range interval specifically includes:

[0023] Obtain historical reporting data of the reporting device according to the information of the reporting device;

[0024] An initial interval is established based on historical reported data, and an error value is set. Based on the error value and the initial interval, the analysis range interval is obtained.

[0025] As a further solution of the present invention, the end time of the time window is updated according to the reporting time of the device, and the data record table is updated according to the data content reported by the device, which specifically includes:

[0026] Query the existing time window in the data record table, that is, the time difference between time_window_end and time_window_start. If no existing time window is found, create a new time window with the current reporting time as time_window_start.

[0027] Check the data reported by the device in the time window. If the data is within the analysis range, update current_value to the currently reported data.

[0028] After the cumulative_production calculation, add one to the data_points value.

[0029] As a further solution of the present invention, the determining of the abnormal cause of the abnormal data, establishing a new time window based on the reporting time of the abnormal data, and recording the reported data in a data recording table specifically includes:

[0030] If the data is not within the analysis range, an abnormal flag is added to the data and flag_anomaly is set to true;

[0031] Read the reported device information, verify the data, and determine whether the abnormal data is caused by parameter adjustment or device abnormality;

[0032] If the cause of the abnormal data is parameter adjustment, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and the data reported by the subsequent device is recorded in the new time window;

[0033] If the cause of the abnormal data is device abnormality, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained. If the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

[0034] As a further solution of the present invention, the calculation formula for obtaining the total output value is:

[0035] cumulative_production=current_value-initial_value.

[0036] Another object of the present invention is to provide a cleaning system for device reported data, the system comprising:

[0037] The form creation module is used to create a data record table. When a device reports, it obtains the reported data content and reporting time, and obtains the information of the reporting device, and registers it in the data record table;

[0038] The interval limiting module is used to set the analysis range interval, compare the data content reported by the device with the analysis range interval, and determine whether the data content reported by the device falls within the analysis range interval;

[0039] The value update module is used to obtain the existing time window in the data record table if the data content reported by the device falls within the analysis range, and update the end time of the time window according to the reporting time of the device, and then update the data record table according to the data content reported by the device;

[0040] The exception handling module is used to define the data content as abnormal data if the data content reported by the device does not fall within the analysis range, read the reporting device information, verify the data, determine the abnormal cause of the abnormal data, and establish a new time window based on the reporting time of the abnormal data, and record the reported data in the data recording table;

[0041] The output value calculation module is used to calculate the difference between the data content reported by the equipment and the data content before the equipment reported, so as to obtain the total output value.

[0042] As a further solution of the present invention, the form creation module includes:

[0043] The establishing unit is used to establish a data record table, wherein the data record table specifically includes:

[0044] Device identifier (device_id):

[0045] The value at the beginning of the detection time window (initial_value):

[0046] The latest value (current_value):

[0047] Cumulative production of the last batch in the time window (cumulative_production):

[0048] The start timestamp of the time window (time_window_start):

[0049] The end timestamp of the time window of the current data (time_window_end):

[0050] The number of data reported within the time window (data_points):

[0051] Flag indicating whether an anomaly is detected (flag_anomaly):

[0052] The device recording unit is used to obtain the reported data content, reporting time and information of the reporting device when the device reports, and record the information of the reporting device as device_id.

[0053] As a further solution of the present invention, the value updating module includes:

[0054] The window update unit is used to query the existing time window in the data record table, that is, the time difference between time_window_end and time_window_start. If the existing time window is not found, a new time window is created with the current reporting time as time_window_start.

[0055] The interval verification unit is used to check the data reported by the device in the time window. If the data is within the analysis range value interval, the current_value is updated to the currently reported data;

[0056] A count unit used to increment the data_points value by one after cumulative_production is calculated.

[0057] As a further solution of the present invention, the exception handling module includes:

[0058] The abnormal marking unit is used to add an abnormal flag to the data if the data is not within the analysis range value interval, and set flag_anomaly to true;

[0059] The abnormality analysis unit is used to read the reported device information, verify the data, and determine whether the abnormal data is caused by parameter adjustment or device abnormality;

[0060] The segmented recording unit is used to create a new time window for the data with the abnormal flag if the cause of the abnormal data is parameter adjustment, use the data with the abnormal flag as the initial_value of the new time window, retain the previous time window and the data information reported in the previous time window, and record the data reported by the subsequent device in the new time window;

[0061] An independent recording unit is used to create a new time window for the data with an abnormal flag if the cause of the abnormal data is a device abnormality, and use the data with the abnormal flag as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and if the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

[0062] The beneficial effects of the present invention are:

[0063] This application records the data reported by the equipment in real time and updates relevant information, so that production management decisions have more efficient and timely data support. Previous systems may not be agile and accurate enough in real-time detection and analysis of data. It also has the ability to identify abnormalities in the data reported by the equipment. If the data changes due to parameter adjustments, the system can intelligently adjust and record the start of a new evaluation cycle, rather than blindly marking abnormalities or ignoring actual changes in the production process. The complete retention of old and new data records provides perfect support for tracking the production process and analyzing historical data, and can provide a richer data analysis foundation compared to current solutions in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for cleaning device reported data provided by an embodiment of the present invention;

[0065] Figure 2 A flow chart of establishing a data record table provided in an embodiment of the present invention;

[0066] Figure 3 A flowchart of setting an analysis range interval and comparing the data content reported by the device with the analysis range interval provided in an embodiment of the present invention;

[0067] Figure 4 A flowchart of updating the end time of the time window according to the reporting time of the device and then updating the data record table according to the data content reported by the device provided in the embodiment of the present invention;

[0068] Figure 5 A flowchart for determining the abnormal cause of the abnormal data provided by an embodiment of the present invention, establishing a new time window based on the reporting time of the abnormal data, and recording the reported data in a data recording table;

[0069] Figure 6 A structural block diagram of a cleaning system for device reported data provided by an embodiment of the present invention;

[0070] Figure 7 A structural block diagram of a form creation module provided in an embodiment of the present invention;

[0071] Figure 8 A structural block diagram of a value updating module provided in an embodiment of the present invention;

[0072] Fig. 9 A structural block diagram of an exception handling module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0075] Figure 1 A flowchart of a method for cleaning device reported data provided by an embodiment of the present invention, such as Figure 1 As shown, a method for cleaning data reported by a device, characterized in that the method comprises:

[0076] S100, establish a data record table, when the device reports, obtain the reported data content and reporting time, and obtain the information of the reporting device, and register it in the data record table;

[0077] This step is the core starting point of the entire data cleaning process. It builds a structured data recording platform to collect and register the data and related information reported by IoT devices. This step is powerful, including the ability to collect data in real time and provide the raw data basis for further analysis and processing. In this step, the first thing to do is to create a data recording table, which not only records the identification of each device for tracking management, but also stores a series of key data points, including the initial and final values ​​in each time window, the cumulative count of data entries, production aggregation, two timestamps of the time window (start and end), and any abnormal data flags.

[0078] Such a data layout greatly facilitates monitoring every moment of the production process, supports micro-review of operations, and enables managers to gain insight into the precise overview of production data at any given moment. In addition, a key feature of the S100 step is the ability to dynamically update when data is reported, which means that table building is not just a static behavior, but a process that continuously responds to new information reported by the equipment. When the equipment reports data, the system will capture the content and time of the report and reflect this new information in the data table in a timely manner. By implementing this step, not only is the real-time and dynamic nature of the data ensured, but a strong foundation is also set, creating possibilities for all subsequent steps of this method, such as abnormal monitoring and processing, output calculation, and even the optimization and improvement of the entire production control system.

[0079] S200, setting an analysis range interval, comparing the data content reported by the device with the analysis range interval, and determining whether the data content reported by the device falls within the analysis range interval;

[0080] In this step, a reliable analysis range (threshold range) is determined based on the historical reported data of the device, providing a clear dividing line to identify whether each reported data is normal.

[0081] First, the system goes back and aggregates the device's past reported data. The collection of this historical data is critical because it provides valuable information about the device's performance, while revealing the device's regular operating mode and the general range of its data reporting. This historical data is then used to define an "initial interval," which is the statistical distribution interval of the device's historical uploaded values ​​under normal operating conditions. This initial interval is then supplemented with a tolerance that is designed to recognize and adapt to a certain level of data reporting fluctuations or accidental variations without misjudging them as anomalies - this makes it easy to avoid unnecessary anomalies that are marked due to extremely small or irrelevant data disturbances, thereby reducing misjudgments.

[0082] Furthermore, the analysis range interval formed in this step is used to implement real-time comparison of the data of the equipment under actual working conditions to determine whether the data is normal or reflects some potential problems, which lays a solid foundation for the stability of equipment operation and the reliability of results.

[0083] S300, if the data content reported by the device falls within the analysis range, then obtain the existing time window in the data recording table, and update the end time of the time window according to the reporting time of the device, and then update the data recording table according to the data content reported by the device;

[0084] This step is the core of the data cleaning process, ensuring the latest information and the accuracy of relevant data. When executing this step, the system first queries the existing time window in the data record table. If there is no existing time window, a new time period is created with the current reporting time. This approach flexibly adapts to the needs of immediacy of data collection, and can manage the continuous reporting data flow, and can also set a new time base for intermittent data transmission.

[0085] The system then performs a quality check on the reported data. This verification process ensures that the values ​​in the original data record table are updated only when they are within the value range of the new data, avoiding data contamination caused by errors or abnormal values. In addition, after confirming that the data is valid, in addition to updating the current value and end time, the accumulated production value will be further calculated, and the data point counter will be incremented to carefully record the number of data collections, which is equivalent to strengthening the quantitative management of data. This not only builds a dynamic data table, but also provides a powerful tool for accurately tracking data changes and monitoring the production process.

[0086] Compared with the prior art, the comprehensive performance of this step can more accurately reflect the situation, has a fast processing speed, and can be adjusted in real time to adapt to changes in data reporting, making it more effective and practical for actual production and data analysis applications.

[0087] S400, if the data content reported by the device does not fall within the analysis range, the data content is defined as abnormal data, and the reporting device information is read, the data is verified, the abnormal cause of the abnormal data is determined, and a new time window is established based on the reporting time of the abnormal data, and the reported data is recorded in the data recording table;

[0088] This step provides a multi-dimensional anomaly management mechanism for the data cleaning process. It not only identifies data that falls outside the normal operating parameter range, but also further determines the outlier nature and possible causes of this data in actual operation, such as adjustment of operating parameters or equipment failure, and carefully classifies and records this information. This process ensures that the data set remains clear and useful, helps isolate problems, and allows targeted operations and adjustments to operating conditions.

[0089] When data is marked as anomaly, it indicates that the data point is not within the normal range. This immediate mark can help subsequent analysis focus on this data and quickly identify potential problems.

[0090] Then, we need to determine the reasons behind the abnormal data. By reading the information reported by the equipment and verifying the data, we can understand whether the data abnormality is caused by configuration adjustments or actual equipment failures. This information is crucial to maintaining the stable operation of the equipment. The project team can use this information to arrange maintenance work, set the original production parameters, or make necessary technical improvements.

[0091] The subsequent action process is then determined based on the specific cause of the anomaly. If it is a parameter adjustment, it means that the abnormal value can continue to be used as normal data, and a new time window will be created to more accurately reflect the equipment performance after the parameter change, and the data record will be updated. This allows the data flow to transition naturally as the operating conditions change, and the subsequent reported data will be updated and recorded in the newly created time window, which is convenient for classifying and processing data with different parameters.

[0092] If it is caused by equipment failure, a new time window is also established, and the data before the abnormal phenomenon occurs is stored in case of emergency. This helps to reveal the context of the abnormality and the operating conditions before the equipment failure. The abnormalities caused by different reasons have different processing of time windows, ensuring that the data is not only recorded, but also can be put into further analysis in a way that retains the previous and subsequent environmental information. Since the data is caused by the abnormality, the value will be saved in a separate time window for recording. When the subsequent device reports new data, if it is detected as non-abnormal data, the data will be recorded in the original time window to ensure the consistency of the data.

[0093] S500, performing a difference calculation between the data content reported by the device and the data content before the device reports, and obtaining the total output value.

[0094] Figure 2 A flowchart of establishing a data record table provided by an embodiment of the present invention, such as Figure 2 As shown, the acquisition of the reported data content and reporting time, and the acquisition of the reporting device information, and registration in the data record table specifically include:

[0095] S110, establishing a data record table, wherein the data record table specifically includes:

[0096] Device identifier (device_id):

[0097] The value at the beginning of the detection time window (initial_value):

[0098] The latest value (current_value):

[0099] Cumulative production of the last batch in the time window (cumulative_production):

[0100] The start timestamp of the time window (time_window_start):

[0101] The end timestamp of the time window of the current data (time_window_end):

[0102] The number of data reported within the time window (data_points):

[0103] Flag indicating whether an anomaly is detected (flag_anomaly):

[0104] S120, when the device reports, obtain the reported data content, reporting time and information of the reporting device, and record the information of the reporting device as device_id.

[0105] Figure 3 The flowchart of setting the analysis range interval provided by the embodiment of the present invention and comparing the data content reported by the device with the analysis range interval is as follows: Figure 3 As shown, the setting of the analysis range interval and comparing the data content reported by the device with the analysis range interval specifically includes:

[0106] S210, acquiring historical reporting data of the reporting device according to the information of the reporting device;

[0107] S220, establishing an initial interval based on historical reported data, and setting an error value, and obtaining an analysis range interval based on the error value and the initial interval.

[0108] Figure 4 A flowchart of updating the end time of the time window according to the reporting time of the device and then updating the data record table according to the data content reported by the device is provided in an embodiment of the present invention, such as Figure 4 As shown, the end time of the time window is updated according to the reporting time of the device, and then the data record table is updated according to the data content reported by the device, which specifically includes:

[0109] S310, querying the existing time window in the data record table, that is, the time difference between time_window_end and time_window_start, if no existing time window is found, creating a new time window with the current reporting time as time_window_start;

[0110] S320, checking the data reported by the device in the time window, if the data is within the analysis range value interval, if it is within the analysis range value interval, then updating current_value to the currently reported data;

[0111] S330, after cumulative_production is calculated, the value of data_points is increased by one.

[0112] Figure 5 The flowchart of determining the abnormal cause of the abnormal data provided by the embodiment of the present invention, establishing a new time window based on the reporting time of the abnormal data, and recording the reported data in the data recording table is as follows: Figure 5 As shown, the abnormal cause of the abnormal data is determined, and a new time window is established based on the reporting time of the abnormal data, and the reported data is recorded in the data recording table, specifically including:

[0113] S410, if the data is not within the analysis range, an abnormal flag is added to the data, and flag_anomaly is changed to true;

[0114] S420, reading the reported device information, verifying the data, and determining whether the abnormal data is caused by parameter adjustment or device abnormality;

[0115] S430, if the cause of the abnormal data is parameter adjustment, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and the data reported by the subsequent device is recorded in the new time window;

[0116] S440, if the abnormal data is caused by device abnormality, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and if the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

[0117] In the embodiment of the present invention, the calculation formula for obtaining the total output value is:

[0118] cumulative_production=current_value-initial_value.

[0119] Embodiment 1:

[0120] (device_id): Arm001

[0121] (initial_value): 0 components

[0122] (current_value): 0 components (initial state, no record)

[0123] (cumulative_production): 0 parts

[0124] (time_window_start): 2023-04-01T08:00:00Z (timestamp of 8 am)

[0125] (time_window_end): 2023-04-01T08:00:00Z (initial state, same as start timestamp)

[0126] (data_points): 0 times

[0127] (flag_anomaly): false

[0128] Historical data shows that Arm001 should complete the assembly of about 100 components per hour on average. Therefore, the hourly output range during normal production is set to: [95, 105].

[0129] At 9:00 AM on April 1, 2023, Arm001 reported that 101 parts had been manufactured. The details are as follows:

[0130] initial_value: 0

[0131] current_value: 101

[0132] Cumulative_production: 101 (because this is the first report, there is no accumulation)

[0133] time_window_start:2023-04-01T08:00:00Z

[0134] time_window_end:2023-04-01T09:00:00Z

[0135] data_points: 1 time

[0136] flag_anomaly: false

[0137] At 10:00 AM on April 1, 2023, Arm001 reported that 110 parts had been manufactured. The details are as follows:

[0138] initial_value: 0

[0139] current_value: 101

[0140] cumulative_production: 101

[0141] time_window_start:2023-04-01T08:00:00Z

[0142] time_window_end:2023-04-01T09:00:00Z

[0143] data_points: 2 times

[0144] flag_anomaly: true (because 110 exceeds the previously set upper limit of 105)

[0145] Create a new counting time window time_window_start: 2023-04-01T10:00:00Z, and use the new time window method to record the newly reported data current_value: 110 for current_value.

[0146] Figure 6 A structural block diagram of a cleaning system for device reported data provided by an embodiment of the present invention, such as Figure 6 As shown, a cleaning system for device reported data, characterized in that the system comprises:

[0147] The form creation module 100 is used to create a data record table. When a device reports, the reported data content and reporting time are obtained, and the information of the reporting device is obtained, and registered in the data record table;

[0148] The interval limiting module 200 is used to set the analysis range interval, compare the data content reported by the device with the analysis range interval, and determine whether the data content reported by the device falls within the analysis range interval;

[0149] The value updating module 300 is used to obtain the existing time window in the data recording table if the data content reported by the device falls within the analysis range, and update the end time of the time window according to the reporting time of the device, and then update the data recording table according to the data content reported by the device;

[0150] The exception handling module 400 is used to define the data content as abnormal data if the data content reported by the device does not fall within the analysis range, and read the reporting device information, verify the data, determine the abnormal cause of the abnormal data, and establish a new time window based on the reporting time of the abnormal data, and record the reported data in the data recording table;

[0151] The output value calculation module 500 is used to calculate the difference between the data content reported by the device and the data content before the device reported, so as to obtain the total output value.

[0152] Figure 7 The structural block diagram of the form creation module provided in the embodiment of the present invention is as follows: Figure 7 As shown, the form creation module includes:

[0153] The establishing unit 110 is used to establish a data record table, and the data record table specifically includes:

[0154] Device identifier (device_id):

[0155] The value at the beginning of the detection time window (initial_value):

[0156] The latest value (current_value):

[0157] Cumulative production of the last batch in the time window (cumulative_production):

[0158] The start timestamp of the time window (time_window_start):

[0159] The end timestamp of the time window of the current data (time_window_end):

[0160] The number of data reported within the time window (data_points):

[0161] Flag indicating whether an anomaly is detected (flag_anomaly):

[0162] The device recording unit 120 is used to obtain the reported data content, reporting time and information of the reporting device when the device reports, and record the information of the reporting device as device_id.

[0163] Figure 8 A structural block diagram of a value updating module provided in an embodiment of the present invention, such as Figure 8 As shown, the value updating module includes:

[0164] The window updating unit 310 is used to query the existing time window in the data recording table, that is, the time difference between time_window_end and time_window_start. If no existing time window is found, a new time window is created with the current reporting time as time_window_start.

[0165] The interval verification unit 320 is used to check the data reported by the device in the time window, and if the data is within the analysis range value interval, update the current_value to the currently reported data;

[0166] The times recording unit 330 is used to increase the value of data_points by one after the cumulative_production is calculated.

[0167] Fig. 9 The structural block diagram of the exception handling module provided by the embodiment of the present invention is as follows: Fig. 9 As shown, the exception handling module includes:

[0168] The abnormal marking unit 410 is used to add an abnormal flag to the data if the data is not within the analysis range value interval, and set flag_anomaly to true;

[0169] The abnormality analysis unit 420 is used to read the reported device information, verify the data, and determine whether the abnormal data is caused by parameter adjustment or device abnormality;

[0170] The segmented recording unit 430 is used to create a new time window for the data with the abnormal flag if the cause of the abnormal data is parameter adjustment, use the data with the abnormal flag as the initial_value of the new time window, retain the previous time window and the data information reported in the previous time window, and record the data reported by the subsequent device in the new time window;

[0171] The independent recording unit 440 is used to create a new time window for the data with an abnormal flag if the cause of the abnormal data is a device abnormality, and use the data with the abnormal flag as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and if the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

[0172] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0174] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for cleaning data reported by a device, characterized in that: The method comprises: Establish a data record table. When the device reports, obtain the reported data content and reporting time, and obtain the information of the reporting device, and register it in the data record table; Set the analysis range, compare the data content reported by the device with the analysis range, and determine whether the data content reported by the device falls within the analysis range; If the data content reported by the device falls within the analysis range, the existing time window in the data record table is obtained, and the end time of the time window is updated according to the reporting time of the device, and then the data record table is updated according to the data content reported by the device; If the data content reported by the device does not fall within the analysis range, the data content is defined as abnormal data, and the reporting device information is read to verify the data, determine the abnormal cause of the abnormal data, and establish a new time window based on the reporting time of the abnormal data, and record the reported data in the data recording table; Calculate the difference between the data reported by the equipment and the data before the equipment reported, and obtain the total output value; Determining the abnormal cause of the abnormal data, establishing a new time window based on the reporting time of the abnormal data, and recording the reported data in a data recording table specifically includes: If the data is not within the analysis range, an abnormal flag is added to the data and flag_anomaly is set to true; Read the reported device information, verify the data, and determine whether the abnormal data is caused by parameter adjustment or device abnormality; If the cause of the abnormal data is parameter adjustment, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and the data reported by the subsequent device is recorded in the new time window; If the cause of the abnormal data is device abnormality, a new time window is created for the data with the abnormal flag, and the data with the abnormal flag is used as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained. If the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

2. The method according to claim 1, characterized in that The obtaining of the reported data content and reporting time, and obtaining the information of the reporting device, and registering them in the data record table specifically include: Establish a data record table, the data record table specifically includes: Device identifier (device_id); The value at the beginning of the detection time window (initial_value); The latest value (current_value); The cumulative production of each batch at the end of the time window (cumulative_production); The start timestamp of the time window (time_window_start); The end timestamp of the time window of the current data (time_window_end); The number of times data is reported within the time window (data_points); Flag indicating whether an anomaly is detected (flag_anomaly); When the device reports, the reported data content, reporting time and information of the reporting device are obtained, and the information of the reporting device is recorded as device_id.

3. The method according to claim 2, characterized in that The setting of the analysis range and comparing the data content reported by the device with the analysis range specifically includes: Obtain historical reporting data of the reporting device according to the information of the reporting device; An initial interval is established based on historical reported data, and an error value is set. Based on the error value and the initial interval, the analysis range interval is obtained.

4. The method according to claim 3, characterized in that The step of updating the end time of the time window according to the reporting time of the device, and then updating the data record table according to the data content reported by the device, specifically includes: Query the existing time window in the data record table, that is, the time difference between time_window_end and time_window_start. If no existing time window is found, create a new time window with the current reporting time as time_window_start. Check the data reported by the device in the time window. If the data is within the analysis range, update current_value to the currently reported data. After the cumulative_production calculation, add one to the data_points value.

5. The method according to claim 1, characterized in that The calculation formula for obtaining the total output value is: cumulative_production=current_value-initial_value.

6. A cleaning system for device reported data, characterized in that: The system comprises: The form creation module is used to create a data record table. When a device reports, it obtains the reported data content and reporting time, and obtains the information of the reporting device, and registers it in the data record table; The interval limiting module is used to set the analysis range interval, compare the data content reported by the device with the analysis range interval, and determine whether the data content reported by the device falls within the analysis range interval; The value update module is used to obtain the existing time window in the data record table if the data content reported by the device falls within the analysis range, and update the end time of the time window according to the reporting time of the device, and then update the data record table according to the data content reported by the device; The exception handling module is used to define the data content as abnormal data if the data content reported by the device does not fall within the analysis range, read the reporting device information, verify the data, determine the abnormal cause of the abnormal data, and establish a new time window based on the reporting time of the abnormal data, and record the reported data in the data recording table; The output value calculation module is used to calculate the difference between the data content reported by the equipment and the data content before the equipment reported, and obtain the total output value; The exception handling module includes: The abnormal marking unit is used to add an abnormal flag to the data if the data is not within the analysis range value interval, and set flag_anomaly to true; The abnormality analysis unit is used to read the reported device information, verify the data, and determine whether the abnormal data is caused by parameter adjustment or device abnormality; The segmented recording unit is used to create a new time window for the data with the abnormal flag if the cause of the abnormal data is parameter adjustment, use the data with the abnormal flag as the initial_value of the new time window, retain the previous time window and the data information reported in the previous time window, and record the data reported by the subsequent device in the new time window; An independent recording unit is used to create a new time window for the data with an abnormal flag if the cause of the abnormal data is a device abnormality, and use the data with the abnormal flag as the initial_value of the new time window. At the same time, the previous time window and the data information reported in the previous time window are retained, and if the data reported by the subsequent device is not abnormal data, it is recorded in the previous time window.

7. The system according to claim 6, characterized in that The form creation module includes: The establishing unit is used to establish a data record table, wherein the data record table specifically includes: Device identifier (device_id); The value at the beginning of the detection time window (initial_value); The latest value (current_value); The cumulative production of each batch at the end of the time window (cumulative_production); The start timestamp of the time window (time_window_start); The end timestamp of the time window of the current data (time_window_end); The number of times data is reported within the time window (data_points); Flag indicating whether an anomaly is detected (flag_anomaly); The device recording unit is used to obtain the reported data content, reporting time and information of the reporting device when the device reports, and record the information of the reporting device as device_id.

8. The system according to claim 7, characterized in that The value updating module comprises: The window update unit is used to query the existing time window in the data record table, that is, the time difference between time_window_end and time_window_start. If the existing time window is not found, a new time window is created with the current reporting time as time_window_start. The interval verification unit is used to check the data reported by the device in the time window. If the data is within the analysis range value interval, the current_value is updated to the currently reported data; A count unit used to increment the data_points value by one after cumulative_production is calculated.

Images