A real-time data cleaning method based on historical data

Abstract

The application discloses a kind of real-time data cleaning methods based on historical data, more particularly to edge internet of things data processing field, including edge internet of things gateway reads the real-time record of same measuring point group in current cleaning period, each measuring point real-time record is arranged according to record arrival time, and each measuring point historical record in historical cleaning result is extracted, generates historical surface sequence according to historical time, generates to be cleaned surface according to current cleaning period;The application is by comparing the real-time record of multiple measuring points in current cleaning period with historical surface sequence, and according to the common pointing result of all drop point sequences, the problem that the false real-time data combination formed by normal record splicing of different time historical is difficult to identify in the process of edge internet of things real-time data cleaning is solved.

Description

Technical Field

[0001] This invention relates to the field of edge IoT data processing technology, and more specifically, to a real-time data cleaning method based on historical data. Background Technology

[0002] In the process of edge IoT data processing, existing real-time cleaning processes mainly revolve around duplicate reporting, out-of-bounds fluctuations, missing fields, and out-of-order arrival. During deployment, edge nodes typically filter, correct, or rewrite each arriving record after receiving various measurement point data, using methods such as time alignment, record deduplication, window statistics, historical interval comparison, or prediction deviation verification. In the integrated electromechanical access scenario of the park, electricity meters, temperature controllers, pump group controllers, valve position collectors and vibration sensors are simultaneously connected to the same edge gateway. The sampling period of each measuring point is different, and network jitter, disconnection and retransmission and batch forwarding occur repeatedly. The edge side also needs to send the cleaning results directly to the alarm and linkage link under limited buffer and limited computing power. At this time, historical normal records from different times will be spliced ​​into a group of data combinations with values ​​falling within their respective normal ranges and no obvious conflicts in field relationships within the same short window. Each item can be checked by individual measuring points and can pass, but when corresponding to the equipment status section at the same time, it cannot fall into any real operating state. On-site observation shows that the curves do not exceed the limits when checked separately, but the combined interpretation triggers false alarms, false linkages or misidentification of status. The problem is not that the value of a single record is distorted, but that although the combination of multiple measuring points in the current window comes from historical normal records, they are not data sections formed at the same time. The existing cleaning process lacks the ability to identify this point. Therefore, how to identify pseudo-real-time data combinations formed by splicing together historical normal records from different times during the real-time data cleaning process of edge IoT has become the technical problem to be solved in this application. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a real-time data cleaning method based on historical data. This method compares the real-time records of multiple measurement points within the current cleaning cycle with the historical surface sequence point by point, and separates pseudo-real-time data combinations based on the common pointing result of all the point sequences, thereby solving the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a real-time data cleaning method based on historical data, comprising: S1. The edge IoT gateway reads the real-time records of the same measurement point group within the current cleaning cycle, arranges the real-time records of each measurement point according to the arrival time of the record, extracts the historical records of each measurement point in the historical cleaning results, generates a historical surface sequence according to the historical time, and generates a surface to be cleaned according to the current cleaning cycle. S2. For each measurement point record in the surface to be cleaned, extract the historical records of measurement points with the same name in the historical surface sequence, calculate the value difference, fluctuation ratio and deviation ratio, arrange the historical surface numbers according to the sum of fluctuation ratio and deviation ratio and value difference, and generate the landing point sequence. S3. Compare the historical surface numbers in each landing sequence, extract the common numbers of all landing sequences, and generate a common number set. When the common number set is not empty, take the first number as the target surface number according to the sequence number and sorting of each number in each landing sequence. When the common number set is empty, delete one measurement point record of the surface to be cleaned in turn and extract it repeatedly. Take the deleted measurement point record that makes the common number set non-empty for the first time as the conflict record. S4. When the target surface number is obtained, calculate the sum of the value difference and time difference between the surface to be cleaned and the historical surface, and generate a valid surface; when the conflict record is obtained, delete the conflict record, generate a valid surface according to the remaining measurement point records, and generate a pseudo real-time group according to the sum of the absolute values ​​of the value differences between the conflict record and each measurement point record in the valid surface. S5. Append the effective surface to the end of the historical cleaning results, update the historical records of each measuring point according to the measuring point records in the effective surface, write the pseudo real-time group to the isolation results, and output the edge IoT real-time data cleaning results of the current cleaning cycle.

[0005] In a preferred embodiment, S1 includes: S1-1. The edge IoT gateway sorts the real-time records in the current cleaning cycle according to the measurement point identifier, sorts them according to the arrival time and reverses the numbering, and gathers the real-time records of each measurement point according to the same reverse sequence number to generate real-time candidate surfaces. It counts the number of missing measurement points and the arrival span of each real-time candidate surface, sorts them according to the number of missing measurement points and the arrival span, and takes the first real-time candidate surface as the surface to be cleaned. S1-2. The edge IoT gateway extracts the historical records of the same measurement point group, counts the number of historical records of each measurement point and sorts them by the number of records. The first measurement point is taken as the anchor measurement point. For each historical record in the anchor measurement point, the historical record with the highest absolute value of time difference is selected from the historical records of the other measurement points. This historical record is combined with the historical record of the anchor measurement point to generate a historical candidate surface. The median of the time of each record in each historical candidate surface is taken as the surface time. The sum of the absolute values ​​of the time difference between each record time and the surface time is taken as the aggregation value. S1-3. Compare the historical records of each candidate face one by one. If there are the same historical records, retain the candidate face with the highest aggregation value and delete the other candidate face. Then sort the remaining candidate faces by face time to generate a historical face sequence.

[0006] In a preferred embodiment, S2 includes: S2-1. For each measurement point record in the surface to be cleaned, extract the historical records of measurement points with the same name in the historical surface sequence, arrange them according to historical time and generate a sequence with the same name, calculate the absolute value of the difference between each historical record in the sequence with the previous historical record, and generate a fluctuation sequence. S2-2. Read the measurement point records and the historical records in the same sequence, calculate the absolute value of the difference between each measurement point record and each historical record, generate the value difference sequence, and count the number of fluctuation items in the fluctuation sequence that are not greater than each value difference. Calculate the fluctuation ratio of each historical record based on the number of fluctuation items and the number of items in the fluctuation sequence.

[0007] In a preferred embodiment, S2 further includes: S2-3. Read three adjacent historical records in the same name sequence according to the historical time sequence. The historical deviation value is generated by subtracting the absolute value of the average of the historical records before and after from the value of the middle historical record. The real-time deviation value is generated by subtracting the absolute value of the average of the historical records before and after from the value of the measurement point record. The number of historical deviation items that are not greater than the real-time deviation value is counted in all historical deviation values. The deviation ratio of each historical record is calculated based on the number of historical deviation items and the number of historical deviation values. S2-4. Calculate the sum of the fluctuation ratio and deviation ratio of each historical record, arrange the historical surface numbers of each historical record in ascending order of the sum, and arrange the historical surface numbers in ascending order of the value difference sequence when the sums are the same, so as to generate the landing point sequence of the measurement point records.

[0008] In a preferred embodiment, S3 includes: S3-1. Compare the historical face numbers in each landing sequence, extract the common numbers of all landing sequences and generate a common number set; for each historical face number in the common number set, read the occurrence sequence number in each landing sequence, generate a sequence number group, and calculate the sum of the sequence numbers, the sum of the absolute values ​​of the differences between adjacent sequence numbers and the sum of the reverse sequence numbers, where the sum of the reverse sequence numbers is the sum of the reciprocals of each occurrence sequence number. S3-2. When the common number set is not empty, calculate the convergence value for each historical face number. The convergence value is the sum of the reverse sequence number and the sum of the absolute values ​​of the differences between the two adjacent sequence numbers. Then, arrange the historical face numbers in descending order according to the convergence value. When the convergence values ​​are the same, arrange the historical face numbers in ascending order according to the sequence number. Take the first historical face number as the target face number.

[0009] In a preferred embodiment, S3 further includes: S3-3. When the common number set is empty, delete one measurement point record from the surface to be cleaned in sequence, and repeat the common number set extraction, sequence group generation, and convergence value calculation for the deleted landing point sequence; when the common number set is not empty after deletion, read the first historical surface number, the number of common numbers, the convergence value of the first historical surface number, and the sequence number sum to generate a rollback item; otherwise, compare each landing point sequence after deletion pairwise, count the number of common historical surface numbers for each sequence and sum them to generate a cross sum, and at the same time write the first historical surface number as empty, the number of common numbers as zero, the convergence value as zero, and the sequence number sum as zero to generate a rollback item; S3-4. Compare each rollback item. First, sort them in descending order by the number of common numbers, then in descending order by convergence value, then in descending order by crossover and crossover, and finally in ascending order by sequence number. Take the deleted measurement point record in the first rollback item as the conflict record.

[0010] In a preferred embodiment, S4 includes: S4-1. When obtaining the target surface number, extract the historical surface pointed to by the target surface number in the historical surface sequence, match the measurement point records in the surface to be cleaned with the historical records in the historical surface according to the measurement point name, calculate the absolute value of the difference between the recorded values ​​and the absolute value of the time difference between the records one by one, and sum them to generate the sum of the value difference and the sum of the time difference respectively. S4-2. Write the sum of the value difference and the sum of the time difference into the surface to be cleaned under the target surface number, and record all the measurement points in the surface to be cleaned to generate a valid surface.

[0011] In a preferred embodiment, S4 further includes: S4-3. When obtaining conflict records, delete the conflict records from the surface to be cleaned, arrange the remaining measurement point records according to the measurement point name, count the sequence number of the conflict records in the surface to be cleaned and write it into the separation position, and then combine the remaining measurement point records with the separation position to generate an effective surface. S4-4. Calculate the absolute value of the difference between the conflict record and each measurement point record in the effective surface, and sum the absolute values ​​of all the difference between the record values ​​to generate the separation sum. Then combine the conflict record, the separation position and the separation sum to generate a pseudo-real-time group.

[0012] In a preferred embodiment, S5 includes: S5-1. Read the records of each measuring point in the effective surface, arrange the records of each measuring point by the name of the measuring point, extract the recording time of each measuring point record and sum them, and then divide the sum of the recording times by the number of measuring point records to generate the surface time. S5-2. Read the historical face number at the end of the historical cleaning result, add one to the historical face number to generate the current face number, and then append the current face number, face time, and valid face to the end of the historical cleaning result.

[0013] In a preferred embodiment, S5 further includes: S5-3. Read the records of each measuring point and the historical records of each measuring point in the effective surface according to the measuring point name, append each measuring point record to the end of the historical records of the same measuring point, and rearrange the historical records of each measuring point according to the recording time. S5-4. When a pseudo-real-time group exists, extract the records of each measurement point in the pseudo-real-time group, arrange the records of each measurement point by the measurement point name, extract the recording time of each measurement point record and sum them, then divide the sum of the recording times by the number of measurement point records to generate the isolation time, and write the pseudo-real-time group and the isolation time into the isolation result; when no pseudo-real-time group exists, retain the isolation result and output the edge IoT real-time data cleaning result of the current cleaning cycle.

[0014] The technical effects and advantages of this invention are as follows: 1. By collecting real-time records in reverse order to form a surface to be cleaned, and reorganizing the historical surface sequence according to historical records, it is possible to establish a direct comparison relationship between the current window multi-measurement point combination and the historical similar cross sections, thereby identifying pseudo-real-time data combinations formed by splicing normal records from different times, and relatively suppressing false alarms, false linkages and status misidentification. 2. By calculating the value difference, fluctuation ratio, and deviation ratio of the historical records of the same measurement points, and generating the landing point sequence according to the sum of the fluctuation ratio and the deviation ratio and the value difference, it is possible to link the proximity relationship of single point values ​​with the historical fluctuation position and the local deviation position to make the landing point sorting relatively close to the real historical trend. 3. By extracting the historical surface numbers shared by all landing point sequences and combining the sum of the serial numbers, the sum of the reverse serial numbers, and the sum of the absolute values ​​of the differences between adjacent serial numbers, the historical surfaces with a higher degree of convergence can be screened out from the common pointing relationships of multiple measurement points, which relatively improves the matching results of the same surface. 4. By sequentially deleting individual measurement point records and generating rollback items when the common number set is empty, and then separating measurement point records that disrupt common landing points from the surface to be cleaned by the number of common numbers, convergence value, cross sum and sequence number and location conflict records, the probability of pseudo-combinations mixed into the main result stream can be relatively reduced. 5. By generating valid surfaces with value difference and time difference sums under the target surface number branch, and generating valid surfaces and pseudo-real-time groups under the conflict record branch, and then writing historical cleaning results and isolation results respectively, the retained data and isolated data can be stored separately, which is convenient for subsequent cleaning cycles to continue to call. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Refer to the instruction manual appendix Figure 1 The present invention provides a real-time data cleaning method based on historical data, comprising: S1. The edge IoT gateway reads the real-time records of the same measurement point group within the current cleaning cycle, arranges the real-time records of each measurement point according to the arrival time of the record, extracts the historical records of each measurement point in the historical cleaning results, generates a historical surface sequence according to the historical time, and generates a surface to be cleaned according to the current cleaning cycle. In this embodiment, S1 is used to select the surfaces to be cleaned for subsequent processing within the current cleaning cycle, and to construct a historical surface sequence from the historical cleaning results that can be directly called for subsequent landing point calculations; wherein, each record entering the edge IoT gateway carries a measurement point group identifier, measurement point name, recording time, and recording value; the same measurement point group identifier is bound to a fixed set of measurement point names; only records with the same measurement point group identifier are processed within the current cleaning cycle; the recording time uses the timestamp value received by the edge IoT gateway; the historical cleaning results are stored in the order of historical surface number, and each measurement point name in each historical surface retains only one historical record; Based on this organizational method, the surfaces to be cleaned are first formed from real-time records, and then a sequence of historical surfaces is formed from historical records. This implementation process includes the following steps: After receiving real-time records belonging to the same measurement point group within the current cleaning cycle, the edge IoT gateway first sorts the real-time records by measurement point name, forming a set of real-time records for each measurement point. For each set of real-time records, they are sorted from earliest to latest by recording time. After sorting, reverse numbers are assigned sequentially from the last record backwards, with the last record numbered 1, the second-to-last record numbered 2, and the remaining records numbered sequentially. Then, the reverse numbers of all measurement points are read, and real-time records with the same reverse number are grouped into a real-time candidate surface. If a real-time record with a certain measurement point name is missing under a certain reverse number, that measurement point is recorded as a missing measurement point in the real-time candidate surface. For each real-time candidate surface, statistics are... The number of missing measurement points is determined, and the maximum and minimum values ​​are extracted from all recorded times within the real-time candidate surface. The difference between the two values ​​is recorded as the arrival span. The real-time candidate surfaces are then arranged in ascending order of the number of missing measurement points. If the number of missing measurement points is the same, they are arranged in ascending order of arrival span. The real-time candidate surface ranked first is designated as the surface to be cleaned. A reverse numbering method is used to ensure that the surface to be cleaned falls on the set of records where each measurement point has arrived most recently. A dual arrangement method using the number of missing measurement points and the arrival span is used to ensure that the surface to be cleaned falls on the set of records where the measurement points are more complete and the arrivals within the set are more concentrated. If there is only one real-time candidate surface in the current cleaning cycle, then that real-time candidate surface is directly designated as the surface to be cleaned. After generating the surface to be cleaned, the edge IoT gateway extracts historical records of the same measurement point group from the historical cleaning results and forms a historical record set for each measurement point according to the measurement point name. The number of records in each historical record set is counted, and then sorted from largest to smallest. The measurement point name at the top of the list is designated as the anchor point. If multiple measurement points have the same number of records, the first measurement point name is selected as the anchor point. Subsequently, the historical records in the anchor point historical record set are read one by one. For each anchor point historical record read, the historical record with the highest absolute value of the time difference between the current anchor point and the previous one is searched in the historical record sets of all other measurement points. This anchor point historical record is then combined with the top-ranked historical records to form a candidate historical surface. The absolute value of the time difference is calculated by subtracting the record time of the anchor point historical record from the record time of the candidate historical record. The time difference is calculated using the "absolute value" method. If a certain measurement point has no historical record, it is recorded as missing in the historical candidate face and does not participate in the recording time statistics of the historical candidate face. For each historical candidate face, the recording times of all historical records within the face are extracted and arranged in ascending order. If the number of records is odd, the recording time in the middle position is taken as the face time. If the number of records is even, the sum of the two middle recording times divided by 2 is taken as the face time, and the integer value under the original precision of the timestamp is retained. Then, the absolute value of the time difference between the recording time of each historical record and the face time is calculated one by one, and all the absolute values ​​of the time difference are summed. The result is written as the aggregate value. Anchor point measurement points are used to expand outward one by one so that each historical candidate face revolves around a real historical record. The face time and aggregate value are used to distinguish the degree of time clustering among multiple candidate results. After obtaining all historical candidate surfaces, the historical records of each candidate surface are compared one by one. During the comparison, the combination of measurement point name and recording time in the historical records is used as the identification item. When all identification items of two historical candidate surfaces are the same, the two historical candidate surfaces are judged as the same historical candidate surface. For the same historical candidate surface, only the historical candidate surface with the higher aggregation value is retained, and the other historical candidate surface is deleted. If the aggregation values ​​are the same, the historical candidate surface with the higher surface time is retained. If the surface times are still the same, the historical candidate surface generated by the earlier anchor measurement point is retained. After deletion, the remaining historical candidate surfaces are sorted from earliest to latest according to surface time, and the historical surface numbers are written in sequence, starting from 1 and incrementing sequentially. All the sorted historical candidate surfaces are written as a historical surface sequence. After this processing, each historical surface in the historical surface sequence corresponds to a set of historical records formed by time aggregation, and the same set of historical records will not be repeatedly written to multiple historical surfaces, providing a basic sequence for the subsequent generation of landing point sequences. Through the above processing, the generation of the surface to be cleaned is based on the intra-group aggregation relationship between the most recently arrived records of each measuring point within the current cleaning cycle, and the generation of the historical surface sequence is based on the temporal proximity relationship between historical records. Both adopt the same organization method in the four types of fields: measuring point group identifier, measuring point name, recording time, and recording value. Subsequently, the calculation of value difference, fluctuation ratio, deviation ratio, and landing point sequence can be directly performed. The reverse numbering method is used to aggregate the surface to be cleaned, which can avoid intra-group misalignment caused by directly truncating the last record. The use of anchor measuring points to expand the historical candidate surface and remove duplicates according to the aggregation value can reduce the interference of repeated combinations of historical records on subsequent landing point judgment. In practical applications: If an edge IoT gateway in a certain park connects to four measuring points—electricity meter, temperature controller, pump controller, and valve position collector—and the four measuring points reach 3, 2, 3, and 2 real-time records respectively during the current cleaning cycle, then they are first sorted by arrival time and numbered in reverse order. Then, they are aggregated according to the same reverse order number to obtain multiple real-time candidate surfaces. Among them, the candidate surface with reverse order number 1 consists of the most recent record of the four measuring points, the candidate surface with reverse order number 2 consists of the next most recent record of the four measuring points, and the candidate surface with reverse order number 3 consists only of the measuring points that have reached the 3rd record. After counting the number of missing measurement points and the reach span of these real-time candidate surfaces, the first one in the ranking is selected as the surface to be cleaned. Then, the historical records of four measurement points are extracted from the historical cleaning results. If the number of the electricity meter historical records is ranked first, the electricity meter is used as the anchor measurement point. The electricity meter historical records are read one by one, and the records with the highest absolute value of the time difference with the electricity meter historical records are found in the historical records of the temperature controller, pump group controller, and valve position collector, forming historical candidate surfaces. Then, duplicate candidate surfaces are removed according to the surface time and aggregation value and sorted to obtain the historical surface sequence for subsequent steps.

[0018] S2. For each measurement point record in the surface to be cleaned, extract the historical records of measurement points with the same name in the historical surface sequence, calculate the value difference, fluctuation ratio and deviation ratio, arrange the historical surface numbers according to the sum of fluctuation ratio and deviation ratio and value difference, and generate the landing point sequence. In this embodiment, S2 is used to establish a point-by-point relationship between the surface to be cleaned and the historical surface sequence, so that the subsequent extraction of common numbers no longer directly depends on a single value difference, but simultaneously considers the historical fluctuation position and the local deviation position. During processing, a single measurement point record in the surface to be cleaned is used as the calculation object. The historical records of the same measurement point are extracted from the historical surface sequence to form a sequence of the same name expanded according to the historical time. Then, the fluctuation sequence, value difference sequence, historical deviation value and real-time deviation value are obtained in sequence. Finally, the fluctuation ratio and deviation ratio are written together into the sorting process of the historical surface number, thereby outputting the point sequence of the measurement point record. Among them, the measurement point record and the historical record both use the same numerical value type. The sequence of the same name is arranged from early to late according to the historical time. The fluctuation ratio and the deviation ratio are both obtained by dividing the number of terms. When the length of the sequence involved in the calculation is insufficient, the result is supplemented according to the predetermined writing method. The implementation process includes the following steps: For each measurement point record in the surface to be cleaned, first read the measurement point name of the record, then extract all historical records with the same name from the historical surface sequence. During extraction, read the historical surface one by one according to the historical surface number. After finding the historical record with the same measurement point name in each historical surface, write it into the same-name sequence, and write the historical surface number to which the historical record belongs into the corresponding position in the same-name sequence. After the same-name sequence is generated, arrange it from earliest to latest historical time. Then, starting from the second historical record in the same-name sequence, read the current historical record and the previous historical record one by one. Take the absolute value of the difference between the record value of the current historical record and the record value of the previous historical record, and use it as a fluctuation value. Write it into the fluctuation sequence in the order of the same-name sequence. If the same-name sequence contains only one historical record, the fluctuation sequence is written as an empty sequence. Through this process, each subsequent measurement point record corresponds to a same-name sequence with a historical surface number, and a fluctuation sequence reflecting the adjacent change amplitude of the same-name sequence. After obtaining the corresponding sequence and fluctuation sequence, the measured point records in the surface to be cleaned are read, and the historical records in the corresponding sequence are read one by one. The absolute value of the difference between the measured point record and the historical record is taken to generate a value difference sequence. Each value difference in the value difference sequence corresponds one-to-one with a historical record in the corresponding sequence. For each value difference in the value difference sequence, it is compared item by item in the fluctuation sequence, and the number of fluctuation values ​​not greater than the value difference is counted and written as the number of fluctuation items. Then, the number of fluctuation items is read, and the fluctuation ratio of the historical record is obtained by dividing the number of fluctuation items by the number of fluctuation items. If the number of fluctuation items is zero, the fluctuation ratio of all historical records in the corresponding sequence is written as zero. Through this process, the value difference between the current measured point record and each historical record can be counted within the range of adjacent historical change amplitudes. When the value difference falls into more historical fluctuation items, the resulting fluctuation ratio is larger; when the value difference falls into fewer historical fluctuation items, the resulting fluctuation ratio is smaller. When calculating the deviation ratio, three adjacent historical records in the same-name sequence are read sequentially according to historical time, forming the previous historical record, the middle historical record, and the next historical record. The absolute value of the difference between the middle historical record value and the average of the previous and next historical records is taken to generate a historical deviation value. Then, the absolute value of the difference between the measurement point record value on the surface to be cleaned and the average of the previous and next historical records is taken to generate a real-time deviation value corresponding to these three adjacent historical records. As the three adjacent historical records move forward along the same-name sequence, all historical deviation values ​​and their corresponding real-time deviation values ​​are obtained sequentially. Subsequently, for each historical record in the same-name sequence... The system reads two adjacent historical records at the current location. If the historical record is at the beginning or end of the same sequence, its deviation ratio is set to zero. If the historical record is in the middle, the system reads the real-time deviation value obtained with the historical record as the middle item, counts the number of historical deviation values ​​that are not greater than the real-time deviation value, and divides this number by the total number of historical deviation values ​​to obtain the deviation ratio of the historical record. If the number of historical deviation values ​​is zero, the deviation ratio of all historical records in the same sequence is set to zero. Through this process, the current measurement point record is no longer compared with a single historical record, but is further placed in a local three-point structure to determine its deviation position. After obtaining the fluctuation ratio, deviation ratio, and value difference sequence, the historical records in the same sequence are read one by one. The sum of the fluctuation ratio and deviation ratio of the historical record is calculated and written under the historical surface number to which the historical record belongs. Then, all historical surface numbers are arranged in ascending order of sum. If the sums are the same, the historical surface numbers are arranged in ascending order of value difference in the value difference sequence, forming the landing sequence of the measurement point record. The historical surface number that comes first indicates that the current measurement point record is closer to the historical record of the same measurement point in the historical surface in terms of historical fluctuation position and historical deviation position. If there is only one historical record in the same sequence, the historical surface number to which the historical record belongs is directly written as the first place of the landing sequence. If the same sequence is empty, the landing sequence of the measurement point record is written as an empty sequence. After all the measurement point records in the surface to be cleaned have been processed in the above way, all landing sequences corresponding one-to-one with the surface to be cleaned are obtained, which are used for subsequent extraction of common number set and location of conflict records. Through the above processing, each measurement point record no longer generates only a list of historical surface numbers directly arranged by value difference. Instead, it undergoes two layers of calculation: adjacent fluctuation counting and three-point deviation counting, before being written into the landing point sequence generation process. This allows single-point value difference, historical adjacent changes, and local structural offsets to participate in the landing point sorting. After this processing, even if a measurement point record has a small value difference from a historical record, it will still be moved to the back of the landing point sequence as long as it falls outside the historical fluctuation range or has an abnormal deviation position. Conversely, even if a measurement point record does not have the minimum value difference, it will still be ranked at the front as long as its fluctuation position and deviation position are closer to the historical trend. In practical applications: If the surface to be cleaned contains a temperature measurement point record, five historical records for that temperature measurement point are extracted sequentially from the historical surface sequence, with record values ​​of 20, 23, 25, 24, and 26 respectively. First, a sequence with the same name is formed, and then the fluctuation sequence 3, 2, 1, 2 is obtained. If the current measurement point record value is 24, the value difference sequence is 4, 1, 1, 0, 2 respectively. Then, the number of fluctuation values ​​not greater than 4, not greater than 1, not greater than 1, not greater than 0, and not greater than 2 in the fluctuation sequence is counted respectively, and divided by the number of fluctuation sequence items 4 to obtain the fluctuation ratio of each historical record. Continue to form a historical deviation value with three adjacent historical records, and then form a real-time deviation value with the current measurement point record value and the average of each group of three points to obtain the deviation ratio of each historical record. Finally, the fluctuation ratio and the deviation ratio are added together, and the historical surface numbers are arranged according to the sum and the value difference to obtain the landing point sequence of the temperature measurement point record. After the same processing is repeated for the remaining measurement point records in the cleaned surface, the entire landing point sequence is obtained for subsequent steps to continue.

[0019] S3. Compare the historical surface numbers in each landing sequence, extract the common numbers of all landing sequences, and generate a common number set. When the common number set is not empty, take the first number as the target surface number according to the sequence number and sorting of each number in each landing sequence. When the common number set is empty, delete one measurement point record of the surface to be cleaned in turn and extract it repeatedly. Take the deleted measurement point record that makes the common number set non-empty for the first time as the conflict record. In this embodiment, S3 is used to determine the historical surface number that is commonly pointed to among all landing point sequences, and to reverse locate the measurement point record that caused the split when there is no common pointing. During processing, the landing point sequence of each of the measurement point records in the surface to be cleaned is read first, and then the historical surface number that is simultaneously contained in all sequences is searched in these landing point sequences. If multiple historical surface numbers that coexist are found, the positions of these historical surface numbers in each landing point sequence are compared, and the one with a higher degree of convergence is selected as the target surface number. If no historical surface number that coexists is found, the measurement point records are deleted one by one according to the arrangement order of the measurement point records in the surface to be cleaned, and the above process is repeated after deletion. The conflict record is deduced from the recovery result after deletion. The measurement point records in the surface to be cleaned are first arranged in ascending order of measurement point name, and this arrangement order is used as the writing order and deletion order of the subsequent sequence number group. The historical surface number in each landing point sequence is written according to the original arrangement order of the landing point sequence, and the sequence number is incremented from 1. If a certain landing point sequence is empty, the comparison is directly written as the common number set is empty. The implementation process includes the following steps: After reading all the landing point sequences corresponding to the surface to be cleaned, first read the historical surface numbers in the first landing point sequence one by one, and then determine whether the historical surface number appears simultaneously in all the other landing point sequences; if they appear simultaneously, write the historical surface number into the common number set; if they do not appear simultaneously, do not write it into the common number set; after checking all the historical surface numbers in the first landing point sequence in the above manner, the common number set is obtained; then, for each historical surface number in the common number set, according to the arrangement order of the measurement point records in the surface to be cleaned, read the position of the historical surface number in each landing point sequence, and then follow the same... The sequence numbers are written sequentially into the sequence number group; the sum of all the sequence numbers in the sequence number group is obtained; the difference between two adjacent items in the sequence number group is taken and the absolute value is calculated, and then the absolute values ​​of all the difference are summed to obtain the sum of the absolute values ​​of the differences between adjacent sequence numbers; the reciprocal of each sequence number in the sequence number group is taken, and then the reciprocals are summed to obtain the reverse sequence number sum; if the common number set is empty, the sequence number group, the sequence number sum, the sum of the absolute values ​​of the differences between adjacent sequence numbers, and the reverse sequence number sum will not be generated in this segment, and the subsequent deletion process will proceed directly; through this process, each historical face number that co-occurs corresponds to a set of positional information that can be further compared, and it no longer only stays at the level of "whether they co-occur"; When the common number set is not empty, read the historical face numbers in the common number set one by one, along with the sum of their reversed numbers and the sum of the absolute values ​​of the differences between adjacent numbers. Calculate the convergence value by dividing the sum of the reversed numbers by the sum of the absolute values ​​of the differences between adjacent numbers. The smaller the sum of the absolute values ​​of the differences between adjacent numbers, the closer the points in each landing sequence are to the historical face number; the larger the sum of the reversed numbers, the earlier the historical face number appears in each landing sequence. After generating the convergence values ​​for all historical face numbers, first arrange the historical face numbers from largest to smallest. If the convergence values ​​are the same, then arrange the historical face numbers from smallest to largest by their sum of numbers. If the sums of numbers are still the same, then arrange the historical face numbers from smallest to largest. After arrangement, the historical face number at the top is written as the target face number. If there is only one historical face number in the common number set, then that historical face number is directly written as the target face number, and no further parallel comparisons are performed. After this processing, the target face number is determined by both the co-occurrence relationship and the positional convergence relationship. When the common number set is empty, delete each measurement point record in the cleaned surface sequentially, starting from the first record. For each deleted record, simultaneously delete the corresponding landing point sequence. Then, repeat the common number set extraction, sequence group generation, and aggregation value calculation for the remaining landing point sequences. If the common number set is not empty after deletion, read the first historical surface number as the first historical surface number, count the number of historical surface numbers in the current common number set as the common number count, and read the aggregation value and sequence number sum of the first historical surface number. Write the deleted measurement point record, the first historical surface number, the common number count, the aggregation value, and the sequence number sum into a backoff item, while setting the cross sum to zero. If the common number set is still empty after deletion, read the records in pairs... For each remaining landing point sequence after deletion, the number of common historical facet numbers in each pair of landing point sequences is counted, and the sum of the counts for all sequence pairs is written as the cross sum. If only one landing point sequence remains after deletion, the cross sum is written as zero. Subsequently, the deleted measurement point record is written into the backtracking item, and the first historical facet number is written as null, the number of common numbers is written as zero, the convergence value is written as zero, and the sequence number sum is written as zero. After all measurement point records have been deleted, a backtracking item corresponding to each deleted measurement point record is obtained. Through this process, if the common pointing can be restored after deleting a measurement point record, the measurement point record will have the restoration result in the backtracking item. If the common pointing still cannot be restored, the pairwise intersection results between the remaining landing point sequences are retained for subsequent comparisons. After obtaining all rollback items, first arrange the rollback items in descending order of the number of common numbers; if the number of common numbers is the same, arrange the rollback items in descending order of the convergence value; if the convergence values ​​are the same, arrange the rollback items in descending order of the cross sum; if the cross sums are the same, arrange the rollback items in ascending order of the sequence number sum; if the sequence number sums are still the same, arrange the rollback items from front to back according to the original arrangement position of the deleted measurement point record in the surface to be cleaned; after all rollback items are arranged, the deleted measurement point record in the rollback item at the top is written as the conflict record; if a rollback item has a non-empty first historical surface number, while the first historical surface numbers of the other rollback items are all empty, then the rollback item is directly placed at the top; if the first historical surface numbers of all rollback items are all empty, then the final sorting is completed according to the cross sum and the original arrangement position; after this processing, the conflict record is determined by the recovery capability after deletion and the local convergence degree of the remaining landing point sequence after deletion; Through the above processing, we first search for common historical surface numbers among all landing point sequences, and then compare the degree of positional convergence among the common historical surface numbers. When the common number set is not empty, we can directly write the target surface number. When the common number set is empty, we generate backtracking items according to the recovery results after deleting a single measurement point record, and compare each backtracking item in a progressive manner of common number, convergence value, cross sum and sequence number sum. This can locate the measurement point record that destroys the common landing point from the surface to be cleaned. After this processing, the acquisition of the target surface number no longer depends only on simple intersection, and the acquisition of conflict records no longer stops at directly selecting the first recovery item after trial deletion. Instead, the common pointing degree after recovery and the local convergence degree before recovery are included in the comparison process. In practical applications: If the surface to be cleaned contains four measurement point records (A, B, C, and D), and the four measurement point records yield landing point sequences, with the landing point sequence for A being {3,5,7,9}, for B {2,5,7,8}, for C {1,5,6,7}, and for D {4,5,7,10}, then the historical surface numbers shared by all landing point sequences are 5 and 7. For historical surface number 5, reading the sequence numbers in the order of A, B, C, and D yields the sequence number group {2,2,2,2}, which calculates to be 8. The sum of the sequence numbers is 0, the sum of the absolute values ​​of the differences between adjacent sequence numbers is 0, and the sum of the reverse sequence numbers is 2. For historical surface number 7, the sequence number group is { Given the sequence 3, 3, 4, 3, the sum of the serial numbers is 13, the sum of the absolute values ​​of the differences between adjacent serial numbers is 1, and the sum of the reverse serial numbers is 1 / 3 + 1 / 3 + 1 / 4 + 1 / 3. Based on this, the convergence values ​​are calculated and sorted. The historical face number 5 is listed first, so 5 is written as the target face number. If there are no historical face numbers that appear in common among the four landing point sequences, the corresponding measurement point records are deleted in the order of A, B, C, and D, and the common number set is recalculated. Assuming that after deleting C, a non-empty common number set is obtained for the first time, and the historical face number at the top is 7, and the number of common numbers, the convergence value, and the sum of serial numbers are all better than the other backtracking items, then C is written as a conflict record for subsequent steps.

[0020] S4. When the target surface number is obtained, calculate the sum of the value difference and time difference between the surface to be cleaned and the historical surface, and generate a valid surface; when the conflict record is obtained, delete the conflict record, generate a valid surface according to the remaining measurement point records, and generate a pseudo real-time group according to the sum of the absolute values ​​of the value differences between the conflict record and each measurement point record in the valid surface. In this embodiment, S4 is used to generate the data results used for subsequent write-back under the target surface number branch and the conflict record branch respectively. When the target surface number has been obtained in the previous stage, it means that all the measurement point records in the surface to be cleaned can fall on the same historical surface. At this time, the numerical offset and time offset between the surface to be cleaned and the historical surface need to be written into the effective surface. When the conflict record has been obtained in the previous stage, it means that a certain measurement point record in the surface to be cleaned has destroyed the overall common landing point. At this time, the conflict record needs to be separated from the surface to be cleaned, and the remaining measurement point records are generated into the effective surface. Then, the separated conflict record is written into the pseudo real-time group for subsequent isolation writing. The entire process revolves around the surfaces to be cleaned, historical surfaces, valid surfaces, conflict records, and pseudo-real-time groups. First, it completes the matching of identical names and the calculation of differences, then it generates valid surfaces and pseudo-real-time groups. This implementation process includes the following steps: After obtaining the target surface number, first search the historical surface sequence one by one according to the historical surface number. Stop the search when a historical surface number with the same target surface number is found, and write that historical surface as the target historical surface. Then read all the measurement point records in the surface to be cleaned, and search for the same historical record in the target historical surface one by one according to the measurement point name. When a historical record with the same name is found, pair the measurement point record with the historical record. If no historical record with the same name is found, the measurement point record will not participate in the calculation of the current value difference and time difference. For each pair of records, read the record value of the measurement point record and the record value of the historical record, subtract the two and take the absolute value. The absolute value of the difference between recorded values ​​is obtained; then the recording time of the measuring point record and the recording time of the historical record are read, the two are subtracted and the absolute value is taken to obtain the absolute value of the time difference between records; as all paired records are calculated in groups, the absolute values ​​of the differences between all records are summed to obtain the sum of the differences, and the absolute values ​​of the time differences between all records are summed to obtain the sum of the time differences; if all measuring point records in the surface to be cleaned have a corresponding historical record in the target historical surface, then the sum of the differences and the sum of the time differences cover all measuring point records in the surface to be cleaned; if only some measuring point records can be paired, then the sum of the differences and the sum of the time differences are generated only from the paired measuring point records. After obtaining the sum of value differences and the sum of time differences, the target surface number, the sum of value differences, the sum of time differences, and the surface to be cleaned are combined and written into a surface result structure. This surface result structure retains all measurement point records in the surface to be cleaned, and the target surface number is written as the history pointer item of this surface result structure, the sum of value differences is written as the numerical offset item, and the sum of time differences is written as the time offset item. After writing, the surface result structure generates a valid surface. If a measurement point record in the surface to be cleaned cannot find a historical record with the same name in the target historical surface, the measurement point record is still retained in the valid surface, but it does not participate in the generation of the sum of value differences and the sum of time differences. If all measurement point records in the surface to be cleaned are matched with the same name, the numerical offset item and the time offset item in the valid surface cover all measurement point records. Through this process, the valid surface under the target surface number branch retains the original group structure of the surface to be cleaned and also carries the numerical offset and time offset results between it and the target historical surface. It can be directly used as a new surface in the historical cleaning results when writing back later. After obtaining the conflict records, the measurement point records are read one by one from the surface to be cleaned, and the measurement point records are compared with the conflict records. If the measurement point record and the conflict record are the same record, they are not written into the set of remaining measurement point records. If the measurement point record and the conflict record are not the same record, they are written into the set of remaining measurement point records. This yields the remaining measurement point records after deleting the conflict records. Then, the remaining measurement point records are arranged in ascending order of measurement point name, and all measurement point records are read in the order they were arranged in the surface to be cleaned before deletion. The position number of the conflict record is counted, and this position number is written as the separation position. The position number of the first measurement point record in the surface to be cleaned is written as 1, and the subsequent measurement point records are incremented sequentially. After the separation position is counted, the remaining measurement point records and the separation positions are combined and written into a surface result structure, and this surface result structure is used to generate a valid surface. This valid surface no longer contains conflict records, but only retains the remaining measurement point records after deleting the conflict records, while also retaining the position of the conflict records in the original surface to be cleaned, for subsequent isolation result writing and review. After obtaining the valid surface under the conflict record branch, each measurement point record in the valid surface is read one by one. The record value of the conflict record and the record value of the measurement point record are read separately. The absolute value of the difference between the two is obtained by subtracting them. As all measurement point records in the valid surface are calculated one by one, the absolute values ​​of all record value differences are summed to obtain the separation sum. Then, the conflict record, separation bit and separation sum are written into an isolation result structure, and this isolation result structure is used to generate a pseudo-real-time group. If there is only one measurement point record left in the valid surface, the separation sum is composed of the absolute value difference between the conflict record and that measurement point record. If there are multiple measurement point records left in the valid surface, the separation sum is composed of the absolute value difference between the conflict record and all the remaining measurement point records. Through this process, not only is the valid surface generated under the conflict record branch, but the numerical separation degree between the conflict record and the remaining measurement point records is also written into the pseudo-real-time group, which can directly enter the isolation result writing process later. After the above processing, the processing results under the target surface number branch are as follows: the surface to be cleaned is retained as a whole, with the sum of the value difference and time difference between it and the target historical surface; the processing results under the conflict record branch are as follows: the surface to be cleaned is split into two parts: the effective surface and the pseudo-real-time group. The remaining measurement point records are entered into the effective surface, and the conflict records are entered into the pseudo-real-time group. At the same time, the separation bit and separation sum are written. After this processing, the target surface number and conflict records given in the previous stage can be placed on the data structure that can be directly written to later. The subsequent updates of historical cleaning results and the writing of isolation results have direct objects to accept. In practical applications: If a surface to be cleaned contains four measurement point records—temperature, pressure, flow rate, and vibration—and the target surface number is 12 obtained in the previous stage, then first locate the target historical surface with historical surface number 12 in the historical surface sequence. Then, pair the four measurement point records (temperature, pressure, flow rate, and vibration) with the corresponding historical records in the target historical surface according to their measurement point names. Calculate the absolute value of the difference between the recorded values ​​and the absolute value of the time difference for each record, and sum them to obtain the sum of the value differences and the sum of the time differences. Finally, combine the target surface number, the sum of the value differences, the sum of the time differences, and the sum of the four measurement point records. Write to the valid surface; if the conflict record obtained in the previous stage is a vibration measurement point record, first delete the vibration measurement point record from the surface to be cleaned, retain the three measurement point records of temperature, pressure and flow, arrange them according to the measurement point name to generate the valid surface, then count the sequence number of the vibration measurement point record in the original surface to be cleaned as the separation position, and calculate the absolute value of the record value difference between the vibration measurement point record and the three measurement point records of temperature, pressure and flow for each record, sum them to obtain the separation sum, and finally write the vibration measurement point record, separation position and separation sum together into the pseudo real-time group for the next stage of processing.

[0021] S5. Append the effective surface to the end of the historical cleaning results, update the historical records of each measuring point according to the measuring point records in the effective surface, write the pseudo real-time group to the isolation results, and output the edge IoT real-time data cleaning results of the current cleaning cycle. In this embodiment, S5 is used to write the valid surfaces obtained in the previous stage back to the historical cleaning results and write the pseudo-real-time group into the isolation results, thereby forming the output results of the current cleaning cycle. During processing, firstly, the surface time and the current surface number that can be written into the historical cleaning results are generated around the valid surfaces, and then the records of each measurement point in the valid surfaces are written back to the historical records of each measurement point. If the pseudo-real-time group has been generated in the previous stage, the isolation time is further calculated and the pseudo-real-time group is written into the isolation results. If the pseudo-real-time group has not been generated in the previous stage, the isolation results retain the original storage content. The entire process revolves around the effective surface, historical cleaning results, historical records of each measuring point, pseudo-real-time groups, and isolation results. The data generated in the previous stage is transformed into continuously accumulating historical data and isolation data in this stage. The implementation process includes the following steps: Before writing back the historical cleaning results, all measurement point records in the valid surface are read and sorted by measurement point name from smallest to largest. After sorting, the recording time of each measurement point record is read one by one, and all recording times are summed to obtain the time sum. Then, the number of measurement point records in the valid surface is counted, and the time sum is divided by the number of measurement point records to obtain the surface time. If the division result has a decimal, the integer value is retained according to the original precision of the timestamp. If the recording time itself is an integer timestamp, the surface time is written as an integer timestamp. If there is only one measurement point record in the valid surface, the recording time of that measurement point record is directly written as the surface time. Through this process, the valid surface is no longer just a set of measurement point records to be written, but also has a surface time that can be used to represent the time position of the set of measurement point records in the historical cleaning results. After obtaining the face time, the last historical face number in the historical cleaning result is read. If a historical face already exists in the historical cleaning result, the last historical face number is incremented by one to obtain the current face number. If no historical face has been written in the historical cleaning result, the current face number is set to 1. Subsequently, the current face number, face time, and valid face are combined into a new historical face, and this new historical face is appended to the end of the historical cleaning result. After appending, the current face number is used as the number field of the new historical face in the historical cleaning result, the face time is used as the time field of the new historical face, and the measurement point records in the valid face are used as the record fields of the new historical face. If the valid face contains any of the following: target face number, sum of value difference, sum of time difference, or separation position, it is written into the new historical face along with the valid face. If the valid face only contains measurement point records, the set of measurement point records is directly written into the new historical face. Through this process, the valid face obtained in the current cleaning cycle is transformed into a new historical face in the historical cleaning result, providing a new source of historical data for subsequent cleaning cycles. After writing the new historical surface, the measurement point records in the valid surfaces are read one by one according to the measurement point name, and a set of historical records with the same measurement point name is searched in the historical records of each measurement point. After finding the historical record with the same name, the current measurement point record is appended to the end of the historical record of the set, and then the historical records of the set are rearranged from earliest to latest according to the recording time. If no historical record with the same name is found, a new set of historical records named with the measurement point name is created, and the current measurement point record is written to the historical record set. As all measurement point records in the valid surfaces complete this process one by one, the historical records of all measurement points are updated synchronously. If a measurement point already has multiple historical records in the historical cleaning results, they are appended and then re-sorted according to the recording time. If a measurement point enters the historical cleaning results for the first time, its historical record set only contains the measurement point record written this time. Through this process, the appending of historical surfaces in the historical cleaning results and the updating of the historical records of each measurement point are completed simultaneously, and subsequent steps can directly obtain the sequence after adding this result when reading the historical records of the same measurement point. After the effective surface is written back, it is determined whether a pseudo-real-time group was generated in the previous stage. If a pseudo-real-time group exists, all measurement point records in the pseudo-real-time group are read first and sorted by measurement point name from smallest to largest. Then, the recording time of each measurement point record is extracted one by one, and the recording times of all records are summed to obtain the isolation time sum. Then, the number of measurement point records in the pseudo-real-time group is counted, and the isolation time sum is divided by the number of measurement point records to obtain the isolation time. If the division result has a decimal, the integer value is retained according to the original precision of the timestamp. If there is only one measurement point record in the pseudo-real-time group, the recording time of that measurement point record is directly written as the isolation time. Subsequently, the pseudo-real-time group and the isolation time are combined. The time-series data is grouped into an isolation unit and written into the isolation result. If the pseudo-real-time group also contains a separation bit and a separation sum, these are written into the isolation result along with the pseudo-real-time group. If no pseudo-real-time group exists, no new content is appended to the isolation result; the current storage content of the isolation result is directly retained. After completing the above processing, the edge IoT real-time data cleaning result for the current cleaning cycle is output. This cleaning result includes at least the appended historical cleaning result and the appended isolation result. Through this processing, the mainstream write-back path and the isolation write-back path are closed simultaneously within the same cleaning cycle, and the processing result of the current cycle can be called by subsequent cycles. Through the above processing, the effective surfaces are transformed into new historical surfaces in this stage and synchronously written into the historical records of each measuring point. The pseudo-real-time groups are transformed into new isolation units in this stage and written into the isolation results. The results of the current cleaning cycle are thus entered into a continuously accumulating data structure. The surface time and isolation time are both obtained by summing the recorded times within the group and dividing by the number of records, so that both historical surfaces and isolation units have a time position that can continue to participate in subsequent processing. The current surface number is generated by adding one to the last historical surface number, so that the historical surfaces in the historical cleaning results can be arranged in ascending order of writing. In practical applications: If the effective surface obtained in a certain cleaning cycle contains three measurement point records for temperature, pressure, and flow rate, with recording times of 100, 102, and 101 respectively, then first arrange the three measurement point records according to the measurement point name, then sum 100, 102, and 101 to obtain 303, and divide 303 by 3 to obtain the surface time 101; if the last historical surface number in the historical cleaning result is 18, then write 19 as the current surface number, and then append the current surface number 19, surface time 101, and the effective surface to the end of the historical cleaning result; subsequently, the temperature, pressure, and flow rate records are respectively... Three measurement point records are appended to their respective measurement point history records with the same name and rearranged according to the recording time. If a pseudo-real-time group is also obtained in the same cleaning cycle, and there is only one vibration measurement point record in the pseudo-real-time group with a recording time of 103, then 103 is directly written as the isolation time, and the pseudo-real-time group and the isolation time 103 are written into the isolation result. If the pseudo-real-time group also has a separation position and a separation sum, then they are written together with the pseudo-real-time group. After processing, a new historical surface is added to the historical cleaning result, and a new isolation unit is added to the isolation result for subsequent cleaning cycles to continue reading.

[0022] Working principle: First, within the current cleaning cycle, real-time records of the same measurement point group are organized into a surface to be cleaned according to their arrival order. Then, from the historical cleaning results, the records previously retained by the same measurement point group are reorganized into a historical surface sequence. Subsequently, for each measurement point record in the surface to be cleaned, the historical records of the same measurement point are searched in the historical surface sequence, and a landing point sequence is generated from three perspectives: value difference, adjacent fluctuation, and local deviation. Then, all landing point sequences are compared. If there is a common historical surface number, the target surface number is determined. If not, measurement point records are deleted one by one for backtracking comparison to find conflicting records that disrupt the overall common landing point. After that, if the target surface number is obtained, the entire surface to be cleaned is retained as a valid surface. If conflicting records are obtained, the conflicting records are separated to form a pseudo-real-time group, and the remaining records are retained as valid surfaces. Finally, the valid surfaces are written back to the historical cleaning results, and the pseudo-real-time groups are written to the isolation results as the output of the current cleaning cycle for continued use in the next cycle. For example, in an edge IoT access scenario within a park, electricity meters, temperature controllers, pump controllers, and vibration sensors are simultaneously connected to the edge gateway. However, their reporting times are not always perfectly synchronized, and network jitter or retransmissions can cause records from different times to be pieced together. In this case, this solution first organizes the most recently arrived set of records within this short period into a cleansing surface, and then looks for common combinations of these measurement points in the historical cleansing results. If these new records can historically fall on a historical surface near the same time, the entire set of records is retained. If one of the records is mixed in, causing the other records to no longer find a common landing point, this record is identified as a conflict record, separated into a pseudo-real-time group, while the other records are retained and written back to the historical results. After this processing, pseudo-real-time combinations that appear normal on their own but are actually pieced together from data from different times in the edge IoT environment will no longer be mixed into the subsequent analysis and control links.

[0023] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time data cleaning method based on historical data, characterized in that, include: S1. The edge IoT gateway reads the real-time records of the same measurement point group within the current cleaning cycle, arranges the real-time records of each measurement point according to the arrival time of the record, extracts the historical records of each measurement point in the historical cleaning results, generates a historical surface sequence according to the historical time, and generates a surface to be cleaned according to the current cleaning cycle. S2. For each measurement point record in the surface to be cleaned, extract the historical records of measurement points with the same name in the historical surface sequence, calculate the value difference, fluctuation ratio and deviation ratio, arrange the historical surface numbers according to the sum of fluctuation ratio and deviation ratio and value difference, and generate the landing point sequence. S3. Compare the historical surface numbers in each landing point sequence, extract the common numbers of all landing point sequences, and generate a common number set; if the common number set is not empty, take the first number as the target surface number according to the sequence number and order of each number in each landing point sequence; if the common number set is empty, delete one measurement point record of the surface to be cleaned in sequence and extract repeatedly, and take the deleted measurement point record that makes the common number set non-empty for the first time as the conflict record; S3 includes: S3-1. Compare the historical face numbers in each landing sequence, extract the common numbers of all landing sequences and generate a common number set; for each historical face number in the common number set, read the occurrence sequence number in each landing sequence, generate a sequence number group, and calculate the sum of the sequence numbers, the sum of the absolute values ​​of the differences between adjacent sequence numbers and the sum of the reverse sequence numbers, where the sum of the reverse sequence numbers is the sum of the reciprocals of each occurrence sequence number. S3-2. When the common number set is not empty, calculate the convergence value for each historical face number. The convergence value is the sum of the reverse sequence number and the sum of the absolute values ​​of the differences between the number and the adjacent sequence number. Then, arrange the historical face numbers in descending order according to the convergence value. When the convergence values ​​are the same, arrange the historical face numbers in ascending order according to the sequence number. Take the first historical face number as the target face number. S3-3. When the common number set is empty, delete one measurement point record from the surface to be cleaned in sequence, and repeat the common number set extraction, sequence group generation, and convergence value calculation for the deleted landing point sequence; when the common number set is not empty after deletion, read the first historical surface number, the number of common numbers, the convergence value of the first historical surface number, and the sequence number sum to generate a rollback item; otherwise, compare each landing point sequence after deletion pairwise, count the number of common historical surface numbers for each sequence and sum them to generate a cross sum, and at the same time write the first historical surface number as empty, the number of common numbers as zero, the convergence value as zero, and the sequence number sum as zero to generate a rollback item; S3-4. Compare each rollback item. First, sort them in descending order by the number of common numbers, then in descending order by the convergence value, then in descending order by the crossover and the sum, and finally in ascending order by the sequence number. Take the deleted measurement point record in the first rollback item as the conflict record. S4. When the target surface number is obtained, calculate the sum of the value difference and time difference between the surface to be cleaned and the historical surface, and generate a valid surface; when the conflict record is obtained, delete the conflict record, generate a valid surface according to the remaining measurement point records, and generate a pseudo real-time group according to the sum of the absolute values ​​of the value differences between the conflict record and each measurement point record in the valid surface. S5. Append the effective surface to the end of the historical cleaning results, update the historical records of each measuring point according to the measuring point records in the effective surface, write the pseudo real-time group to the isolation results, and output the edge IoT real-time data cleaning results of the current cleaning cycle.

2. The real-time data cleaning method based on historical data according to claim 1, characterized in that: S1 includes: S1-1. The edge IoT gateway sorts the real-time records in the current cleaning cycle according to the measurement point identifier, sorts them according to the arrival time and reverses the numbering, and gathers the real-time records of each measurement point according to the same reverse sequence number to generate real-time candidate surfaces. It counts the number of missing measurement points and the arrival span of each real-time candidate surface, sorts them according to the number of missing measurement points and the arrival span, and takes the first real-time candidate surface as the surface to be cleaned. S1-2. The edge IoT gateway extracts the historical records of the same measurement point group, counts the number of historical records of each measurement point and sorts them by the number of records. The first measurement point is taken as the anchor measurement point. For each historical record in the anchor measurement point, the historical record with the highest absolute value of time difference is selected from the historical records of the other measurement points. This historical record is combined with the historical record of the anchor measurement point to generate a historical candidate surface. The median of the time of each record in each historical candidate surface is taken as the surface time. The sum of the absolute values ​​of the time difference between each record time and the surface time is taken as the aggregation value. S1-3. Compare the historical records of each candidate face one by one. If there are the same historical records, retain the candidate face with the highest aggregation value and delete the other candidate face. Then sort the remaining candidate faces by face time to generate a historical face sequence.

3. The real-time data cleaning method based on historical data according to claim 2, characterized in that: S2 includes: S2-1. For each measurement point record in the surface to be cleaned, extract the historical records of measurement points with the same name in the historical surface sequence, arrange them according to historical time and generate a sequence with the same name, calculate the absolute value of the difference between each historical record in the sequence with the previous historical record, and generate a fluctuation sequence. S2-2. Read the measurement point records and the historical records in the same sequence, calculate the absolute value of the difference between each measurement point record and each historical record, generate the value difference sequence, and count the number of fluctuation items in the fluctuation sequence that are not greater than each value difference. Calculate the fluctuation ratio of each historical record based on the number of fluctuation items and the number of items in the fluctuation sequence.

4. The real-time data cleaning method based on historical data according to claim 3, characterized in that: S2 further includes: S2-3. Read three adjacent historical records in the same name sequence according to the historical time sequence. The historical deviation value is generated by subtracting the absolute value of the average of the historical records before and after from the value of the middle historical record. The real-time deviation value is generated by subtracting the absolute value of the average of the historical records before and after from the value of the measurement point record. The number of historical deviation items that are not greater than the real-time deviation value is counted in all historical deviation values. The deviation ratio of each historical record is calculated based on the number of historical deviation items and the number of historical deviation values. S2-4. Calculate the sum of the fluctuation ratio and deviation ratio of each historical record, arrange the historical surface numbers of each historical record in ascending order of the sum, and arrange the historical surface numbers in ascending order of the value difference sequence when the sums are the same, so as to generate the landing point sequence of the measurement point records.

5. The real-time data cleaning method based on historical data according to claim 1, characterized in that: S4 includes: S4-1. When obtaining the target surface number, extract the historical surface pointed to by the target surface number in the historical surface sequence, match the measurement point records in the surface to be cleaned with the historical records in the historical surface according to the measurement point name, calculate the absolute value of the difference between the recorded values ​​and the absolute value of the time difference between the records one by one, and sum them to generate the sum of the value difference and the sum of the time difference respectively. S4-2. Write the sum of the value difference and the sum of the time difference into the surface to be cleaned under the target surface number, and record all the measurement points in the surface to be cleaned to generate a valid surface.

6. The real-time data cleaning method based on historical data according to claim 5, characterized in that: S4 further includes: S4-3. When obtaining conflict records, delete the conflict records from the surface to be cleaned, arrange the remaining measurement point records according to the measurement point name, count the sequence number of the conflict records in the surface to be cleaned and write it into the separation position, and then combine the remaining measurement point records with the separation position to generate an effective surface. S4-4. Calculate the absolute value of the difference between the conflict record and each measurement point record in the effective surface, and sum the absolute values ​​of all the difference between the record values ​​to generate the separation sum. Then combine the conflict record, the separation position and the separation sum to generate a pseudo-real-time group.

7. The real-time data cleaning method based on historical data according to claim 6, characterized in that: S5 includes: S5-1. Read the records of each measuring point in the effective surface, arrange the records of each measuring point by the name of the measuring point, extract the recording time of each measuring point record and sum them, and then divide the sum of the recording times by the number of measuring point records to generate the surface time. S5-2. Read the historical face number at the end of the historical cleaning result, add one to the historical face number to generate the current face number, and then append the current face number, face time, and valid face to the end of the historical cleaning result.

8. The real-time data cleaning method based on historical data according to claim 7, characterized in that: The S5 also includes: S5-3. Read the records of each measuring point and the historical records of each measuring point in the effective surface according to the measuring point name, append each measuring point record to the end of the historical records of the same measuring point, and rearrange the historical records of each measuring point according to the recording time. S5-4. When a pseudo-real-time group exists, extract the records of each measurement point in the pseudo-real-time group, arrange the records of each measurement point by the measurement point name, extract the recording time of each measurement point record and sum them, then divide the sum of the recording times by the number of measurement point records to generate the isolation time, and write the pseudo-real-time group and the isolation time into the isolation result; when no pseudo-real-time group exists, retain the isolation result and output the edge IoT real-time data cleaning result of the current cleaning cycle.

Images