A data preprocessing guard method for a big data processing system

Abstract

The present invention relates to a data preprocessing guard method for a big data processing system, which belongs to the field of big data processing technology, and solves the problems of low precision and low efficiency of subsequent data processing and the inability of the interface to display all data points caused by existing big data preprocessing methods. The method includes: filtering and time-aligning the data to obtain filtered and aligned data; performing hierarchical sampling on the filtered and aligned data to form multi-level sampling data; re-sampling the data that failed to be sampled to obtain multi-level sampling data; compressing the multi-level sampling data to obtain compressed multi-level sampling data; creating a data import process to batch import the compressed multi-level sampling data into a database. The data preprocessing method of the present invention can ensure the integrity of the data, improve the processing precision and efficiency of subsequent big data processing, and at the same time can display the multi-level sampling data without distortion, and can be widely used in the field of big data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data preprocessing guard method for a big data processing system. Background Art

[0002] In actual use, the domestic big data processing system will receive data from multiple redundant data forwarding computers composed of the same single machine. Therefore, there is a large amount of redundant data received by the big data processing system.

[0003] At the same time, when the big data processing system synchronizes time with the host computer, there are three situations between the synchronized system time and the original system time: the synchronized system time and the original system time are unchanged, the synchronized system time is faster than the original system time, and the synchronized system time is slower than the original system time. When the synchronized system time is slower than the original system time, since the system will use the current system time as the timestamp of the received data, the timestamp of the first received data frame will be later than the timestamp of the later received data frame, that is, the timing relationship of the data is misaligned.

[0004] Big data processing systems are faced with a huge amount of data. In the existing technology, large amounts of data are usually directly stored in the database, which often leads to database congestion and server lag.

[0005] In the big data preprocessing stage, when the data volume is relatively large, the test data points cannot be displayed all at once on the interface, and the full picture of the data cannot be seen.

[0006] The above-mentioned data redundancy, time series misalignment, and direct storage of large amounts of data will reduce the subsequent big data processing accuracy and efficiency; all data points cannot be displayed on the interface, making it impossible to grasp the overall picture of data changes; to solve the above-mentioned technical problems, the present application provides a data preprocessing and protection method for a big data processing system, which can eliminate data redundancy, correct time series, and use batch loading mode for data import, which can greatly improve the subsequent big data processing accuracy and efficiency. At the same time, a hierarchical sampling method is used to form sampling point data of different levels, so that in the data preprocessing stage, the overall picture of the data can be quickly seen at a high level. When the level of data browsing is reduced, more and more data details will be seen. Summary of the invention

[0007] In view of the above analysis, an embodiment of the present invention aims to provide a data preprocessing guard method for a big data processing system, so as to solve the problems of low subsequent data processing accuracy, low efficiency and inability of the interface to display all data points caused by existing big data preprocessing methods.

[0008] On the one hand, an embodiment of the present invention provides a data preprocessing guard method of a big data processing system, the method comprising:

[0009] Filter and time-align the data to obtain filtered and aligned data;

[0010] The filtered and aligned data are sampled in stages to form multi-stage sampling data; the data that failed sampling is re-sampled to obtain multi-stage sampling data;

[0011] Compressing the multi-level sampling data to obtain compressed multi-level sampling data;

[0012] Create a data import process to import the compressed multi-level sampling data into the database in batches.

[0013] Further, filtering the data includes:

[0014] Calculate the MD5 checksum of the received data packet and determine whether the checksum is the same as the MD5 checksum of the received data packet. If they are the same, discard the data packet.

[0015] Further, the time alignment processing of the data includes:

[0016] When the synchronized system time is greater than or equal to the original system time, the synchronized system time is directly used as the timestamp of the received data packet;

[0017] When the synchronized system time is less than the original system time, for the subsequently received data packets, the original system time is used as the starting point, and each fixed time interval Δt is used as the timestamp of the data packet until the synchronized system time is greater than or equal to the timestamp of the currently received data packet;

[0018] When the synchronized system time is greater than or equal to the timestamp of the currently received data packet, the synchronized system time continues to be used as the timestamp of the data packet.

[0019] Further, the step of performing hierarchical sampling on the filtered and aligned data to form multi-level sampling data includes:

[0020] Arranging the filtered and aligned data in chronological order to form a current data curve;

[0021] Based on the peak and trough positions of the current data curve, the data curve is divided into a number of increasing / decreasing sampling intervals and a number of increasing and decreasing mixed sampling intervals;

[0022] For the increasing / decreasing sampling interval, a simple sampling method is used to perform sampling to obtain sampling points of the increasing / decreasing sampling interval;

[0023] For the increasing and decreasing mixed sampling interval, a sampling method based on peak detection is adopted to obtain the sampling points of the increasing and decreasing mixed sampling interval;

[0024] Merging the sampling points of each increasing / decreasing sampling interval and the sampling points of each mixed sampling interval in chronological order to obtain the sampling points of the current data curve;

[0025] The data curve formed by the sampling points of the current data curve is used as the current data curve for the next level of sampling, and the above steps are repeated to achieve hierarchical sampling.

[0026] Further, the sampling method based on peak detection is used to obtain the sampling points of the increasing and decreasing mixed sampling intervals, including:

[0027] A peak sampling method is used to obtain the peak point of the mixed sampling interval;

[0028] The mixed sampling interval is sampled using a simple sampling method to obtain a first sampling point of the mixed sampling interval;

[0029] The peak point and the first sampling point are combined in time sequence, and duplicate data points are removed to obtain sampling points of the mixed sampling interval.

[0030] Further, the adopting the peak sampling method to obtain the peak point of the mixed sampling interval includes:

[0031] According to the time sequence, the mixed sampling interval is divided into a plurality of sampling sub-intervals in sequence with N data points as a group;

[0032] The data point type in each sampling sub-interval is identified, and the maximum and minimum points are determined from each data point type. The maximum and minimum points that can characterize the data curve envelope are retained, and the remaining points are discarded; the maximum and minimum points of each sub-interval are merged in chronological order to form the peak point of the mixed sampling interval.

[0033] Further, the identifying of the data point type in each sampling subinterval and determining the maximum value point and the minimum value point from each data point type includes:

[0034] For the data point set {V1,V2,...,V N},

[0035] If V i-1 >V i And V i >V i+1 , then the data point V i Type is Normal;

[0036] If V i-1 >V i And V i =V i+1 , then the data point V iType: left groove;

[0037] If V i-1 >V i And V i <V i+1 , then the data point V i The type is pointed bottom;

[0038] If V i-1 =V i And V i >V i+1 , then the data point V i Type: left step;

[0039] If V i-1 =V i And V i =V i+1 , then the data point V i Type is Normal;

[0040] If V i-1 =V i And V i <V i+1 , then the data point V i Type: right groove;

[0041] If V i-1 <V i And V i >V i+1 , then the data point V i The type is pointed roof;

[0042] If V i-1 <V i And V i =V i+1 , then the data point V i Type: right step;

[0043] If V i-1 <V i And V i <V i+1 , then the data point V i Type is Normal;

[0044] The above-mentioned sharp bottom point is taken as the minimum point, and the sharp top point is taken as the maximum point; for adjacent left groove points and right groove points, the left groove point is taken as the minimum point, for adjacent left step points and right step points, the right step point is taken as the maximum point, and the first and last data points of the sampling sub-interval are taken as the maximum point or the minimum point.

[0045] Further, the simple sampling method includes:

[0046] According to the time sequence, the sampling interval is divided into multiple sampling sub-intervals with N data points as a group;

[0047] A maximum value point and a minimum value point are extracted in each sampling sub-interval; and the maximum value point and the minimum value point of each sampling sub-interval are combined in time sequence to form a sampling point of the sampling interval.

[0048] Further, the data that failed to be sampled is re-sampled to obtain multi-level sampling data, including:

[0049] Get the data file where sampling failed, change the file name suffix to .txt, and add the system timestamp after the file name;

[0050] Obtain the server with the smallest current load based on the load balancing algorithm;

[0051] A data sampling process is created on the server to perform data sampling and obtain multi-level sampling data.

[0052] Furthermore, the data import process is created to batch import the compressed multi-level sampling data into the database, including:

[0053] When the server's maximum process number limit and data import limit are met, a data import process is created in the server to import data; otherwise, a data import process is created after the current process is completed.

[0054] After the data import process completes the task, it is determined whether the task is completed correctly based on the return value. If so, all temporary files generated are deleted; if not, the data files that failed to be imported are moved to the error file data directory, and the data import process is re-established for import.

[0055] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0056] 1. By filtering the data, it is possible to remove duplicate data obtained from redundant computer hosts, reduce the amount of data processing, and thus improve the efficiency of subsequent data processing;

[0057] 2. Perform time alignment on the data to avoid the situation where the timestamp of the data received later is earlier than the timestamp of the data received earlier, so that the data can be arranged in the actual order of reception, which can improve the accuracy of the received data and further improve the processing accuracy of subsequent big data processing.

[0058] 3. By adopting hierarchical sampling and obtaining multi-level sampling data, you can quickly see the overall picture of the data at a high level. When the level of data browsing is reduced, you will see more and more data details.

[0059] 4. The present invention divides the data curve to be sampled into increasing / decreasing intervals and increasing / decreasing mixed sampling intervals, adopts simple sampling for the increasing / decreasing intervals, and adopts a sampling method based on peak detection for the increasing / decreasing mixed sampling intervals. The sampling method based on peak detection first obtains the peak point of the mixed sampling interval through peak sampling, and the above peak point forms the outline of the data curve, and then obtains the first sampling point of the mixed sampling interval through simple sampling, merges the first sampling point with the peak point, and removes duplicate points to obtain the sampling points of the mixed sampling interval. The above sampling method can retain the data outline and avoid waveform distortion during sampling.

[0060] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0062] Figure 1 A schematic diagram of a data preprocessing process in one embodiment of the present invention; DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0064] A specific embodiment of the present invention discloses a data preprocessing guard method for a big data processing system, such as Figure 1 As shown, the method includes:

[0065] S10, filtering and time-aligning the data to obtain filtered and aligned data;

[0066] S20, performing hierarchical sampling on the filtered and aligned data to form multi-level sampling data; re-sampling the data that failed sampling to obtain multi-level sampling data;

[0067] S30, compressing the multi-level sampling data to obtain compressed multi-level sampling data;

[0068] S40, creating a data import process to batch import the compressed multi-level sampling data into the database.

[0069] Compared with the prior art, the data preprocessing guard method provided in this embodiment can remove duplicate data by filtering and time-aligning the data, and timestamp the received data in the order of reception, which is conducive to improving the subsequent big data processing accuracy and processing efficiency. At the same time, by adopting a hierarchical sampling method to obtain multi-level sampling data, the overall picture of the data can be seen at a high level. When the level of data browsing is reduced, more and more data details will be seen. By compressing the multi-level sampling data, and then establishing a data import process for the compressed sampling data to import the data into the database in batches, the data import efficiency can be improved without causing database congestion and server jams.

[0070] In step S10, firstly, data sent to the designated port of the server needs to be received from the network, and then the data is filtered and time-aligned;

[0071] The big data processing system communicates with the host computer through the UDP protocol. The host computer sends data packets to the big data processing system according to the UDP protocol. When receiving data sent to the specified port of the server, a host computer address list will be set. When the address of the received data packet is not in the list and the data packet is a data packet that needs to be received, the address will be added to the receiving address list. When receiving data packets sent by host computers with multiple IP addresses (the data sent by multiple host computers are the same), select the host computer corresponding to any IP address in the list as the receiving matching address, and the data packets sent by other host computer addresses will be discarded. This is a processing of data redundancy during the receiving process.

[0072] However, it is inevitable that the data actually received will still be repeated. In order to filter out the repeated data, the following method is used to filter the data:

[0073] Calculate the MD5 checksum of the received data packet and determine whether the checksum is the same as the MD5 checksum of the received data packet. If they are the same, discard the data packet.

[0074] During implementation, the check codes of the data frames of the latest 100 packets may be retained. If the check codes are repeated, the data frame is determined to be a repeated data frame.

[0075] During the operation of the big data processing system, time synchronization with the host computer is required. There are three situations between the system time after synchronization and the original system time: the system time after synchronization does not change with the original system time, the system time after synchronization is faster than the original system time, and the system time after synchronization is slower than the original system time. When the system time after synchronization is slower than the original system time, since the system will use the current system time as the timestamp of the received data, the timestamp of the data frame received first will be later than the timestamp of the data frame received later, that is, the timing relationship of the data is misaligned.

[0076] In order to solve the above-mentioned problem of misaligned timing relationship, the present invention adopts the following method to perform time alignment processing on the data:

[0077] When the synchronized system time is greater than or equal to the original system time, the synchronized system time is directly used as the timestamp of the received data packet;

[0078] When the synchronized system time is less than the original system time, for the subsequently received data packets, the original system time is used as the starting point, and each fixed time interval Δt is used as the timestamp of the data packet until the synchronized system time is greater than or equal to the timestamp of the currently received data packet.

[0079] Preferably, the fixed time Δt=the time interval for the host computer to send data / 4.

[0080] When the synchronized system time is greater than or equal to the timestamp of the currently received data packet, the synchronized system time continues to be used as the timestamp of the data packet.

[0081] After the data is time-aligned, the timestamps of the received data packets are guaranteed to be arranged in the order of receipt. When the data is subsequently sampled and otherwise processed, the authenticity of the data can be guaranteed and the accuracy of subsequent data processing can be improved.

[0082] In step S20, in order to solve the problem that all data points cannot be displayed on the interface, the present application adopts a hierarchical sampling method to sample the data, which can obtain sampling point data of different levels, so that in the data preprocessing stage, the overall picture of the data can be quickly seen at a high level. When the level of data browsing is reduced, more and more data details will be seen.

[0083] Specifically, the step of performing hierarchical sampling on the filtered and aligned data to form multi-level sampling data includes:

[0084] S201, arranging the filtered and aligned data in chronological order to form a current data curve;

[0085] S202, dividing the data curve into a plurality of increasing / decreasing sampling intervals and a plurality of increasing and decreasing mixed sampling intervals based on the peak and trough positions of the current data curve;

[0086] S203, for the increasing / decreasing sampling interval, adopt a simple sampling method to perform sampling, and obtain sampling points of the increasing / decreasing sampling interval;

[0087] S204: For the increasing and decreasing mixed sampling interval, a sampling method based on peak detection is used to perform sampling to obtain sampling points of the increasing and decreasing mixed sampling interval;

[0088] S205, merging the sampling points of each increasing / decreasing sampling interval and the sampling points of each mixed sampling interval in chronological order to obtain the sampling points of the current data curve;

[0089] S206: Use the data curve formed by the sampling points of the current data curve as the current data curve for next level sampling, and repeat the above steps to achieve hierarchical sampling.

[0090] When processing big data, we often encounter data with a high frequency of changes, such as sawtooth wave data. In order to achieve high-fidelity sampling of the above types of data, we first need to divide the sampling interval.

[0091] Specifically, in step S202, dividing the data curve into a plurality of increasing / decreasing sampling intervals and a plurality of increasing and decreasing mixed sampling intervals based on the peak and trough positions of the current data curve comprises the following steps:

[0092] S2021. Identify all peak and trough positions of the current data curve, each pair of trough-peak / peak-trough forms a preset monotonically increasing / decreasing interval, and all preset monotonically increasing / decreasing intervals are arranged in chronological order;

[0093] S2022. For one of the preset monotonically increasing / decreasing intervals, if the number of data points in the interval is greater than or equal to a set threshold, the preset monotonically increasing / decreasing interval is used as a monotonically increasing / decreasing sampling interval of the current data curve;

[0094] If the number of data points in a preset monotonically increasing / decreasing interval is less than a set threshold, the preset monotonically increasing / decreasing interval is sequentially merged with the next preset monotonically increasing / decreasing interval until the number of data points after merging is greater than or equal to the set threshold; the interval after merging the multiple preset monotonically increasing / decreasing intervals is used as an increasing and decreasing mixed sampling interval of the current data curve;

[0095] S2023. According to the above step S2022, each preset monotonically increasing / decreasing interval is traversed to form a plurality of increasing / decreasing sampling intervals and a plurality of increasing / decreasing mixed sampling intervals.

[0096] During implementation, the threshold is preferably set to 8. When the threshold is set to the above threshold, the sampling data points at each level after graded sampling can well reflect the contour of the overall data curve, and the sampling effect is better.

[0097] Exemplarily, assuming that the number of data points in the first preset increasing / decreasing interval is greater than 8, the first preset increasing / decreasing interval is used as an increasing / decreasing sampling interval of the current data curve; next, if the number of data points in the second preset increasing / decreasing interval is less than 8, the second preset increasing / decreasing interval is merged with the third preset increasing / decreasing interval. If the total number of data points after the merger of the two is still less than 8, the fourth preset increasing / decreasing interval is merged with the second and third increasing / decreasing intervals. Assuming that the total number of data points after the second to fourth preset increasing / decreasing intervals are merged is greater than 8, which meets the requirements, The above-mentioned second to fourth preset increasing / decreasing intervals are merged as an increasing / decreasing mixed sampling interval for the current data curve; according to the above method, continue to determine whether the number of data points in the fifth preset increasing / decreasing interval meets the requirements; if so, use it as an increasing / decreasing sampling interval for the current data curve; if not, merge it with the next preset increasing / decreasing interval until the requirements are met; and process each preset increasing / decreasing interval in turn according to the above method until the last preset increasing / decreasing interval. Finally, the current data curve can be divided into several increasing / decreasing sampling intervals and several increasing / decreasing mixed sampling intervals.

[0098] In step S203, since the increasing / decreasing sampling intervals are monotonically increasing or decreasing intervals, the data changes slowly, and thus a simple sampling method can be used to achieve accurate sampling of the interval.

[0099] Specifically, the step of sampling the increasing / decreasing sampling interval by using a simple sampling method and obtaining the sampling points of the increasing / decreasing sampling interval comprises the following steps:

[0100] S2031, dividing the increasing / decreasing sampling interval into a plurality of sampling sub-intervals in sequence according to the chronological order and taking N data points as a group;

[0101] S2032, extracting the maximum value point and the minimum value point in each sampling sub-interval;

[0102] S2033: The maximum value point and the minimum value point of each sampling sub-interval are combined in time sequence to form the sampling points of the increasing / decreasing sampling interval.

[0103] When implementing, the N value can be set to 6-9;

[0104] Preferably, the value of N is 8. When N is 8, the obtained sampling points at all levels can well reflect the profile of the data curve.

[0105] In step S204, since the data trends in the increasing and decreasing mixed sampling interval are both increasing and decreasing, and the data changes rapidly, if a simple sampling method is used, the changing contour of the data curve in this interval cannot be obtained, and data distortion may occur. Therefore, a sampling method based on peak detection is used for the increasing and decreasing mixed sampling interval in this application.

[0106] Specifically, the sampling method based on peak detection is used to obtain the sampling points of the increasing and decreasing mixed sampling intervals, including:

[0107] S2041, using a peak sampling method to obtain a peak point of the mixed sampling interval;

[0108] S2042, sampling the mixed sampling interval using a simple sampling method to obtain a first sampling point in the mixed sampling interval;

[0109] S2043: Merge the peak point and the first sampling point in chronological order, and remove duplicate data points to obtain sampling points in the mixed sampling interval.

[0110] Specifically, the adopting the peak sampling method to obtain the peak point of the mixed sampling interval includes:

[0111] S2041-1, dividing the mixed sampling interval into a plurality of sampling sub-intervals in sequence according to the chronological order and taking N data points as a group;

[0112] S2041-2. Identify the data point type in each sampling sub-interval, and determine the maximum and minimum points from each data point type, retain the maximum and minimum points that can characterize the data curve envelope, and discard all other points; the maximum and minimum points of each sub-interval are merged in chronological order to form the peak point of the mixed sampling interval.

[0113] Specifically, identifying the data point type in each sampling subinterval and determining the maximum value point and the minimum value point from each data point type includes:

[0114] For the data point set {V1,V2,...,V N},

[0115] If V i-1 >V i And V i >V i+1 , then the data point V i Type is Normal;

[0116] If V i-1 >V i And V i =V i+1, then the data point V i Type: left groove;

[0117] If V i-1 >V i And V i <V i+1 , then the data point V i The type is pointed bottom;

[0118] If V i-1 =V i And V i >V i+1 , then the data point V i Type: left step;

[0119] If V i-1 =V i And V i =V i+1 , then the data point V i Type is Normal;

[0120] If V i-1 =V i And V i <V i+1 , then the data point V i Type: right groove;

[0121] If V i-1 <V i And V i >V i+1 , then the data point V i The type is pointed roof;

[0122] If V i-1 <V i And V i =V i+1 , then the data point V i Type: right step;

[0123] If V i-1 <V i And V i <V i+1 , then the data point V i Type is Normal;

[0124] The above-mentioned sharp bottom point is taken as the minimum point, and the sharp top point is taken as the maximum point; for adjacent left groove points and right groove points, the left groove point is taken as the minimum point, and for adjacent left step points and right step points, the right step point is taken as the maximum point, and the first and last data points of the sampling sub-interval are retained as maximum points or minimum points.

[0125] During implementation, in step S3011, after the mixed sampling increasing and decreasing mixed sampling interval is divided into multiple sampling sub-intervals according to N points as a group, if the number of points remaining in the final mixed sampling interval is less than N, all the remaining data points are retained as the next level sampling data points.

[0126] Specifically, the step S2042 uses a simple sampling method to sample the mixed sampling interval to obtain the first sampling point of the mixed sampling interval, including the following steps:

[0127] S2042-1. Divide the mixed sampling interval into multiple sampling sub-intervals in sequence according to the time sequence and with N data points as a group;

[0128] S2042-2, extracting the maximum value point and the minimum value point in each sampling sub-interval;

[0129] S2042-3. The maximum value point and the minimum value point of each sampling sub-interval are combined in chronological order to form the first sampling point of the mixed sampling interval.

[0130] During implementation, after the mixed sampling increasing and decreasing mixed sampling interval is divided into multiple sampling sub-intervals according to N points as a group in step S2042-1, if the number of points remaining in the final mixed sampling interval is less than N, all the remaining data points are retained as the first sampling points.

[0131] In step S2043, after obtaining the peak point and the first sampling point of the mixed sampling interval through the above steps, the peak point and the first sampling point are merged in chronological order, and considering that the above peak point and the first sampling point will be repeated, it is also necessary to remove the repeated data points. After removing the repeated data points, the sampling points of the mixed sampling interval can be obtained.

[0132] The present application first obtains the peak points of the mixed sampling interval through the peak sampling method, and the above peak points form the data outline of the mixed sampling interval. After obtaining the above data outline, the mixed sampling interval is simply sampled to obtain the first sampling point, and the above peak point and the first sampling point are merged to obtain the sampling point of the mixed sampling interval. Since the peak points are retained in the above sampling points, even if the peak points are omitted during simple sampling and data distortion occurs, a complete data outline will be formed after merging with the peak points without distortion.

[0133] In step S205, after obtaining the sampling points of the increasing / decreasing sampling interval and the sampling points of the mixed sampling interval, the sampling points of each increasing / decreasing sampling interval and the sampling points of each mixed sampling interval are merged in chronological order to obtain the sampling points of the current data curve.

[0134] To achieve hierarchical sampling, in step S206, the data curve formed by the sampling points of the current data curve is used as the current data curve for the next level of sampling, and the above steps S202-S206 are repeated to obtain multi-level sampling points and achieve multi-level sampling.

[0135] In order to ensure that hierarchical sampling can be achieved for all data, the data files that failed sampling once need to be sampled again.

[0136] If a data file fails to be sampled, the system will modify its file name suffix to .unfinished, and the timestamp in the file name will be erased. To achieve re-sampling of the above data file, the following method is used:

[0137] Get the data file where sampling failed, change the file name suffix to .txt, and add the system timestamp after the file name;

[0138] Obtain the server with the smallest current load based on the load balancing algorithm;

[0139] A data sampling process is created on the server to perform data sampling and obtain multi-level sampling data.

[0140] During implementation, firstly, the filemonitor program is executed on the data processing server specified in [server] in the configuration file data.conf to complete the modification of the data file name.

[0141] According to the acquired data files of each failed sampling, the server load is collected on each server, the server with the smallest load is selected, and a data sampling process is created on the server to perform data sampling. When the maximum number of processes is reached, it is necessary to wait for the current process to complete before scheduling again.

[0142] By adopting a load balancing algorithm to select the server with the smallest load, the sampling efficiency can be improved.

[0143] In order to improve the efficiency of data import, the sampling data at all levels are first compressed before data import;

[0144] Specifically, a dictionary compression method may be used to compress sampling data at each level.

[0145] Since directly importing a large amount of data into the database will cause congestion in the database and server lag, a batch loading mode is used to set the batch loading time, periodically receive the data, parse it, and store it on the disk in text form. When importing data, the files on the disk are read in batches and imported into the database. In this way, the efficiency of database import is greatly optimized, and the low efficiency of database import will not affect data reception and analysis.

[0146] Specifically, the process of creating a data import process to batch import the compressed multi-level sampling data into the database includes:

[0147] When the server's maximum process number limit and data import limit are met, a data import process is created in the server to import data; otherwise, a data import process is created after the current process is completed.

[0148] After the data import process completes the task, it is determined whether the task is completed correctly based on the return value. If so, all temporary files generated are deleted; if not, the data files that failed to be imported are moved to the error file data directory, and the data import process is re-established for import.

[0149] During implementation, the data files generated by data sampling have the file name naming rule of data identifier tablename attribute_timestamp.txt or data identifier tablename attribute_sampling level_timestamp.txt, where the data identifier is the parameter type, such as voltage, current, etc.

[0150] The name of the database table corresponding to the tablename attribute of the data identifier is written to the database table or the tablename attribute of the data identifier _ the database table of the sampling level.

[0151] The system has two process limits: the maximum number of server processes and the data import limit.

[0152] After finding the data file in the import data directory, first determine whether the file has started to be imported. If it has not started, and the two process quantity restrictions are met at the same time, start to create and schedule the data import execution. When the process quantity limit is reached, it is necessary to wait for the current process to complete data sampling before scheduling again.

[0153] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0154] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A data preprocessing guard method for a big data processing system, characterized in that: The method comprises: Filtering and time-aligning the data to obtain filtered and aligned data; wherein the filtering of the data includes: calculating the MD5 checksum of the received data packet, determining whether the checksum is the same as the MD5 checksum of the received data packet, and if the checksum is the same, discarding the data packet; The filtered and aligned data are sampled in stages to form multi-stage sampling data, including: arranging the filtered and aligned data in chronological order to form a current data curve; dividing the data curve into a number of increasing / decreasing sampling intervals and a number of increasing / decreasing mixed sampling intervals based on the peak and trough positions of the current data curve; for the increasing / decreasing sampling intervals, a simple sampling method is used to perform sampling to obtain sampling points of the increasing / decreasing sampling intervals; for the increasing / decreasing mixed sampling intervals, a sampling method based on peak detection is used to perform sampling to obtain sampling points of the increasing / decreasing mixed sampling intervals; the sampling points of each increasing / decreasing sampling interval and the sampling points of each mixed sampling interval are merged in chronological order to obtain sampling points of the current data curve; the data curve formed by the sampling points of the current data curve is used as the current data curve at the next level of sampling, and the above steps are repeated to achieve hierarchical sampling; Resample the data that failed to be sampled to obtain multi-level sampling data; Compressing the multi-level sampling data to obtain compressed multi-level sampling data; Create a data import process to import the compressed multi-level sampling data into the database in batches.

2. The data preprocessing guard method according to claim 1, characterized in that: The time alignment processing of the data comprises: When the synchronized system time is greater than or equal to the original system time, the synchronized system time is directly used as the timestamp of the received data packet; When the synchronized system time is less than the original system time, for the subsequently received data packets, the original system time is used as the starting point, and each fixed time interval Δt is used as the timestamp of the data packet until the synchronized system time is greater than or equal to the timestamp of the currently received data packet; When the synchronized system time is greater than or equal to the timestamp of the currently received data packet, the synchronized system time continues to be used as the timestamp of the data packet.

3. The data preprocessing guard method according to claim 1, characterized in that: The sampling method based on peak detection is used to perform sampling to obtain the sampling points of the increasing and decreasing mixed sampling intervals, including: A peak sampling method is used to obtain the peak point of the mixed sampling interval; The mixed sampling interval is sampled using a simple sampling method to obtain a first sampling point of the mixed sampling interval; The peak point and the first sampling point are combined in time sequence, and duplicate data points are removed to obtain sampling points of the mixed sampling interval.

4. The data preprocessing guard method according to claim 3 is characterized in that: The adopting the peak sampling method to obtain the peak point of the mixed sampling interval includes: According to the time sequence, the mixed sampling interval is divided into a plurality of sampling sub-intervals in sequence with N data points as a group; The data point type in each sampling sub-interval is identified, and the maximum and minimum points are determined from each data point type. The maximum and minimum points that can characterize the data curve envelope are retained, and the remaining points are discarded; the maximum and minimum points of each sub-interval are merged in chronological order to form the peak point of the mixed sampling interval.

5. The data preprocessing guarding method according to claim 4, characterized in that: The step of identifying the data point type in each sampling subinterval and determining the maximum value point and the minimum value point from each data point type includes: For the data point set {V1,V2,...,V N }, If V i-1 >V i And V i >V i+1 , then the data point V i Type is Normal; If V i-1 >V i And V i =V i+1 , then the data point V i Type: left groove; If V i-1 >V i And V i <V i+1 , then the data point V i The type is pointed bottom; If V i-1 =V i And V i >V i+1 , then the data point V i Type: left step; If V i-1 =V i And V i =V i+1 , then the data point V i Type is Normal; If V i-1 =V i And V i <V i+1 , then the data point V i Type: right groove; If V i-1 <V i And V i >V i+1 , then the data point V i The type is pointed roof; If V i-1 <V i And V i =V i+1 , then the data point V i Type: right step; If V i-1 <V i And V i <V i+1 , then the data point V i Type is normal; The above-mentioned sharp bottom point is taken as the minimum point, and the sharp top point is taken as the maximum point; for adjacent left groove points and right groove points, the left groove point is taken as the minimum point, for adjacent left step points and right step points, the right step point is taken as the maximum point, and the first and last data points of the sampling sub-interval are taken as the maximum point or the minimum point.

6. The data preprocessing guard method according to claim 5, characterized in that: The simple sampling method includes: According to the time sequence, the sampling interval is divided into multiple sampling sub-intervals with N data points as a group; A maximum value point and a minimum value point are extracted in each sampling sub-interval; and the maximum value point and the minimum value point of each sampling sub-interval are combined in time sequence to form a sampling point of the sampling interval.

7. The data preprocessing guard method according to claim 1, characterized in that: The method of re-sampling the data that failed to be sampled to obtain multi-level sampling data includes: Get the data file where sampling failed, change the file name suffix to .txt, and add the system timestamp after the file name; Obtain the server with the smallest current load based on the load balancing algorithm; A data sampling process is created on the server to perform data sampling and obtain multi-level sampling data.

8. The data preprocessing guard method according to claim 1, characterized in that: The process of creating a data import process, importing the compressed multi-level sampling data into the database in batches, includes: When the server's maximum process number limit and data import limit are met, a data import process is created in the server to import data; otherwise, a data import process is created after the current process is completed. After the data import process completes the task, it is determined whether the task is completed correctly based on the return value. If so, all temporary files generated are deleted; if not, the data files that failed to be imported are moved to the error file data directory, and the data import process is re-established for import.

Images