A component-based method for processing streaming data

Abstract

The present invention provides a component-based real-time streaming data processing method, belonging to the technical field of data processing, including: reading the received real-time streaming data through a data source component; performing first data processing on the received real-time streaming data through an operator component to determine first streaming data; performing second data processing on the first streaming data through a window operation component to determine third streaming data; and sending the third streaming data to a target system or external storage through a connector component to achieve data output or storage. It can improve the quality, consistency, and integrity of streaming data, easily adjust, replace, or expand each component according to requirements, improve the flexibility, timeliness, and accuracy of streaming data processing, ensure the efficiency and reliability of data output or storage, and achieve real-time analysis, scalability, and maintainability of streaming data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a component-based streaming data processing method. Background Art

[0002] Streaming data processing has a wide range of applications and can adapt to application scenarios of different scales and complexities. For example, in the financial field, it is widely used in real-time transaction monitoring, fraud detection and risk control to ensure the efficiency and accuracy of data processing; in smart home systems, real-time data processing is used to optimize the interaction between devices and provide a personalized user experience; in smart manufacturing production lines, real-time analysis of production data is used to detect faults and bottlenecks on the production line, optimize the production process, and improve production efficiency; however, streaming data processing has the disadvantages of untimely data processing, low processing efficiency, poor processing flexibility and reliability.

[0003] Therefore, the present invention provides a component-based streaming data processing method. Summary of the invention

[0004] The present invention provides a component-based streaming data processing method, which performs a first processing on received real-time streaming data to determine first streaming data, performs a second data processing on the first streaming data to determine third streaming data, and sends the third streaming data to a target system or external storage to achieve data output or storage. The quality, consistency and integrity of streaming data can be improved, and each component can be easily adjusted, replaced or expanded according to needs, thereby improving the flexibility, timeliness and accuracy of streaming data processing, ensuring the efficiency and reliability of data output or storage, and achieving real-time analysis, scalability and maintainability of streaming data processing.

[0005] The present invention provides a component-based streaming data processing method, comprising:

[0006] 101: read the received real-time streaming data through the data source component;

[0007] 102: Performing first data processing on the received real-time streaming data through an operator component to determine first streaming data;

[0008] 103: Perform second data processing on the first streaming data through a window operation component to determine third streaming data;

[0009] 104: Send the third streaming data to the target system or external storage through the connector component to achieve data output or storage.

[0010] According to a component-based streaming data processing method provided by the present invention, real-time streaming data received is read through a data source component, including:

[0011] The data source component receives real-time streaming data from a plurality of data sources, wherein the real-time streaming data includes sub-real-time streaming data received by the data source component from each data source;

[0012] The data source component identifies the data format of each sub-real-time streaming data.

[0013] According to a component-based streaming data processing method provided by the present invention, first data processing is performed on received real-time streaming data by an operator component to determine first streaming data, including:

[0014] The first data processing includes format conversion, missing value processing and data filtering;

[0015] Performing format conversion on the received real-time streaming data, and calculating the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after the format conversion;

[0016] Based on the comprehensive consistency of all the sub-real-time streaming data, missing value processing and data filtering are performed on the real-time streaming data after format conversion to determine the first streaming data.

[0017] According to a component-based streaming data processing method provided by the present invention, format conversion is performed on received real-time streaming data, including:

[0018] Extract all data formats appearing in the real-time streaming data, determine the number of sub-real-time streaming data of each data format, and select the data format with the largest number of sub-real-time streaming data as the first format;

[0019] Format conversion is performed on all sub-real-time streaming data that are not in the first format.

[0020] According to a component-based streaming data processing method provided by the present invention, the comprehensive consistency of each sub-real-time streaming data is calculated based on the real-time streaming data after format conversion, including:

[0021] Determine the number of data records of each sub-real-time streaming data after format conversion, the data field set of each data record, and the data name and data type of each data field in the data field set, wherein the data field set includes all data field names in the corresponding data record;

[0022] Determining the comprehensive consistency of the sub-real-time streaming data based on the data fields included in all data records in the same sub-real-time streaming data;

[0023] in, represents the first consistency of the k-th sub-real-time streaming data, w1 represents the first weight of the first consistency of the real-time streaming data, N1 represents the number of sub-real-time streaming data in the real-time streaming data, kN2 represents the number of data records of the k-th sub-real-time streaming data, Represents the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the data field set of the jth data record in the kth sub-real-time streaming data. Indicates the first sub-consistency of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data, represents the second consistency of the kth sub-real-time streaming data, w2 represents the second weight of the second consistency of the real-time streaming data, Indicates the number of data fields in the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the average number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the variance of the number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the third consistency of the k-th sub-real-time streaming data, w3 represents the third weight of the third consistency of the real-time streaming data, Indicates the number of different data fields contained in all data records in the k-th sub-real-time streaming data. represents the number of data fields that appear in all data records of the k-th sub-real-time streaming data and have the same data field type as the p-th data field, Indicates the comprehensive consistency of the k-th sub-real-time streaming data.

[0024] According to a component-based streaming data processing method provided by the present invention, missing value processing and data filtering are performed on the real-time streaming data after format conversion based on the comprehensive consistency of all sub-real-time streaming data to determine the first streaming data, including:

[0025] Sort all sub-real-time streaming data in ascending order based on their comprehensive consistency, and select the The comprehensive consistency is used as the missing value threshold, where [] is the rounding symbol;

[0026] Perform missing value processing on all data records in the sub-real-time streaming data based on comprehensive consistency and the missing value threshold, wherein all data records in the sub-real-time streaming data whose comprehensive consistency is lower than the missing value threshold are deleted from the missing values;

[0027] Fill missing values ​​for all data records in the sub-real-time streaming data whose comprehensive consistency is higher than the missing value threshold;

[0028] Determining a filter factor of the sub-real-time streaming data based on a first consistency of the sub-real-time streaming data and all first sub-consistencies in the sub-real-time streaming data;

[0029] Extracting first sub-consistencies associated with each data record in the sub-real-time streaming data respectively, and determining whether to delete the data record based on all the extracted first sub-consistencies, the first consistency of the sub-real-time streaming data and the filter factor;

[0030] Determine first sub-streaming data of the sub-real-time streaming data based on all data records that have been processed for missing values ​​and have not been deleted in the sub-real-time streaming data;

[0031] The first streaming data is determined based on all the first sub-streaming data.

[0032] According to a component-based streaming data processing method provided by the present invention, second data processing is performed on second streaming data through a window operation component to determine third streaming data, including:

[0033] The second data processing includes diversion processing and aggregation processing;

[0034] Extracting all data records of each first sub-streaming data in the first streaming data and a data field set of each data record, and obtaining a key field set in each data field set, wherein the key field set may include one or more data fields of the corresponding data record;

[0035] Determine a diversion rule based on all key field sets in the first streaming data, perform data diversion on all data records of each first sub-streaming data in the first streaming data based on the diversion rule, and determine a second sub-data stream corresponding to each diversion type;

[0036] The second streaming data is determined based on all the second sub-data streams, and the second streaming data is aggregated to determine the third streaming data.

[0037] According to a component-based streaming data processing method provided by the present invention, second streaming data is aggregated to determine third streaming data, including:

[0038] Determine a window length and a sliding interval of a sliding time window of the second streaming data;

[0039] Applying a sliding time window to each second sub-data stream in the second streaming data to divide each second sub-stream data into a plurality of sub-window data;

[0040] Aggregating the data of each sub-window divided based on the second sub-stream data to determine the third sub-stream data of each sub-window;

[0041] The third streaming data is determined based on the third sub-streaming data of all the sub-windows.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] By performing a first processing on the received real-time streaming data to determine the first streaming data, performing a second data processing on the first streaming data to determine the third streaming data, and sending the third streaming data to the target system or external storage, the output or storage of data can be achieved, the quality, consistency and integrity of the streaming data can be improved, and each component can be easily adjusted, replaced or expanded according to needs, thereby improving the flexibility, timeliness and accuracy of streaming data processing, ensuring the efficiency and reliability of data output or storage, and achieving real-time analysis, scalability and maintainability of streaming data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 It is a flowchart of a component-based streaming data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Embodiment 1:

[0048] The embodiment of the present invention provides a component-based streaming data processing method, such as Figure 1 As shown, including:

[0049] 101: read the received real-time streaming data through the data source component;

[0050] 102: Performing first data processing on the received real-time streaming data through an operator component to determine first streaming data;

[0051] 103: Performing second data processing on the first streaming data through a window operation component to determine third streaming data;

[0052] 104: Send the third streaming data to the target system or external storage through the connector component to achieve data output or storage.

[0053] In this embodiment, the data source component is an entry in the real-time stream processing system for receiving external data. It is responsible for reading real-time streaming data from various data sources and introducing it into the stream processing system for subsequent processing.

[0054] In this embodiment, real-time streaming data is data that is continuously generated and input into the processing system in the form of a stream. Such data is usually continuous and arrives in real time, such as sensor data, user activity logs, financial transaction data, etc.

[0055] In this embodiment, the operator component is a basic unit for operating and processing data in a real-time stream processing system. Each operator receives real-time streaming data, executes processing logic, and outputs first streaming data.

[0056] In this embodiment, the first data processing means performing format conversion, missing value processing and data filtering processing on all sub-real-time streaming data in the real-time streaming data.

[0057] In this embodiment, the window operation component is used to window the first streaming data so as to perform aggregation operations within these windows.

[0058] In this embodiment, the third streaming data is a data stream that has undergone diversion processing and aggregation processing, contains the aggregation result of the data in the window, and is the final output stream in the data processing process.

[0059] In this embodiment, the connector component is used to output the processed third streaming data to the target system or external storage. It is the last link in the data processing chain and is responsible for sending the result data to a database, message queue, file system or other external services.

[0060] In this embodiment, the target system or external storage is the final destination of streaming data processing and is used to receive and save processed data. The target system or external storage may be a traditional relational database, NoSQL database, data warehouse, or a real-time analysis system, reporting system, etc.

[0061] The beneficial effects of the above technical solution are as follows: by performing a first processing on the received real-time streaming data to determine the first streaming data, performing a second data processing on the first streaming data to determine the third streaming data, and sending the third streaming data to the target system or external storage, the data output or storage is realized, which can improve the quality, consistency and integrity of the streaming data, easily adjust, replace or expand each component according to needs, improve the flexibility, timeliness and accuracy of streaming data processing, ensure the efficiency and reliability of data output or storage, and realize real-time analysis, scalability and maintainability of streaming data processing.

[0062] Embodiment 2:

[0063] The embodiment of the present invention provides a component-based streaming data processing method, which reads received real-time streaming data through a data source component, including:

[0064] The data source component receives real-time streaming data from a plurality of data sources, wherein the real-time streaming data includes sub-real-time streaming data received by the data source component from each data source;

[0065] The data source component identifies the data format of each sub-real-time streaming data.

[0066] In this embodiment, the data source component includes multiple data sources, such as a message queue, a database, a file system, an API, etc.

[0067] In this embodiment, each data source corresponds to a sub-real-time streaming data.

[0068] In this embodiment, the data source component can support multiple types of data sources, which can be structured (such as relational databases), semi-structured (such as JSON files, XML files) or unstructured (such as text files, log data), etc.

[0069] The beneficial effect of the above technical solution is that the flexibility and adaptability of streaming data processing can be improved by reading the received real-time streaming data through the data source component.

[0070] Embodiment 3:

[0071] An embodiment of the present invention provides a component-based streaming data processing method, which performs first data processing on received real-time streaming data through an operator component to determine first streaming data, including:

[0072] The first data processing includes format conversion, missing value processing and data filtering;

[0073] Performing format conversion on the received real-time streaming data, and calculating the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after the format conversion;

[0074] Based on the comprehensive consistency of all the sub-real-time streaming data, missing value processing and data filtering are performed on the real-time streaming data after format conversion to determine the first streaming data.

[0075] In this embodiment, after the format conversion, the data formats of all the sub-real-time streaming data in the real-time streaming data are all in a unified first format.

[0076] In this embodiment, missing value processing refers to identifying and processing missing values ​​in the real-time streaming data after format conversion, and the measures taken include missing value deletion and missing value filling.

[0077] In this embodiment, data filtering means filtering out data records with symbol conditions according to preset conditions, and only retaining records that meet the conditions.

[0078] In this embodiment, the first streaming data represents real-time streaming data after format conversion, missing value processing, and data filtering processing.

[0079] The beneficial effects of the above technical solution are as follows: by performing first data processing on the received real-time streaming data through the operator component and determining the first streaming data, the quality, consistency and integrity of the streaming data can be improved, thereby optimizing the flexibility and accuracy of the entire streaming data processing.

[0080] Embodiment 4:

[0081] The embodiment of the present invention provides a component-based streaming data processing method, which performs format conversion on received real-time streaming data, including:

[0082] Extract all data formats appearing in the real-time streaming data, determine the number of sub-real-time streaming data of each data format, and select the data format with the largest number of sub-real-time streaming data as the first format;

[0083] Format conversion is performed on all sub-real-time streaming data that are not in the first format.

[0084] In this embodiment, all data formats appearing in the received real-time streaming data are identified and extracted, and these formats may include JSON, XML, CSV, binary, etc.

[0085] In this embodiment, for each data format, the number of sub-real-time streaming data belonging to the format is counted.

[0086] In this embodiment, after counting the number of sub-real-time streaming data in each data format, the data format with the largest number is selected and called the "first format". This format dominates the entire real-time streaming data and will therefore be used as the standard format in subsequent processing.

[0087] In this embodiment, for those sub-real-time streaming data that are not in the first format, they are converted into the first format.

[0088] The beneficial effects of the above technical solution are as follows: by converting the format of the received real-time streaming data, the streaming data from different data sources can be subsequently processed in a unified format, ensuring data consistency, thereby avoiding processing errors or data loss due to format differences and improving the accuracy of data processing.

[0089] Embodiment 5:

[0090] The embodiment of the present invention provides a component-based streaming data processing method, which calculates the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after format conversion, including:

[0091] Determine the number of data records of each sub-real-time streaming data after format conversion, the data field set of each data record, and the data name and data type of each data field in the data field set, wherein the data field set includes all data field names in the corresponding data record;

[0092] Determining the comprehensive consistency of the sub-real-time streaming data based on the data fields included in all data records in the same sub-real-time streaming data;

[0093] in, represents the first consistency of the k-th sub-real-time streaming data, w1 represents the first weight of the first consistency of the real-time streaming data, N1 represents the number of sub-real-time streaming data in the real-time streaming data, kN2 represents the number of data records of the k-th sub-real-time streaming data, Represents the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the data field set of the jth data record in the kth sub-real-time streaming data. Indicates the first sub-consistency of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data, represents the second consistency of the kth sub-real-time streaming data, w2 represents the second weight of the second consistency of the real-time streaming data, Indicates the number of data fields in the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the average number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the variance of the number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the third consistency of the k-th sub-real-time streaming data, w3 represents the third weight of the third consistency of the real-time streaming data, Indicates the number of different data fields contained in all data records in the k-th sub-real-time streaming data. represents the number of data fields that appear in all data records of the k-th sub-real-time streaming data and have the same data field type as the p-th data field, Indicates the comprehensive consistency of the k-th sub-real-time streaming data.

[0094] In this embodiment, the first consistency Indicates the consistency of the data field names in the data field set contained in all data records in the k-th sub-real-time streaming data.

[0095] In this embodiment, the second consistency Indicates the consistency of the number of data fields in the data field set contained in all data records in the k-th sub-real-time streaming data.

[0096] In this embodiment, the third consistency Indicates the consistency of data field types in the data field set contained in all data records in the k-th sub-real-time streaming data.

[0097] In this embodiment, Indicates the number of sub-real-time streaming data pairs in the real-time streaming data.

[0098] In this embodiment, It means traversing the first data record to the second to last data record in the k-th sub-real-time streaming data. It means to traverse the i+1th data record to the last data record in the kth sub-real-time streaming data, ensuring that each record is It is calculated only once and will not be repeated, because when the consistency calculation is performed on the data record, and are identical (symmetry), so only need to be considered once.

[0099] In this embodiment, Represents the number of data fields in the intersection of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data.

[0100] In this embodiment, Represents the number of data fields in the union of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data.

[0101] In this embodiment, It represents the consistency of the intersection and union of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data, that is, the degree of overlap, and represents the proportion of the intersection in the union.

[0102] In this embodiment, Indicates the data type consistency of the p-th data field in the k-th sub-real-time streaming data in the kN2 data records.

[0103] In this embodiment, the comprehensive consistency is an indicator value for measuring the comprehensive consistency of the internal data of the corresponding sub-real-time streaming data.

[0104] The beneficial effect of the above technical solution is: by calculating the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after format conversion, a data basis can be provided for missing value processing and data filtering to improve the flexibility, timeliness and accuracy of streaming data processing.

[0105] Embodiment 6:

[0106] The embodiment of the present invention provides a component-based streaming data processing method, which performs missing value processing and data filtering on real-time streaming data after format conversion based on the comprehensive consistency of all sub-real-time streaming data, and determines the first streaming data, including:

[0107] Sort all sub-real-time streaming data in ascending order based on their comprehensive consistency, and select the The comprehensive consistency is used as the missing value threshold, where [] is the rounding symbol;

[0108] Perform missing value processing on all data records in the sub-real-time streaming data based on comprehensive consistency and the missing value threshold, wherein all data records in the sub-real-time streaming data whose comprehensive consistency is lower than the missing value threshold are deleted from the missing values;

[0109] Fill missing values ​​for all data records in the sub-real-time streaming data whose comprehensive consistency is higher than the missing value threshold;

[0110] Determining a filter factor of the sub-real-time streaming data based on a first consistency of the sub-real-time streaming data and all first sub-consistencies in the sub-real-time streaming data;

[0111] Extracting first sub-consistencies associated with each data record in the sub-real-time streaming data respectively, and determining whether to delete the data record based on all the extracted first sub-consistencies, the first consistency of the sub-real-time streaming data and the filter factor;

[0112] Determine first sub-streaming data of the sub-real-time streaming data based on all data records that have been processed for missing values ​​and have not been deleted in the sub-real-time streaming data;

[0113] The first streaming data is determined based on all the first sub-streaming data.

[0114] In this embodiment, if the comprehensive consistency of a sub-data stream is lower than the missing value threshold, the missing values ​​of all data records in the sub-data stream will be directly deleted.

[0115] In this embodiment, if the comprehensive consistency of a sub-data stream is higher than the missing value threshold, the missing values ​​of all data records in the sub-data stream will be filled. The filling method may use mean, median, forward filling or interpolation techniques.

[0116] In this embodiment, a filter factor is determined based on the first consistency of the sub-data stream and all the first sub-consistencies, and the filter factor is used to evaluate whether a data record needs to be retained or deleted.

[0117] In this embodiment, the first sub-consistency indicates the consistency of two data records in the same sub-real-time streaming data.

[0118] In this embodiment, the filter factor can be dynamically adjusted according to the conditions of different sub-data streams. For all sub-data streams with higher first sub-consistency in the sub-real-time streaming data, the filter factor can be stricter, while for all sub-data streams with lower first sub-consistency in the sub-real-time streaming data, the filter standard can be appropriately relaxed.

[0119] In this embodiment, a statistical analysis is performed on the first sub-consistency related to each data record (statistics such as the average value, standard deviation, and median may be calculated to understand the consistency distribution of the data), and by comparing it with the first consistency of the sub-data stream, it is determined whether to delete the data record. For example, if the statistical result of the first sub-consistency of the data record deviates too much from the first consistency (such as exceeding a certain percentage or multiple standard deviations), the record may be considered to be a record with poor consistency and may need to be deleted.

[0120] In this embodiment, each sub-real-time streaming data corresponds to a first sub-streaming data.

[0121] The beneficial effects of the above technical solution are as follows: missing value processing and data filtering are performed on the real-time streaming data after format conversion according to the comprehensive consistency of all sub-real-time streaming data to determine the first streaming data, which can improve the quality of the second data processing, reduce the impact of erroneous data on the overall processing results, and improve the quality, consistency and integrity of streaming data.

[0122] Embodiment 7:

[0123] An embodiment of the present invention provides a component-based streaming data processing method, which performs second data processing on second streaming data through a window operation component to determine third streaming data, including:

[0124] The second data processing includes diversion processing and aggregation processing;

[0125] Extracting all data records of each first sub-streaming data in the first streaming data and a data field set of each data record, and obtaining a key field set in each data field set, wherein the key field set may include one or more data fields of the corresponding data record;

[0126] Determine a diversion rule based on all key field sets in the first streaming data, perform data diversion on all data records of each first sub-streaming data in the first streaming data based on the diversion rule, and determine a second sub-data stream corresponding to each diversion type;

[0127] The second streaming data is determined based on all the second sub-data streams, and the second streaming data is aggregated to determine the third streaming data.

[0128] In this embodiment, the key fields are usually key fields of real-time streaming data, such as user ID, event type, product category, geographic location, etc.

[0129] In this embodiment, the diversion rules are rules defined based on a key field set and are used to assign data records to different sub-data streams. The diversion rules can generally be based on the value, range, category, etc. of the field. The diversion rules are determined based on all data fields that appear in all key field sets. For example, records with a user ID in a certain range and an event type of "purchase" are diverted to one module, while records with a user ID in the same range but an event type of "browse" are diverted to another module.

[0130] In this embodiment, if there is a conflict between the diversion rules, the priority of the rules is defined to ensure that the first sub-stream data can be correctly classified and processed.

[0131] In this embodiment, all data records in each first sub-stream data are classified using the determined diversion rule, and each record is assigned to a different diversion type, namely, the second sub-data stream, according to the matching of its key field with the corresponding diversion rule.

[0132] The beneficial effects of the above technical solution are as follows: by performing second data processing on the second streaming data through the window operation component and determining the third streaming data, the efficiency of streaming data processing can be improved, real-time analysis of streaming data processing can be achieved, and the reliability and maintainability of streaming data processing can be enhanced.

[0133] Embodiment 8:

[0134] An embodiment of the present invention provides a component-based streaming data processing method, which performs aggregation processing on second streaming data to determine third streaming data, including:

[0135] Determine a window length and a sliding interval of a sliding time window of the second streaming data;

[0136] Applying a sliding time window to each second sub-data stream in the second streaming data to divide each second sub-stream data into a plurality of sub-window data;

[0137] Aggregating the data of each sub-window divided based on the second sub-stream data to determine the third sub-stream data of each sub-window;

[0138] The third streaming data is determined based on the third sub-streaming data of all the sub-windows.

[0139] In this embodiment, the window length is the duration of the sliding time window, that is, the time range included in each window. For example, the window length can be set to 5 minutes, 1 hour, etc.

[0140] In this embodiment, the sliding interval is the step size of the sliding window movement, which determines the data range covered by the new window each time the window moves. The sliding interval can be less than, equal to or greater than the window length. If the sliding interval is less than the window length, there will be overlap between the windows. For example: if the window length is 10 minutes and the sliding interval is 5 minutes, a new window is generated every 5 minutes, in which there may be 5 minutes of data overlap.

[0141] In this embodiment, for each second sub-data stream in the second streaming data, a sliding time window is applied to divide the data into multiple sub-windows, and each sub-window contains data records within a certain time range.

[0142] In this embodiment, the sub-window data in each sub-window is aggregated, and the aggregation result of each sub-window data will form a new data stream, which is called the third sub-stream data.

[0143] The beneficial effects of the above technical solution are as follows: by performing aggregation processing on the second streaming data to determine the third streaming data, the efficiency of streaming data processing can be further improved, real-time analysis of streaming data processing can be achieved, the efficiency and reliability of data output or storage can be ensured, and real-time analysis, scalability and maintainability of streaming data processing can be achieved.

[0144] The method embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A component-based streaming data processing method, characterized in that: include: 101: read the received real-time streaming data through the data source component; 102: Performing first data processing on the received real-time streaming data through an operator component to determine first streaming data; 103: Performing second data processing on the first streaming data through a window operation component to determine third streaming data; 104: Sending the third stream data to a target system or external storage through a connector component to achieve data output or storage; The method further includes: performing format conversion on the received real-time streaming data, and calculating the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after the format conversion, specifically including: Determining the comprehensive consistency of the sub-real-time streaming data based on the data fields included in all data records in the same sub-real-time streaming data; in, represents the first consistency of the k-th sub-real-time streaming data, w1 represents the first weight of the first consistency of the real-time streaming data, N1 represents the number of sub-real-time streaming data in the real-time streaming data, kN2 represents the number of data records of the k-th sub-real-time streaming data, Represents the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the data field set of the jth data record in the kth sub-real-time streaming data. Indicates the first sub-consistency of the data field set of the i-th data record and the data field set of the j-th data record in the k-th sub-real-time streaming data, represents the second consistency of the kth sub-real-time streaming data, w2 represents the second weight of the second consistency of the real-time streaming data, Indicates the number of data fields in the data field set of the i-th data record in the k-th sub-real-time streaming data. Represents the average number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the variance of the number of data fields in the data field set of kN2 data records in the kth sub-real-time streaming data. represents the third consistency of the k-th sub-real-time streaming data, w3 represents the third weight of the third consistency of the real-time streaming data, Indicates the number of different data fields contained in all data records in the k-th sub-real-time streaming data. represents the number of data fields that appear in all data records of the k-th sub-real-time streaming data and have the same data field type as the p-th data field, Indicates the comprehensive consistency of the k-th sub-real-time streaming data.

2. A component-based streaming data processing method according to claim 1, characterized in that: Read the received real-time streaming data through the data source component, including: The data source component receives real-time streaming data from a plurality of data sources, wherein the real-time streaming data includes sub-real-time streaming data received by the data source component from each data source; The data source component identifies the data format of each sub-real-time streaming data.

3. A component-based streaming data processing method according to claim 2, characterized in that: Performing first data processing on the received real-time streaming data through an operator component to determine the first streaming data includes: The first data processing includes format conversion, missing value processing and data filtering; Based on the comprehensive consistency of all the sub-real-time streaming data, missing value processing and data filtering are performed on the real-time streaming data after format conversion to determine the first streaming data.

4. A component-based streaming data processing method according to claim 3, characterized in that: Format conversion of received real-time streaming data, including: Extract all data formats appearing in the real-time streaming data, determine the number of sub-real-time streaming data of each data format, and select the data format with the largest number of sub-real-time streaming data as the first format; Format conversion is performed on all sub-real-time streaming data that are not in the first format.

5. A component-based streaming data processing method according to claim 3, characterized in that: Before calculating the comprehensive consistency of each sub-real-time streaming data based on the real-time streaming data after format conversion, it also includes: Determine the number of data records of each sub-real-time streaming data after format conversion, the data field set of each data record, and the data name and data type of each data field in the data field set, wherein the data field set includes all data field names in the corresponding data record.

6. A component-based streaming data processing method according to claim 5, characterized in that: Based on the comprehensive consistency of all sub-real-time streaming data, missing value processing and data filtering are performed on the real-time streaming data after format conversion to determine the first streaming data, including: Sort all sub-real-time streaming data in ascending order based on their comprehensive consistency, and select the The comprehensive consistency is used as the missing value threshold, where [] is the rounding symbol; Perform missing value processing on all data records in the sub-real-time streaming data based on comprehensive consistency and the missing value threshold, wherein all data records in the sub-real-time streaming data whose comprehensive consistency is lower than the missing value threshold are deleted from the missing values; Fill missing values ​​for all data records in the sub-real-time streaming data whose comprehensive consistency is higher than the missing value threshold; Determining a filter factor of the sub-real-time streaming data based on a first consistency of the sub-real-time streaming data and all first sub-consistencies in the sub-real-time streaming data; Extracting first sub-consistencies associated with each data record in the sub-real-time streaming data respectively, and determining whether to delete the data record based on all the extracted first sub-consistencies, the first consistency of the sub-real-time streaming data and the filter factor; Determine first sub-streaming data of the sub-real-time streaming data based on all data records that have been processed for missing values ​​and have not been deleted in the sub-real-time streaming data; The first streaming data is determined based on all the first sub-streaming data.

7. A component-based streaming data processing method according to claim 1, characterized in that: Performing second data processing on the second streaming data through the window operation component to determine third streaming data includes: The second data processing includes diversion processing and aggregation processing; Extracting all data records of each first sub-streaming data in the first streaming data and a data field set of each data record, and obtaining a key field set in each data field set, wherein the key field set may include one or more data fields of the corresponding data record; Determine a diversion rule based on all key field sets in the first streaming data, perform data diversion on all data records of each first sub-streaming data in the first streaming data based on the diversion rule, and determine a second sub-data stream corresponding to each diversion type; The second streaming data is determined based on all the second sub-data streams, and the second streaming data is aggregated to determine the third streaming data.

8. A component-based streaming data processing method according to claim 7, characterized in that: Aggregating the second streaming data to determine third streaming data includes: Determine a window length and a sliding interval of a sliding time window of the second streaming data; Applying a sliding time window to each second sub-data stream in the second streaming data to divide each second sub-stream data into a plurality of sub-window data; Aggregating the data of each sub-window divided based on the second sub-stream data to determine the third sub-stream data of each sub-window; The third streaming data is determined based on the third sub-streaming data of all the sub-windows.

Images