Micro-batch-based real-time statistics method, device, equipment and storage medium

Abstract

The application discloses a micro-batch-based real-time statistical method and device, equipment and storage medium, and relates to the field of data statistics, which comprises the following steps: analyzing the received data to be processed, filtering the analyzed data, and obtaining processed data; dividing the processed data into several parts of processed data and inputting them into several preset concurrent statistical subunits to perform data statistics on the data input into the subunits to obtain temporary statistical results; determining several index identifiers corresponding to the temporary statistical results to group the temporary statistical results by the index identifiers, using a preset summary statistical subunit to perform summary statistics on the grouped temporary statistical results, and outputting target statistical results corresponding to the grouped temporary statistical results. In this way, the calculation capacity of statistical indicators can be optimized by different concurrent statistical subunits to support simultaneous statistical calculation under different dimensions and different time windows.

Description

Technical Field

[0001] This invention relates to the field of data statistics, and in particular to a real-time statistical method, apparatus, device, and storage medium based on micro-batch processing. Background Technology

[0002] Apache Flink is a general-purpose distributed real-time processing engine in the big data field. Its high throughput and low latency characteristics make it widely applicable to the development of various large-scale stream processing applications. In big data analytics scenarios, real-time dimensional statistics are a common and frequently encountered task. A unified definition of statistical metrics can cover various business scenarios, and a unified implementation of these metrics can support various dimensional statistical needs. However, when there are many statistical metrics with different dimensions and windows, multiple windows will consume data multiple times, leading to significant performance bottlenecks with limited resources. Typically, with existing technologies, statistical metrics for different time windows are processed independently, making simultaneous merging impossible. Furthermore, when data is received, all window data is usually maintained and stored in memory for computation. When the window duration is long, the amount of data maintained in memory increases linearly with the window duration, thus creating a memory resource bottleneck for stream computing tasks. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a real-time statistical method, apparatus, device, and storage medium based on micro-batch processing. This method can optimize the computational capabilities of statistical indicators through different concurrent statistical computing units, supporting simultaneous statistical calculations across multiple dimensions and time windows, thereby further reducing memory resource consumption and improving the computational efficiency of statistical indicators. The specific solution is as follows:

[0004] In a first aspect, this application discloses a real-time statistical method based on micro-batch processing, applied to a target framework, including:

[0005] The received data to be processed is parsed, and the parsed data is filtered to obtain the processed data.

[0006] The processed data is divided into several parts and input into several preset concurrent statistical calculators respectively. Each of the preset concurrent statistical calculators performs data statistics on the parts of processed data input to it to obtain temporary statistical results.

[0007] Determine several indicator identifiers corresponding to the temporary statistical results, and group the temporary statistical results by the several indicator identifiers to obtain grouped temporary statistical results;

[0008] The pre-defined aggregate calculator is used to summarize and statistically analyze the temporary statistical results after grouping, so as to output the target statistical result corresponding to each group of temporary statistical results in the temporary statistical results after grouping.

[0009] Optionally, the step of parsing the received data to be processed and filtering the parsed data to obtain processed data includes:

[0010] The data to be processed generated by the target framework is accessed, and the data to be processed is parsed using a preset parsing operator and / or a preset parser, so as to determine the target field obtained through data parsing as the parsed data;

[0011] The filter configuration is obtained from the local database, and a hash calculation is performed on the filter configuration to obtain the filter identifier corresponding to the filter configuration.

[0012] Based on the filtering configuration, target data is filtered from the parsed data, and the filtering identifier is added to the target data to obtain filtered data.

[0013] Optionally, before dividing the processed data into several parts and inputting them respectively into several preset concurrent statistical calculators, so as to obtain a provisional statistical result by performing data statistics on the input parts of the processed data through each of the several preset concurrent statistical calculators, the method further includes:

[0014] The statistical dimension identifier, statistical object, and statistical temporary variable of the processed data are determined; the statistical dimension identifier includes the filter identifier, statistical field identifier, preset data processing time threshold, and statistical field concatenation value;

[0015] For each of the plurality of preset concurrent statistical computation sub-sub ...

[0016] Optionally, the process of dividing the processed data into several parts and inputting them respectively into several preset concurrent statistical sub-tasks, so that each of the preset concurrent statistical sub-tasks performs data statistics on the parts of processed data input to itself to obtain temporary statistical results, including:

[0017] The processed data is divided into several parts and input into several preset concurrent statistical operators respectively, so that the preset concurrent operators can determine whether there are the same statistical dimension identifiers in the outer Map based on the parts of processed data input to them.

[0018] If it exists, then determine whether the same statistical object exists in the inner Map;

[0019] If no identical statistical object exists in the inner Map, the temporary statistical variable corresponding to the statistical object will be determined as a temporary statistical result.

[0020] If the same statistical object exists in the inner Map, then the statistical temporary variable corresponding to the statistical object and the historical statistical temporary variable are progressively calculated to obtain temporary statistical data; collect the temporary statistical data of each of the several preset concurrent statistical calculation sub-sub ...

[0021] Optionally, obtaining the temporary statistical result based on the statistical dimension identifier and the target temporary statistical data includes:

[0022] Determine whether the data processing time of the plurality of preset concurrent statistical calculators is not less than a preset data processing time threshold. If so, obtain a temporary statistical result based on the statistical dimension identifier and the target temporary statistical data.

[0023] Optionally, determining several indicator identifiers corresponding to the temporary statistical results, and grouping the temporary statistical results using the several indicator identifiers to obtain grouped temporary statistical results, includes:

[0024] Determine several indicator identifiers corresponding to the temporary statistical results, and set the several indicator identifiers as several grouping keys respectively;

[0025] The temporary statistical results are grouped by keyby according to the several group keys using a preset selector to obtain the grouped temporary statistical results.

[0026] Optionally, the step of summarizing and statistically analyzing the grouped temporary statistical results using a preset aggregate calculator to output a target statistical result corresponding to each group of temporary statistical results in the grouped temporary statistical results includes:

[0027] The grouped temporary statistical results are input into a preset aggregate calculation unit to perform progressive calculation on each group of temporary statistical results to obtain the target statistical result;

[0028] Register a trigger time point for the target statistical result. If the trigger time point is met, output the target statistical result through the OnTimer method.

[0029] Secondly, this application discloses a real-time statistical device based on micro-batch processing, applied to a target framework, comprising:

[0030] The data preprocessing module is used to parse the received data to be processed and filter the parsed data to obtain the processed data.

[0031] The data statistics module is used to divide the processed data into several parts of processed data and input them into several preset concurrent statistical calculation units respectively, so that each of the preset concurrent statistical calculation units can perform data statistics on the parts of processed data input to itself to obtain temporary statistical results.

[0032] The data grouping module is used to determine several indicator identifiers corresponding to the temporary statistical results, so as to group the temporary statistical results by the several indicator identifiers to obtain the grouped temporary statistical results;

[0033] The data output module is used to summarize and statistically analyze the temporary statistical results after grouping using a preset aggregate calculation unit, so as to output the target statistical result corresponding to each group of temporary statistical results in the temporary statistical results after grouping.

[0034] Thirdly, this application discloses an electronic device, including:

[0035] Memory, used to store computer programs;

[0036] A processor is used to execute the computer program to implement the real-time statistical method based on micro-batch processing as described above.

[0037] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned real-time statistical method based on micro-batch processing.

[0038] In this application, the received data to be processed is first parsed and then filtered to obtain processed data. The processed data is then divided into several parts, each input to a set of preset concurrent statistical calculators. Each preset concurrent statistical calculator performs data statistics on its input processed data to obtain temporary statistical results. Several indicator identifiers corresponding to the temporary statistical results are determined, and the temporary statistical results are grouped using these indicator identifiers to obtain grouped temporary statistical results. Finally, a preset aggregate statistical calculator is used to summarize and statistically analyze the grouped temporary statistical results to output a target statistical result corresponding to each group of temporary statistical results in the grouped temporary statistical results. Therefore, the real-time statistical method based on micro-batch processing described in this application can parse and filter the received data, divide the processed data into several parts, and input them into several preset concurrent statistical sub-tasks. These sub-tasks then perform statistical analysis on their input data, obtaining temporary statistical results. These temporary results are grouped, and a summary statistical sub-task is used to aggregate the grouped temporary results, yielding several summary statistical results. In this way, the computational capabilities of statistical indicators can be optimized through different concurrent statistical sub-tasks to support simultaneous statistical calculations across multiple dimensions and time windows, further reducing memory resource consumption and improving the computational efficiency of statistical indicators. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 A flowchart of a real-time statistical method based on micro-batch processing is provided for this application;

[0041] Figure 2 A time series diagram of a real-time statistical method based on micro-batch processing provided in this application;

[0042] Figure 3 A flowchart of a specific real-time statistical method based on micro-batch processing is provided for this application;

[0043] Figure 4 A schematic diagram of a real-time statistical device based on micro-batch processing is provided for this application;

[0044] Figure 5This application provides a structural diagram of an electronic device. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In existing technologies, statistical indicators for each different time window are processed independently, making it impossible to merge and process them simultaneously. Furthermore, when data is received, all window data is typically maintained in memory for storage and computation. When the window time is too long, the amount of data maintained in memory will increase linearly with the increase of the window time, thus creating a bottleneck in memory resources for stream computing tasks.

[0047] To address the aforementioned technical issues, this application provides a real-time statistical method, apparatus, device, and storage medium based on micro-batch processing. It can optimize the computational capabilities of statistical indicators through different concurrent statistical computing sub-modules to support simultaneous statistical calculations under multiple dimensions and time windows, thereby further reducing memory resource consumption and improving the computational efficiency of statistical indicators.

[0048] See Figure 1 As shown, this embodiment of the invention discloses a real-time statistical method based on micro-batch processing, applied to a target framework, including:

[0049] Step S11: Parse the received data to be processed and filter the parsed data to obtain the processed data.

[0050] In this embodiment, the received data to be processed is parsed, and the parsed data is filtered to obtain processed data. This includes: accessing the data to be processed generated by the target framework, and parsing the data using a preset parsing operator and / or a preset parser to determine the target fields obtained through data parsing as parsed data; obtaining a filtering configuration from a local database and performing a hash calculation on the filtering configuration to obtain a filtering identifier corresponding to the filtering configuration; filtering target data from the parsed data based on the filtering configuration, and adding the filtering identifier to the target data to obtain filtered data. That is, this embodiment uses the Flink framework to consume real-time middleware data, such as... Figure 2As shown, once data access is complete, the data to be processed can be parsed using custom preset parsing operators and / or preset parsers. For example, a custom Map operator can be defined, which receives the input data stream and parses and transforms it according to specific rules, mapping each element in the input data stream to a parsed data object. Additionally, a parser can be defined to parse the raw data into the required fields and attributes based on the data's format and structure; the parser can be a function or a class. In this way, the data can be efficiently processed and analyzed using preset operators and / or preset parsers.

[0051] Furthermore, after parsing the data to be processed, the parsed data needs to be filtered. Specifically, this can be done by retrieving the filtering configurations for all statistical indicators from the database, as well as the mapping relationship between indicator identifiers and filter identifiers. The filter identifiers can be obtained by hashing the filtering configurations. Then, the expression engine in the filtering operator calculates whether a single data point meets the filtering configurations for one or more statistical indicators. If none of the criteria are met, the data will not be included in the current statistical indicator calculation and can be discarded. If one or more filtering configurations are met, a filtering result field is added to the data, and the corresponding one or more filter identifiers are assigned. After assignment, the data is output downstream for the next step of calculation. In this way, data can be filtered to select the necessary data, and data that does not meet the filtering conditions can be directly discarded, avoiding the processing of unnecessary data during statistics and thus improving data statistics efficiency.

[0052] Step S12: Divide the processed data into several parts of processed data, and input them into several preset concurrent statistical sub-sub ...

[0053] In this embodiment, as Figure 2As shown, after data filtering is complete, the processed data needs to be aggregated independently in small windows and micro-batches. This means that a single concurrent operator only processes the data it receives and performs small-window statistics. This leverages the multi-concurrency capabilities of the Flink framework to improve overall processing performance. Specifically, the statistical dimension identifier, statistical object, and statistical temporary variable of the processed data can be determined. It should be noted that the statistical dimension identifier includes the filter identifier, statistical field identifier, preset data processing time threshold, and the concatenated value of the statistical field. Then, for each of the preset concurrent statistical operators, an outer Map is created with the statistical dimension identifier as the key and the statistical object and the statistical temporary variable as the value, and an inner Map is created with the statistical object as the key and the statistical temporary variable as the value. After creating two layers of Maps, the processed data needs to be divided into several parts and input into several preset concurrent statistical operators. These operators then determine whether the outer Map contains the same statistical dimension identifier based on the input processed data. If so, they determine whether the inner Map contains the same statistical object. If the inner Map does not contain the same statistical object, the corresponding temporary statistical variable is designated as a temporary statistical result. If the inner Map contains the same statistical object, the corresponding temporary statistical variable and historical temporary statistical variables are statistically analyzed according to preset statistical rules to obtain temporary statistical data. Finally, the temporary statistical data of each of the preset concurrent statistical operators is collected to obtain the target temporary statistical data. Based on the statistical dimension identifier and the target temporary statistical data, several temporary statistical results are obtained.

[0054] Step S13: Determine several indicator identifiers corresponding to the temporary statistical results, and group the temporary statistical results by the several indicator identifiers to obtain grouped temporary statistical results.

[0055] In this embodiment, several indicator identifiers corresponding to the temporary statistical results are determined, and the temporary statistical results are grouped using these indicator identifiers to obtain grouped temporary statistical results. This includes: determining several indicator identifiers corresponding to the temporary statistical results and setting each of the indicator identifiers as a grouping key; and using a preset selector to perform key-by grouping of the temporary statistical results according to the grouping keys to obtain grouped temporary statistical results. That is, after determining the temporary statistical results, as... Figure 2As shown, a secondary aggregation of large windows is required. First, the indicator identifier corresponding to the temporary statistical results needs to be determined. This is done by looking up the filter identifier, statistical field identifier, statistical object, statistical method, and preset data processing time threshold of the upstream data to determine the unique indicator identifier for each data point. Then, the indicator identifier is used as the key, and the Flink KeySelector in the Flink framework is used to group data by key by key. This allows temporary statistical data with the same indicator identifier from various operators in the temporary statistical results to be uniformly aggregated. In this way, data with the same indicator identifier can be aggregated into a single partition, achieving global data aggregation and effectively improving data statistical efficiency.

[0056] Step S14: Use the preset aggregate calculation function to summarize and statistically analyze the temporary statistical results after grouping, so as to output the target statistical result corresponding to each group of temporary statistical results in the temporary statistical results after grouping.

[0057] In this embodiment, a preset aggregate operator is used to summarize and statistically analyze the grouped temporary statistical results, outputting the target statistical result corresponding to each group of temporary statistical results. That is, after the temporary statistical results are grouped, the preset aggregate operator, specifically the `keyedProcess` operator, can be used to receive the grouped temporary statistical results and summarize and statistically analyze them, achieving secondary aggregation and statistical analysis within a large window to obtain the statistical result corresponding to the grouped temporary statistical results. In this way, after performing small-window micro-batch processing through several preset concurrent operators, a secondary aggregation and summary statistical analysis can be performed on the processed data within a large window, ultimately achieving high-performance real-time statistical calculations across multiple statistical dimensions and longer time windows, while reducing resource consumption and improving overall statistical processing capabilities.

[0058] Therefore, in this embodiment, the received data to be processed is first parsed and then filtered to obtain processed data. The processed data is then divided into several parts and input to several preset concurrent statistical calculators. Each preset concurrent statistical calculator performs data statistics on its input processed data to obtain temporary statistical results. Several indicator identifiers corresponding to the temporary statistical results are determined, and the temporary statistical results are grouped using these indicator identifiers to obtain grouped temporary statistical results. Finally, a preset aggregate statistical calculator is used to summarize and statistically analyze the grouped temporary statistical results to output the target statistical result corresponding to each group of temporary statistical results in the grouped temporary statistical results. Therefore, the real-time statistical method based on micro-batch processing described in this application can parse and filter the received data, divide the processed data into several parts, and input them into several preset concurrent statistical sub-tasks. These sub-tasks then perform statistical analysis on their input data, obtaining temporary statistical results. These temporary results are grouped, and a summary statistical sub-task is used to aggregate the grouped temporary results, yielding several summary statistical results. In this way, the computational capabilities of statistical indicators can be optimized through different concurrent statistical sub-tasks to support simultaneous statistical calculations across multiple dimensions and time windows, further reducing memory resource consumption and improving the computational efficiency of statistical indicators.

[0059] As can be seen from the foregoing embodiments, after parsing and filtering the data to be processed, micro-batch processing is required to obtain temporary statistical results. After grouping the temporary statistical results, a summary statistical analysis is needed on the grouped temporary statistical results. Therefore, this embodiment provides a detailed description of how to perform micro-batch processing and data aggregation. See [link to documentation]. Figure 3 As shown, this embodiment of the invention discloses a real-time statistical method based on micro-batch processing, including:

[0060] Step S21: Divide the processed data into several parts of processed data and input them into several preset concurrent statistical operators respectively, so that the several preset concurrent operators can determine whether there are the same statistical dimension identifiers in the outer Map based on the parts of processed data input to themselves.

[0061] In this embodiment, the processed data is divided into several parts and input into several preset concurrent statistical sub-tasks. Before each of the preset concurrent statistical sub-tasks performs data statistics on the input parts of the processed data to obtain temporary statistical results, the method further includes: determining the statistical dimension identifier, statistical object, and statistical temporary variable of the processed data; the statistical dimension identifier includes a filter identifier, a statistical field identifier, a preset data processing time threshold, and a statistical field concatenation value. For each of the preset concurrent statistical sub-tasks, an outer Map is created with the statistical dimension identifier as the key, the statistical object and the statistical temporary variable as the value, and an inner Map is created with the statistical object as the key and the statistical temporary variable as the value. In other words, since micro-batch processing of data in small windows requires processing the data input to each operator through different concurrent operators, a two-level Map object needs to be constructed first to achieve micro-batch processing in small windows. The statistical dimension identifier is designated as Key1, and the statistical object and its corresponding temporary statistical variable are designated as Value1. Furthermore, the statistical object is designated as Key2, and the temporary statistical variables such as count, sum, avg, max, min, and distinctCount are designated as Value2. The statistical dimension identifier Key1 consists of a filter identifier, a statistical field identifier, a preset data processing time threshold, and the concatenated value of the statistical fields. The preset data processing time threshold is the end point of the small window time.

[0062] Furthermore, after the second-level Map object is constructed, the processed data needs to be divided into several parts of processed data corresponding to the preset number of concurrent statistical calculation sub-sub ...

[0063] Step S22: If it exists, determine whether the same statistical object exists in the inner Map.

[0064] In this embodiment, if the concurrent statistical calculator determines that a statistical dimension identifier identical to statistical dimension identifier Key1 exists in the outer Map, it retrieves the statistical object and corresponding temporary statistical variable corresponding to the same statistical dimension identifier Key1. Based on the obtained statistical object and corresponding temporary statistical variable, a second calculation is performed. It should be noted that this second calculation requires first determining whether a statistical object identical to statistical object Key2 exists in the inner Map. It should also be noted that if a statistical dimension identifier identical to statistical dimension identifier Key1 does not exist in the outer Map, then an initialized statistical object and corresponding temporary statistical variable need to be added.

[0065] Step S23: If there is no identical statistical object in the inner Map, then the temporary statistical variable corresponding to the statistical object is determined as the temporary statistical result.

[0066] In this embodiment, if there is no statistical object in the inner Map that is the same as the statistical object Key2, then the temporary statistical variable corresponding to the statistical object Key2 needs to be determined as the temporary statistical result.

[0067] Step S24: If the same statistical object exists in the inner Map, then the statistical temporary variable corresponding to the statistical object and the historical statistical temporary variable are cumulatively calculated to obtain temporary statistical data.

[0068] In this embodiment, if the inner Map contains a statistical object identical to the statistical object Key2, then the corresponding statistical temporary variable and historical statistical temporary variable need to be cumulatively calculated. The specific logic of the cumulative calculation is as follows: when the statistical method is count, the cumulative calculation is count statistical temporary variable + 1; when the statistical method is sum, the cumulative calculation is sum statistical temporary variable and the value of the corresponding statistical object; when the statistical method is avg, sum_avg and count_avg statistical temporary variables are calculated separately, and the cumulative calculation is sum_avg statistical temporary variable and the value of the corresponding statistical object, and count_avg statistical temporary variable + 1; when the statistical method is max, the cumulative calculation is the maximum value between max statistical temporary variable and the value of the corresponding statistical object; when the statistical method is min, the cumulative calculation is the minimum value between min statistical temporary variable and the value of the corresponding statistical object; when the statistical method is distinctCount, first calculate the hash value converted from the value of the corresponding statistical object, and then add the distinctCount temporary statistical object HLL to the converted hash value. After the above progressive calculation, the temporary statistical data in the preset concurrent statistical calculation unit is obtained.

[0069] Step S25: Collect temporary statistical data of each of the several preset concurrent statistical sub-sub ...

[0070] In this embodiment, it is necessary to collect temporary statistical data from each preset concurrent statistical sub-calculator and summarize the collected temporary statistical data to obtain the target temporary statistical data.

[0071] Step S26: Obtain temporary statistical results based on the statistical dimension identifier and the target temporary statistical data.

[0072] In this embodiment, obtaining temporary statistical results based on the statistical dimension identifier and the target temporary statistical data includes: determining whether the data processing time of the plurality of preset concurrent statistical calculators is not less than a preset data processing time threshold; if so, obtaining temporary statistical results based on the statistical dimension identifier and the target temporary statistical data. That is, after summarizing the temporary statistical data obtained through progressive calculation to obtain the target temporary statistical data, a small window trigger time needs to be set through a custom thread, i.e., a preset data processing time threshold. If the data processing time meets or exceeds the small window trigger time, the values ​​of the statistical dimension identifier, data processing time, statistical method, statistical object, and corresponding temporary statistical variables constitute the temporary statistical results. The temporary statistical variables include the count temporary statistical variable, sum temporary statistical variable, sum_avg temporary statistical variable, count_avg temporary statistical variable, max temporary statistical variable, min temporary statistical variable, and HLL temporary statistical object.

[0073] Step S27: Determine several indicator identifiers corresponding to the temporary statistical results, and group the temporary statistical results by the several indicator identifiers to obtain grouped temporary statistical results.

[0074] Step S28: Input the grouped temporary statistical results into the preset aggregate calculation unit to perform progressive calculation on each group of temporary statistical results in the grouped temporary statistical results to obtain the target statistical results.

[0075] In this embodiment, a preset aggregation operator, namely the keyedProcess operator, can be used to receive the temporary statistical results after grouping, so as to summarize and statistically analyze the temporary statistical results after grouping, realize secondary aggregation statistics in a large window, and summarize all temporary statistical results with the same indicator. Progressive calculation is then performed according to the statistical methods corresponding to different indicators, and the logic of the progressive calculation is the same as that in step S26 above. Finally, the target statistical result is obtained through progressive calculation.

[0076] Step S29: Register a trigger time point for the target statistical result. If the trigger time point is met, output the target statistical result through the OnTimer method.

[0077] In this embodiment, a timer trigger point for a large window can be registered for the target statistical results. When the time meets the registered timer trigger point, the target statistical results can be output through the OnTimer method, such as the count statistical value, sum statistical value, avg statistical value, max statistical value, min statistical value, distinctCount statistical value (base value of HLL object) and the corresponding indicator identifier.

[0078] Therefore, this embodiment can achieve independent aggregation of small windows and micro-batches through preset concurrent statistical calculation units. Each preset concurrent statistical calculation unit performs internal statistics and outputs temporary statistical results, thereby further compressing the amount of secondary statistical data to be collected. Furthermore, the internal statistics of preset concurrent statistical calculation units can avoid the full data exchange of global statistics, further reducing the large amount of network consumption caused by data exchange. Finally, the aggregated temporary statistical results after grouping can be summarized by the aggregated calculation unit, and the OnTimer method can be used to determine whether the window time is met. If the window time is met, the statistical results are output. In this way, network consumption is effectively reduced and statistical efficiency is improved.

[0079] See Figure 4 As shown, this embodiment of the invention discloses a real-time statistical device based on micro-batch processing, applied to a target framework, comprising:

[0080] The data preprocessing module 11 is used to parse the received data to be processed and filter the parsed data to obtain the processed data.

[0081] The data statistics module 12 is used to divide the processed data into several parts of processed data and input them into several preset concurrent statistical calculation units respectively, so that each of the several preset concurrent statistical calculation units can perform data statistics on the parts of processed data input to itself to obtain temporary statistical results.

[0082] The data grouping module 13 is used to determine several indicator identifiers corresponding to the temporary statistical results, so as to group the temporary statistical results by the several indicator identifiers to obtain the grouped temporary statistical results;

[0083] Data output module 14 is used to summarize and statistically analyze the temporary statistical results after grouping using a preset summary calculator, so as to output the target statistical result corresponding to each group of temporary statistical results in the temporary statistical results after grouping.

[0084] Therefore, in this embodiment, the received data to be processed is first parsed and then filtered to obtain processed data. The processed data is then divided into several parts and input to several preset concurrent statistical calculators. Each preset concurrent statistical calculator performs data statistics on its input processed data to obtain temporary statistical results. Several indicator identifiers corresponding to the temporary statistical results are determined, and the temporary statistical results are grouped using these indicator identifiers to obtain grouped temporary statistical results. Finally, a preset aggregate statistical calculator is used to summarize and statistically analyze the grouped temporary statistical results to output the target statistical result corresponding to each group of temporary statistical results in the grouped temporary statistical results. Therefore, the real-time statistical method based on micro-batch processing described in this application can parse and filter the received data, divide the processed data into several parts, and input them into several preset concurrent statistical sub-tasks. These sub-tasks then perform statistical analysis on their input data, obtaining temporary statistical results. These temporary results are grouped, and a summary statistical sub-task is used to aggregate the grouped temporary results, yielding several summary statistical results. In this way, the computational capabilities of statistical indicators can be optimized through different concurrent statistical sub-tasks to support simultaneous statistical calculations across multiple dimensions and time windows, further reducing memory resource consumption and improving the computational efficiency of statistical indicators.

[0085] In some embodiments, the data preprocessing module 11 may specifically include:

[0086] The data parsing unit is used to access the data to be processed generated by the target framework, and to parse the data to be processed using a preset parsing operator and / or a preset parser, so as to determine the target field obtained through data parsing as the parsed data.

[0087] The first identifier determination unit is used to obtain the filter configuration from the local database and perform a hash calculation on the filter configuration to obtain the filter identifier corresponding to the filter configuration.

[0088] A data filtering unit is used to filter target data from the parsed data based on the filtering configuration, and add the filtering identifier to the target data to obtain filtered data.

[0089] In some embodiments, the real-time statistical device based on micro-batch processing may further include:

[0090] The data feature determination unit is used to determine the statistical dimension identifier, statistical object, and statistical temporary variable of the processed data; the statistical dimension identifier includes a filter identifier, a statistical field identifier, a preset data processing time threshold, and a statistical field concatenation value.

[0091] The relationship building unit is used to create an outer Map for each of the plurality of preset concurrent statistical calculation sub-sub ...

[0092] In some embodiments, the data statistics module 12 may specifically include:

[0093] The similarity identifier determination submodule is used to divide the processed data into several parts of processed data and input them into several preset concurrent statistical operators respectively, so that the several preset concurrent operators can determine whether there are the same statistical dimension identifiers in the outer Map based on the parts of processed data input to themselves;

[0094] The similarity object determination submodule is used to determine whether the same statistical object exists in the inner Map if it exists.

[0095] The first temporary statistical result determination submodule is used to determine the temporary statistical variable corresponding to the statistical object as a temporary statistical result if there is no identical statistical object in the inner Map.

[0096] The temporary statistics determination submodule is used to perform progressive calculations on the temporary statistical variables corresponding to the statistical objects and the historical temporary statistical variables if the same statistical object exists in the inner Map, so as to obtain temporary statistics.

[0097] The temporary statistics data aggregation submodule is used to collect temporary statistics data of each of the several preset concurrent statistics calculation submodules to obtain the target temporary statistics data.

[0098] The second temporary statistical result determination submodule is used to obtain temporary statistical results based on the statistical dimension identifier and the target temporary statistical data.

[0099] In some embodiments, the second temporary statistical result determination submodule may specifically include:

[0100] The temporary statistical result determination unit is used to determine whether the data processing time of the plurality of preset concurrent statistical calculators is not less than a preset data processing time threshold. If so, a temporary statistical result is obtained based on the statistical dimension identifier and the target temporary statistical data.

[0101] In some embodiments, the data grouping module 13 may specifically include:

[0102] The second identifier determination unit is used to determine several indicator identifiers corresponding to the temporary statistical results, and set the several indicator identifiers as several grouping keys respectively.

[0103] The result grouping unit is used to perform keyby grouping of the temporary statistical results according to the several grouping keys through a preset selector, so as to obtain the grouped temporary statistical results.

[0104] In some embodiments, the data output module 14 may specifically include:

[0105] The statistical result determination unit is used to input the grouped temporary statistical results into a preset aggregate calculation unit to perform progressive calculation on each group of temporary statistical results in the grouped temporary statistical results to obtain the target statistical result;

[0106] The result output unit is used to register a trigger time point for the target statistical result. If the trigger time point is met, the target statistical result is output through the OnTimer method.

[0107] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the real-time statistical method based on micro-batch processing disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0109] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0110] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0111] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the real-time statistical method based on micro-batch processing disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0112] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned real-time statistical method based on micro-batch processing. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0114] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0116] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A real-time statistical method based on micro-batch processing, characterized in that, Applied to the target framework, including: The received data to be processed is parsed, and the parsed data is filtered to obtain the processed data. The processed data is divided into several parts and input into several preset concurrent statistical sub-sub ... Determine several indicator identifiers corresponding to the temporary statistical results, and group the temporary statistical results by the several indicator identifiers to obtain grouped temporary statistical results; The pre-defined aggregate calculation function is used to summarize and statistically analyze the temporary statistical results after grouping, so as to output the target statistical result corresponding to each group of temporary statistical results in the temporary statistical results after grouping; The step of dividing the processed data into several parts and inputting them respectively into several preset concurrent statistical calculators, so that each of the preset concurrent statistical calculators can perform data statistics on the input parts of the processed data to obtain temporary statistical results, further includes: The statistical dimension identifier, statistical object, and statistical temporary variable of the processed data are determined; the statistical dimension identifier includes the filter identifier, statistical field identifier, preset data processing time threshold, and statistical field concatenation value; For each of the plurality of preset concurrent statistical calculation sub-sub ... The process involves dividing the processed data into several parts and inputting each part into several preset concurrent statistical sub-tasks. Each of these sub-tasks performs statistical analysis on its input part of the processed data to obtain temporary statistical results, including: The processed data is divided into several parts and input into several preset concurrent statistical calculators respectively, so that the preset concurrent statistical calculators can determine whether there are the same statistical dimension identifiers in the outer Map based on the parts of processed data input into themselves. If it exists, then determine whether the same statistical object exists in the inner Map; If no identical statistical object exists in the inner Map, the temporary statistical variable corresponding to the statistical object will be determined as a temporary statistical result. If the same statistical object exists in the inner Map, then the statistical temporary variable corresponding to the statistical object and the historical statistical temporary variable are progressively calculated to obtain temporary statistical data; collect the temporary statistical data of each of the several preset concurrent statistical calculation sub-sub ...

2. The real-time statistical method based on micro-batch processing according to claim 1, characterized in that, The process of parsing the received data to be processed and filtering the parsed data to obtain processed data includes: The data to be processed generated by the target framework is accessed, and the data to be processed is parsed using a preset parsing operator and / or a preset parser, so as to determine the target field obtained through data parsing as the parsed data; The filter configuration is obtained from the local database, and a hash calculation is performed on the filter configuration to obtain the filter identifier corresponding to the filter configuration. Based on the filtering configuration, target data is filtered from the parsed data, and the filtering identifier is added to the target data to obtain processed data.

3. The real-time statistical method based on micro-batch processing according to claim 1, characterized in that, The process of obtaining temporary statistical results based on the statistical dimension identifier and the target temporary statistical data includes: Determine whether the data processing time of the plurality of preset concurrent statistical calculators is not less than a preset data processing time threshold. If so, obtain a temporary statistical result based on the statistical dimension identifier and the target temporary statistical data.

4. The real-time statistical method based on micro-batch processing according to claim 1, characterized in that, The step of determining several indicator identifiers corresponding to the temporary statistical results, and grouping the temporary statistical results using the several indicator identifiers to obtain grouped temporary statistical results, includes: Determine several indicator identifiers corresponding to the temporary statistical results, and set the several indicator identifiers as several grouping keys respectively; The temporary statistical results are grouped by keyby according to the several group keys using a preset selector to obtain the grouped temporary statistical results.

5. The real-time statistical method based on micro-batch processing according to any one of claims 1 to 4, characterized in that, The step of summarizing and statistically analyzing the grouped temporary statistical results using a preset aggregate calculator to output a target statistical result corresponding to each group of temporary statistical results includes: The grouped temporary statistical results are input into a preset aggregate calculation unit to perform progressive calculation on each group of temporary statistical results to obtain the target statistical result; Register a trigger time point for the target statistical result. If the trigger time point is met, output the target statistical result through the OnTimer method.

6. A real-time statistical device based on micro-batch processing, characterized in that, Applied to the target framework, including: The data preprocessing module is used to parse the received data to be processed and filter the parsed data to obtain the processed data. The data statistics module is used to divide the processed data into several parts of processed data and input them into several preset concurrent statistical calculation units respectively, so that each of the preset concurrent statistical calculation units can perform data statistics on the parts of processed data input to itself to obtain temporary statistical results. The data grouping module is used to determine several indicator identifiers corresponding to the temporary statistical results, so as to group the temporary statistical results by the several indicator identifiers to obtain the grouped temporary statistical results; The data output module is used to summarize and statistically analyze the grouped temporary statistical results using a preset aggregate calculation unit, so as to output the target statistical result corresponding to each group of temporary statistical results in the grouped temporary statistical results; The real-time statistical device based on micro-batch processing further includes: The data feature determination unit is used to determine the statistical dimension identifier, statistical object, and statistical temporary variable of the processed data; the statistical dimension identifier includes a filter identifier, a statistical field identifier, a preset data processing time threshold, and a statistical field concatenation value. The relationship building unit is used to create an outer Map for each of the plurality of preset concurrent statistical calculation sub-sub ... The data statistics module includes: The similarity identifier determination submodule is used to divide the processed data into several parts of processed data and input them into several preset concurrent statistical calculation units respectively, so that the several preset concurrent statistical calculation units can determine whether there are the same statistical dimension identifiers in the outer Map based on the parts of processed data input to them; The similarity object determination submodule is used to determine whether the same statistical object exists in the inner Map if it exists. The first temporary statistical result determination submodule is used to determine the temporary statistical variable corresponding to the statistical object as a temporary statistical result if there is no identical statistical object in the inner Map. The temporary statistics determination submodule is used to perform progressive calculations on the temporary statistical variables corresponding to the statistical objects and the historical temporary statistical variables if the same statistical object exists in the inner Map, so as to obtain temporary statistics. The temporary statistics data aggregation submodule is used to collect temporary statistics data of each of the several preset concurrent statistics calculation submodules to obtain the target temporary statistics data. The second temporary statistical result determination submodule is used to obtain temporary statistical results based on the statistical dimension identifier and the target temporary statistical data.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the real-time statistical method based on micro-batch processing as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the real-time statistical method based on micro-batch processing as described in any one of claims 1 to 5.

Images