Configurable dynamic data aggregation method based on group code

Abstract

The technical scheme of the invention discloses a configurable dynamic data aggregation method based on a group code. In a traditional code data processing process, abstract definition is not performed on data processing logic. In the invention, the data source and the data processing logic are abstractly defined as operators, and can be configured through a panel. And if a filtering logic condition needs to be added to certain interface data, only one filtering operator needs to be added, and the filtering condition is configured, so that a data filtering function can be realized. According to the method, operators are configured in a flow chart mode, data processing services are completed in a flow chart configuration mode, and a large amount of repeated development and test work is reduced. In a traditional data processing mode, data flow logic needs to be automatically processed in codes, and the reliability of program execution depends on a large amount of development and test work.

Description

Technical Field

[0001] This invention relates to a data aggregation method. Background Art

[0002] In internet business, when developing systems that use data from other systems, data reuse is typically achieved through HTTP interfaces. However, due to the customization of business requirements, the raw data format of third-party interfaces cannot be directly applied to downstream services and requires a series of additional processing steps such as filtering, sorting, and aggregation before it can be used. Traditionally, the downstream system's code needs to implement the processing logic for third-party data, filtering and transforming the raw data into the data required by the business system. As business grows, the system's processing logic becomes increasingly complex, and its dependence on third-party systems gradually increases, leading to a sharp rise in the cost of understanding and maintaining the system.

[0003] Existing downstream service systems use HTTP interfaces to obtain raw data and write data processing flows in their code to filter, transform, and merge the raw data to obtain usable final data. However, this processing method lacks unified maintenance, and different service systems perform repetitive work. In summary, the traditional methods of processing third-party data in internet business applications have the following main problems:

[0004] 1. Low efficiency in handling changes to data processing logic. Data processing logic added to various services cannot be reused, requiring a large amount of repetitive work and testing.

[0005] 2. High system maintenance costs. As business needs change, system iterations increase complexity. Each developer has different coding habits and design approaches, making system maintenance extremely difficult as subsequent maintainers need to understand the iterative process of the business system.

[0006] 3. Low hardware resource utilization. Each business system requires project deployment, consuming server hardware resources. To avoid peak service pressure, redundant deployment is necessary, resulting in even greater resource waste. Summary of the Invention

[0007] The purpose of this invention is to propose a dynamically configurable and modifiable, efficient and flexible data processing solution to reduce system maintenance costs and respond to business needs more quickly.

[0008] To achieve the above objectives, the technical solution of this invention provides a configurable dynamic data aggregation method based on Groovy code. The method is characterized by defining each row of the data to be aggregated as a data entry and each column as a field. The configurable dynamic data aggregation method includes the following steps:

[0009] Step 1: Based on the current data aggregation task, select the required operators, configure each operator, and determine the execution order of each configured operator according to the overall data processing logic of the current data aggregation task. Determine the dependencies between different operators based on the execution order, where:

[0010] Operators include data source operators and non-data source operators. Data source operators are used to obtain data through the configured network interface and generate output data, while non-data source operators are used to process the input data according to the configuration and generate output data.

[0011] If the input data of operator one is the output data generated by operator two, then operator one depends on operator two, and the dependencies between all operators form a directed acyclic graph.

[0012] Initialize the state of all operators to unexecuted;

[0013] Step 2: Calculate the in-degree of each operator, where the in-degree of the current operator is the number of all operators whose states the current operator depends on are not yet executed;

[0014] Step 3: Determine if there exists an operator with an in-degree of 0 and a state of not executed.

[0015] If it exists, execute the current operator and update the state of the current operator to complete. Update the in-degree of all operators that depend on the current operator. The updated in-degree is the original in-degree minus one. Return to step 2.

[0016] If it does not exist, proceed to step 4;

[0017] Step 4: Obtain the result of the last executed operator and store the result in the storage device.

[0018] Preferably, the data source operator obtains JSON format data through an HTTP interface or Kafka message, and extracts data values ​​from the JSON format data according to the configured jsonpath variable: if the JSON format data is an array, the number of array elements M in the JSON format data determines the number of rows of data, and the value N of the jsonpath variable determines the number of columns of data; if the JSON format data is not an array, then M=1, and the value N of the jsonpath variable determines the number of columns of data; the data source operator outputs an M×N data.

[0019] Preferably, the non-data source operator includes:

[0020] Data filtering operators:

[0021] The data filtering operator processes the input data row by row, replacing the `{field}` in the Boolean expression with the value of the `field` field of the current data row. In the Boolean expression, `{field}` represents the value of a field. The Groovy script performs logical calculations on `{field}` to obtain the Boolean value of the current Boolean expression. The data filtering operator then checks if the obtained Boolean value is true: if yes, it adds the current data row to the output data; otherwise, it discards the current data row. Finally, the data filtering operator outputs the filtered data.

[0022] Data-derived operators:

[0023] The data derivation field operator processes the input data row by row. It replaces the `{field}` column in the field expression with the value of the `field` column from the current record. In the field expression, `{field}` represents the value of a field. The Groovy script performs logical operations and string manipulation on `{field}` to obtain the result of the current field expression. The data derivation operator adds an extra column to the current record, with the field name being the configured field name and the value being the result of the current field expression. This extra column is then added to the output data. Ultimately, the data derivation operator outputs data with an added column of derived fields.

[0024] Data merging operator:

[0025] The data merging operator uses different merging methods to merge multiple input data, including appending row merging and appending column merging.

[0026] If the merging method is append row merging, and the fields of multiple input data are consistent, then each data item from the multiple input data will be added to the output data in turn;

[0027] If the merging method is append column merging, multiple input data have the same unique field as the associated field. The associated field can contain multiple fields to ensure uniqueness. The data merging operator processes each row of each input data in turn, obtains the associated field of the current data, and checks whether there is a data row with the same associated field in the output data. If yes, a new column is added to the current output data row, and the new column is any field in the input data row other than the associated field. If no, the current input data row is added to the output data.

[0028] Finally, the data merging operator outputs the merged data;

[0029] Data grouping operator:

[0030] The data grouping operator adds each row of input data to a different grouping list based on the grouping field. The grouping field can be one or more fields. Data rows with the same grouping field are merged into a single row. Within each group, the data grouping operator processes each column of data according to the field processing type, then merges multiple rows into a single row and adds it to the output data. The field processing type includes retrieving the first value, retrieving the maximum value, retrieving the minimum value, retrieving the average value, and merging into an array. Finally, the data grouping operator outputs the grouped and merged data.

[0031] Data sorting operators:

[0032] The data sorting operator reorders the input data rows according to the sorting field value and sorting direction, and adds them to the output data in order. The sorting field is a comparable field, and the sorting direction is ascending or descending. Finally, the data sorting operator outputs the sorted data.

[0033] Preferably, the non-data source operator also includes a custom operator, which is implemented by writing Groovy scripts and configuring the operator to meet non-standard business requirements that cannot be handled by the data filtering operator, data derivation operator, data merging operator, data grouping operator, and data sorting operator.

[0034] Compared with existing technical solutions, the present invention has the following beneficial effects:

[0035] 1. Abstract definition of data processing logic

[0036] Traditional code processing of data often lacks an abstract definition of the data processing logic. In this invention, both the data source and the data processing logic are abstracted and defined as operators, which can be configured via a control panel. If filtering logic conditions need to be added to data from a specific interface, simply add a filtering operator and configure the filtering conditions to achieve the data filtering function.

[0037] 2. Data processing via configuration

[0038] This invention enables operators to be configured via flowcharts, allowing data processing to be completed through flowchart configuration, thus reducing a significant amount of repetitive development and testing work. In traditional data processing methods, data flow logic needs to be handled manually in the code, and the reliability of program execution depends heavily on extensive development and testing.

[0039] 3. This invention enables visual management of the data processing node's processing flow, allowing for data tracking during system operation. Traditional data processing methods often lack intuitive data visualization, requiring log printing for tracking the data processing logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Configure the graph for filtering conditions;

[0041] Figure 2 This illustrates data processing in a configured manner;

[0042] Figure 3 This illustrates visual management;

[0043] Figure 4 A flowchart of a data aggregation method provided in an embodiment of the present invention;

[0044] Figure 5 The operator dependency relationship diagram provided for embodiments of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0046] If each row of the data to be aggregated is defined as a data point and each column is defined as a field, then... Figure 1 As shown, the overall execution flow of the configurable dynamic data aggregation method based on Groovy code disclosed in this embodiment of the invention specifically includes the following steps:

[0047] Step S11: After obtaining all operators, configure each operator and specify the dependencies between them.

[0048] In this invention, data aggregation logic is defined as operators, and data is transformed and processed through these operators. Operators typically take one or more data points to be aggregated as input and output aggregated data. In special cases, the data source operator serves as the starting operator, with no input data and only output data. Through a series of pre-defined operators, basic functions such as data filtering, field derivation, merging, grouping, and sorting can be implemented. More complex data processing is achieved by executing user-defined Groovy scripts using custom operators.

[0049] In this invention, the operator includes:

[0050] 1) Data source operator:

[0051] The data source operator includes an HTTP interface or Kafka message address and a jsonpath variable configuration. It retrieves JSON data via the HTTP interface or Kafka message and extracts data from the JSON data according to the configured jsonpath variable. The number of array elements M in the JSON data determines the number of rows, and the number of columns N in the jsonpath variable determines the number of columns. An exception is if the JSON data is not an array, in which case the number of rows is 1. The data source operator ultimately outputs an M×N data set.

[0052] 2) Data filtering operators:

[0053] The data filtering operator consists of a Boolean expression and input data. In the Boolean expression, `{field}` represents the value of a field. Logical calculations are performed on `{field}`, and a Boolean value is output. The execution of the Boolean expression is implemented using a Groovy script. The data filtering operator processes the input data line by line, replacing `{field}` in the Boolean expression with the value of the `field` field of that line, and then executing the Groovy script to obtain the Boolean value of the expression. The data filtering operator checks if the obtained Boolean value is true: if yes, the current line of data is added to the output data; otherwise, the current line of data is discarded. Finally, the data filtering operator outputs the filtered data.

[0054] 3) Data derivation operators:

[0055] The data derivation operator takes one input data set, along with field names and field expressions. In the field expression, `{field}` represents the value of a field, and logical operations and string manipulation are performed on `{field}`. The data derivation field operator processes the input data row by row, replacing `{field}` in the field expression with the value of the current row's `field` field, and executing the Groovy script to obtain the result of the current field expression. The data derivation operator adds an extra column to the current row's data, with the field name being the configured field name and the value being the result of the current field expression. The data derivation operator adds this extra column to the output data. Finally, the data derivation operator outputs data with an added column of derived fields.

[0056] 4) Data merging operator:

[0057] The data merging operator includes multiple input data, a merging method, and an associated field. There are two merging methods: append row merging and append column merging. When merging with append rows, the fields of all input data must be consistent, and each row of data from each input data set is sequentially added to the output data. When merging with append columns, all input data sets must have a common unique field as the associated field. In special cases, the associated field can contain multiple fields to ensure uniqueness. The data merging operator processes each row of data from each input data set sequentially, obtaining the associated field for that row, and checking if a row with the same associated field exists in the output data. If it does, a new column is added to the current output data row, containing all fields from the input data row except the associated field. If not, the current input data row is added to the output data. Finally, the data merging operator outputs the merged data.

[0058] 5) Data grouping operator:

[0059] A data grouping operator includes input data, grouping fields, and the processing type for each field. The grouping fields can be one or more fields; rows with the same grouping field are merged into a single row. Based on the grouping field, the data grouping operator adds each row of input data to a different grouping list. Field processing types include taking the first value, taking the maximum value, taking the minimum value, taking the average value, and merging into an array. Within each group, based on the field processing type, the data grouping operator processes each column of data, then merges multiple rows into a single row and adds it to the output data. Finally, the data grouping operator outputs the grouped and merged data.

[0060] 6) Data sorting operators:

[0061] A data sorting operator takes one input data set, a sort field, and a sort direction. The sort field must be a comparable field; unsortable fields such as arrays or objects cannot be specified as the sort field. The sort direction is either ascending or descending. Based on the sort field value and sort direction, the data sorting operator reorders the input data rows and adds them to the output data in the correct order. Finally, the data sorting operator outputs the sorted data.

[0062] 7) Custom operators:

[0063] Users can write their own Groovy scripts and configure them as custom operators to meet non-standard business requirements that the general operators described in 1) to 6) cannot handle.

[0064] Step S12: Calculate the execution order of all operators. The execution order of operators is determined by calculating the in-degree of each operator, where the number of unexecuted operators that the current operator depends on is defined as the in-degree.

[0065] Step S13: Determine if there exists an operator with an in-degree of 0 that has not yet been executed.

[0066] If so, execute the current operator, update the state of the current operator to complete, set the in-degree of all subsequent operators that depend on the current operator to decrement by 1, and return to step S12;

[0067] If not, proceed to step S14.

[0068] Step S14: Obtain the result of the last executed operator and store the data in a storage device, which can be one of a database, a disk, or a hard disk.

[0069] The dependencies between operators in this invention are represented by a directed acyclic graph to ensure that all operators can be executed.

[0070] To clearly explain the data aggregation process in this embodiment and to facilitate a more comprehensive understanding of the specific aggregation process by those skilled in the art, a specific example is provided. In this example, the dependencies between different operators are as follows: Figure 2 As shown:

[0071] On the first execution, the in-degree of all operators is calculated: operators S21 and S22 have no dependencies, so their in-degree is 0; operator S23 depends on operators S21 and S22, so its in-degree is 2; operator S24 depends on operator S23, so its in-degree is 1; operator S25 depends on operator S24, so its in-degree is 1; operator S26 depends on operator S25, so its in-degree is 1; and operator S27 depends on operator S26, so its in-degree is 1.

[0072] The initial execution shows the states of all operators as shown in the table below.

[0073] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 2 1 1 1 1 Executed no no no no no no no

[0074] Operators S21 and S22 are the ones with an in-degree of 0 that have not been executed.

[0075] Step S21: Request the HTTP interface to obtain JSON data. For ease of description, the data used in this example is relatively small, as follows:

[0076]

[0077]

[0078] The jsonpath used is $.data.stock[*] and $.data.name[*]. The data for 'stock' and 'name' is extracted respectively, and the field names are then renamed to 'stock' and 'name', resulting in the following data:

[0079] Table 1: HTTP Data from Data Source Operator S21

[0080] stock name 000001.sh Company No. 1 000002.sz Company No. 2 000003.sz Company No. 3 000004.sz Company No. 4

[0081] Step S22: Request Kafka data. For ease of description, the data used in this example is relatively small, as follows:

[0082]

[0083]

[0084] The jsonpath used is $.data.stock[*] and $.data.pe[*]. We extracted data from stock and pe respectively, and named their fields stock and pe, resulting in the following data:

[0085] Table 2: Data from the S2KAFKA Data Source Operator

[0086]

[0087] The states of operators S21 and S22 are updated to "completed," and the in-degree of the subsequent operators is decremented by 1. Since operator S23 depends on both operators S21 and S22, its in-degree is decremented by 2, and its in-degree is updated to 0. After the update, the states of all operators are shown in the table below:

[0088] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 0 1 1 1 1 Executed yes yes no no no no no

[0089] Operator S23 with an in-degree of 0 and not yet executed:

[0090] Step S23: Merge the data from operators S21 and S22. For ease of description, the merging method in this example is append column merging. The data fields for operator S21 are stock and name; the data fields for operator S22 are stock and pe. The stock field is selected as the associated field. Data with the same stock value are saved in the same row, resulting in the following data:

[0091] Table 3: Data for the S23MERGE Merge Operator

[0092]

[0093] The state of operator S23 is updated to "completed," and the in-degree of its successor operator is decremented by 1. Operator S24 depends on operator S23, its in-degree is decremented by 1, and its in-degree is updated to 0. After the update, the states of all operators are shown in the table below:

[0094] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 0 0 1 1 1 Executed yes yes yes no no no no

[0095] Operator S24, which has an in-degree of 0 and has not been executed:

[0096] Step S24: Filter the data in operator S23 using an expression. For ease of description, this example uses the simple expression: {pe}<20. Each row of data in operator S23 is checked sequentially. The value of pe is substituted into the expression to determine if the expression is true. If yes, the data row is added to the output data; otherwise, the data is discarded. In this example, only 000002.sz has pe>20, so the expression result is false, and this data is discarded, resulting in the following data:

[0097] Table 4: Data from the S24FILTER filter operator

[0098]

[0099]

[0100] The status of S24 is updated to "completed," and the in-degree of the post-operator is reduced by 1: the in-degree of operator S25, which depends on operator S24, is reduced by 1 and updated to 0. After the update, the status of all operators is shown in the table below:

[0101] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 0 0 0 1 1 Executed yes yes yes yes no no no

[0102] Operator S25, which has an in-degree of 0 and has not been executed:

[0103] Step S25: Generate new derived fields based on existing fields. For ease of description, this example generates the market code field "market" for each row of data, using the rule {stock}.substring(7,9). `substring(7,9)` extracts the 8th and 9th characters of the string. This example uses a built-in Groovy function to implement string extraction. Substitute the "stock" value from each row of data into the expression, execute the Groovy script, and generate the value of the derived field "market," resulting in the following data:

[0104] Table 5: FIELD Data for S25 Derived Field Operators

[0105] stock name pe market 000001.sh Company No. 1 10 sh 000003.sz Company No. 3 13 sz 000004.sz Company No. 4 5 sz

[0106] The status of operator S25 is updated to "completed," and the in-degree of its successor operator is decremented by 1. Operator S26 depends on operator S25, its in-degree is decremented by 1, and its in-degree is updated to 0. After the update, the status of all operators is shown in the table below:

[0107] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 0 0 0 0 1 Executed yes yes yes yes yes no no

[0108] Operator S26, which has an in-degree of 0 and has not been executed:

[0109] Step S26: Custom operators allow users to implement non-standard operations using Groovy scripts. For ease of description, the stock and name fields are combined into an info object in this example. The Groovy script is as follows:

[0110]

[0111] Substitute the data into the script variable {data}, execute the script to generate new data. The result is the following data:

[0112] Table 6: Custom Operator S26CUSTOM Data

[0113]

[0114] The state of operator S26 is updated to "completed," and the in-degree of its successor operator is decremented by 1. Operator S27 depends on operator S26, its in-degree is decremented by 1, and its in-degree is updated to 0. After the update, the states of all operators are shown in the table below:

[0115] Operator S21 S22 S23 S24 S25 S26 S27 degree 0 0 0 0 0 0 0 Executed yes yes yes yes yes yes no

[0116] Operator S27, which has an in-degree of 0 and has not been executed:

[0117] Step S27: Group the data according to the grouping field "market". In this example, the data is divided into two groups by the "market" field. The "pe" field is processed as an average, and the "info" field is processed as merged data. The following data is obtained:

[0118] Table 7: S27 Grouping Operator GROUP Data

[0119]

[0120] Update the state of operator S27 to complete, and decrement the in-degree of the follower operator by 1: there is no follower operator, and the in-degree of all operators remains unchanged.

[0121]

[0122]

[0123] There are no operators with an in-degree of 0 that have not yet been executed; all operators have been executed. The output data of the last executed operator S27 is the final data, which is stored in the storage device.

Claims

1. A configurable dynamic data aggregation method based on groovy code, characterized in that: Each row of the data to be aggregated is defined as a piece of data, and each column is defined as a field. The configurable dynamic data aggregation method includes the following steps: Step 1: According to the current data aggregation task, select the operator to be used, configure each operator, and determine the execution order of each configured operator according to the overall data processing logic of the current data aggregation task, and determine the dependency between different operators according to the execution order, where: Operators include data source operators and non-data source operators. Data source operators are used to generate output data after obtaining data through the configured network interface, and non-data source operators are used to generate output data after performing data aggregation logic processing in response to input data according to the configuration; If the input data of operator one is the output data generated by operator two, then operator one depends on operator two, and the dependency relationships between all operators form a directed acyclic graph; Initialize the status of all operators to unexecuted; Step 2: Calculate the in-degree of each operator, where the in-degree of the current operator is the number of all operators that the current operator depends on and whose status is not executed; Step 3: Determine whether there is an operator with in-degree 0 and status not executed: If it exists, execute the current operator and update the status of the current operator to completed. Update the in-degree of all operators that depend on the current operator. The updated in-degree is the original in-degree minus one. Return to step 2. If it does not exist, go to step 4; Step 4: Get the result of the last executed operator and store the result in a storage device.

2. A configurable dynamic data aggregation method based on groovy code as claimed in claim 1, characterized in that: The data source operator obtains json format data through the http interface or kafka message, and extracts the data value in the json format data according to the configured jsonpath variable: if the json format data is an array, the number of array elements M in the json format data determines the number of rows of the data, and the value N of the jsonpath variable determines the number of columns of the data; if the json format data is not an array, M=1, and the value N of the jsonpath variable determines the number of columns of the data; The data source operator outputs an M×N data.

3. A configurable dynamic data aggregation method based on groovy code as claimed in claim 1, characterized in that: The non-data source operator includes: Data filtering operator: The data filter operator processes the input data row by row, and replaces {field} in the Boolean expression with the value of the field field of the current data. In the Boolean expression, {field} represents the value of a field. The groovy script is executed to perform logical calculations on {field} to obtain the Boolean value of the current Boolean expression. The data filter operator determines whether the obtained Boolean value is true: if so, the current data is added to the output data; if not, the current data is discarded; finally, the data filter operator outputs the filtered data; Data Derivation Operators: The data-derived field operator processes the input data row by row, and replaces {field} in the field expression with the value of the field field of the current data. In the field expression, {field} represents the value of a field. The groovy script is executed to perform logical operations and string processing on {field} to obtain the result of the current field expression. The data-derived operator adds an additional column to the current data. The field name is the configured field name, and the value is the execution result of the current field expression. The data-derived operator adds the additional column to the output data. Finally, the data-derived operator outputs data with an additional column of derived fields. Data merging operator: The data merge operator merges multiple input data using different merging methods, including append row merging and append column merging: If the merge mode is to append rows, the fields of multiple input data are consistent, and each piece of data of the multiple input data is added to the output data in sequence; If the merging method is append column merging, multiple input data have the same unique field as the associated field domain, where the associated field domain can contain multiple fields to ensure uniqueness; the data merging operator processes each row of each input data in turn, obtains the associated field domain of the current data, and searches whether there is a data row with the same associated field domain in the output data: if so, directly add column data to the current output data row, and the newly added column data is the other fields in the input data row except the associated field domain; if not, add the current input data row to the output data; Finally, the data merging operator outputs the merged data; Data grouping operator: The data grouping operator adds each row of input data to a different grouping list according to the grouping field domain, where the grouping field domain is one field or multiple fields, and the data rows with the same grouping field domain are finally merged into one row of data; in each group, the data grouping operator processes each column of data according to the field processing type, and then merges multiple rows of data into one row and adds it to the output data, where the field processing types are divided into taking the first, taking the maximum, taking the minimum, taking the average, and merging into an array; finally, the data grouping operator outputs the grouped and merged data; Data sorting operator: The data sorting operator reorders the input data rows according to the sorting field value and sorting direction, and adds them to the output data in order. The sorting field is a comparable field, and the sorting direction is ascending or descending. Finally, the data sorting operator outputs the sorted data.

4. A configurable dynamic data aggregation method based on groovy code as claimed in claim 3, characterized in that: The non-data source operators also include custom operators, which are implemented by writing groovy scripts and configuring operators to meet non-standard business requirements that cannot be handled by the data filtering operator, data derivation operator, data merging operator, data grouping operator, and data sorting operator.

Images