Data processing method and related equipment

Abstract

The embodiment of the invention provides a data processing method and related device.The data processing method comprises the steps that in response to a first operation instruction, the first operation instruction comprises a second index field selected in first index fields, and a first dimension field is determined in dimension fields corresponding to the second index field; responding to a second operation instruction, wherein the second operation instruction comprises a second dimension field selected from the first dimension fields; reading a first dimension value meeting the filtering rule from the dimension values of the second dimension field, and reading a first index value meeting the filtering rule from the index values of the second index field; performing aggregation calculation on the first index value and the first dimension value to obtain an aggregation calculation result; and generating a first aggregation table based on the aggregation calculation result, the second index field and the second dimension field. By adopting the embodiment of the invention, the flexibility of aggregation calculation can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related equipment. Background Art

[0002] With the continuous development of Internet technology, more and more business scenarios use indicator data to measure business operations, which makes the importance of indicator data in business scenarios gradually increase; and with the continuous expansion of business scenarios, the amount of indicator data in business scenarios continues to increase, and the forms of indicator data become more and more diverse, resulting in increasing application requirements for indicator data and the amount of analysis and calculation of indicator data. In this process, higher indicator data processing requirements are also put forward to the business side. Summary of the Invention

[0003] In a first aspect, an embodiment of the present application provides a data processing method, comprising: In response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; In response to a second operation instruction, the second operation instruction includes a second dimension field selected from the first dimension field; Reading a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and reading a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; Performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; A first aggregate table is generated based on the aggregate calculation result, the second indicator field, and the second dimension field.

[0004] It can be seen that in an embodiment of the present application, first, on the one hand, in response to the first operation instruction, the first operation instruction includes the second indicator field selected in the first indicator field, and the first dimension field is determined in the dimension field corresponding to the second indicator field; on the other hand, in response to the second operation instruction, the second operation instruction includes the second dimension field selected in the first dimension field. Secondly, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field. By selecting fields in the indicator table and the dimension table and inputting field values ​​to construct filtering rules, the flexibility of constructing filtering rules is improved. On this basis, the first indicator value and the first dimension value are aggregated to obtain the aggregation calculation result, and the first aggregation table is generated by combining the aggregation calculation result, the second indicator field and the second dimension field. By flexibly selecting the second indicator field and flexibly selecting the second dimension field, and then combining the filtering rules to filter the first indicator value of the second indicator field and the first dimension value of the second dimension field, the flexibility of the aggregation calculation is improved, and the diverse needs of users for aggregation calculation are met. The aggregation calculation results are displayed in a visual way through the first aggregation table, and the intuitiveness of the aggregation calculation is improved.

[0005] In a second aspect, an embodiment of the present application provides a data processing device, including: a field determination module, configured to, in response to a first operation instruction, wherein the first operation instruction includes a second indicator field selected in the first indicator field, determine a first dimension field in a dimension field corresponding to the second indicator field; An instruction response module, configured to respond to a second operation instruction, wherein the second operation instruction includes a second dimension field selected from the first dimension field; a reading module, configured to read a first dimension value satisfying a filtering rule from the dimension value of the second dimension field, and to read a first index value satisfying the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; an aggregation calculation module, configured to perform aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; An aggregation table generation module is used to generate a first aggregation table based on the aggregation calculation result, the second indicator field and the second dimension field.

[0006] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the data processing method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the data processing method as described in the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive work. Figure 1 A data processing method processing flow chart provided in an embodiment of the present application; Figure 2 A schematic diagram of an aggregate calculation process provided in an embodiment of the present application; Figure 3 A flowchart of a data processing method applied to a table scenario provided in an embodiment of the present application; Figure 4 A schematic diagram of a data processing device provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0011] This specification provides an embodiment of a data processing method: Currently, when processing indicator data, business personnel are often required to write fixed SQL (Structured Query Language) to query and filter relevant indicator data and then perform indicator-related calculations, which has poor flexibility.

[0012] In response to this, the data processing method provided by this embodiment responds to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining the first dimension field in the dimension field corresponding to the second indicator field; responds to a second operation instruction, the second operation instruction includes selecting a second dimension field in the first dimension field; then, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field, and the filtering rule is constructed by selecting fields in the indicator table and the dimension table and inputting field values ​​to improve the flexibility of constructing the filtering rule. Finally, the aggregation calculation result is obtained by performing aggregation calculation on the first indicator value and the first dimension value, and a first aggregation table is generated according to the aggregation calculation result, the second indicator field and the second dimension field, so as to realize flexible selection of the second indicator field and flexible selection of the second dimension field, and then the first indicator value of the second indicator field and the first dimension value of the second dimension field are screened in combination with the filtering rule to improve the flexibility of the aggregation calculation, and the aggregation calculation result is displayed in a visual way through the first aggregation table to improve the intuitiveness of the aggregation calculation.

[0013] The data processing method provided in this embodiment can be executed by a server, and the server can run on a server, wherein the server can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0014] Reference Figure 1 The data processing method provided in this embodiment specifically includes steps S102 to S110.

[0015] Step S102 , in response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field.

[0016] The first operation instruction may be a user instruction issued through gestures, voice, etc., or may be sent by a client, terminal, or other device that has established a communication connection with the server. The following specific embodiments are described using a client that has established a communication connection with the server. This embodiment is applicable to the server.

[0017] The first indicator field in this embodiment may include a first indicator item, that is, a field at the indicator level, and the first indicator field may be one or more; the second indicator field may include a second indicator item, and the second indicator field may also be one or more, for example, the first indicator field is sales_amount (sales) and sell_total (sales volume), and the second indicator field is sales_amount (sales); for another example, the first indicator field is the number of client logins, the number of resource applicants, the number of successful resource applicants, the resource application success rate, and the resource application rate, and the second indicator field is the resource application rate and the resource application success rate; it should be noted that the first indicator field and the second indicator field here are merely schematic, and the first indicator field and the second indicator field may be indicator fields of any one or multiple fields, that is, they may be any one or more indicator fields. The first indicator field and the second indicator field may be determined based on the actual application scenario, and this embodiment does not make specific limitations here.

[0018] Optionally, the first indicator field is selected in the indicator table, and the dimension field corresponding to the first indicator field is selected in the indicator table and / or the dimension table; the indicator table and the dimension table here may be associated with each other, that is, the indicator table and the dimension table may belong to the same data graph, and the indicator table in the data graph may be one or more, and the dimension table in the data graph may also be one or more, and the indicator table and the dimension table in the data graph may be associated with each other; that is, the data graph may be a graph composed of one or more indicator tables and one or more dimension tables.

[0019] An indicator table may contain indicator fields and / or dimension fields, and a dimension table may contain dimension fields. In addition, an indicator table may contain field types of indicator fields and / or dimension fields, and a dimension table may contain field types of dimension fields. Such field types may include Integer (integer type), String (string type) and / or Decimal (decimal type). In addition, the field type may also be other field types. An indicator table and a dimension table in a data graph may contain the same dimension fields, that is, an indicator table and a dimension table in a data graph may contain common dimension fields. An indicator table may contain any number of indicator fields and / or dimension fields, and a dimension table may contain any number of dimension fields. For example, an indicator table in a data graph is shown in Table 1 below:

[0020] Table 1 The indicator table shown in Table 1 contains the indicator field sales_amount, which is sales volume. The field type of sales volume is decimal. The dimension fields contained in the indicator table are dt, product_type, and batch_no, which are time, product type, and batch code. The field type of time is integer, the field type of product type is integer, and the field type of batch code is string. Another indicator table in the data map shown in Table 2 below:

[0021] Table 2 Among them, the indicator table shown in Table 2 contains the indicator field sell_total, that is, sales volume, and the field type of sales volume is integer type. The dimension fields contained in the indicator table are dt, product_type, and batch_no, that is, time, product type, and batch code. The field type of time is integer type, the field type of product type is integer type, and the field type of batch code is string type; the indicator tables shown in Table 1 and Table 2 have common dimension fields, that is, the same dimension fields are time, product type, and batch code.

[0022] As shown above, a data graph can contain indicator tables. In addition, a data graph can also contain dimension tables. The indicator table and the dimension table can contain the same dimension fields. For example, the dimension table in the data graph to which the indicator tables shown in Table 1 and Table 2 belong is shown in Table 3 below:

[0023] Table 3 Among them, the dimension table shown in Table 3 contains dimension fields dt, product_type, and batch_no, namely time, product type, and batch code. The field type of time is integer type, the field type of product type is integer type, and the field type of batch code is string type.

[0024] The dimension field corresponding to the second indicator field can be the dimension field set to the second indicator field. The dimension field corresponding to the second indicator field here can be selected in the indicator table and / or dimension table, that is, the dimension field corresponding to the second indicator field can be the dimension field set by the user to the second indicator field. For example, the first indicator field selected from the indicator tables shown in Table 1 and Table 2 is sales_amount and sell_total, the second indicator field selected from the first indicator field is sales_amount and sell_total, the dimension fields corresponding to sales_amount are dt, product_type and batch_no, and the dimension fields corresponding to sales volume are dt, product_type and batch_no.

[0025] The first dimension field may include first dimension items, that is, fields at the dimension level. The first dimension field may be one or more, and the second dimension field may also be one or more. The first dimension field may be a dimension field of any one or multiple fields, that is, it may be any one or multiple dimension fields. For example, the first dimension field is dt, product_type, and batch_no, and the second dimension field is dt and product_type.

[0026] In a specific implementation, in response to the first operation instruction, the first dimension field is determined in the dimension field corresponding to the second indicator field; the first operation instruction includes the second indicator field selected in the first indicator field, and the first operation instruction may be sent by the client. Specifically, the client may send the first operation instruction to the server, and the first operation instruction includes the second indicator field selected in the first indicator field. The server determines the first dimension field in the dimension field corresponding to the second indicator field in response to the first operation instruction; more specifically, the client may obtain the second indicator field selected in the first indicator field and send it to the server, and the server may receive the second indicator field selected in the first indicator field sent by the client; the client in this embodiment may be a resource client, which may be used to apply for resources or perform other resource-related task processing; the client may be an application or a subroutine. In addition, the client may also be other types of clients.

[0027] For example, the first indicator field is sales_amount and sell_total, the client obtains the second indicator field selected by the user in the first indicator field as sales_amount and sell_total, sends the first operation instruction to the server, and the server receives the first operation instruction, which includes the second indicator field selected in the first indicator field.

[0028] During the specific execution process, it is mentioned above that the first indicator field is selected in the indicator table. In order to improve the convenience and flexibility of performing aggregation calculations, specifically, after the server receives the first indicator field selected in the indicator table sent by the client, it can select the dimension field corresponding to the first indicator field in the indicator table and / or dimension table, and set filtering conditions and / or aggregation algorithms to the first indicator field, so as to improve the efficiency and convenience of subsequent actual aggregation calculations; specifically, in the process of selecting the first indicator field in the indicator table, the client can respond to the indicator setting instruction, display the data map list, and send the data map selected in the data map list to the server; the data map list here may contain one or more data maps, and the data maps in the data map list may be constructed in advance. Each data map may contain one or more indicator tables and one or more dimension tables. The indicator table and dimension table in each data map may contain the same dimension field; After the client sends the data map selected in the data map list to the server, the server can send the indicator fields contained in the indicator table in the data map to the client, and the client can send the first indicator field selected in the indicator fields contained in the indicator table to the server, and the server can receive the first indicator field selected in the indicator fields contained in the indicator table; thereafter, the server can receive the dimension field corresponding to the first indicator field selected in the indicator table and / or dimension table, and construct the filtering rule of the first indicator field based on the fields selected in the indicator table and / or dimension table and the field values ​​input, and receive the aggregation algorithm of the first indicator field selected in the aggregation algorithm list; therefore, since the indicator table in the data map only contains indicator fields but does not contain indicator tables, and the dimension table in the data map only contains dimension fields but does not contain dimension values, the simple data structure of the data map can be used to conveniently determine the first indicator field, conveniently select fields and input field values, and then construct filtering rules to reduce interference between indicator values ​​and dimension values.

[0029] Among them, the first indicator field selected in the indicator table, the dimension field corresponding to the first indicator field selected in the indicator table and / or dimension table, the indicator table and dimension table used to set the filtering conditions and / or aggregation algorithm for the first indicator field can belong to the same data graph; the data graph can be obtained in the following manner: determining the indicator table containing the same dimension field in the indicator table set, and determining the dimension table matching the same dimension field in the dimension table set, and constructing a data graph based on the indicator table and the dimension table; the dimension table matching the same dimension field here can be a dimension table containing dimension fields that are completely consistent or partially consistent with the same dimension field.

[0030] It should be noted that the indicator table set here can be a set consisting of one or more indicator tables, the dimension table set can be a set consisting of one or more dimension tables, the indicator tables in the indicator table set can be pre-generated, specifically generated based on business scenario requirements or constructed by users according to requirements, and the dimension tables in the dimension table set can also be pre-generated, specifically generated based on business scenario requirements or constructed by users according to requirements.

[0031] In actual applications, the second indicator field selected in the first indicator field may be one or more. In order to improve the flexibility of determining the first dimension field and meet the diverse needs of determining the first dimension field, an optional implementation provided by this embodiment performs the following operations during the process of determining the first dimension field in the dimension field corresponding to the second indicator field: If the number of fields in the second indicator field is equal to 1, the dimension field corresponding to the second indicator field is used as the first dimension field; If the number of fields is greater than 1, the same dimension field is filtered out from the dimension fields corresponding to the multiple indicator fields included in the second indicator field as the first dimension field.

[0032] For example, if the second indicator field is sales_amount and sell_total, the dimension fields corresponding to sales_amount are dt, product_type, and batch_no, and the dimension fields corresponding to sell_total are dt, product_type, and batch_no. If the number of fields of the second indicator field is greater than 1, then the same dimension fields dt, product_type, and batch_no are filtered out from the dimension fields corresponding to the multiple indicator fields contained in the second indicator field. Then the first dimension fields are dt, product_type, and batch_no. If the second indicator field is sell_total, the dimension fields corresponding to sell_total are dt and product_type. If the number of fields of the second indicator field is equal to 1, then the dimension fields dt and product_type corresponding to sell_total are used as the first dimension fields.

[0033] Step S104 , responding to a second operation instruction, where the second operation instruction includes a second dimension field selected from the first dimension field.

[0034] In this step, in response to the second operation instruction sent by the client, the second operation instruction includes a second dimension field selected in the first dimension field.

[0035] In one embodiment, the above-mentioned server determines the first dimension field in the dimension field corresponding to the second indicator field and sends it to the client. The client can display the first dimension field sent by the server and send the second dimension field selected in the first dimension field to the server, that is, the client can generate a second operation instruction based on the second dimension field and send it to the server. The server receives the second operation instruction, that is, the server can receive the second dimension field selected in the first dimension field sent by the client.

[0036] During specific implementation, the server may send the first dimension field determined in the dimension field corresponding to the second indicator field to the client. After that, the client may display the first dimension field sent by the server, generate a second operation instruction based on the second dimension field selected in the first dimension field, and send it to the server. The server may receive the second operation instruction, or the client may display the first dimension field sent by the server, and send the second dimension field selected in the first dimension field to the server. The server may receive the second dimension field selected in the first dimension field sent by the client.

[0037] For example, the second indicator field is sales_amount and sell_total, the dimension fields corresponding to sales_amount are dt, product_type, and batch_no, and the dimension fields corresponding to sell_total are dt, product_type, and batch_no. In the dimension field corresponding to the second indicator field, the first dimension field is determined to be dt, product_type, and batch_no. The server sends the first dimension field to the client, the client displays the first dimension field sent by the server, and sends the second dimension fields dt and product_type selected in the first dimension field to the server. The server receives the second dimension fields dt and product_type selected in the first dimension field sent by the client.

[0038] Step S106: read the first dimension value that meets the filtering rule from the dimension value of the second dimension field, and read the first index value that meets the filtering rule from the index value of the second index field.

[0039] In this step, the server reads the first dimension value in the dimension value of the second dimension field and reads the first indicator value in the indicator value of the second indicator field according to the filtering rules; optionally, the filtering rules are constructed based on the fields selected in the indicator table and the dimension table and the input field values; the filtering rules can be configured for the second indicator field.

[0040] The indicator value of the second indicator field in this embodiment refers to the value of the second indicator field. For example, if the second indicator field is sales_amount and sell_total, the indicator value of the second indicator field is the specific numerical value of the second indicator field. The indicator value of each indicator field contained in the second indicator field can be one or more. The dimension value of the second dimension field refers to the value of the second dimension field. For example, if the second dimension field is dt and product_type, the dimension value of the second dimension field is the specific numerical value of the second dimension field. The dimension value of each dimension field contained in the second dimension field can be one or more.

[0041] The indicator value of the second indicator field can be stored in a data table, and the dimension value of the second dimension field can also be stored in a data table. The indicator value of the second indicator field and the dimension value of the second dimension field can be stored in the same data table or in different data tables. There can be one or more data tables storing the indicator value of the second indicator field, and there can also be one or more data tables storing the dimension value of the second dimension field. The data table can be a MySQL (Structured Query Language) data table.

[0042] In this embodiment, the fields selected in the indicator table and the dimension table may include the third indicator field selected in the indicator table and / or the third dimension field selected in the indicator table and / or the dimension table; the input field value may include the indicator value of the third indicator field and / or the dimension value of the third dimension field; the indicator table and dimension table here may be the same as the indicator table and dimension table used for selecting the first indicator field and the dimension field corresponding to the first indicator field.

[0043] Specifically, in the process of constructing filtering rules, the client can obtain the fields selected in the indicator table and / or dimension table and the input field values ​​for the second indicator field, build filtering rules based on the fields and field values, and send them to the server. The server can store the filtering rules corresponding to the second indicator field.

[0044] Specifically, in the process of constructing filtering rules based on fields and field values, the fields and field values ​​can be connected based on the input connection relationship to obtain the filtering rules, that is, the connection relationship can be input by the user; wherein, the connection relationship includes greater than, less than, equal to, greater than or equal to, less than or equal to, empty, not empty and / or regular expressions. In addition, the connection relationship can also include other types of connection relationships.

[0045] For example, for the second indicator field sales_amount, the selected fields are the third dimension field dt and the third indicator field sell_total. The connection relationship entered for the third dimension field is greater than or equal to. Based on the connection relationship, the third dimension field dt is connected with the input dimension value t2 / t2 / t2, and the first filtering rule for the second indicator field sales_amount is dt>=2t2 / t2 / t2. The connection relationship entered for the third indicator field is greater than or equal to. Based on the connection relationship, the third indicator field sell_total is connected with the input indicator value 1 million, and the second filtering rule for sales_amount is sell_total>=1 million. The filtering rule for the first indicator field sales_amount is generated based on the first filtering rule and the second filtering rule. Specifically, the first filtering rule and the second filtering rule can be connected according to the preset connection relationship to obtain the filtering rule. The preset connection relationship can be and or or. For the first indicator field sell_total, the constructed filtering rule is dt>=t3 / t3 / t3.

[0046] It should be noted that the filtering rules can be constructed for the second indicator field, that is, each indicator field in the second indicator field may have its own filtering rules, and the filtering rules corresponding to each indicator field in the second indicator field may be one or more. The fields and field values ​​used in the process of constructing the filtering rules for each indicator field in the second indicator field may be the same or different, and the constructed filtering rules may be the same or different.

[0047] During the specific execution process, the indicator value of the second indicator field stored in the data table and the dimension value of the second dimension field stored in the data table can be obtained from the data channel. However, in actual application scenarios, the fields of the data channel may be inconsistent with the fields stored on the server. Therefore, in order to accurately store the indicator value and dimension value obtained from the data channel, a mapping relationship between the first field and the second field can be preset; the first field here can be an indicator field or a dimension field. When the first field is an indicator field, the second field is an indicator field. When the first field is a dimension field, the second field is a dimension field. The first field can be a field stored in the data channel, and the second field can be a field stored in the data table on the server; for example, the mapping relationship between the first field and the second field is shown in Table 4 below:

[0048] Table 4 Among them, the first field is the field of the data channel, the second field is the field stored on the server, the first field dt (time) is mapped to the second field dt (time), the first field id (product code) is mapped to the second field id (product code), the first field name (product name) is mapped to the second field name (product name), the first field price (unit price) is mapped to the second field unitPrice (unit price), the first field salesVolume (sales) is mapped to the second field sales_amount (sales); the field type of the first field dt is Integer (integer type), the field type of the second field dt is Integer (integer type), and the field type of the second field dt is Integer (integer type). teger (integer type), the field type of the first field id is Long (long integer type), the field type of the second field id is Long (long integer type), the field type of the first field name is String (string type), the field type of the second field name is String (string type), the field type of the first field price is String (string type), the field type of the second field unitPrice is Decimal (decimal number type), the field type of the first field salesVolume is String (string type), and the field type of the second field sales_amount is Decimal (decimal number type).

[0049] During specific implementation, the server can obtain the field value of the first field from the data channel, that is, obtain the field value of the indicator field and / or the dimension value of the dimension field from the data channel, and write the field value of the first field into the second field mapped to the first field in the data table; the first field can be an indicator field and / or a dimension field.

[0050] The data channel may include a first data channel and / or a second data channel. The first data channel may be Hive (Hadoop Interactive Visualization and Exploration, a data warehouse tool), and the second data channel may be Kafka (Apache Kafka, a distributed messaging system); if the data channel is the first data channel, then after detecting that the first data channel calls the data acquisition reminder passed in through the interface and the data acquisition cycle task arrives, the field value of the first field is obtained from the first data channel; if the data channel is the second data channel, then after detecting that the subscription cycle arrives, the field value of the first field is obtained from the second data channel; the subscription cycle may be the subscription cycle set by the server for data subscription in the second data channel.

[0051] In the above process of writing the field value of the first field into the second field mapped to the first field in the data table, the writing can be performed in an update mode and / or a detailed mode. Specifically, the field value of the first field can be added to the second field mapped to the first field in the data table, or the dimension value that is the same as the dimension value of the dimension field in the first field can be queried in the data table, and the indicator value corresponding to the dimension value in the queried data table in the data table can be replaced with the indicator value of the indicator field in the first field. Whether to use the update mode or the detailed mode can be determined based on the business scenario, so as to achieve flexibility in data storage.

[0052] In this embodiment, the first indicator value that meets the filtering rules is read in the indicator value of the second indicator field, and the first dimension value that meets the filtering rules is read in the dimension value of the second dimension field; specifically, the first indicator value that meets the filtering rules can be read in the indicator value of the second indicator field stored in the data table, and the first dimension value that meets the filtering rules can be read in the dimension value of the second dimension field stored in the data table.

[0053] Continuing with the above example, the first filtering rule for the second indicator field sales_amount is dt>=t2yeart2montht2day, and the second filtering rule is sell_total>=1 million. The filtering rule constructed based on the first and second filtering rules is dt>=t2yeart2montht2day and sell_total>=1 million. The filtering rule for the second indicator field sell_total is dt>=t3yeart3montht3day. Assume that the first indicator value that meets the sales_amount filtering rule is read from the indicator value of the second indicator field sales_amount stored in the data table, as shown in Table 5 below:

[0054] Table 5 Among them, 1 million yuan, 900,000 yuan, and 850,000 yuan represent the first indicator value of the second indicator field sales_amount (sales); Assume that the first dimension value that satisfies the sales_amount filtering rule is read from the second dimension field dt and the dimension value of product_type stored in the data table, as shown in Table 6 below:

[0055] Table 6 Among them, t2, t2, t2, and t3 represent the first dimension values ​​of the second dimension field dt (time), and e2, e2, and e1 represent the first dimension values ​​of the second dimension field product_type (product type), that is, they represent different product types. The first indicator value that satisfies the filtering rule of sell_total is read from the indicator value of the second indicator field sell_total stored in the data table, which is 6 million. 6 million represents the first indicator value of the second indicator field sell_total (sales volume); the first dimension value that satisfies the filtering rule of sell_total is read from the dimension values ​​of the second dimension fields dt and product_type stored in the data table, which is "t3 year t3 month t3 day, e1". Among them, t3 year t3 month t3 day represents the first dimension value of the second dimension field dt (time); e1 represents the first dimension value of the second dimension field product_type (product type).

[0056] Step S108: performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result.

[0057] The above-mentioned first dimension value that meets the filtering rules is read from the dimension value of the second dimension field, and the first indicator value that meets the filtering rules is read from the indicator value of the second indicator field. In this step, the read first indicator value and first dimension value are aggregated and calculated to obtain the aggregated calculation result. Based on the above-mentioned flexible selection of the second indicator field and the second dimension field, and the flexible construction of the filtering rules, and the reading of the first indicator value and the first dimension value according to the filtering rules, the aggregation calculation can be conveniently performed, and diversified aggregation calculations can be performed according to the user's own needs, thereby improving the flexibility and convenience of the aggregation calculation.

[0058] The aggregation calculation in this embodiment may include sum (summation), count, min (minimum), max (maximum), and / or avg (average). That is, the aggregation calculation may include sum calculation, count calculation, minimum value calculation, maximum value calculation, and / or average value calculation. In addition, the aggregation calculation may also include other types of aggregation calculations, such as calculating the number of rows and / or deduplication.

[0059] In specific implementation, if the number of fields of the second indicator field is 1, the first indicator value and the first dimension value can be merged to obtain the first merged data, and the indicator value contained in the first merged data can be grouped according to the dimension value contained in the first merged data to obtain the second indicator value grouping result, and the indicator value in the second indicator value grouping result is input into the aggregation algorithm of the second indicator field to aggregate the indicator value to obtain the second aggregated indicator value, and the second aggregated indicator value and the dimension value corresponding to the second indicator value grouping result are combined to obtain the aggregated calculation result; in the process of grouping the indicator values ​​contained in the first merged data according to the dimension value contained in the first merged data to obtain the second indicator value grouping result, the indicator value corresponding to the same dimension value in the first merged data can be used as the second indicator value grouping result, that is, the indicator value corresponding to the same dimension value in the first merged data can be divided into the same indicator value group.

[0060] Among them, the aggregation algorithm of the second indicator field can be pre-set, that is, it can be an aggregation algorithm set for the second indicator field; the aggregation algorithm of the second indicator field may include a sum algorithm, a counting algorithm, a minimum value calculation algorithm, a maximum value calculation algorithm, an average value calculation algorithm, a row number calculation algorithm and / or a deduplication algorithm. In addition, the aggregation algorithm corresponding to the second indicator field may also include other aggregation algorithms; the aggregation calculation result in this embodiment refers to the result obtained after the aggregation calculation.

[0061] Continuing with the above example, the second indicator field is sales_amount, the first indicator value and the first dimension value are shown in Table 5 and Table 6, and the number of fields in the second indicator field is 1. Then, the first indicator value and the first dimension value are merged to obtain the first merged data shown in Table 7 below:

[0062] Table 7 Assuming that the aggregation algorithm for the second indicator field sales_amount is sum, that is, the addition algorithm, the indicator values ​​contained in the first merged data are grouped according to the dimension values ​​contained in the first merged data, and the second indicator value grouping results are "1 million yuan, 900,000 yuan" and "850,000 yuan". That is, "1 million yuan and 900,000 yuan" belong to the same indicator value group, and "850,000 yuan" belongs to the same indicator value group. The indicator values ​​"1 million yuan, 900,000 yuan" and "850,000 yuan" in the second indicator value grouping result are input into the aggregation algorithm sum of the second indicator field in sequence to perform indicator table aggregation, resulting in second aggregate indicator values ​​of 1.9 million yuan and 850,000 yuan. The second aggregate indicator value of 1.9 million yuan is combined with the corresponding dimension value "t2 year t2 month t2 day, e2", and the second aggregate indicator value of 850,000 yuan is combined with the corresponding dimension value "t3 year t3 month t3 day, e1". The aggregation calculation results are shown in Table 8 below:

[0063] Table 8 In actual applications, when the first indicator value and the first dimension value are merged to obtain the first merged data, there may be second merged data. During the aggregation calculation process, the user may also need to perform aggregation calculations on the first merged data and the second merged data at the same time. In order to meet the diverse needs of aggregation calculations and improve the comprehensiveness of aggregation calculations, in an optional implementation provided by this embodiment, during the process of performing aggregation calculations on the first indicator value and the first dimension value to obtain the aggregation calculation result, the following operations are performed: Merging the first indicator value and the first dimension value to obtain first merged data; Performing data assembly on the first combined data and the second combined data to obtain assembled combined data; Perform aggregation calculation on the assembled and merged data to obtain the aggregation calculation result.

[0064] The second merged data may be merged data obtained by merging the first indicator value of the third indicator field and the first indicator value of the third dimension field; the second merged data will not be described in detail here.

[0065] On this basis, since only the indicator values ​​corresponding to the same dimension value can be aggregated and calculated, in order to improve the pertinence of subsequent aggregation calculations and reduce the interference of irrelevant data; in an optional implementation provided by this embodiment, in the process of assembling the first merged data and the second merged data to obtain the assembled merged data, the following operations are performed: In the second merged data, searching for matching data and non-matching data of the first merged data; Replace the non-matching data according to the preset characters to obtain the replacement result; The first merged data, the matching data and the replacement processing result are assembled to obtain assembled merged data.

[0066] In which, the matching data may include data in the second merged data that matches the first merged data, and the non-matching data may include data in the second merged data that does not match the first merged data; the preset character may be a pre-set character, such as NULL; the first merged data may be a first merged table, the second merged data may be a second merged table, and the assembled merged data may be an assembled merged table.

[0067] For example, the first merged data is shown in Table 7 above, and the second merged data is shown in Table 9 below:

[0068] Table 9 Among them, t3 represents the dimension value of dt (time), e1 represents the dimension value of product_type (product type), and 6 million represents the indicator value of sell_total (sales volume); In the second merged data, the matching data and non-matching data of the first merged data are searched, and the non-matching data are replaced according to the preset character NULL to obtain the replacement processing result. The first merged data, the matching data and the replacement processing result are assembled to obtain the assembled merged data shown in Table 10 below:

[0069] Table 10 Based on the above-mentioned aggregation calculation on the assembled and merged data to obtain the aggregation calculation result, in an optional implementation provided by this embodiment, in the process of performing the aggregation calculation on the assembled and merged data to obtain the aggregation calculation result, the following operations are performed: Grouping the index values ​​contained in the assembled and merged data according to the dimension values ​​contained in the assembled and merged data to obtain the index value grouping results; Input the indicator values ​​in the indicator value grouping result into the preset aggregation algorithm to aggregate the indicator values ​​to obtain the aggregated indicator value; The aggregated indicator value and the dimension value corresponding to the indicator value grouping result are combined to obtain the aggregate calculation result.

[0070] Among them, the preset aggregation algorithm can be the aggregation algorithm of the second indicator field to which the indicator value in the indicator value grouping result belongs, that is, the aggregation algorithm pre-set for the second indicator field. The preset aggregation algorithm may include a sum algorithm, a counting algorithm, a minimum value calculation algorithm, a maximum value calculation algorithm, an average value calculation algorithm, a row number calculation algorithm and / or a deduplication algorithm. In addition, the preset aggregation algorithm may also include other types of aggregation algorithms.

[0071] Continuing with the above example, assume that the aggregation algorithm for sales_amount is sum, which is a summation algorithm, and the aggregation algorithm for sell_total is max, which is a maximum calculation algorithm. In other words, the preset aggregation algorithms include the sum aggregation algorithm for sales_amount and the max aggregation algorithm for sell_total. The indicator values ​​contained in the assembled and merged data are grouped according to the dimension values ​​contained in the assembled and merged data, and the indicator value grouping results obtained are "1 million yuan, 900,000 yuan", "850,000 yuan", and "6 million yuan". The indicator values ​​"1 million yuan, 900,000 yuan", "850,000 yuan", and "6 million yuan" in the indicator value grouping results are input into the corresponding preset aggregation algorithms in sequence for indicator value aggregation to obtain the aggregated indicator value. The aggregated indicator value and the dimension values ​​corresponding to the indicator value grouping results are combined to obtain the aggregation calculation results shown in Table 11 below:

[0072] Table 11 In addition, in the process of assembling the first merged data and the second merged data to obtain the assembled merged data, the first merged table and the second merged table can also be connected, and the connection result can be used as the assembled merged data; the connection processing here can be a left connection processing, a right connection processing or an inner connection processing, for example, the left connection processing is left join, the right connection processing is right join, and the inner connection processing is inner join.

[0073] For example, metric_table1 represents the first merged table, and metric_table2 represents the second merged table. To perform a left join, right join, or inner join on the first and second merged tables, use the following method: select metric_table1, metric_table2 from metric_table1 left / right / inner join metric_table2 on metric_table1.(primary key)=metric_table2.(association id) as metric_table.

[0074] Step S110: Generate a first aggregate table based on the aggregate calculation result, the second indicator field, and the second dimension field.

[0075] The above-mentioned aggregation calculation is performed on the first indicator value and the first dimension value to obtain the aggregation calculation result. In this step, the first aggregation table is generated according to the aggregation calculation result, the second indicator field and the second dimension field obtained by the aggregation calculation.

[0076] During specific implementation, the server can generate a first aggregate table based on the aggregation calculation results, the second indicator field and the second dimension field, and send the first aggregate table to the client so that the first aggregate table can be displayed on the client, that is, the client can display the first aggregate table.

[0077] Using the above example, the aggregation calculation results are shown in Table 8. Based on the aggregation calculation results, the second indicator field, and the second dimension field, the first aggregation table is generated as shown in Table 12 below:

[0078] Table 12 During the specific execution process, the server may send the first aggregate table to the client, and the client may display the first aggregate table; for example, display the first aggregate table by selecting tmp.* from tmp; at the same time, the server may also store the first aggregate table, for example, by performing insert into select tmp.* from tmp.

[0079] In actual applications, in addition to the first aggregation table, there may be a fourth aggregation table. The fourth aggregation table may be an aggregation table stored historically. The fourth aggregation table may contain the same dimension field as the first aggregation table. In this case, the first aggregation table and the fourth aggregation table can be merged to obtain a merged aggregation table. Specifically, if the first aggregation table and the fourth aggregation table contain the same dimension field, the first aggregation table and the fourth aggregation table can be merged according to the dimension field to obtain a merged aggregation table. For example, insert into select (select * from bus_model1 where dt=20240810union all select * from bus_model2 where dt = 20240810) as tmp can be used for merging to obtain a merged aggregation table.

[0080] For example, the first aggregate table is generated for the second indicator field (sales_amount and sell_total), and the first aggregate table contains the dimension field dt. The fourth aggregate table is generated for other indicator fields (profit) different from the second indicator field, and the fourth aggregate table contains the dimension field dt. Then, the indicator value of the dimension field dt with the preset dimension value t2 on t2, year t2, month t2 is extracted from the first aggregate table, and the indicator value of the dimension field dt with the preset dimension value t2 on t2, year t2, month t2 is extracted from the fourth aggregate table. The indicator values ​​extracted from the first aggregate table and the fourth aggregate table are merged to obtain a merged aggregate table.

[0081] During specific implementation, in order to improve the reusability and availability of the first aggregation table, the server may store the generated first aggregation table. In the process of storing the first aggregation table, in order to reduce the amount of storage data of the storage server and thereby ensure the stability of the server's subsequent analysis and processing using the first aggregation table, a server cluster may be introduced so that the first aggregation table generated each time is evenly distributed on different servers. In this embodiment, the server may also perform the following operations: Perform hash calculation on the dimension fields corresponding to the dimension values ​​contained in the first aggregate table to obtain a hash value; The server identifier is calculated according to the hash value and the number of servers in the server cluster, and the first aggregation table is stored in the server corresponding to the server identifier.

[0082] The number of servers in the server cluster refers to the number of servers included in the server cluster; the server identifier may be an identifier that uniquely represents the server; and the dimension field corresponding to the dimension value included in the first aggregation table may be the dimension field corresponding to all dimension values ​​included in the first aggregation table.

[0083] Specifically, in the process of calculating the server identification based on the hash value and the number of servers in the server cluster, the hash value can be converted, and the number of servers can be modulo the conversion result to obtain the remainder as the server identification; wherein, the conversion process may include performing integer conversion on the hash value to obtain an integer as the conversion result.

[0084] In addition, in order to reduce the repetition rate of the server identification obtained each time, it is ensured that the first aggregate table generated each time can be evenly distributed among the servers in the server cluster; the dimension field to be processed can also be determined in the dimension field corresponding to the dimension value contained in the first aggregate table, and a hash calculation is performed on the dimension field to be processed to obtain a hash value. The server identification is calculated based on the hash value and the number of servers in the server cluster, and the first aggregate table is stored in the server corresponding to the server identification; wherein, the dimension field to be processed can be the dimension field with the least repetition of dimension values ​​in the dimension field.

[0085] In actual applications, the first aggregation table generated for the same data graph may be the latest, and there is a need to overwrite the historical aggregation table. To avoid temporary data unavailability caused by frequent deletion and rewriting when replacing the historical aggregation table with the first aggregation table, and to improve the smoothness of data storage, the server can also perform the following operations: Matching the timestamp of the first aggregate table with the partition identifier of each data partition of the server to be stored in the first aggregate table to obtain a first data partition; optionally, the timestamp is determined based on the generation time of the first aggregate table; A second data partition is created, the first aggregate table is stored in the second data partition, and the first data partition is deleted after the storage is completed, and the timestamp is used as the partition identifier of the stored second data partition.

[0086] Among them, the timestamp can be a timestamp in months, days, hours or minutes, such as the timestamp of xx / xx / xx, or a timestamp in other time units; the server to be stored in the first aggregate table can be a server waiting to store the first aggregate table.

[0087] Specifically, the timestamp of the first aggregate table can be generated based on the generation time of the first aggregate table, and the timestamp of the first aggregate table can be matched with the partition identifier of each data partition of the server to be stored of the first aggregate table. If the matching result is not empty, a second data partition is created on the server to be stored, and the first aggregate table is stored in the second data partition. After the storage is completed, the matched first data partition is deleted, and the timestamp of the first aggregate table is used as the partition identifier of the second data partition after storage; if the matching result is empty, a third data partition is created on the server to be stored, and the first aggregate table is stored in the third data partition, and the timestamp of the first aggregate table is used as the partition identifier of the third data partition after storage.

[0088] In addition, after the first aggregate table is stored in the second data partition as mentioned above, the matched first data partition and the stored second data partition can also be atomically replaced, the first data partition can be deleted after the atomic replacement, and the timestamp can be used as the partition identifier of the stored second data partition.

[0089] It should be added that the partition identifier of the above-mentioned data partition, that is, the timestamp of the data partition, can be stored in redis, and an expiration time can be set for the timestamp. After the expiration time of the timestamp is reached, the data stored in the data partition corresponding to the timestamp can be deleted. By clearing data that may be used less frequently, excessive data storage can be avoided.

[0090] In actual application scenarios, the indicator fields and filtering rules that require aggregation calculations may change continuously. To ensure that the aggregation table remains accessible during the change process of indicator fields and filtering rules, avoid data loss during the change process, and achieve a smooth transition of changes, in an optional implementation provided by this embodiment, the server may also perform the following operations: In response to the aggregation change reminder instruction, performing aggregation processing according to the aggregation change information carried in the aggregation change reminder instruction to obtain a third aggregation table; The data recorded in the first aggregate table is written into the third aggregate table, and the third aggregate table is marked based on the table identifier of the first aggregate table, and the first aggregate table is deleted.

[0091] Among them, the aggregation change reminder instruction can be a reminder instruction for changing the aggregation calculation related data, the aggregation change reminder instruction can be an aggregation change transaction, the aggregation change reminder instruction can carry aggregation change information, the aggregation change information can include the changed second indicator field, the changed second dimension field, the changed filtering rules and / or the changed aggregation algorithm. In addition, the aggregation change information can also include other change information. Aggregation processing based on the aggregation change information may include selecting the second indicator field, selecting the second dimension field, reading the first indicator value, reading the first dimension value, performing aggregation calculation and / or generating a third aggregation table, which is similar to the above-mentioned related content and will not be repeated here. The optional implementation method here can also be performed on the basis of any of the optional implementation methods provided above.

[0092] Specifically, the client can create an aggregation change transaction after detecting an aggregation change instruction. After detecting that the change is successful, the client will send the aggregation change transaction carrying the aggregation change information to the server. The server can perform aggregation processing based on the aggregation change information carried by the aggregation change transaction to obtain a third aggregation table, write the data recorded in the first aggregation table into the third aggregation table, mark the third aggregation table based on the table identifier of the first aggregation table, and delete the first aggregation table; the aggregation change transaction here can be a redis change transaction.

[0093] For example, the client submits a redis change transaction to the server, adds deduplication to the redis change transaction, and only takes the latest aggregate change information, that is, the redis change transaction carries the latest aggregate change information, and performs aggregation processing based on the aggregate change information to obtain a third aggregate table, writes the data recorded in the first aggregate table into the third aggregate table, and marks the third aggregate table based on the table identifier of the first aggregate table, and deletes the first aggregate table.

[0094] In actual applications, the second indicator field may continuously generate new indicator values, and the second dimension field may also continuously generate new dimension values. The new indicator values ​​generated by the second indicator field and the new dimension values ​​generated by the second dimension field may have been aggregated in the first aggregation table. If the aggregation calculation is performed again, errors may be caused. In order to avoid calculation errors, ensure the accuracy of the aggregation calculation results in the first aggregation table, and ensure that the aggregated calculation data is not repeated, in an optional implementation manner provided by this embodiment, the server further performs the following operations: Read the second indicator value of the second indicator field and the second dimension value of the second dimension field that meet the filtering rule, and calculate the version identifier according to the second indicator value and the second dimension value; If the version identifier is inconsistent with the version identifiers corresponding to the first index value and the first dimension value, an aggregation calculation is performed in the first aggregation table based on the second index value and the second dimension value to obtain a second aggregation table.

[0095] The version identifier can be a hash string.

[0096] Specifically, the second indicator value of the second indicator field and the second dimension value of the second dimension field that meet the filtering rules can be read according to a preset time period, and the version identifier can be calculated based on the second indicator value and the second dimension value. If the version identifier is inconsistent with the version identifier corresponding to the first indicator value and the first dimension value, then an aggregation calculation is performed in the first aggregation table based on the second indicator value and the second dimension value to obtain a second aggregation table. If the version identifier is consistent with the version identifier corresponding to the first indicator value and the first dimension value, no processing is required. The process of performing aggregation calculation in the first aggregation table based on the second indicator value and the second dimension value here can be based on the second indicator value, the second dimension value, the indicator value of the second indicator field in the first aggregation table, and the dimension value of the second dimension field in the first aggregation table to obtain a second aggregation table. The process of aggregation calculation here is similar to the above-mentioned process of aggregation calculation of the first indicator value and the first dimension value, and this embodiment will not be repeated here.

[0097] For example, Figure 2 As shown, the field value of the first field is obtained from any one or more data channels of Hive and Kafka, the field value of the first field is written into the second field mapped to the first field in the data table, and then a data map is constructed based on the indicator table set and the dimension table set to form a data map list. The server obtains the data map selected in the data map list, obtains the first indicator field selected from the indicator field contained in the indicator table in the data map, and obtains the dimension field corresponding to the first indicator field selected in the indicator table and / or the dimension table in the data map; it should be noted that the above Figure 2 These contents in the program may be pre-completed or not, and are executed continuously with the subsequent contents; After that, the server can receive the second indicator field selected in the first indicator field sent by the client, and receive the second dimension field selected in the first dimension field determined in the dimension field corresponding to the second indicator field, read the first indicator value that meets the filtering rules in the indicator value of the second indicator field, read the first dimension value that meets the filtering rules in the dimension value of the second dimension field, perform aggregation calculation on the first indicator value and the first dimension value to obtain the aggregation calculation result, generate a first aggregation table based on the aggregation calculation result, the second indicator field and the second dimension field, display the first aggregation table in the client, and store the first aggregation table through the server cluster.

[0098] In summary, the data processing method provided in this embodiment responds to a first operation instruction, the first operation instruction including selecting a second indicator field in a first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; responds to a second operation instruction, the second operation instruction including selecting a second dimension field in the first dimension field; reading a first dimension value that satisfies a filtering rule from a dimension value of the second dimension field, and reading a first indicator value that satisfies a filtering rule from an indicator value of the second indicator field; Then, the first indicator value and the first dimension value are merged to obtain first merged data, the first merged data and the second merged data are assembled to obtain assembled merged data, aggregation calculation is performed on the assembled merged data to obtain the aggregation calculation result, and a first aggregation table is generated based on the aggregation calculation result, the second indicator field and the second dimension field, and the first aggregation table is displayed in the client; in this way, by flexibly selecting the second indicator field, flexibly selecting the second dimension field, and flexibly constructing filtering rules, the flexibility of the aggregation calculation is improved, the diversified needs of the aggregation calculation are met, the comprehensiveness of the aggregation calculation is improved, and the visualization and intuitiveness of the aggregation calculation are improved through the first aggregation table.

[0099] The following uses the application of a data processing method provided by this embodiment in a table scenario as an example to further illustrate the data processing method provided by this embodiment. Figure 3 ,The data processing method applied to the table scenario specifically includes the following steps.

[0100] Step S302, in response to a first operation instruction sent by the client, the first operation instruction includes a second indicator field selected in the first indicator field, determining a first dimension field in a dimension field corresponding to the second indicator field and sending the first dimension field to the client.

[0101] Step S304 , responding to a second operation instruction sent by the client, where the second operation instruction includes a second dimension field selected from the first dimension field.

[0102] Step S306: read the first dimension value that meets the filtering rule from the dimension value of the second dimension field, and read the first index value that meets the filtering rule from the index value of the second index field.

[0103] Optionally, filtering rules are constructed based on the fields selected in the indicator table and dimension table and the entered field values.

[0104] Step S308: merge the first indicator value and the first dimension value to obtain first merged data, and assemble the first merged data and the second merged data to obtain assembled merged data.

[0105] Step S310: perform aggregation calculation on the assembled merged data to obtain the aggregation calculation result, generate a first aggregation table based on the aggregation calculation result, the second indicator field and the second dimension field, and send the first aggregation table to the client to display the first aggregation table on the client.

[0106] Step S312: read the second indicator value of the second indicator field and the second dimension value of the second dimension field that meet the filtering rule, and calculate the version identifier based on the second indicator value and the second dimension value.

[0107] Step S314: If the version identifier is inconsistent with the version identifiers corresponding to the first index value and the first dimension value, an aggregation calculation is performed in the first aggregation table based on the second index value and the second dimension value to obtain a second aggregation table.

[0108] The server can store the second aggregation table.

[0109] It should be noted that any one of steps S302 to step S314 or any combination of multiple steps can be combined with any one of steps S102 to step S110 to form a new implementation method according to the needs of implementation deployment; in addition, according to the needs of actual deployment, any one or multiple technical features can be selected from steps S302 to step S314 and combined with any one or multiple technical features provided by steps S102 to step S110 to form a new implementation method; or, any one or multiple technical features in steps S302 to step S314 can also be replaced by any one or multiple technical features provided by steps S102 to step S110 to form a new implementation method according to the needs of actual deployment, which will not be repeated here.

[0110] An embodiment of a data processing device provided in this specification is as follows: In the above embodiment, a data processing method is provided, and correspondingly, a data processing device is also provided, which will be described below with reference to the accompanying drawings.

[0111] Reference Figure 4 , which shows a schematic diagram of a data processing device provided by this embodiment.

[0112] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0113] This embodiment provides a data processing device, comprising: A field determination module 402 is configured to, in response to a first operation instruction, wherein the first operation instruction includes selecting a second indicator field in the first indicator field, determine a first dimension field in a dimension field corresponding to the second indicator field; An instruction response module 404 is configured to respond to a second operation instruction, where the second operation instruction includes a second dimension field selected from the first dimension field; A reading module 406 is configured to read a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and to read a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected from the index table and the dimension table and the input field values; An aggregation calculation module 408 is configured to perform aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; The aggregation table generation module 410 is used to generate a first aggregation table based on the aggregation calculation result, the second indicator field and the second dimension field.

[0114] In one embodiment, performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregate calculation result is implemented in the following manner: Merging the first indicator value and the first dimension value to obtain first merged data; Performing data assembly on the first combined data and the second combined data to obtain assembled combined data; Aggregate calculation is performed on the assembled and merged data to obtain the aggregate calculation result.

[0115] In one embodiment, the data assembly of the first combined data and the second combined data to obtain assembled combined data is implemented in the following manner: In the second merged data, searching for matching data and non-matching data of the first merged data; Performing a replacement process on the non-matching data according to a preset character to obtain a replacement process result; The first merged data, the matching data and the replacement processing result are assembled to obtain the assembled merged data.

[0116] In one embodiment, performing aggregation calculation on the assembled and merged data to obtain the aggregation calculation result is achieved in the following manner: Grouping the index values ​​contained in the assembled and merged data according to the dimension values ​​contained in the assembled and merged data to obtain an index value grouping result; Inputting the indicator values ​​in the indicator value grouping result into a preset aggregation algorithm to perform indicator value aggregation to obtain an aggregated indicator value; The aggregate index value and the dimension value corresponding to the index value grouping result are combined to obtain the aggregate calculation result.

[0117] In one embodiment, determining the first dimension field in the dimension field corresponding to the second indicator field is implemented in the following manner: If the number of fields of the second indicator field is equal to 1, the dimension field corresponding to the second indicator field is used as the first dimension field; If the number of fields is greater than 1, the same dimension field is filtered out from the dimension fields corresponding to the multiple indicator fields included in the second indicator field as the first dimension field.

[0118] In one embodiment, the apparatus further performs the following operations: Reading the second indicator value of the second indicator field and the second dimension value of the second dimension field that satisfy the filtering rule, and calculating a version identifier based on the second indicator value and the second dimension value; If the version identifier is inconsistent with the version identifiers corresponding to the first index value and the first dimension value, an aggregation calculation is performed in the first aggregation table based on the second index value and the second dimension value to obtain a second aggregation table.

[0119] In one embodiment, the apparatus further performs the following operations: In response to the aggregation change reminder instruction, performing aggregation processing according to the aggregation change information carried in the aggregation change reminder instruction to obtain a third aggregation table; The data recorded in the first aggregate table is written into the third aggregate table, and the third aggregate table is marked based on the table identifier of the first aggregate table, and the first aggregate table is deleted.

[0120] In the data processing device provided in this embodiment, in response to the first operation instruction, the first operation instruction includes selecting the second indicator field in the first indicator field, and determining the first dimension field in the dimension field corresponding to the second indicator field; in response to the second operation instruction, the second operation instruction includes selecting the second dimension field in the first dimension field. Secondly, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field. By selecting fields in the indicator table and the dimension table and inputting field values ​​to construct the filtering rule, the flexibility of constructing the filtering rule is improved. On this basis, the first indicator value and the first dimension value are aggregated to obtain the aggregation calculation result, and the first aggregation table is generated by combining the aggregation calculation result, the second indicator field and the second dimension field. In this way, by flexibly selecting the second indicator field and the flexibly selecting the second dimension field, and then combining the filtering rule to filter the first indicator value of the second indicator field and the first dimension value of the second dimension field, the flexibility of the aggregation calculation is improved, and the diverse needs of users for aggregation calculation are met. The aggregation calculation result is displayed in a visual way through the first aggregation table, and the intuitiveness of the aggregation calculation is improved.

[0121] An embodiment of a computer device provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application further provides a computer device, which is used to execute the data processing method provided above. Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0122] This embodiment provides a computer device, including: like Figure 5 As shown, computer devices can vary significantly depending on their configuration or performance. They may include one or more processors 501 and memory 502. Memory 502 may store one or more applications or data. Memory 502 may be either ephemeral or persistent. Applications stored in memory 502 may include one or more modules (not shown), each of which may include a series of computer-executable instructions within the computer device. Furthermore, processor 501 may be configured to communicate with memory 502 to execute the series of computer-executable instructions within memory 502 on the computer device. The computer device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, and the like.

[0123] In a specific embodiment, a computer device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the computer device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following: In response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; In response to a second operation instruction, the second operation instruction includes a second dimension field selected from the first dimension field; Reading a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and reading a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; Performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; A first aggregate table is generated based on the aggregate calculation result, the second indicator field, and the second dimension field.

[0124] In one embodiment, performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregate calculation result is implemented in the following manner: Merging the first indicator value and the first dimension value to obtain first merged data; Performing data assembly on the first combined data and the second combined data to obtain assembled combined data; Aggregate calculation is performed on the assembled and merged data to obtain the aggregate calculation result.

[0125] In one embodiment, the data assembly of the first combined data and the second combined data to obtain assembled combined data is implemented in the following manner: In the second merged data, searching for matching data and non-matching data of the first merged data; Performing a replacement process on the non-matching data according to a preset character to obtain a replacement process result; The first merged data, the matching data and the replacement processing result are assembled to obtain the assembled merged data.

[0126] In one embodiment, performing aggregation calculation on the assembled and merged data to obtain the aggregation calculation result is achieved in the following manner: Grouping the index values ​​contained in the assembled and merged data according to the dimension values ​​contained in the assembled and merged data to obtain an index value grouping result; Inputting the indicator values ​​in the indicator value grouping result into a preset aggregation algorithm to perform indicator value aggregation to obtain an aggregated indicator value; The aggregate index value and the dimension value corresponding to the index value grouping result are combined to obtain the aggregate calculation result.

[0127] In one embodiment, determining the first dimension field in the dimension field corresponding to the second indicator field is implemented in the following manner: If the number of fields of the second indicator field is equal to 1, the dimension field corresponding to the second indicator field is used as the first dimension field; If the number of fields is greater than 1, the same dimension field is filtered out from the dimension fields corresponding to the multiple indicator fields included in the second indicator field as the first dimension field.

[0128] In one embodiment, the computer device further performs the following operations: Reading the second indicator value of the second indicator field and the second dimension value of the second dimension field that satisfy the filtering rule, and calculating a version identifier based on the second indicator value and the second dimension value; If the version identifier is inconsistent with the version identifiers corresponding to the first index value and the first dimension value, an aggregation calculation is performed in the first aggregation table based on the second index value and the second dimension value to obtain a second aggregation table.

[0129] In one embodiment, the computer device further performs the following operations: In response to the aggregation change reminder instruction, performing aggregation processing according to the aggregation change information carried in the aggregation change reminder instruction to obtain a third aggregation table; The data recorded in the first aggregate table is written into the third aggregate table, and the third aggregate table is marked based on the table identifier of the first aggregate table, and the first aggregate table is deleted.

[0130] In the computer device provided in this embodiment, in response to the first operation instruction, the first operation instruction includes selecting the second indicator field in the first indicator field, and determining the first dimension field in the dimension field corresponding to the second indicator field; in response to the second operation instruction, the second operation instruction includes selecting the second dimension field in the first dimension field. Secondly, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field. By selecting fields in the indicator table and the dimension table and inputting field values ​​to construct the filtering rule, the flexibility of constructing the filtering rule is improved. On this basis, the first indicator value and the first dimension value are aggregated to obtain the aggregation calculation result, and the first aggregation table is generated by combining the aggregation calculation result, the second indicator field and the second dimension field. In this way, by flexibly selecting the second indicator field and the flexibly selecting the second dimension field, and then combining the filtering rule to filter the first indicator value of the second indicator field and the first dimension value of the second dimension field, the flexibility of the aggregation calculation is improved, and the diverse needs of users for aggregation calculation are met. The aggregation calculation result is displayed in a visual way through the first aggregation table, and the intuitiveness of the aggregation calculation is improved.

[0131] An embodiment of a computer-readable storage medium provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium.

[0132] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented: In response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; In response to a second operation instruction, the second operation instruction includes a second dimension field selected from the first dimension field; Reading a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and reading a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; Performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; A first aggregate table is generated based on the aggregate calculation result, the second indicator field, and the second dimension field.

[0133] In the computer-readable storage medium provided in this embodiment, in response to the first operation instruction, the first operation instruction includes selecting the second indicator field in the first indicator field, and determining the first dimension field in the dimension field corresponding to the second indicator field; in response to the second operation instruction, the second operation instruction includes selecting the second dimension field in the first dimension field. Secondly, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field. By selecting fields in the indicator table and the dimension table and inputting field values ​​to construct the filtering rule, the flexibility of constructing the filtering rule is improved. On this basis, the first indicator value and the first dimension value are aggregated to obtain the aggregation calculation result, and the first aggregation table is generated by combining the aggregation calculation result, the second indicator field and the second dimension field. In this way, by flexibly selecting the second indicator field and the flexibly selecting the second dimension field, and then combining the filtering rule to filter the first indicator value of the second indicator field and the first dimension value of the second dimension field, the flexibility of the aggregation calculation is improved, and the diverse needs of users for aggregation calculation are met. The aggregation calculation result is displayed in a visual way through the first aggregation table, and the intuitiveness of the aggregation calculation is improved.

[0134] It should be noted that the embodiment of a computer-readable storage medium in this specification and the embodiment of a data processing method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method mentioned above, and the repeated parts will not be repeated.

[0135] An embodiment of a computer program product provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer program product.

[0136] The computer program product provided in this embodiment includes a computer program. When the computer program is executed by a processor, the following process is implemented: In response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; In response to a second operation instruction, the second operation instruction includes a second dimension field selected from the first dimension field; Reading a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and reading a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; Performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; A first aggregate table is generated based on the aggregate calculation result, the second indicator field, and the second dimension field.

[0137] In the computer program product provided in this embodiment, in response to the first operation instruction, the first operation instruction includes selecting the second indicator field in the first indicator field, and determining the first dimension field in the dimension field corresponding to the second indicator field; in response to the second operation instruction, the second operation instruction includes selecting the second dimension field in the first dimension field. Secondly, according to the filtering rule, the first indicator value is read from the indicator value of the second indicator field, and the first dimension value is read from the dimension value of the second dimension field. By selecting fields in the indicator table and the dimension table and inputting field values ​​to construct the filtering rule, the flexibility of constructing the filtering rule is improved. On this basis, the first indicator value and the first dimension value are aggregated to obtain the aggregation calculation result, and the first aggregation table is generated by combining the aggregation calculation result, the second indicator field and the second dimension field. In this way, by flexibly selecting the second indicator field and the second dimension field, and then combining the filtering rule to filter the first indicator value of the second indicator field and the first dimension value of the second dimension field, the flexibility of the aggregation calculation is improved, and the diverse needs of users for aggregation calculation are met. The aggregation calculation result is displayed in a visual way through the first aggregation table, and the intuitiveness of the aggregation calculation is improved.

[0138] It should be noted that the embodiment of a computer program product in this specification and the embodiment of a data processing method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method mentioned above, and the repeated parts will not be repeated.

[0139] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable test equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable test equipment generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable test equipment to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable test device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process described in the flow. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0145] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0146] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0148] The embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0150] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. A data processing method, characterized in that: The method comprises: In response to a first operation instruction, the first operation instruction includes selecting a second indicator field in the first indicator field, and determining a first dimension field in a dimension field corresponding to the second indicator field; In response to a second operation instruction, the second operation instruction includes a second dimension field selected from the first dimension field; Reading a first dimension value that satisfies a filtering rule from the dimension value of the second dimension field, and reading a first index value that satisfies the filtering rule from the index value of the second index field; the filtering rule is constructed based on the fields selected in the index table and the dimension table and the input field values; Performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result; A first aggregate table is generated based on the aggregate calculation result, the second indicator field, and the second dimension field.

2. The method according to claim 1, characterized in that The performing aggregation calculation on the first indicator value and the first dimension value to obtain an aggregation calculation result includes: Merging the first indicator value and the first dimension value to obtain first merged data; Performing data assembly on the first combined data and the second combined data to obtain assembled combined data; Aggregate calculation is performed on the assembled and merged data to obtain the aggregate calculation result.

3. The method according to claim 2, characterized in that The step of assembling the first combined data and the second combined data to obtain assembled combined data includes: In the second merged data, searching for matching data and non-matching data of the first merged data; Performing a replacement process on the non-matching data according to a preset character to obtain a replacement process result; The first merged data, the matching data and the replacement processing result are assembled to obtain the assembled merged data.

4. The method according to claim 2, characterized in that The performing aggregation calculation on the assembled and merged data to obtain the aggregation calculation result includes: Grouping the index values ​​contained in the assembled and merged data according to the dimension values ​​contained in the assembled and merged data to obtain an index value grouping result; Inputting the indicator values ​​in the indicator value grouping result into a preset aggregation algorithm to perform indicator value aggregation to obtain an aggregated indicator value; The aggregate index value and the dimension value corresponding to the index value grouping result are combined to obtain the aggregate calculation result.

5. The method according to claim 1, wherein The determining the first dimension field in the dimension field corresponding to the second indicator field includes: If the number of fields of the second indicator field is equal to 1, the dimension field corresponding to the second indicator field is used as the first dimension field; If the number of fields is greater than 1, the same dimension field is filtered out from the dimension fields corresponding to the multiple indicator fields included in the second indicator field as the first dimension field.

6. The method according to claim 1, characterized in that The method further comprises: Reading the second indicator value of the second indicator field and the second dimension value of the second dimension field that satisfy the filtering rule, and calculating a version identifier based on the second indicator value and the second dimension value; If the version identifier is inconsistent with the version identifiers corresponding to the first index value and the first dimension value, an aggregation calculation is performed in the first aggregation table based on the second index value and the second dimension value to obtain a second aggregation table.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In response to the aggregation change reminder instruction, performing aggregation processing according to the aggregation change information carried in the aggregation change reminder instruction to obtain a third aggregation table; The data recorded in the first aggregate table is written into the third aggregate table, and the third aggregate table is marked based on the table identifier of the first aggregate table, and the first aggregate table is deleted.

8. A computer device, characterized in that: The device comprises: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program or instructions, and the computer program or instructions are used by a processor to execute the steps of the data processing method according to any one of claims 1 to 7.

Images