Data processing method and system based on indexes and labels

Abstract

The invention discloses a data processing method and system based on indexes and labels, and the method comprises the steps: obtaining an index generation template through configuring calculation logic and dimensions according to an actual application scene; associating an actual task demand with the index generation template to generate a script to be scheduled; according to a preset data source, executing the script to be scheduled to calculate indexes and labels, and generating the indexes and the labels meeting the actual task requirements; and obtaining a demand analysis result according to the indexes and the labels. According to the method, a large number of labels and indexes meeting intertask requirements can be generated through simple operation, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a data processing method and system based on indicators and labels. Background Art

[0002] In today's big data era, enterprises need to analyze their own operating conditions by mining operating data from different angles in their daily business activities in order to make subsequent business decisions and carry out business activities. However, multi-dimensional indicators and labels from different angles often have the characteristics of complex calculations and high flexibility.

[0003] However, the processing flow of data indicators and labels is long and relies on the operation and processing of professional data technicians. Therefore, the output of data indicators and labels often has problems such as long cycle, late launch time, low timeliness, etc. At the same time, as the output time is delayed, the value of some indicators and labels also decreases. Moreover, since various indicators and labels are very complicated and large, it is easy to have management chaos, which is not conducive to the management and construction of indicator and label systems. There is a lot of repeated development and it relies on professional data technicians. The threshold for business personnel to conduct independent exploration is high. Summary of the invention

[0004] This application proposes a data processing method and system based on indicators and labels, which can quickly generate labels and indicators that meet actual task requirements through simple operations, thereby improving the efficiency of data processing.

[0005] The first aspect of the present application provides a data processing method based on indicators and labels, the method comprising:

[0006] According to the actual application scenario, the indicator generation template is obtained by configuring the calculation logic and dimensions;

[0007] Associating the actual task requirements with the indicator generation template to generate a script to be scheduled;

[0008] According to the predetermined data source, the script to be scheduled is executed to calculate the indicators and labels, and the indicators and labels that meet the actual task requirements are generated;

[0009] According to the indicators and labels, the demand analysis results are obtained.

[0010] The above scheme first configures the logic and dimensions according to the actual application scenario, completes the configuration of the mapping relationship between the data source and the indicators and labels to be generated, and obtains the corresponding indicator generation template; then associates the actual task requirements with the indicator generation template, and maps the data analysis tasks to be completed by the actual task requirements to the template, realizes the assignment of tasks, and obtains the script to be scheduled for the executable task. Finally, the data source is selected for the script to be scheduled, and the indicators and labels are calculated based on the mapping relationship between the labels, indicators and data sources in the script to be scheduled, so as to obtain the indicators and labels that meet the actual task requirements without relying on manual operations, and then the demand analysis is completed based on the obtained indicators and labels.

[0011] In a possible implementation method of the first aspect, according to an actual application scenario, by configuring calculation logic and dimensions, an indicator generation template is obtained, specifically:

[0012] According to the actual application scenario, several computing targets and dimensions corresponding to the computing targets are obtained;

[0013] According to the calculation target, generate a first logic unit of each of the calculation targets; wherein each of the first logic units can be used to calculate a number of indicators and labels;

[0014] Based on the dimension, field configuration is performed for each of the first logical units to obtain an indicator generation template.

[0015] The above solution first determines the calculation target and the data dimension required for the calculation target in the scenario according to the actual application scenario, and generates a corresponding first logical unit for each calculation target to provide technical support for the calculation of indicators and labels. Then, the dimension is configured for the first logical unit, and the fields in the first logical unit are set to provide data support for the calculation of indicators and labels.

[0016] In a possible implementation method of the first aspect, based on the dimension, an indicator configuration is performed for each of the first logic units to obtain an indicator generation template, specifically:

[0017] According to the attributes of the dimension, each dimension is divided into a plurality of sub-dimensions, and associated fields corresponding to the sub-dimensions are determined;

[0018] According to the business logic of the actual application scenario, determine the indicator field corresponding to the associated field from a preset data field library;

[0019] Each of the first logic units is configured according to the indicator field to obtain an indicator generation template.

[0020] The above solution first layers the dimension to obtain multiple sub-dimensions. By refining the dimension, the indicator field corresponding to each sub-dimension can be accurately found. In subsequent applications, the data in the data source and the indicator field can be matched more quickly, improving the efficiency of data extraction. By configuring the first logical unit using the indicator field, a configured indicator generation template can be obtained.

[0021] In a possible implementation method of the first aspect, the actual task requirements are associated with the indicator generation template to generate a script to be scheduled, specifically:

[0022] According to the actual task requirements, indicator filtering conditions and dimension matching conditions are set for the indicator generation template to obtain the script to be scheduled;

[0023] Among them, the indicator filtering condition is used to filter out indicators that meet the actual task requirements from the data source; and the dimension matching condition is used to extract tags that meet the actual task requirements from the indicators.

[0024] The above solution sets the indicator filtering conditions based on the actual task requirements, which can efficiently extract the data that meets the requirements from a large amount of data for generating indicators. The dimension matching conditions are used to further find the labels that meet the actual task requirements from the generated indicators, ensuring the accuracy of indicator and label generation.

[0025] In a possible implementation method of the first aspect, the demand analysis result is specifically:

[0026] Obtaining the application frequency of each indicator or label in the demand analysis result;

[0027] When the application frequency exceeds a first threshold, the corresponding label or indicator is solidified;

[0028] The demand analysis task is performed using the solidified labels or indicators at a preset frequency.

[0029] After obtaining the indicators and labels, the above scheme will also monitor their usage frequency to determine the importance of the indicators and labels. When the indicator or label is used frequently, it will be solidified for long-term or regular application in data processing to improve the breadth of data application.

[0030] A second aspect of the present application provides a data processing system based on indicators and labels, the system comprising: a logic configuration module, a script generation module, an indicator and label generation module and a demand analysis result generation module;

[0031] The logic configuration module is used to obtain an indicator generation template by configuring calculation logic and dimensions according to actual application scenarios;

[0032] The script generation module is used to associate the actual task requirements with the indicator generation template to generate a script to be scheduled;

[0033] The indicator and label generation module is used to execute the script to be scheduled to calculate indicators and labels according to a predetermined data source, and generate indicators and labels that meet the actual task requirements;

[0034] The demand analysis result generating module is used to obtain the demand analysis result according to the indicators and labels.

[0035] In a possible implementation of the second aspect, the logic configuration module includes: a data configuration unit;

[0036] The template generation unit is used to obtain a number of computing targets and dimensions corresponding to the computing targets according to an actual application scenario; generate a first logic unit for each computing target according to the computing target; each of the first logic units can be used to generate a number of indicators and labels; based on the dimensions, perform field configuration for each of the first logic units to obtain an indicator generation template.

[0037] In a possible implementation of the second aspect, the logic configuration module includes: a template generation unit;

[0038] Among them, the template generation unit is used to divide each dimension into several sub-dimensions according to the attributes of the dimension, and determine the associated fields corresponding to the sub-dimensions; according to the business logic of the actual application scenario, determine the indicator field corresponding to the associated field from a preset data field library; according to the indicator field, configure each of the first logical units to obtain an indicator generation template.

[0039] In a possible implementation of the second aspect, the script generation module includes: a condition setting unit;

[0040] Among them, the condition setting unit is used to set indicator filtering conditions and dimension matching conditions for the task script according to actual task requirements to obtain the script to be scheduled; wherein the indicator filtering conditions are used to filter out indicators that meet the actual task requirements from the data source; the dimension matching conditions are used to extract labels that meet the actual task requirements from the indicators.

[0041] In a possible implementation of the second aspect, the demand analysis result generating module includes: a data solidification unit;

[0042] The data solidification unit is used to obtain the application frequency of each indicator or label in the demand analysis result; when the application frequency exceeds a first threshold, the corresponding label or indicator is solidified; and the demand analysis task is performed using the solidified label or indicator at a preset frequency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a specific flow chart of a data processing method based on indicators and tags provided in a certain embodiment of the present application;

[0045] Figure 2 It is a logic and dimensional configuration diagram of a data processing method based on indicators and tags provided in a certain embodiment of the present application;

[0046] Figure 3 It is a specific structural diagram of a data processing system based on indicators and tags provided in a certain embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] It should be understood that the step numbers used in this article are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0049] First embodiment

[0050] In many practical application scenarios, data labels are relied on, such as user portraits, recommendation algorithms, data analysis and mining, and refined operations. Labels are used to uniformly label standardized data with classification information on top of the data warehouse. Data indicators are one of the important tools for measuring goals, and together with labels, they play an important role in the process of data analysis. The processing process of existing indicators and labels is long and relies on professional data technicians, resulting in low output efficiency of indicators and labels. Moreover, once the amount of data for indicators and labels is too large, in the absence of a unified management system, it is necessary to study how to unify the mapping relationship between labels, indicators, and data sources, so as to generate a large number of usable labels and indicators based on the data source, which is convenient for effective analysis of data.

[0051] like Figure 1 As shown, Figure 1 A specific flow chart of a data processing method based on indicators and labels is provided for a certain embodiment of the present application. The data processing method based on indicators and labels of this embodiment includes steps S1 to S4, which are described in detail as follows:

[0052] Step S1, according to the actual application scenario, obtain the indicator generation template by configuring the calculation logic and dimensions.

[0053] In the embodiment of the present application, it is based on the typical B / S architecture design, that is, the interaction between the web page end, the application end and the database server. The implementation of the entire embodiment is mainly divided into two parts: the configuration of the script to be scheduled and the use of the configured script to be executed.

[0054] For the script to be executed, the first step is to configure the calculation logic and dimensions to obtain the corresponding indicator generation template. During the configuration process, it is necessary to perform a unified basic maintenance for the most basic calculation logic involved in the actual application scenario, and simply configure the calculation logic unit with multiple algorithms to obtain multiple first logic units. In an embodiment of the present application, the first logic unit is a configured SQL template, and each first logic unit can be used to generate a number of indicators and labels. At the same time, by specifying the analyzable dimensions, associated keywords and syntax of the calculation logic unit, the indicators and label logic generation based on the achievable dimensions of the calculation logic can be completed, and the algorithm logic of multiple calculations can be optionally specified to support subsequent users to configure complex indicators and label logic generation scenarios.

[0055] Then, according to the SQL code in the first logic unit, the corresponding dimensions are determined, and these dimensions are divided into sub-dimensions of major categories and minor categories, and the associated fields corresponding to the sub-dimensions are determined. Then, according to the business logic of the actual application scenario, the indicator field corresponding to the associated field is determined from the preset data field library. Finally, the indicator field is associated with the sub-dimension, so as to complete the construction of the mapping relationship between the indicator and the data source.

[0056] As an improvement of the above solution, based on the dimensions and combined with the business logic caliber of the indicators and labels involved in the actual application scenarios, the calculation logic units used for the indicators and the data fields for statistical analysis are defined to confirm the meaning of the indicators and the subsequent calculation logic, and the indicator fields corresponding to the associated fields are determined from the preset data field library.

[0057] In a broad sense, labels are also a type of dimension, but the special thing about labels is that they are dimensions evolved from indicators. For example, in the fund industry, gender is a common dimension, and the total number of customer transactions was originally an indicator. However, if the total number of customer transactions is greater than 1,000 times, it is considered a high-frequency trading customer. Then, when analyzing customers, a high-frequency trading customer label is derived based on this total number of transactions. Based on this label, the customer's investment behavior and income can be further analyzed. Moreover, many of these labels are very diverse and changeable. It is possible that this time the total number of transactions is greater than 1,000 and it is considered a high-frequency trading customer. The next time, it may be considered that the number of transactions is greater than 5,000 to be a high-frequency trading customer. The next time, it may be considered that the number of transactions in the past year is greater than 100 to be a high-frequency trading customer, and so on. In addition, labels will be combined with conventional dimensions, such as the income and investment of high-frequency trading customers by gender, and similarly, based on a changeable label and conventional and diverse dimension analysis, it is difficult to meet the analysis through conventional data analysis tools and data warehouse construction, so a flexible and changeable framework is needed to meet this type of data analysis.

[0058] For example, in a certain application scenario, it is necessary to summarize the total number of transactions, total transaction amount, and total sales amount of fund users this year. Then the calculation goal of this application scenario is to count the total number of transactions and total transaction amount of users in the year and filter out the sales operations to calculate the total sales amount. The dimensions of the above calculation goals are the calculation data involved, such as sales amount, transaction operations, and each transaction amount.

[0059] Specifically, when obtaining several calculation targets and the dimensions corresponding to the calculation targets through actual application scenarios, it is necessary to consider the business requirements involved in the actual application scenarios. The general business requirements are to describe the data they want to view in text or table form, such as viewing the total number of transactions, total transaction amount, total sales amount, etc. of customers since this year by age group, by seller, and by fund. By converting the description of the business into the language of data analysis, that is, age, seller, fund, and since this year are dimensions, and the total number of transactions, total transaction amount, and total sales amount are indicators, then the corresponding dimension classification is specified, for example, age is classified as other categories, sellers are classified as channel categories, funds are classified as product categories, and since this year are classified as time categories. The dimension classification and dimension template defined in advance have corresponding pre-configured underlying SQL database templates. By selecting a dimension, the corresponding dimension classification and dimension template can be matched, and at the same time, the underlying SQL database template and field are matched, so as to obtain an indicator generation template.

[0060] To better demonstrate how to configure the logic and dimensions, Figure 2 A logical and dimension configuration diagram is provided. As shown in the figure, the indicator generation template in the figure is called Full Test Execution 1, which can analyze dimensions including time, customer and channel. In the template, the corresponding indicator fields are associated according to the dimensions; the middle SQL syntax is the configured first logical unit, and below is the field configuration completed by the first logical unit according to the dimensions: the total number of fund codes and the total number of sellers, so as to obtain the corresponding indicator generation template.

[0061] After obtaining the indicator generation template, you can also configure intermediate grouping and secondary algorithms for the indicator fields to support statistical analysis at the indicator level.

[0062] Step S2, associating the actual task requirements with the indicator generation template to generate a script to be scheduled.

[0063] In the embodiment of the present application, according to the actual task requirements, the overall task is decomposed into multiple small tasks, each of which corresponds to a dynamic SQL code in the indicator generation template, and the overall task is actually also a dynamic SQL code. Then, the linkage configuration of indicators, labels and dimensions is performed for each small task to meet the calculation scenarios of indicators and labels, and the corresponding script to be scheduled is generated.

[0064] In addition, an indicator filtering condition and a dimension matching condition are set for the script to be scheduled. The indicator filtering condition is used to filter out indicators that meet the actual task requirements from the data source; and the dimension matching condition is used to extract tags that meet the actual task requirements from the indicators.

[0065] Step S3, according to the predetermined data source, execute the script to be scheduled to calculate the indicators and labels, and generate the indicators and labels that meet the actual task requirements.

[0066] In an embodiment of the present application, the backend server parses the parameters in the configured script to be scheduled, accesses a predetermined data source, executes the script to be scheduled to calculate indicators and labels, and generates a large number of indicators and labels that meet the actual task requirements.

[0067] Among them, the indicators are calculated and counted based on the basic data in the database, such as the customer's transaction flow data, the customer's asset flow data, the customer's dividend record data, plus the dimensional data, such as the classification of fund products, etc., and then these data are associated and calculated through grouping, summing, accumulation, maximum value, minimum value and other calculation rules. The same is true for labels, which are also calculated and counted based on the basic data in the database. The above indicators can be converted into labels. For example, the above indicator is the number of transactions of the customer in the past year, and then the label can be defined as: if the number of transactions of the customer in the past year exceeds 30 times, the customer will be labeled as a frequent trader. At this time, the label data is a customer ID, plus a label of frequent transactions of the customer. However, in real applications, the scope of indicators and labels often changes, and there is no way to calculate the corresponding indicators and labels in advance. For example, this time the number of transactions of the customer in the past year exceeds 30 times, which is defined as frequent transactions of the customer. The next time may be 50 times, and the next time may be 30 times with conditions, excluding redemption transactions, and so on. Therefore, it is necessary to automatically generate indicators and labels that meet current needs through the script to be scheduled in the embodiment of the present application to reduce the manual estimation of indicators and labels in each calculation.

[0068] Step S4, obtaining demand analysis results according to the indicators and labels.

[0069] In an embodiment of the present application, according to actual business needs, the generated indicators and tags are used to analyze the existing data to obtain corresponding demand analysis results.

[0070] After long-term use, these indicators and labels will be evaluated for their value, and high-value indicators or labels will be selected for solidification. Then, the solidified labels or indicators will be used to perform demand analysis tasks at a preset frequency.

[0071] In the embodiments of the present application, the value of indicators and labels is mainly reflected by their application frequency.

[0072] Specifically, since the generated indicators and labels are actually data obtained at one time, they are usually some indicators that the business suddenly thinks of or tries to use. They must be verified by the business for multiple rounds before they can be judged to be of great value. For example, a certain demand content is to count the customer yields of more than 30 transactions in each year in the past three years, and observe whether such indicators are really helpful for investor education. In the future, the business will try to propose customer yields from other angles, such as the customer yields of more than 30 transactions in each year of ICBC, etc., to obtain the corresponding indicators and labels, and apply them to the corresponding demand analysis business. After business verification and observation, if it is really helpful to customers, the business will propose to solidify this indicator, from a one-time report to a regular weekly or monthly report, and use it as a reference for actual operations.

[0073] Implementing the embodiments of the present application has the following beneficial effects:

[0074] The embodiment of the present application first configures the logic and dimensions according to the actual application scenario, completes the configuration of the mapping relationship between the data source and the indicators and labels to be generated, and obtains the corresponding indicator generation template; then the actual task requirements and the indicator generation template are associated, and the data analysis tasks to be completed by the actual task requirements are mapped to the template, so as to achieve the allocation of tasks and obtain the script to be scheduled for the executable task. Finally, the data source is selected for the script to be scheduled, and the indicators and labels are calculated based on the mapping relationship between the labels, indicators and data sources in the script to be scheduled, and the indicators and labels that meet the actual task requirements are quickly obtained without relying on manual operation. The obtained indicators and labels are then used to perform a comprehensive data analysis on the business of the actual task requirements, and the logical generation of complex indicators, labels and dimensions is quickly completed without the need for manual intervention, and the indicators and labels are quickly obtained to comprehensively process the data, thereby improving the efficiency of data processing. In addition, the generated indicators and labels can also be valued to meet more customer needs.

[0075] Second embodiment

[0076] Further, in order to execute the data processing system based on indicators and tags corresponding to the above method embodiment to achieve corresponding functions and technical effects, Figure 3 A structural diagram of a data processing system based on indicators and labels is provided. For ease of explanation, only the parts related to this embodiment are shown. The data processing system based on indicators and labels provided in the embodiment of the present application includes:

[0077] The logic configuration module 201 is used to obtain an indicator generation template by configuring calculation logic and dimensions according to actual application scenarios.

[0078] In an embodiment of the present application, the logic configuration module 201 also includes a data configuration unit, which is used to obtain a number of computing targets and dimensions corresponding to the computing targets according to an actual application scenario; generate a first logic unit for each of the computing targets according to the computing targets; wherein each of the first logic units can be used to generate a number of indicators and labels; based on the dimensions, perform field configuration for each of the first logic units to obtain an indicator generation template.

[0079] The script generation module 202 is used to associate the actual task requirements with the indicator generation template to generate a script to be scheduled.

[0080] In an embodiment of the present application, the script generation module 202 also includes a condition setting unit, which is used to set indicator filtering conditions and dimension matching conditions for the task script according to actual task requirements to obtain a script to be scheduled; wherein the indicator filtering conditions are used to filter out indicators that meet the actual task requirements from the data source; and the dimension matching conditions are used to extract labels that meet the actual task requirements from the indicators.

[0081] The indicator and label generation module 203 is used to execute the script to be scheduled to calculate indicators and labels according to a predetermined data source, and generate indicators and labels that meet the actual task requirements.

[0082] In an embodiment of the present application, the backend server parses the parameters in the configured script to be scheduled, accesses a predetermined data source, executes the script to be scheduled to calculate indicators and labels, and generates a large number of indicators and labels that meet the actual task requirements.

[0083] Among them, the indicators are calculated and counted based on the basic data in the database, such as the customer's transaction flow data, the customer's asset flow data, the customer's dividend record data, plus the dimensional data, such as the classification of fund products, etc., and then these data are associated and calculated through grouping, summing, accumulation, maximum value, minimum value and other calculation rules. The same is true for labels, which are also calculated and counted based on the basic data in the database. The above indicators can be converted into labels. For example, the above indicator is the number of transactions of the customer in the past year, and then the label can be defined as: if the number of transactions of the customer in the past year exceeds 30 times, the customer will be labeled as a frequent trader. At this time, the label data is a customer ID, plus a label of frequent transactions of the customer. However, in real applications, the caliber of indicators and labels often changes, and there is no way to calculate the corresponding indicators and labels in advance. For example, this time the number of transactions of the customer in the past year exceeds 30 times, which is defined as frequent transactions of the customer. The next time may be 50 times, and the next time may be added with conditions, 30 times except for redemption transactions, and so on. Therefore, it is necessary to automatically generate indicators and labels that meet current needs through the script to be scheduled in the embodiment of the present application to reduce the manual estimation of indicators and labels in each calculation.

[0084] The demand analysis result generating module 204 is used to obtain the demand analysis result according to the indicators and tags.

[0085] In an embodiment of the present application, the demand analysis result generation module 204 also includes a data solidification unit, which is used to obtain the application frequency of each indicator or label in the demand analysis result; when the application frequency exceeds a first threshold, the corresponding label or indicator is solidified; and the demand analysis task is performed using the solidified label or indicator at a preset frequency.

[0086] In some embodiments, the logic configuration module 201 further includes:

[0087] It is based on a typical B / S architecture design, that is, the interaction between the web page end, the application end and the database server. The implementation of the entire embodiment is mainly divided into two parts: the configuration of the script to be scheduled and the use of the configured script to be executed.

[0088] For the script to be executed, the first step is to configure the calculation logic and dimensions to obtain the corresponding indicator generation template. During the configuration process, it is necessary to perform a unified basic maintenance for the most basic calculation logic involved in the actual application scenario, and simply configure the calculation logic unit with multiple algorithms to obtain multiple first logic units. In an embodiment of the present application, the first logic unit is a configured SQL template, and each first logic unit can be used to generate a number of indicators and labels. At the same time, by specifying the analyzable dimensions, associated keywords and syntax of the calculation logic unit, the indicators and label logic generation based on the achievable dimensions of the calculation logic can be completed, and the algorithm logic of multiple calculations can be optionally specified to support subsequent users to configure complex indicators and label logic generation scenarios.

[0089] Then, according to the SQL code in the first logic unit, the corresponding dimensions are determined, and these dimensions are divided into sub-dimensions of major categories and minor categories, and the associated fields corresponding to the sub-dimensions are determined. Then, according to the business logic of the actual application scenario, the indicator field corresponding to the associated field is determined from the preset data field library. Finally, the indicator field is associated with the sub-dimension, so as to complete the construction of the mapping relationship between the indicator and the data source.

[0090] As an improvement of the above solution, based on the dimensions and combined with the business logic caliber of the indicators and labels involved in the actual application scenarios, the calculation logic units used for the indicators and the data fields for statistical analysis are defined to confirm the meaning of the indicators and the subsequent calculation logic, and the indicator fields corresponding to the associated fields are determined from the preset data field library.

[0091] In a broad sense, labels are also a type of dimension, but the special thing about labels is that they are dimensions evolved from indicators. For example, in the fund industry, gender is a common dimension, and the total number of customer transactions was originally an indicator. However, if the total number of customer transactions is greater than 1,000 times, it is considered a high-frequency trading customer. Then, when analyzing customers, a high-frequency trading customer label is derived based on this total number of transactions. Based on this label, the customer's investment behavior and income can be further analyzed. Moreover, many of these labels are very diverse and changeable. It is possible that this time the total number of transactions is greater than 1,000 and it is considered a high-frequency trading customer. The next time, it may be considered that the number of transactions is greater than 5,000 to be a high-frequency trading customer. The next time, it may be considered that the number of transactions in the past year is greater than 100 to be a high-frequency trading customer, and so on. In addition, labels will be combined with conventional dimensions, such as the income and investment of high-frequency trading customers by gender, and similarly, based on a changeable label and conventional and diverse dimension analysis, it is difficult to meet the analysis through conventional data analysis tools and data warehouse construction, so a flexible and changeable framework is needed to meet this type of data analysis.

[0092] For example, in a certain application scenario, it is necessary to summarize the total number of transactions, total transaction amount, and total sales amount of fund users this year. Then the calculation goal of this application scenario is to count the total number of transactions and total transaction amount of users in the year and filter out the sales operations to calculate the total sales amount. The dimensions of the above calculation goals are the calculation data involved, such as sales amount, transaction operations, and each transaction amount.

[0093] Specifically, when obtaining several calculation targets and the dimensions corresponding to the calculation targets through actual application scenarios, it is necessary to consider the business requirements involved in the actual application scenarios. The general business requirements are to describe the data they want to view in text or table form, such as viewing the total number of transactions, total transaction amount, total sales amount, etc. of customers since this year by age group, by seller, and by fund. By converting the description of the business into the language of data analysis, that is, age, seller, fund, and since this year are dimensions, and the total number of transactions, total transaction amount, and total sales amount are indicators, then the corresponding dimension classification is specified, for example, age is classified as other categories, sellers are classified as channel categories, funds are classified as product categories, and since this year are classified as time categories. The dimension classification and dimension template defined in advance have corresponding pre-configured underlying SQL database templates. By selecting a dimension, the corresponding dimension classification and dimension template can be matched, and at the same time, the underlying SQL database template and field are matched, so as to obtain an indicator generation template.

[0094] After obtaining the indicator generation template, you can also configure intermediate grouping and secondary algorithms for the indicator fields to support statistical analysis at the indicator level.

[0095] In some embodiments, the script generation module 202 further includes:

[0096] In the embodiment of the present application, according to the actual task requirements, the overall task is decomposed into multiple small tasks, each of which corresponds to a dynamic SQL code in the indicator generation template, and the overall task is actually also a dynamic SQL code. Then, the linkage configuration of indicators, labels and dimensions is performed for each small task to meet the calculation scenarios of indicators and labels, and the corresponding script to be scheduled is generated.

[0097] In addition, an indicator filtering condition and a dimension matching condition are set for the script to be scheduled. The indicator filtering condition is used to filter out indicators that meet the actual task requirements from the data source; and the dimension matching condition is used to extract tags that meet the actual task requirements from the indicators.

[0098] Step S3, according to the predetermined data source, execute the script to be scheduled to calculate the indicators and labels, and generate the indicators and labels that meet the actual task requirements.

[0099] In an embodiment of the present application, the backend server parses the parameters in the configured script to be scheduled, accesses a predetermined data source, executes the script to be scheduled to calculate indicators and labels, and generates a large number of indicators and labels that meet the actual task requirements.

[0100] Among them, the indicators are calculated and counted based on the basic data in the database, such as the customer's transaction flow data, the customer's asset flow data, the customer's dividend record data, plus the dimensional data, such as the classification of fund products, etc., and then these data are associated and calculated through grouping, summing, accumulation, maximum value, minimum value and other calculation rules. The same is true for labels, which are also calculated and counted based on the basic data in the database. The above indicators can be converted into labels. For example, the above indicator is the number of transactions of the customer in the past year, and then the label can be defined as: if the number of transactions of the customer in the past year exceeds 30 times, the customer will be labeled as a frequent trader. At this time, the label data is a customer ID, plus a label of frequent transactions of the customer. However, in real applications, the scope of indicators and labels often changes, and there is no way to calculate the corresponding indicators and labels in advance. For example, this time the number of transactions of the customer in the past year exceeds 30 times, which is defined as frequent transactions of the customer. The next time may be 50 times, and the next time may be 30 times with conditions, excluding redemption transactions, and so on. Therefore, it is necessary to automatically generate indicators and labels that meet current needs through the script to be scheduled in the embodiment of the present application to reduce the manual estimation of indicators and labels in each calculation.

[0101] In some embodiments, the demand analysis result generating module 204 further includes:

[0102] In an embodiment of the present application, according to actual business needs, the generated indicators and tags are used to analyze the existing data to obtain corresponding demand analysis results.

[0103] After long-term use, these indicators and labels will be evaluated for their value, and high-value indicators or labels will be selected for solidification. Then, the solidified labels or indicators will be used to perform demand analysis tasks at a preset frequency.

[0104] In the embodiments of the present application, the value of indicators and labels is mainly reflected by their application frequency.

[0105] Implementing the embodiments of the present application has the following beneficial effects:

[0106] The embodiment of the present application first configures the logic and dimensions according to the actual application scenario, completes the configuration of the mapping relationship between the data source and the indicators and labels to be generated, and obtains the corresponding indicator generation template; then the actual task requirements and the indicator generation template are associated, and the data analysis tasks to be completed by the actual task requirements are mapped to the template, so as to achieve the allocation of tasks and obtain the script to be scheduled for the executable task. Finally, the data source is selected for the script to be scheduled, and the indicators and labels are calculated based on the mapping relationship between the labels, indicators and data sources in the script to be scheduled, and the indicators and labels that meet the actual task requirements are quickly obtained without relying on manual operation. The obtained indicators and labels are then used to perform a comprehensive data analysis on the business of the actual task requirements, and the logical generation of complex indicators, labels and dimensions is quickly completed without the need for manual intervention, and the indicators and labels are quickly obtained to comprehensively process the data, thereby improving the efficiency of data processing. In addition, the generated indicators and labels can also be valued to meet more customer needs.

[0107] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A data processing method based on indicators and labels, characterized in that: include: According to the actual application scenario, the indicator generation template is obtained by configuring the calculation logic and dimensions; Associating the actual task requirements with the indicator generation template to generate a script to be scheduled; According to the predetermined data source, the script to be scheduled is executed to calculate the indicators and labels, and the indicators and labels that meet the actual task requirements are generated; According to the indicators and labels, the demand analysis results are obtained.

2. The data processing method based on indicators and labels according to claim 1 is characterized in that: According to the actual application scenario, the indicator generation template is obtained by configuring the calculation logic and dimensions, specifically: According to the actual application scenario, several computing targets and dimensions corresponding to the computing targets are obtained; According to the calculation target, generate a first logic unit of each of the calculation targets; wherein each of the first logic units can be used to calculate a number of indicators and labels; Based on the dimension, field configuration is performed for each of the first logical units to obtain an indicator generation template.

3. The data processing method based on indicators and labels according to claim 2 is characterized in that: Based on the dimension, an indicator configuration is performed for each of the first logic units to obtain an indicator generation template, specifically: According to the attributes of the dimension, each dimension is divided into a plurality of sub-dimensions, and associated fields corresponding to the sub-dimensions are determined; According to the business logic of the actual application scenario, determine the indicator field corresponding to the associated field from a preset data field library; Each of the first logic units is configured according to the indicator field to obtain an indicator generation template.

4. The data processing method based on indicators and labels according to claim 1 is characterized in that: The actual task requirements are associated with the indicator generation template to generate a script to be scheduled, specifically: According to the actual task requirements, indicator filtering conditions and dimension matching conditions are set for the indicator generation template to obtain the script to be scheduled; Among them, the indicator filtering condition is used to filter out indicators that meet the actual task requirements from the data source; and the dimension matching condition is used to extract tags that meet the actual task requirements from the indicators.

5. The data processing method based on indicators and labels according to claim 1 is characterized in that: The demand analysis results are specifically as follows: Obtaining the application frequency of each indicator or label in the demand analysis result; When the application frequency exceeds a first threshold, the corresponding label or indicator is solidified; The demand analysis task is performed using the solidified labels or indicators at a preset frequency.

6. A data processing system based on indicators and labels, characterized in that: include: Logical configuration module, script generation module, indicator and label generation module and demand analysis result generation module; The logic configuration module is used to obtain an indicator generation template by configuring calculation logic and dimensions according to actual application scenarios; The script generation module is used to associate the actual task requirements with the indicator generation template to generate a script to be scheduled; The indicator and label generation module is used to execute the script to be scheduled to calculate indicators and labels according to a predetermined data source, and generate indicators and labels that meet the actual task requirements; The demand analysis result generating module is used to obtain the demand analysis result according to the indicators and labels.

7. The data processing system based on indicators and tags according to claim 6 is characterized in that: The logic configuration module includes: a data configuration unit; The template generation unit is used to obtain a number of computing targets and dimensions corresponding to the computing targets according to an actual application scenario; generate a first logic unit for each computing target according to the computing target; each of the first logic units can be used to calculate a number of indicators and labels; and based on the dimensions, perform field configuration for each of the first logic units to obtain an indicator generation template.

8. The data processing system based on indicators and tags according to claim 6 is characterized in that: The logic configuration module includes: a template generation unit; Among them, the template generation unit is used to divide each dimension into several sub-dimensions according to the attributes of the dimension, and determine the associated fields corresponding to the sub-dimensions; according to the business logic of the actual application scenario, determine the indicator field corresponding to the associated field from a preset data field library; according to the indicator field, configure each of the first logical units to obtain an indicator generation template.

9. The data processing system based on indicators and tags according to claim 6, characterized in that: The script generation module includes: a condition setting unit; Among them, the condition setting unit is used to set indicator filtering conditions and dimension matching conditions for the indicator generation template according to actual task requirements to obtain the script to be scheduled; wherein the indicator filtering conditions are used to filter out indicators that meet the actual task requirements from the data source; the dimension matching conditions are used to extract labels that meet the actual task requirements from the indicators.

10. The data processing system based on indicators and tags according to claim 6, characterized in that: The demand analysis result generating module comprises: a data solidification unit; The data solidification unit is used to obtain the application frequency of each indicator or label in the demand analysis result; when the application frequency exceeds a first threshold, the corresponding label or indicator is solidified; and the demand analysis task is performed using the solidified label or indicator at a preset frequency.

Images