Variable one-key to-online method and system based on Hadoop ecological component

Abstract

The invention relates to the technical field of informatization, and discloses a Hadoop ecological component-based variable one-key-to-online method and system, and the method comprises the steps: defining a universal variable SQL template according to an international unified standard, calling a corresponding variable SQL template according to an obtained business demand, and generating a variable SQL meeting the business demand; uploading the generated variable SQL to an HDFS (Hadoop Distributed File System), and carrying out grammar inspection on the uploaded SQL; analyzing the relationship in the variable SQL, and constructing a blood relationship map between the variable tables; according to the configured variable execution frequency, generating a corresponding scheduling task, and submitting the scheduling task to Oozie for automatic execution; synchronizing data of the scheduling execution result to a high-performance database; automatically generating an API (Application Program Interface) by taking the dimension as an input parameter and the variable column as a result; registering a variable to a variable query platform, and packaging variable metadata of the variable; registering the release variable to a variable calculation system; the technical problems that in the prior art, in the variable-to-online process, efficiency is low, and program development is complex are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and in particular to a method and system for one-key online conversion of variables based on Hadoop ecological components. BACKGROUND

[0002] In today's digital age, banks, as an important part of the financial industry, generate and accumulate massive amounts of data every day. These data contain a wealth of information that is of great value to business decision-making, risk control, and marketing.

[0003] Currently, in the process of using bank data, valuable data features in the data need to be published online for use, which usually requires a series of cumbersome steps, including: Data development colleagues need to generate variable logic based on business logic and use SQL, ql expression, etc. to express the logic and generate a table. The API is introduced as a variable, and the input parameters and calls of the variable are sorted out.

[0004] This process requires data developers to not only be familiar with various programming languages and data processing techniques, but also have a deep understanding of the various business processes and rules of the bank. However, due to the complexity and diversity of business logic, data developers often cannot be comprehensive, and understanding biases are prone to occur, resulting in generated tables that do not meet actual needs.

[0005] When introducing the API as a variable, multiple departments and systems need to work together, and due to the difficulty of communication and coordination between different departments and systems, information synchronization is often not timely and accurate, and problems such as inconsistent understanding of variable input parameters and incorrect calling methods may occur.

[0006] The entire process is complex and complicated, resulting in low overall efficiency. From the conception of variable logic to the final online use of variables, it needs to go through multiple approval and verification links, and each link needs to consume a certain amount of time, making the technology development investment cost large. For variable input production strategy and model use, a relatively long process needs to be experienced, and the rapid changes of business cannot be responded to in a timely manner. SUMMARY

[0007] The present application aims to provide a method and system for one-key online conversion of variables based on Hadoop ecological components to solve the technical problems of low efficiency and complicated program development in the process of converting variables to online in the prior art.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solution: a method for one-key online conversion of variables based on Hadoop ecological components, comprising: According to the international standard, the general variable SQL template is defined and stored in the system, and the business requirements input by the user are obtained through the interactive interface; the corresponding variable SQL template is called according to the obtained business requirements, and the variable SQL template is adjusted and converted to generate a variable SQL that meets the business requirements; The generated variable SQL is uploaded to the specified directory on HDFS, and the SQL parser of Hive or Spark is called to perform syntax checking on the uploaded SQL. If the SQL syntax is correct, the subsequent steps are continued; otherwise, the error information is returned to the user interface, prompting the user to modify the SQL; The table association relationship and field reference relationship in the variable SQL are parsed, and the blood graph between the variable tables is constructed. The generated blood relationship data is stored in the graph database; According to the configured variable execution frequency, the corresponding scheduling task is generated and submitted to Oozie for automatic execution; the scheduling execution result data is synchronized to the Hbase high-performance database; According to the dimension as the input parameter, the variable column as the result, the API is generated; the variable is registered to the variable query platform, and the variable metadata of the variable is encapsulated; the variable is registered and published to the variable calculation system, which can be called at any time online.

[0009] The principle and advantages of the scheme are: technical personnel only need to design a general variable SQL template in the pre-stage, and business personnel can use it by inputting relevant information through the interactive interface according to actual business requirements. Business personnel do not need to master complex programming languages and data processing techniques, which reduces the deviation of technical personnel in understanding business logic and solves the problem that the generated table does not meet the requirements due to complex and diverse business logic.

[0010] The whole process realizes high automation, from the generation, verification, scheduling execution of variable SQL to the generation of API and the publication of variable, each link is closely connected, reducing the time consumption of manual intervention and approval verification. Compared with the traditional cumbersome process, the time from the conception of variable to the online use is greatly shortened, which can respond to the rapid changes of business in time.

[0011] By constructing the blood graph between the variable tables and storing it in the graph database, the dependency relationship between the data is clearly presented, which is convenient for managing and tracking the data. At the same time, strict SQL syntax checking and standardized process design reduce the risk of information synchronization error, avoid the problems of inconsistent understanding of variable input parameters and incorrect calling mode, and improve the data quality and accuracy.

[0012] Due to the simplification and automation of the process, a large number of manual operations and communication coordination costs are reduced. Frequent information synchronization and collaboration between multiple departments and systems are no longer needed, reducing errors and rework caused by poor communication, thereby effectively controlling the cost of technology development investment.

[0013] Preferably, as an improvement, the dimensions of the fields contained in the general variable SQL template are unified in addition to the variable name. The user inputs variable definition information and SQL logic through the interactive interface, and when unified dimension information is involved, the specified unified field name must be used.

[0014] The improvement has the beneficial effect that in the data system of the bank, different business scenarios may involve some common dimension information, such as customer identification and time dimension. Unifying the field names of these dimension information makes the data consistent in various processing links and tables. For example, no matter which business generates a variable table, as long as it involves customer number, the unified field name "cust_no" is used, which facilitates subsequent data integration, correlation and analysis. When full-row unified query or statistics of data are needed, complex field mapping and conversion are no longer needed, improving the maintainability of the data.

[0015] Preferably, as an improvement, the workflow is defined by using the XML configuration file of Oozie to convert the SQL into the ETL processing statement of the table according to the generated scheduling task information, and dynamically generate an XML file according to the configured scheduling period. The system submits the generated XML file to the Oozie server through the Java API of Oozie. The Oozie server automatically executes the scheduling task according to the configuration information in the workflow.xml file and the predetermined time arrangement.

[0016] The improvement has the beneficial effect that by using the workflow defined by the XML configuration file of Oozie, the task flow can be flexibly configured according to different scheduling task information, such as task name, execution frequency, and SQL file path. The dynamic generation of the XML file enables the system to quickly adjust the scheduling task according to the actual business demand without modifying a large number of underlying codes. For example, when the business demand changes and the calculation frequency of a variable needs to be adjusted, only the corresponding configuration information needs to be modified, and the XML file needs to be regenerated and submitted.

[0017] The Oozie server automatically executes the scheduled tasks according to the configuration information in the workflow.xml file and the predetermined time schedule. This ensures the accuracy and timeliness of task execution, avoiding omissions or delays that may occur when tasks are executed manually. At the same time, Oozie provides perfect task monitoring and error handling mechanisms, which can record error information and take corresponding measures in a timely manner when an exception occurs during task execution, improving the reliability of task execution.

[0018] Preferably, as an improvement, when generating the API, the variable definition information and the dimension information are used to fill in the variable name, variable type and request parameter information using a preset API template to generate a complete API code; the generated API code is compiled and packaged through a build tool and deployed to an API gateway server through an automated deployment tool.

[0019] The improvement has the beneficial effects that: using a preset API template to fill in relevant content according to the variable definition information and the dimension information can quickly generate a complete API code. This avoids the tedious process of starting from scratch to write API code, greatly improving development efficiency. At the same time, the preset template ensures that the structure and function of the API meet certain standards, making the generated API have good consistency and maintainability.

[0020] Compiling and packaging the generated API code through a build tool can detect syntax errors and potential problems in the code, ensuring the quality of the code. The build tool can also optimize and compress the code to improve the execution efficiency of the code. The automated deployment tool can quickly and accurately deploy the packaged API code to the API gateway server, reducing errors that may occur during manual deployment and improving the stability of the system.

[0021] A variable one-key online conversion system based on Hadoop ecological components is used to implement a variable one-key online conversion method based on Hadoop ecological components, comprising: A variable template management module is used to define and store a general variable SQL template, which complies with international unified standards. A user interface module is used for users to input business requirements. An SQL processing and verification module receives SQL statements from the user interface module and uploads them to a designated directory on HDFS; calls the SQL parser of Hive or Spark to perform syntax verification on the uploaded SQL; parses the table association relationship and field reference relationship in the SQL, constructs the blood graph between the variable tables, and stores it in the graph database; The scheduling task generation and execution module generates corresponding scheduling tasks according to the variable execution frequency configured by the user, generates an XML file and submits it to the Oozie server, and calls the Oozie server to automatically execute the scheduling task according to the predetermined time arrangement; The data synchronization and storage module listens to the execution result of the scheduling task, and synchronizes the result data to HBase and Hive when the task execution is completed. The API generation and deployment module generates the code framework of the API using a template engine according to the variable definition information and dimension information, compiles and packages the generated API code, and deploys it to the API gateway server through an automated deployment tool. The variable definition information received from the data synchronization and storage module is used to generate the API code, and the deployed API information is passed to the subsequent module for registration and publishing. The variable registration and publishing module registers the generated API information in the API market system, registers the variable information in the variable query platform, encapsulates the metadata information of the variable, and publishes the variable to the variable calculation system after the user confirms that the variable information is correct. The variable calculation and calling module receives the variable information from the variable registration and publishing module and stores and caches it, and other systems obtain the variable value by calling the API in the API market system.

[0022] The improved system has the following advantages: the variable one-key online system based on the Hadoop ecological components realizes the full-process automatic processing of the variable from definition to online use through the cooperative work of various modules. Compared with the traditional manual operation mode, the process is greatly simplified, the manual intervention is reduced, the accuracy and efficiency of data processing are improved, the cost of scientific and technological development is reduced, and the demand of banks for data rapid application and value mining in the digital era can be better met.

[0023] Preferably, as an improvement, the API market system forwards the request to the variable calculation system after receiving the request, the variable calculation system queries the corresponding variable value from HBase or Hive according to the request parameter, and returns the result to the API market system; receives and stores the published variable information, and provides variable value query service in response to the request of the API market system.

[0024] The improved beneficial effect is that the variable calculation system queries the corresponding variable value from HBase or Hive according to the request parameters. HBase is suitable for fast querying of small amounts of data, and Hive is suitable for large-scale data analysis. The system can select the appropriate storage system for querying according to the actual situation. At the same time, the variable calculation system stores and caches the published variable information. When receiving the same or similar request again, the data can be directly obtained from the cache, reducing the data query time and improving the system response speed.

[0025] Preferably, as an improvement, in the variable template management module, the templates are stored in the Hive or Spark related metadata storage system for retrieval and use. The user interface module allows users to input variable definition information, including variable name, type, dimension, and description, as well as write SQL logic.

[0026] The improved beneficial effect is that the templates are stored in the Hive or Spark related metadata storage system, which is convenient for retrieval and use. Data developers can quickly find suitable templates according to different business scenarios, reducing the time for template searching and selection. At the same time, the metadata storage system can perform version management and permission control on the templates, ensuring the security and consistency of the templates. The user interface module allows users to input variable definition information, including variable name, type, dimension, and description, as well as write SQL logic. This allows users to provide more comprehensive information when defining variables, which helps subsequent SQL processing and variable generation. At the same time, the interface can perform certain verification and prompts on user input, reducing input errors and improving data accuracy.

[0027] Preferably, as an improvement, the scheduling task generation and execution module uses Oozie's XML configuration file to define the workflow and dynamically generates Oozie's workflow.xml file. The generated workflow.xml file is submitted to the Oozie server through Oozie's Java API, and the scheduling task is automatically executed according to the predetermined time schedule. The Oozie server is called to execute the task, and the data sources include real-time data from the submission center database and data obtained from external third-party systems through the data gateway.

[0028] The improved beneficial effect is that the automatic execution of the scheduling task is realized, reducing manual intervention and improving the accuracy and timeliness of task execution. The system can integrate data from multiple sources for variable calculation, meeting the complex business needs of banks. For example, in risk assessment business, bank internal customer transaction data and external credit data can be used to calculate risk variables, improving the accuracy of risk assessment.

[0029] As an improvement, the user can replace the placeholders in the SQL template according to the actual business requirements.

[0030] The advantage of this improvement is that for business personnel or developers who are not familiar with complex SQL writing, generating SQL statements by replacing placeholders reduces the development difficulty. They do not need to master the syntax and details of SQL, but only need to fill in the corresponding parameters according to the business requirements. This reduces the development time and cost, and improves the development efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The structure diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be further described in detail through specific embodiments: EMBODIMENT Basically as shown in the accompanying drawings, a method for one-key conversion of variables online based on Hadoop ecological components includes: Figure 1 According to international unified standards, define general variable SQL templates. Store these templates in Hive or Spark related metadata storage systems, and wait to be called. In Hive, templates can be stored in specific databases or tables, and retrieved and used conveniently through metadata management. Spark can also store and access SQL templates through similar metadata management methods. SQL templates are standard SQL structures predefined to facilitate users to quickly generate variable SQLs that meet business requirements. The following is an example of a SQL template for calculating the total amount of transactions of a customer within a certain period of time:

[0033] -- Calculate the total amount of transactions of a customer within a specified time period -- Parameter description: --start_date: Start date --end_date: End date SELECT cust_no,-- Unified field name, representing the customer number corresponding to the ID number SUM(amount) AS total_amount-- Variable name: total_amount, representing the total amount of transactions FROM transactions-- Transaction data table WHERE ​transaction_date BETWEEN '{start_date}' AND '{end_date}'-- Time range condition GROUP BY cust_no; In this template, {start_date} and {end_date} are placeholders that need to be replaced by the user according to specific business requirements when actually used.

[0034] The transactions table involved in the SQL template example can contain the following fields: Table 1: Field table of variable SQL template example

[0035] The fields in the table in the SQL template should have uniform field names except for variable names.

[0036] A visual or command line interface is provided for users to input business requirements and save them. The system retrieves and matches the appropriate template from the pre-stored SQL templates according to the user's input business requirements. The system adjusts and transforms this template according to actual requirements to generate the corresponding variable SQL.

[0037] The user's input interface should include fields for inputting variable definition information, including variable name, variable type, variable dimension, and variable description information. Variable types include integers, strings, and floating-point numbers, and variable dimensions include time dimension and regional dimension.

[0038] Users also write SQL logic through the interface, which needs to follow the SQL syntax specification to ensure that the written SQL statement meets the Hive or Spark SQL syntax requirements. When it comes to unified dimension information such as ID numbers, the specified unified field name must be used. If there are placeholders in the SQL template, the user needs to correctly replace them according to the actual business requirements.

[0039] The system starts a timing task or real-time listens to the user's operation of saving SQL in the background, and uploads the generated variable SQL to the specified directory on HDFS through the Hadoop Java API. During the upload process, the system records the upload time, file size, and other information, and stores these metadata in the database for subsequent query and management.

[0040] Call the SQL parser of Hive or Spark to perform syntax checks on the uploaded SQL. For example, use HiveClient to obtain HiveMetaStoreClient, and then use related classes such as GetTablesRequest and GetTablesResponse to parse the SQL and check for syntax errors.

[0041] If the SQL syntax is correct, proceed to the next step; otherwise, return an error message to the user interface, prompting the user to modify the SQL. The error message should include detailed information such as the error location and type to help the user quickly locate and resolve the problem.

[0042] The table relationships and field references in the SQL query are analyzed to construct a lineage graph between the variable tables. The generated lineage data is then stored in a graph database for subsequent querying and analysis.

[0043] For example, if the SQL statement references tables A and B and links them through a certain field, the lineage analysis module will record the relationship between tables A and B, as well as the dependency relationship between the variables generated by the SQL statement and the fields in tables A and B.

[0044] Based on the execution frequency of variables configured by the user during the variable definition phase, corresponding scheduled tasks are generated. Scheduled task information includes task name, task type, execution frequency, path to the SQL file to be executed, and input parameters. The system stores the generated scheduled task information in the database and links it to the corresponding variable definition information.

[0045] Oozie's XML configuration file defines the workflow. Based on the generated scheduling task information, it transforms SQL into ETL processing statements for tables and dynamically generates an Oozie workflow.xml file according to the configured scheduling cycle. The workflow.xml file configures the task execution nodes, specifying the path to the SQL file to be executed, input parameters, and other information. The system submits the generated workflow.xml file to the Oozie server via the Oozie Java API. The Oozie server automatically executes the scheduled tasks according to the configuration information in the workflow.xml file and the predetermined time schedule.

[0046] After the Oozie server triggers a scheduled task, it calls the corresponding computing engine, such as Hive or Spark, to execute SQL statements according to the configuration in the workflow.xml file. The computing engine reads input data from HDFS, executes the SQL logic, and stores the result data in a temporary directory.

[0047] The execution result of the monitoring scheduling task is listened to, and when the task is executed, the result data is synchronized to HBase and Hive. In HBase, the system synchronizes the result data in the temporary directory to the HBase table through the HBase API of Hadoop. During the synchronization process, according to the dimension information in the variable definition, the data is stored in HBase according to the appropriate row key, that is, RowKey, to improve the data query performance.

[0048] For example, if the variable dimension includes time and customer ID, the row key can be designed in the form of time_customer ID, which facilitates fast query according to time and customer ID.

[0049] When the result data is stored in the Hive data warehouse, the data is written into the Hive table through the LOAD DATA command of Hive or using Spark, so as to perform subsequent data analysis and query.

[0050] According to the variable definition information and the dimension information, a template engine such as Freemarker is used to generate a code framework of the RESTful API. According to the preset API template, the variable name, variable type, request parameter and other information are filled in to generate a complete API code. The generated API code includes a controller class (Controller), a service class (Service), a data access class (DAO), etc., which realizes the request processing, business logic processing and data access functions of the API.

[0051] The generated API code is compiled and packaged through a build tool such as Maven or Gradle, and is deployed on the API gateway server through an automated deployment tool. The automated deployment tool can use Jenkins.

[0052] The generated API information is registered in the API market system. The API information includes API name, API description, request URL, request method, request parameter, response format, etc. At the same time, the variable information is registered in the variable query platform, and the metadata information of the variable is encapsulated, such as variable type, input parameter, creation time, modification time, etc., to facilitate user query and management of variables. In the variable query platform, the user can view the registered variable information.

[0053] After the user confirms that the variable information is correct, the user clicks the publish button to publish the variable to the variable calculation system. The system updates the variable state to "published", and synchronizes the variable information to the cache of the variable calculation system to improve the variable calling performance.

[0054] Other systems obtain variable values by calling APIs in the API marketplace system. When calling, the corresponding request parameters such as customer ID, time range, etc. need to be passed in. After receiving the request, the API marketplace system forwards the request to the variable calculation system. The variable calculation system queries the corresponding variable value from HBase or Hive according to the request parameters, and returns the result to the API marketplace system. The API marketplace system returns the variable value to the caller according to the agreed response format.

[0055] The system uses dubbo protocol for communication between systems, so that users can directly convert SQL variable logic to online by one key and publish it to online strategy and model.

[0056] A variable one-key online conversion system based on Hadoop ecological components, for implementing a "variable one-key online conversion method", comprising: A variable template management module defines and stores general variable SQL templates, which follow international unified standards. The templates are stored in Hive or Spark related metadata storage systems for easy retrieval and use. The module provides template selection for users, who can select appropriate templates according to business needs for subsequent operations.

[0057] A user interface module provides a visual or command line interface for users to input business requirements. Users can input variable definition information such as variable name, type, dimension and description, as well as write SQL logic. The interface should include syntax highlighting, auto-completion and other functions to help users write correct SQL statements.

[0058] The user interface module is used to receive user input and pass the information to subsequent modules for processing. It also displays SQL syntax checking results and error information for user modification.

[0059] The SQL processing and verification module uploads the variable SQL generated by the user to the specified directory on HDFS. It calls the SQL parser of Hive or Spark to perform syntax verification on the uploaded SQL. It parses the table association relationship and field reference relationship in the SQL, constructs the bloodline graph between variable tables, and stores it in the graph database.

[0060] SQL processing and verification module receives SQL statements from user interface module, processes and verifies them. The verification results and blood relationship data are passed to subsequent modules. The scheduling task generation and execution module generates corresponding scheduling tasks according to the variable execution frequency configured by the user. The workflow is defined using the XML configuration file of Oozie, and the workflow.xml file of Oozie is dynamically generated. The generated workflow.xml file is submitted to the Oozie server through the Java API of Oozie, and the scheduling task is automatically executed according to the predetermined time schedule.

[0061] The SQL processing and verification module receives task information from the SQL processing and verification module, generates scheduling tasks, calls the Oozie server to execute the tasks, and passes the execution results to subsequent modules. The calculation data sources include real-time data from the submission center database and data obtained from external third-party systems in real time through the data gateway.

[0062] The data synchronization and storage module listens to the execution results of the scheduling tasks, and synchronizes the result data to HBase and Hive when the task execution is completed. In HBase, the row key is designed according to the dimension information in the variable definition to improve data query performance. In Hive, data is written into the Hive table through the LOAD DATA command or using Spark, so as to facilitate subsequent data analysis and query.

[0063] The data synchronization and storage module receives the execution results from the scheduling task generation and execution module, performs data synchronization and storage, and provides data query services for subsequent modules.

[0064] The API generation and deployment module generates the code framework of RESTful API using a template engine according to the variable definition information and dimension information. The generated API code is compiled and packaged, and deployed to the API gateway server through an automated deployment tool.

[0065] The API generation and deployment module receives the variable definition information from the data synchronization and storage module, generates the API code, and passes the deployed API information to the subsequent module for registration and publication.

[0066] The variable registration and publication module registers the generated API information in the API market system. The variable information is registered in the variable query platform, and the metadata information of the variable is encapsulated. After the user confirms that the variable information is correct, the variable is published to the variable calculation system.

[0067] The variable registration and publication module receives the API information and variable definition information from the API generation and deployment module. It provides a user interface for the user to confirm and publish the variable. The published variable information is passed to the variable calculation system.

[0068] The variable calculation and calling module receives the variable information from the variable registration and publishing module, and stores and caches the variable information. Other systems obtain the variable value by calling the API in the API market system. After receiving the request, the API market system forwards the request to the variable calculation system, the variable calculation system queries the corresponding variable value from HBase or Hive according to the request parameter, and returns the result to the API market system.

[0069] The variable calculation and calling module receives and stores the published variable information. In response to the request of the API market system, the variable value query service is provided.

[0070] From the input of the business requirement by the user to the final online calling of the variable, the whole process realizes the functions of the rapid definition, generation, scheduling, storage, API generation and deployment, registration and publishing, and online calling of the variable through the cooperative work of the various modules.

[0071] The above is only an embodiment of the present application, and the specific technical solutions and / or common knowledge of the scheme are not described in detail. It should be noted that for those skilled in the art, without departing from the technical solutions of the present application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

Claims

1. A method for Hadoop eco-component based variable one-click transfer online, characterized in that, Comprise: According to international standards, define a common variable SQL template, and store the variable SQL template in the system, obtain the user input business requirements through the interactive interface; According to the obtained business requirements, call the corresponding variable SQL template, and adjust and transform the variable SQL template to generate a variable SQL that meets the business requirements; Upload the generated variable SQL to the specified directory on HDFS, call the SQL parser of Hive or Spark to perform syntax checking on the uploaded SQL, and if the SQL syntax is correct, continue to execute the subsequent steps; Otherwise, return the error information to the user interface and prompt the user to modify the SQL; Parse the table association relationship and field reference relationship in the variable SQL, and construct the blood relationship graph between the variable tables; Store the generated blood relationship data into the graph database; According to the configured variable execution frequency, generate the corresponding scheduling task, and submit it to Oozie automatic execution; Synchronize the data of the scheduling execution result to the Hbase high-performance database; Automatically generate API according to the dimension as the input parameter and the variable column as the result; Register the variable to the variable query platform and encapsulate the variable metadata of the variable; Register and publish the variable to the variable calculation system for online calling at any time. 2.The method of claim 1, wherein the Hadoop eco-component-based variable one-key transfer online method is characterized in that: The fields contained in the definition of the common variable SQL template except the variable name are unified field names; When the user inputs variable definition information and SQL logic through the interactive interface, the unified field name must be used when the unified dimension information is involved.

3. The method for Hadoop eco-component-based variable one-key transfer online according to claim 2, characterized in that: Use the XML configuration file of Oozie to define the workflow according to the generated scheduling task information, convert the SQL into table ETL processing statements, and dynamically generate XML files according to the configured scheduling period. The system submits the generated XML file to the Oozie server through the Java API of Oozie. The Oozie server automatically executes the scheduling task according to the configuration information in the workflow.xml file and according to the predetermined time schedule.

4. The method and system of Hadoop eco-component based variable one-key transfer online according to claim 3, characterized in that: When generating API, use the preset API template to fill in the variable name, variable type and request parameter information according to the variable definition information and dimension information, and generate the complete API code; Compile and package the generated API code through the build tool, and deploy it to the API gateway server through the automated deployment tool.

5. A system for implementing the method of any one of claims 1-4 based on Hadoop eco-components based variable one-click online, characterized by, Comprise: Variable template management module, used for defining and storing common variable SQL templates, and the variable SQL template conforms to international standards; User interface module, used for inputting business requirements by users; SQL processing and checking module, receiving SQL statements from the user interface module, and uploading them to the specified directory on HDFS; Call the SQL parser of Hive or Spark to perform syntax checking on the uploaded SQL; Parse the table association relationship and field reference relationship in the SQL, construct the blood relationship graph between the variable tables, and store it into the graph database; The scheduling task generation and execution module generates corresponding scheduling tasks according to the variable execution frequency configured by the user, generates an XML file and submits it to the Oozie server, and calls the Oozie server to automatically execute the scheduling task according to the predetermined time schedule; The data synchronization and storage module listens to the execution result of the scheduling task, and synchronizes the result data to HBase and Hive when the task execution is completed; The API generation and deployment module generates the code framework of the API using a template engine according to the variable definition information and dimension information; The generated API code is compiled and packaged, and deployed to the API gateway server through an automated deployment tool; The variable registration and publishing module registers the generated API information in the API market system; The variable information is registered in the variable query platform, and the metadata information of the variable is encapsulated; After the user confirms that the variable information is correct, the variable is published to the variable calculation system and delivered to the variable calculation system; The variable calculation and calling module receives the variable information from the variable registration and publishing module, and stores and caches it; Other systems obtain variable values by calling APIs in the API market system.

6. The system of claim 5, wherein the system is based on Hadoop eco-components. After receiving the request, the API market system forwards the request to the variable calculation system, which queries the corresponding variable value from HBase or Hive according to the request parameters, and returns the result to the API market system; Receive and store the published variable information, and provide variable value query service in response to the request of the API market system.

7. The system of claim 6, wherein the system is based on Hadoop eco-components. In the variable template management module, the template is stored in the metadata storage system related to Hive or Spark, which is used for retrieval and use; The user interface module allows users to input variable definition information, including variable name, type, dimension, and description, as well as write SQL logic. 8.The system of claim 7, wherein: The scheduling task generation and execution module uses the Oozie XML configuration file to define the workflow, and dynamically generates the Oozie workflow.xml file; The generated workflow.xml file is submitted to the Oozie server through the Oozie Java API, and the scheduling task is automatically executed according to the predetermined time schedule; The Oozie server executes the task, and the data source includes real-time data from the submission center database and data obtained from external third-party systems through the data gateway. 9.The system of claim 8, wherein: Users can replace placeholders in the SQL template according to actual business needs.

Images