Business processing plug-in generation method and device, equipment, medium and product
By dynamically configuring the initial business processing plug-in developed by Kettle, the target business processing plug-in is generated, which solves the problem of time-consuming, labor-intensive and error-prone traditional data processing processes, and improves the stability and flexibility of the business system.
Patent Information
- Application Number
- CN202411188630.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2025-02-11
AI Technical Summary
Traditional data processing processes require manual writing of complex data processing scripts, which are time-consuming and labor-intensive, prone to errors, and are difficult to deal with changes in different data sources and processing processes.
By analyzing the metadata information and processing logic of the target business, dynamically configure the initial business processing plug-in developed by Kettle, the data integration tool, and generate the target business processing plug-in to improve the maintainability and flexibility of the plug-in.
It reduces the time and errors of manual scripting, improves the stability and flexibility of the business system, and can cope with changes in different data sources and processing processes.
Smart Images
Figure CN119415166A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of big data processing technology, and in particular to a method, device, equipment, medium and product for generating a business processing plug-in. Background Art
[0002] In the era of big data, the processing flow of business data has become increasingly complex and large. Currently, programmers usually write data processing scripts and execute them to process business data. However, traditional data processing processes usually require manual writing of complex data processing scripts, which is not only time-consuming and labor-intensive, but also prone to errors. In addition, as data processing requirements change, existing data processing scripts are difficult to cope with changes in different data sources and processing flows, and may need to be frequently modified and maintained. However, manual maintenance of scripts is prone to omissions or errors, resulting in system instability. Summary of the invention
[0003] The embodiments of the present invention provide a method, apparatus, device, medium and product for generating a business processing plug-in, which can automatically generate a target business processing plug-in, improve the maintainability and flexible adaptability of the target business processing plug-in, and further improve the stability of the business system.
[0004] In a first aspect, an embodiment of the present invention provides a method for generating a business processing plug-in, the method comprising:
[0005] Based on the business type of the target business, an initial business processing plug-in corresponding to the business type is obtained, where the initial business processing plug-in is developed using the data integration tool Kettle;
[0006] Analyze the target business to obtain metadata information and processing logic corresponding to the target business, wherein the metadata information at least includes data source metadata, output metadata, and scheduling metadata;
[0007] configuring a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic;
[0008] configuring a target data output component for the initial business processing plug-in based on the output metadata;
[0009] Configuring a target service scheduling component for the initial service processing plug-in based on the scheduling metadata;
[0010] A target business processing plug-in for processing the target business is determined according to the target data processing component, the target data output component and the target business scheduling component.
[0011] In a second aspect, an embodiment of the present invention further provides a device for generating a business processing plug-in, the device comprising:
[0012] An initial plug-in acquisition module, used to acquire an initial business processing plug-in corresponding to a target business type based on the business type of the target business, wherein the initial business processing plug-in is developed using the data integration tool Kettle;
[0013] A business data analysis module, used to analyze the target business to obtain metadata information and processing logic corresponding to the target business, wherein the metadata information at least includes data source metadata, output metadata and scheduling metadata;
[0014] A first component configuration module, configured to configure a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic;
[0015] A second component configuration module, configured to configure a target data output component for the initial business processing plug-in based on the output metadata;
[0016] A third component configuration module, configured to configure a target business scheduling component for the initial business processing plug-in based on the scheduling metadata;
[0017] The target plug-in generation module is used to determine the target business processing plug-in for processing the target business according to the target data processing component, the target data output component and the target business scheduling component.
[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for generating a business processing plug-in as described in any one of the embodiments of the present invention is implemented.
[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating a business processing plug-in as described in any one of the embodiments of the present invention.
[0020] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a method for generating a business processing plug-in as described in any one of the embodiments of the present invention.
[0021] In an embodiment of the present invention, an initial business processing plug-in corresponding to a business type is obtained based on the business type of the target business; the target business is analyzed to obtain metadata information and processing logic corresponding to the target business; a target data processing component is configured for the initial business processing plug-in based on the data source metadata and processing logic; a target data output component is configured for the initial business processing plug-in based on the output metadata; a target business scheduling component is configured for the initial business processing plug-in based on the scheduling metadata; and a target business processing plug-in for processing the target business is determined based on the target data processing component, the target data output component, and the target business scheduling component. The present application constructs a corresponding initial business processing plug-in including an infrastructure for each business type in the business system in advance; then the data structure and business rules of the target business are analyzed to obtain metadata information and processing logic describing the target business; the target data processing component, the target data output component, and the target business scheduling component are dynamically configured for the initial business processing plug-in based on the metadata information and processing logic, thereby obtaining a target business processing plug-in for processing the target business. This application analyzes the data processing requirements of the target business and dynamically generates a target business processing plug-in in the form of a plug-in in Kettle, thereby solving the problem in the prior art that the data processing process usually requires manual writing of complex data processing scripts, which is time-consuming, labor-intensive and error-prone. This application generates a target business processing plug-in by driving the metadata and processing logic of the target business, which can cope with changes in different data sources and processing processes, improves the maintainability and flexible adaptability of the target business processing plug-in, and thereby improves the stability of the business system. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A first flow chart of a method for generating a business processing plug-in provided in an embodiment of the present application;
[0024] Figure 2 A second flow chart of a method for generating a business processing plug-in provided in an embodiment of the present application;
[0025] Figure 3 A third flow chart of a method for generating a business processing plug-in provided in an embodiment of the present application;
[0026] Figure 4 A schematic diagram of the structure of a device for generating a business processing plug-in provided in an embodiment of the present application;
[0027] Figure 5 It is a block diagram of an electronic device used to implement a method for generating a business processing plug-in in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0029] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. The purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or must use the relevant content of the solution.
[0030] Figure 1 The first flow chart of a method for generating a business processing plug-in provided in an embodiment of the present application is applicable to the case where a corresponding target business processing plug-in is dynamically generated according to the metadata information and processing logic of the target business. The method for generating a business processing plug-in provided in this embodiment can be executed by a business processing plug-in generating device provided in an embodiment of the present application, which can be implemented by software and / or hardware and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a microprocessor of a business system.
[0031] See also Figure 1 The method of this embodiment includes but is not limited to the following steps:
[0032] S110 . Acquire an initial service processing plug-in corresponding to the service type based on the service type of the target service.
[0033] The target business refers to a user-oriented business in the business system. The business type depends on the business system, and the business types corresponding to different business systems are divided differently. If the business system is a financial system, the business types may include investment business, agency business, settlement business, and loan business. The embodiment of the present application takes the financial system as an example to explain the business processing plug-in generation method of the present application.
[0034] In the embodiment of the present application, a corresponding initial business processing plug-in is pre-built for each business type, and one business type in the business system corresponds to one initial business processing plug-in. The initial business processing plug-in is a universal exclusive template plug-in for processing a specific business type, and is a plug-in that only contains the infrastructure but is not instantiated. The microprocessor first determines the business type corresponding to the target business, and then obtains the initial business processing plug-in corresponding to the business type from a storage unit for storing plug-ins or a database for maintaining plug-ins based on the business type.
[0035] Optionally, the initial business processing plug-in in this embodiment can be developed using a data integration tool (Pentaho Data Integration, Kettle) to build a basic template plug-in for a general Kettle transformation and job. The plug-in can include multiple reusable components and support customized configuration components according to business needs to adapt to different usage scenarios.
[0036] S120: Analyze the target business to obtain metadata information and processing logic corresponding to the target business.
[0037] In the embodiment of the present application, the microprocessor analyzes the data structure and business rules of the target business to determine the metadata information and processing logic describing the target business. The purpose of analyzing the target business in this step is to dynamically configure the initial business processing plug-in according to the metadata information and processing logic in the subsequent steps to obtain the target business processing plug-in for processing the target business.
[0038] The metadata information in this embodiment may include at least data source metadata, output metadata and scheduling metadata. Processing logic refers to the method and steps for processing data of the target business, such as data sorting, data cleaning, data calculation, data conversion, data analysis and data visualization.
[0039] Specifically, data source metadata may include information such as business data tables, locations of business data tables (such as database table names, file paths, etc.), data types, etc. Output metadata may include definitions of target output formats and target storage paths for data processing results, such as database tables, file paths, etc. Scheduling metadata may include definitions of scheduling policies for data processing, such as the frequency and time of executing business processing tasks every day, month, or year.
[0040] Optionally, the metadata information can be stored in a centralized configuration table, such as a configuration table in a database or a configuration file (such as an Object Notation (JSON) or Extensible Markup Language (XML) file), which can be easily read dynamically by the Kettle process at runtime.
[0041] S130: configuring a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic.
[0042] The initial business processing plug-in in this embodiment may include an initial data extraction component and an initial data processing component.
[0043] Specifically, the microprocessor configures the target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic, including: first, reading the data source metadata through the Table Input or File Input component, configuring the first component parameter for the initial data extraction component based on the data source metadata through the Table Input component, that is, instantiating the initial data extraction component to obtain the target data extraction component; wherein the first component parameter may include a structured query language (SQL) query statement, and the target data extraction component is used to extract relevant data from the business data table related to the target business. Then, the target data extraction component is automatically executed to extract relevant data from the data warehouse corresponding to the target business, and the relevant data is summarized into business wide table data. Finally, the initial data processing component is configured with the second component parameter based on the processing logic and the business wide table data through the User Defined Java Expression or Calculator component to obtain the target data processing component; wherein the second component parameter includes a mapping relationship and a data processing task, and the data processing task refers to which data in the business wide table data is specifically processed by which data processing method, and the data processing method may include any combination of at least one of data sorting, data cleaning, data calculation, data conversion, data analysis and data visualization.
[0044] S140: configuring a target data output component for the initial business processing plug-in based on the output metadata.
[0045] The initial business processing plug-in in this embodiment may also include an initial data output component, which is used to format the business processing result of the target business, dynamically output it and store it in a specified location.
[0046] Specifically, the microprocessor configures a target data output component for the initial business processing plug-in based on the output metadata, including: first reading the output metadata through the Table Input or File Input component, and determining the target output format and the target storage path based on the output metadata; and then configuring the third component parameter for the initial data output component based on the target output format and the target storage path through the Table Output or File Output component, that is, instantiating the initial data output component to obtain the target data output component. The third component parameter can be a parameter for instantiating the initial data output component.
[0047] S150: configuring a target service scheduling component for the initial service processing plug-in based on the scheduling metadata.
[0048] The initial service processing plug-in in this embodiment may further include an initial service scheduling component, which is used to automatically generate scheduled tasks according to the scheduling configuration in the scheduling metadata and integrate the scheduled tasks into the operating system or scheduling system.
[0049] Specifically, the microprocessor configures the target business scheduling component for the initial business processing plug-in based on the scheduling metadata, including: first, reading the scheduling metadata through the Table Input or File Input component, and formulating a scheduling strategy for the target business based on the scheduling metadata. The scheduling strategy depends on the business type of the target business. The scheduling strategy may include scheduling time and scheduling frequency, for example: executing the target business processing plug-in of the target business at 3 am every 5 days. Then, based on the scheduling strategy, the fourth component parameter is configured for the initial business scheduling component, that is, the initial business scheduling component is instantiated to obtain the target business scheduling component; wherein the fourth component parameter may be a parameter for instantiating the initial business scheduling component.
[0050] Optionally, use Kettle's command line tools Kitchen and Pan, combined with the operating system's scheduling tools, to automate the scheduling of target business processing plug-ins. Scheduling tasks can include scheduled scheduling subtasks and monitoring subtasks; the scheduled scheduling subtask is to trigger Kettle jobs on a regular basis through Cron (Linux) or Task Scheduler (Windows); the monitoring subtask is to monitor the automated process and notify exceptions through Kettle's Mail component combined with the log component.
[0051] S160: Determine a target business processing plug-in for processing a target business according to a target data processing component, a target data output component, and a target business scheduling component.
[0052] In the embodiment of the present application, after the initial data processing component, the initial data output component and the initial business scheduling component are instantiated respectively through the above steps, the target data processing component, the target data output component and the target business scheduling component are obtained, and the target business processing plug-in for processing the target business is obtained. Processing the target business through the target business processing plug-in can effectively reduce duplication of work, especially when processing large-scale data integration tasks, which can improve the efficiency of data processing.
[0053] The technical solution provided in this embodiment obtains the initial business processing plug-in corresponding to the business type based on the business type of the target business; analyzes the target business to obtain the metadata information and processing logic corresponding to the target business; configures the target data processing component for the initial business processing plug-in based on the data source metadata and processing logic; configures the target data output component for the initial business processing plug-in based on the output metadata; configures the target business scheduling component for the initial business processing plug-in based on the scheduling metadata; determines the target business processing plug-in for processing the target business according to the target data processing component, the target data output component and the target business scheduling component. This application pre-constructs a corresponding initial business processing plug-in containing an infrastructure for each business type in the business system; then analyzes the data structure and business rules of the target business to obtain the metadata information and processing logic describing the target business; dynamically configures the target data processing component, the target data output component and the target business scheduling component for the initial business processing plug-in according to the metadata information and processing logic, thereby obtaining the target business processing plug-in for processing the target business. This application analyzes the data processing requirements of the target business and dynamically generates a target business processing plug-in in the form of a plug-in in Kettle, thereby solving the problem in the prior art that the data processing process usually requires manual writing of complex data processing scripts, which is time-consuming, labor-intensive and error-prone. This application generates a target business processing plug-in by driving the metadata and processing logic of the target business, which can cope with changes in different data sources and processing processes, improves the maintainability and flexible adaptability of the target business processing plug-in, and thereby improves the stability of the business system.
[0054] The following further describes the method for generating a business processing plug-in provided in an embodiment of the present application. Figure 2 A second flow chart of a method for generating a business processing plug-in provided in an embodiment of the present application is provided. The embodiment of the present application is optimized on the basis of the above embodiments, and the specific optimization is as follows: the present embodiment explains the configuration process of the target data processing component in detail.
[0055] See also Figure 2 The method of this embodiment includes but is not limited to the following steps:
[0056] S210: Determine a business primary key of the target business based on the data source metadata.
[0057] In this embodiment, the business primary key is the identification number (Identity document, ID) of the most critical or core data table of the target business. For example, if the target business is a budget business, the business primary key may be a budget preparation table ID.
[0058] S220. Generate an SQL query statement based on the business primary key and a preset script editing tool.
[0059] The preset script editing tool in this embodiment may be a Python script editing tool.
[0060] Specifically, the microprocessor generates an SQL query statement based on the business primary key and a preset script editing tool, including: the microprocessor determines the business data table associated with the business primary key and the key field corresponding to the business data table; generates an SQL query statement through the preset script editing tool, and the SQL query statement is used to extract the key value corresponding to the key field from the business data table.
[0061] For example, if the target business is a budget business, the business primary key may be the budget preparation table ID, and the business data tables associated with the budget preparation table may include: budget details table, budget unit configuration management table, organization information table, budget cycle type table, budget system master table, and budget system project setting table. Among them, the budget preparation table and the budget details table are associated through the budget preparation table ID and the budget_id of the budget details table, the unit_id of the budget preparation table is associated with the ID of the budget unit configuration management table, the budget_unit_id of the budget preparation table is associated with the ID of the organization information table, the cycle type ID (type_id) of the budget preparation table is associated with the ID of the budget cycle type table, the budget system ID (system_id) of the budget preparation table is associated with the ID of the budget system master table, and the ID of the budget system master table is associated with the father_id of the budget system project setting table. Therefore, the key fields corresponding to the business data table may include: organization code, organization name, budget year, execution amount, budget preparation amount, carry-over amount from the previous period, adjustment amount, transfer amount, budget type, budget project level, and non-business field pure data processing field.
[0062] S230. Configure the SQL query statement in the initial data extraction component to obtain the target data extraction component.
[0063] In the embodiment of the present application, the microprocessor configures the SQL query statement in the initial data extraction component through the Table Input component, that is, instantiates the initial data extraction component, and then obtains the target data extraction component. The target data extraction component is used to extract the key value corresponding to the key field from the business data table related to the target business.
[0064] S240 . Determine the business wide table data corresponding to the target business by executing the target data extraction component.
[0065] In the embodiment of the present application, the microprocessor first executes the target data extraction component to extract the key value corresponding to the key field. After that, the microprocessor summarizes the key field and its corresponding key value to obtain the business wide table data of the target business.
[0066] S250: Determine at least one business field and at least one data processing task for the business field corresponding to the processing logic.
[0067] The processing logic in this embodiment refers to the methods and steps for processing data of the target business, such as data sorting, data cleaning, data calculation, data conversion, data analysis and data visualization. Business fields refer to the fields in the target business that need to be processed. Data processing tasks refer to how to perform which data processing methods on these business fields. Data processing methods can include any combination of at least one of data sorting, data cleaning, data calculation, data conversion, data analysis and data visualization.
[0068] In an embodiment of the present application, the microprocessor analyzes the processing logic of the target business, determines which data processing steps (ie, data processing tasks) correspond to the target business, and the business fields included in each data processing step.
[0069] S260: Determine a mapping relationship of at least one business field in business wide table data.
[0070] In the embodiment of the present application, the microprocessor determines a one-to-one correspondence between each business field and a key field and its corresponding key value in the business wide table data to obtain a mapping relationship. The mapping relationship is used to represent the correspondence between the business field and the key field in the business wide table data.
[0071] S270: Configure a mapping relationship and a data processing task for the initial data processing component to obtain a target data processing component.
[0072] In the embodiment of the present application, the microprocessor configures the mapping relationship and data processing tasks for the initial data processing component through the Table Input component, that is, instantiates the initial data processing component to obtain the target data processing component.
[0073] The technical solution provided in this embodiment determines the business primary key of the target business based on the data source metadata; generates an SQL query statement based on the business primary key and a preset script editing tool; configures the SQL query statement in the initial data extraction component to obtain the target data extraction component; determines the business wide table data corresponding to the target business by executing the target data extraction component; determines at least one business field and at least one data processing task of the business field corresponding to the processing logic; determines the mapping relationship of at least one business field in the business wide table data; configures the mapping relationship and the data processing task for the initial data processing component to obtain the target data processing component.
[0074] This application first determines the business primary key of the target business through the data source metadata, and then determines the business data table associated with the business primary key and the key fields corresponding to the business data table, and then generates an SQL query statement for extracting the key values corresponding to the key fields from the business data table, and finally obtains the target data extraction component; obtains the business wide table data corresponding to the target business through the target data extraction component, and then obtains the mapping relationship and data processing tasks in combination with the processing logic, and finally obtains the target data processing component. This application obtains the data source metadata and processing logic by analyzing the data processing requirements of the target business, and configures the target data processing component for the initial business processing plug-in based on the data source metadata and processing logic, which solves the problem in the prior art that the data processing process usually requires manual writing of complex data processing scripts, which is time-consuming, labor-intensive and prone to errors; the target business processing plug-in generated by this application can cope with changes in different data sources and processing processes, improves the maintainability and flexible adaptability of the target business processing plug-in, and thus can improve the stability of the business system.
[0075] In an optional embodiment, when the target business is a budget business, at least one business field includes at least the budget execution rate, the total budget amount, the budget execution amount, the original preparation amount, the previous period carry-over amount and the adjustment amount, and the data processing task is to determine the budget execution rate.
[0076] The data processing tasks are determined in the following way: First, the original preparation amount, the carry-over amount from the previous period, and the adjustment amount are calculated to obtain the budget amount corresponding to a single budget system in the target budget unit. The calculation formula can be: budget amount = original preparation amount + carry-over amount from the previous period + adjustment amount. Then, when the status of multiple budget systems in the target budget unit is effective, the highest level (that is, when there are annual, quarterly, and monthly budgets, the annual budget is taken) is taken from the budget amounts corresponding to the multiple budget systems as the total budget amount. Secondly, the budget execution amount corresponding to the target budget unit is counted from the business data table corresponding to the budget business. It can be: the sum of the amounts occupied by the budget execution details table of the selected budget unit is counted according to the time period. Finally, the budget execution amount and the total budget amount are calculated to obtain the budget execution rate. The calculation formula can be: budget execution rate = sum of budget execution amounts / total budget amount.
[0077] In another optional embodiment, when the target business is a fund concentration business, at least one business field includes at least account information, account balance and exchange rate value, and the data processing task is to determine the collected funds, uncollected funds and restricted funds of the target account.
[0078] The data processing tasks are determined in the following manner: the account balance of the target account within the preset transaction time period is counted from the business data table corresponding to the funds concentration business; the account balance and exchange rate value are calculated based on the preset calculation formula to obtain the collected funds, uncollected funds and restricted funds.
[0079] The following further describes the method for generating a business processing plug-in provided in an embodiment of the present application. Figure 3 The third flow diagram of a method for generating a business processing plug-in provided in an embodiment of the present application is as follows: the present embodiment is optimized on the basis of the above embodiments, and the specific optimization is as follows: the present embodiment explains the construction and deployment process of the initial business processing plug-in in detail.
[0080] See also Figure 3 The method of this embodiment includes but is not limited to the following steps:
[0081] S310: Construct a first business processing plug-in, and embed an execution script and a call operation on the execution script into the first business processing plug-in to obtain a second business processing plug-in.
[0082] The first business processing plug-in in this embodiment can be a custom plug-in in Kettle. Kettle supports custom plug-ins. When automatically generating scripts, custom logic or functions can be integrated according to specific needs.
[0083] Exemplarily, the construction process of the first business processing plug-in can be: Use an integrated development environment (IDE) tool such as Eclipse or IntelliJ IDEA to create a new Java project in the IDE for developing a Kettle plug-in. Add the Kettle SDK and other related libraries to the project's build path. Create a Java class that inherits BaseStep, which can implement the core functions of the custom plug-in. Define the plug-in's metadata, such as a description, icon, etc.
[0084] The embodiment of the present application does not limit the scripting language of the execution script. Preferably, Python can be selected as the scripting language. It is necessary to write a Python script (i.e., an execution script) in the first business processing plug-in to generate a dynamic SQL query statement, and store the SQL query statement in the BUDGET_SQL variable of Kettle. It is also necessary to embed a call operation for the execution script in the first business processing plug-in, such as calling a Python script through Java to obtain a second business processing plug-in.
[0085] S320: After configuring control properties for the second business processing plug-in, a compile operation is performed to obtain an initial business processing plug-in.
[0086] Exemplarily, a custom control is created for the second business processing plug-in to implement the development of the custom plug-in, the implementation of the control logic, and the process of integration into the Kettle environment. The second business processing plug-in is compiled into a software package file format (Java Archive, JAR) file to obtain an initial business processing plug-in. Among them, the custom control can be a Kettle step for generating and executing a dynamic budget indicator script, which can be implemented through the Kettle plug-in framework and support Java development.
[0087] S330. Deploy the initial business processing plug-in to Kettle.
[0088] Exemplarily, copy the JAR file to the plugins / steps directory of Kettle and restart Kettle. Optionally, the initial business processing plug-in needs to be compatible with the core framework of Kettle to ensure that it can run normally in different versions of Kettle. The implementation logic of the initial business processing plug-in usually relies on Kettle's plug-in interface (such as StepInterface, JobEntryInterface, etc.). By implementing these interfaces, the plug-in can be seamlessly integrated with Kettle's core framework.
[0089] Run the initial business processing plug-in in the Kettle development environment to verify whether the script generation and data processing are correct. You can also set up job scheduling tasks to automatically execute business processing tasks for the target business and verify the business processing results.
[0090] In some optional embodiments, after building the target business processing plug-in, the user triggers a processing request for the target business in the client of the business system. When the client receives the processing request for the target business, the microprocessor obtains the target business processing plug-in corresponding to the business type based on the business type of the target business; runs the target business processing plug-in to obtain the business processing result of the target business; determines the risk level of the target business based on the business processing result, wherein the target business is a business to be risk managed; when the risk level exceeds the preset risk standard, generates risk warning information, and displays the risk warning information to the user.
[0091] The technical solution provided in this embodiment is to obtain a second business processing plug-in by constructing a first business processing plug-in, embedding an execution script and a call operation on the execution script for the first business processing plug-in; after configuring the control properties for the second business processing plug-in, a compilation operation is performed to obtain an initial business processing plug-in; and the initial business processing plug-in is deployed to Kettle. This application pre-builds the basic structure of the plug-in for each business type in the business system, writes Java code to dynamically generate SQL and processing logic, and defines plug-in metadata for the plug-in, packages it into a JAR file and deploys it to Kettle, so that the initial business processing plug-in can be dynamically configured according to the specific target business to obtain a target business processing plug-in for processing the target business. This application creates a highly customized Kettle plug-in to meet specific business needs, and various configuration items of the target business can be flexibly selected and integrated into the data processing workflow, which can solve the problem that the data processing process in the prior art usually requires manual writing of complex data processing scripts, which is time-consuming and error-prone; the target business processing plug-in generated by this application can cope with changes in different data sources and processing processes, improve the maintainability and flexible adaptability of the target business processing plug-in, and thus improve the stability of the business system.
[0092] Figure 4 A schematic diagram of a structure of a device for generating a business processing plug-in provided in an embodiment of the present application, such as Figure 4 As shown, the apparatus 400 may include:
[0093] An initial plug-in acquisition module 410 is used to acquire an initial business processing plug-in corresponding to a target business type based on the business type of the target business, wherein the initial business processing plug-in is developed using a data integration tool Kettle;
[0094] The business data analysis module 420 is used to analyze the target business to obtain metadata information and processing logic corresponding to the target business, wherein the metadata information at least includes data source metadata, output metadata, and scheduling metadata;
[0095] A first component configuration module 430, configured to configure a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic;
[0096] A second component configuration module 440, configured to configure a target data output component for the initial business processing plug-in based on the output metadata;
[0097] A third component configuration module 450 is used to configure a target business scheduling component for the initial business processing plug-in based on the scheduling metadata;
[0098] The target plug-in generation module 460 is used to determine a target business processing plug-in for processing the target business according to the target data processing component, the target data output component and the target business scheduling component.
[0099] Optionally, the initial business processing plug-in includes an initial data extraction component and an initial data processing component;
[0100] Furthermore, the above-mentioned first component configuration module 430 can be specifically used to: configure the first component parameters for the initial data extraction component based on the data source metadata to obtain the target data extraction component; determine the business wide table data corresponding to the target business by executing the target data extraction component; configure the second component parameters for the initial data processing component based on the processing logic and the business wide table data to obtain the target data processing component.
[0101] Optionally, the first component parameter includes a structured query language SQL query statement;
[0102] Furthermore, the above-mentioned first component configuration module 430 can also be specifically used to: determine the business primary key of the target business based on the data source metadata; generate the SQL query statement based on the business primary key and a preset script editing tool; configure the SQL query statement in the initial data extraction component to obtain the target data extraction component.
[0103] Optionally, the second component parameters include a mapping relationship and a data processing task;
[0104] Furthermore, the above-mentioned first component configuration module 430 can also be specifically used to: determine at least one business field corresponding to the processing logic and the data processing task of the at least one business field; determine the mapping relationship of the at least one business field in the business wide table data; configure the mapping relationship and the data processing task for the initial data processing component to obtain the target data processing component.
[0105] Furthermore, the above-mentioned first component configuration module 430 can also be specifically used to: determine the business data table associated with the business primary key and the key field corresponding to the business data table; generate the SQL query statement through the preset script editing tool, and the SQL query statement is used to extract the key value corresponding to the key field from the business data table.
[0106] In one embodiment, when the target business is a budget business, the at least one business field includes at least a budget execution rate, a total budget quota, a budget execution amount, an original preparation amount, an amount carried over from the previous period, and an adjustment amount, and the data processing task is to determine the budget execution rate; the data processing task is determined in the following manner: data processing is performed on the original preparation amount, the amount carried over from the previous period, and the adjustment amount to obtain the budget amount corresponding to a single budget system in the target budget unit; when the status of multiple budget systems in the target budget unit is effective, the total budget quota is determined from the budget amounts corresponding to the multiple budget systems; the budget execution amount corresponding to the target budget unit is counted from the business data table corresponding to the budget business; data processing is performed on the budget execution amount and the total budget quota to obtain the budget execution rate.
[0107] In one embodiment, when the target business is a fund concentration business, the at least one business field includes at least account information, account balance and exchange rate value, and the data processing task is to determine the collected funds, uncollected funds and restricted funds of the target account; the data processing task is determined in the following manner: counting the account balance of the target account within a preset transaction time period from the business data table corresponding to the fund concentration business; performing data processing on the account balance and the exchange rate value to obtain the collected funds, the uncollected funds and the restricted funds.
[0108] Furthermore, the above-mentioned device for generating a business processing plug-in may also include: an initial plug-in generating module;
[0109] The initial plug-in generation module is used to build a first business processing plug-in before obtaining the initial business processing plug-in corresponding to the business type based on the business type of the target business, and embed an execution script and a call operation on the execution script for the first business processing plug-in to obtain a second business processing plug-in; configure control properties for the second business processing plug-in and then perform a compilation operation to obtain the initial business processing plug-in; deploy the initial business processing plug-in to Kettle.
[0110] Optionally, the initial business processing plug-in further includes an initial data output component;
[0111] Furthermore, the above-mentioned second component configuration module 440 can be specifically used to: determine the target output format and the target storage path based on the output metadata; configure the third component parameters for the initial data output component based on the target output format and the target storage path to obtain the target data output component.
[0112] Optionally, the initial service processing plug-in further includes an initial service scheduling component;
[0113] Furthermore, the third component configuration module 450 can be specifically used to: formulate a scheduling strategy for the target business based on the scheduling metadata; configure fourth component parameters for the initial business scheduling component based on the scheduling strategy to obtain the target business scheduling component.
[0114] Furthermore, the above-mentioned business processing plug-in generating device may also include: a risk warning module;
[0115] The risk warning module is used to, when receiving a processing request for the target business, obtain a target business processing plug-in corresponding to the business type based on the business type of the target business; run the target business processing plug-in to obtain a business processing result of the target business; determine the risk level of the target business based on the business processing result, the target business being a business to be risk managed; and generate risk warning information when the risk level exceeds a preset risk standard.
[0116] The device for generating a business processing plug-in provided in this embodiment can be applied to the method for generating a business processing plug-in provided in any of the above embodiments, and has corresponding functions and beneficial effects.
[0117] Figure 5 It is a block diagram of an electronic device for implementing a method for generating a business processing plug-in of an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0118] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for generating a business processing plug-in.
[0121] In some embodiments, the method for generating a business processing plug-in may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for generating a business processing plug-in described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for generating a business processing plug-in in any other appropriate manner (e.g., by means of firmware).
[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0123] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0124] In the context of the present application, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0126] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server) or a computing system that includes middleware components (e.g., an application server) or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein) or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0128] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a method for generating a business processing plug-in as provided in any embodiment of the present application.
[0129] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0130] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A method for generating a business processing plug-in, characterized in that: include: Based on the business type of the target business, an initial business processing plug-in corresponding to the business type is obtained, where the initial business processing plug-in is developed using the data integration tool Kettle; Analyze the target business to obtain metadata information and processing logic corresponding to the target business, wherein the metadata information at least includes data source metadata, output metadata, and scheduling metadata; configuring a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic; configuring a target data output component for the initial business processing plug-in based on the output metadata; Configuring a target service scheduling component for the initial service processing plug-in based on the scheduling metadata; A target business processing plug-in for processing the target business is determined according to the target data processing component, the target data output component and the target business scheduling component.
2. The method according to claim 1, characterized in that The initial business processing plug-in includes an initial data extraction component and an initial data processing component; Configuring a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic includes: Based on the data source metadata, configure a first component parameter for the initial data extraction component to obtain the target data extraction component; Determine the business wide table data corresponding to the target business by executing the target data extraction component; The second component parameters are configured for the initial data processing component based on the processing logic and the business wide table data to obtain the target data processing component.
3. The method according to claim 2, characterized in that The first component parameter includes a structured query language SQL query statement; configuring the first component parameter for the initial data extraction component based on the data source metadata to obtain the target data extraction component includes: Determine the business primary key of the target business based on the data source metadata; Generate the SQL query statement based on the business primary key and a preset script editing tool; The SQL query statement is configured in the initial data extraction component to obtain the target data extraction component.
4. The method according to claim 2, characterized in that: The second component parameters include a mapping relationship and a data processing task; configuring the second component parameters for the initial data processing component based on the processing logic and the business wide table data to obtain the target data processing component includes: Determine at least one business field corresponding to the processing logic and a data processing task for the at least one business field; Determine a mapping relationship of the at least one business field in the business wide table data; The mapping relationship and the data processing task are configured for the initial data processing component to obtain the target data processing component.
5. The method according to claim 3, characterized in that: Generating the SQL query statement based on the business primary key and a preset script editing tool includes: Determine a business data table associated with the business primary key and a key field corresponding to the business data table; The SQL query statement is generated by the preset script editing tool, and the SQL query statement is used to extract the key value corresponding to the key field from the business data table.
6. The method according to claim 4, characterized in that When the target business is a budget business, the at least one business field includes at least a budget execution rate, a total budget quota, a budget execution amount, an original budget amount, a previous period carry-over amount, and an adjustment amount, and the data processing task is to determine the budget execution rate; the data processing task is determined in the following manner: Performing data processing on the original budget amount, the previous period carry-over amount and the adjustment amount to obtain the budget amount corresponding to a single budget system in the target budget unit; When the status of multiple budget systems in the target budget unit is effective, determining the total budget amount from the budget amounts corresponding to the multiple budget systems; Counting the budget execution amount corresponding to the target budget unit from the business data table corresponding to the budget business; The budget execution amount and the total budget amount are processed to obtain the budget execution rate.
7. The method according to claim 4, characterized in that When the target business is a fund concentration business, the at least one business field includes at least account information, account balance and exchange rate value, and the data processing task is to determine the collected funds, uncollected funds and restricted funds of the target account; the data processing task is determined in the following manner: Counting the account balance of the target account within a preset transaction time period from the business data table corresponding to the fund concentration business; Data processing is performed on the account balance and the exchange rate value to obtain the collected funds, the uncollected funds and the restricted funds.
8. The method according to claim 1, characterized in that Before acquiring the initial service processing plug-in corresponding to the service type based on the service type of the target service, the method further includes: Constructing a first business processing plug-in, and embedding an execution script and a calling operation on the execution script into the first business processing plug-in to obtain a second business processing plug-in; After configuring the control properties for the second business processing plug-in, a compile operation is performed to obtain the initial business processing plug-in; Deploy the initial business processing plug-in into Kettle.
9. The method according to claim 1, characterized in that: The initial business processing plug-in also includes an initial data output component; Configuring a target data output component for the initial business processing plug-in based on the output metadata includes: Determine a target output format and a target storage path based on the output metadata; A third component parameter is configured for the initial data output component based on the target output format and the target storage path to obtain the target data output component.
10. The method according to claim 1, characterized in that The initial service processing plug-in also includes an initial service scheduling component; Configuring a target business scheduling component for the initial business processing plug-in based on the scheduling metadata includes: formulating a scheduling strategy for the target service based on the scheduling metadata; A fourth component parameter is configured for the initial service scheduling component based on the scheduling strategy to obtain the target service scheduling component.
11. The method according to claim 1, characterized in that: The method further comprises: When receiving a processing request for the target service, obtaining a target service processing plug-in corresponding to the service type based on the service type of the target service; Run the target business processing plug-in to obtain the business processing result of the target business; Determine the risk level of the target business according to the business processing result, the target business is the business to be risk managed; When the risk level exceeds the preset risk standard, risk warning information is generated.
12. A device for generating a business processing plug-in, characterized in that: include: An initial plug-in acquisition module, used to acquire an initial business processing plug-in corresponding to a target business type based on the business type of the target business, wherein the initial business processing plug-in is developed using the data integration tool Kettle; A business data analysis module, used to analyze the target business to obtain metadata information and processing logic corresponding to the target business, wherein the metadata information at least includes data source metadata, output metadata and scheduling metadata; A first component configuration module, configured to configure a target data processing component for the initial business processing plug-in based on the data source metadata and the processing logic; A second component configuration module, configured to configure a target data output component for the initial business processing plug-in based on the output metadata; A third component configuration module, configured to configure a target business scheduling component for the initial business processing plug-in based on the scheduling metadata; The target plug-in generation module is used to determine the target business processing plug-in for processing the target business according to the target data processing component, the target data output component and the target business scheduling component.
13. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a business processing plug-in according to any one of claims 1 to 11 when executing the computer program.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a business processing plug-in as claimed in any one of claims 1 to 11 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for generating a business processing plug-in according to any one of claims 1 to 11.