A method and system for realizing a task of quickly deploying a visual large screen index

Abstract

The present application relates to the technical field of computer, specifically to a method and system for realizing rapid deployment of visual large-screen index task, comprising an export step and an import step, wherein: in the source data governance platform, the following operations are performed: selecting an index space to be exported, and calling a data development export interface to asynchronously obtain confirmation information of successful interface call; task grouping information, task information, task configuration information and associated data source information under the index space are exported, and the information is organized into a task information file in json format; beneficial effects are as follows: the method and system for realizing rapid deployment of visual large-screen index task proposed by the present application, through the implementation of exporting index platform task templates and importing customer platforms, rapidly and efficiently solve the requirement of customers to visually display platform operation indexes, improve the use function of the platform, and truly achieve one-key deployment and rapid use effect.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a method and system for rapidly deploying visual large-screen metrics tasks. Background Technology

[0002] With the advent of the big data era, data development and sharing have become mainstream, and data visualization is becoming increasingly important across all industries. In the big data era, the operational data of data platforms need to be displayed on large screens, and indicators such as the amount of data managed by various systems, access volume, and the amount of data exchanged daily, weekly, and monthly all need to be visualized.

[0003] After deploying the platform, when it's necessary to use visualization functions to display the governance platform's business data, it's often necessary to reconfigure the relevant metrics and tasks before visualization can be performed. This implementation often consumes a lot of manpower and carries a high error rate. To address the complexity of reconfiguration, there is an urgent need to provide a set of export and import functions for visualization metrics and tasks, so that customers can use a large visualization dashboard to display operational data after the governance platform is deployed. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for rapidly deploying visual large-screen metrics tasks, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for rapidly deploying visualization dashboard metrics tasks, the method comprising an export step and an import step, wherein:

[0006] Export steps: In the source data governance platform, perform the following operations:

[0007] Select the indicator space to be exported, call the data development export interface, and asynchronously obtain the confirmation information that the interface has been successfully called.

[0008] Export task grouping information, task information, task configuration information and their associated data source information under the indicator space, and organize the information into a JSON format task information file;

[0009] Export the NIFI workflow template files for the workflow orchestration tasks. Each task template is named with the task's NIFI group ID and saved as an XML file.

[0010] Compress all files into a single zip file and upload the zip file to the minIo server;

[0011] The exported results and minIo upload results are pushed to the Redis server for later use.

[0012] Preferably, the JSON-formatted task information file jobInfo.json includes, but is not limited to: task grouping information, task information, task configuration information, and anonymized associated data source information.

[0013] Preferably, the following import steps are also included and executed in the target data governance platform:

[0014] Retrieve information about the zip file stored in minIo from Redis and download the zip file;

[0015] Select a target indicator space in the target data governance platform, and call the data development import interface, passing in the downloaded zip file and the correspondence between the original indicator space and the target space as parameters;

[0016] Based on the principle of minimizing the number of tasks and components, the NIFI node with the best resources is automatically selected for subsequent operations.

[0017] Save the data source information in the target space and create or update the NIFI controller information;

[0018] Parse the contents of the zip file, including task grouping information, task information, task configuration information, and NIFI process template file, update this information to adapt to the target environment, such as replacing group ID, NIFI group id, and controller ID, and import the updated information into the target data governance platform;

[0019] For process orchestration tasks, it is also necessary to replace the original controller ID in the NIFI process template with the controller ID of the new NIFI instance, and redeploy these processes on the NIFI node.

[0020] The message indicating successful import is pushed to the Redis server so that the management center can query the import status.

[0021] Preferably, in the import step, the update of task information includes, but is not limited to: updating the space to which the task belongs, the nifi instance, user information, replacing the group ID and nifi group id, and updating the data source ID and nifiControllerServiceId in the task configuration information when necessary.

[0022] Preferably, the method enables rapid migration and deployment of visualization dashboard metrics tasks across different data governance platform environments by providing one-click export and one-click import functions, ensuring task consistency and continuity.

[0023] A system for rapidly deploying visualization metrics on large screens, the system comprising:

[0024] A data export module is used in the source data governance platform to call the data development export interface according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and associated data source information under the indicator space, generating a JSON format task information file.

[0025] A data import module is used to download a zip file based on the minIo file information stored in Redis in the target data governance platform, select the target indicator space, and call the import interface developed by the data development team. The downloaded zip file and the correspondence between the original indicator space and the target space data source are passed in as parameters to perform the import operation.

[0026] Preferably, the data export module exports the NIFI process template file of the process orchestration task, compresses the NIFI process template file into a zip file and uploads it to the minIo server, and finally pushes the export result information and the minIo upload result to the Redis server.

[0027] Preferably, the JSON-formatted task information file jobInfo.json includes, but is not limited to, task grouping information, task information, task configuration information, and anonymized associated data source information.

[0028] Preferably, the data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of minimizing the number of tasks and components, saving or updating the data source and controller information of the target space, parsing the contents of the zip file, updating the task group, task, task configuration and NIFI process template information, replacing the necessary ID information to adapt to the target environment, and finally pushing the import success information to the Redis server.

[0029] Preferably, during the data import process, the data import module adds the target space, NIFI instance, and user information for data development SQL tasks, and replaces the group ID for import; for process orchestration tasks, it replaces the original controller ID in the NIFI process template with the controller ID of the new NIFI instance, and records the new NIFI group ID and the mapping relationship between the original component ID and the new component ID; at the same time, it updates the relevant information in the di_job and di_job_config tables, including space, NIFI instance, user information, group ID, NIFI group ID, and data source ID.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] This invention proposes a method and system for rapidly deploying visualized dashboard metrics tasks. By exporting metric platform task templates and importing them into the client's platform, it quickly and efficiently solves the client's requirement to visualize platform operational metrics, enhances platform functionality, and truly achieves one-click deployment and rapid use. The import function enables the deployment of governance platform projects and the visualization of platform metrics. This significantly improves work efficiency, reduces the complexity and error rate of manual operations, and ensures that clients can view the platform's operational status. It also promotes the deployment of the governance platform, enhances platform functionality, and increases the success rate of bidding. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the process of deriving the index space for this invention.

[0033] Figure 2 This is a flowchart of the target space import process for this invention;

[0034] Figure 3 This is a diagram showing the exported file structure of the present invention;

[0035] Figure 4 Generate a zip image for the exported file of this invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1, please refer to Figures 1 to 4 This invention provides a technical solution: a method for rapidly deploying visualization large-screen indicator tasks, the method comprising the following steps:

[0038] Exporting metrics from the platform:

[0039] 1. The management center calls the data development and export interface.

[0040] The data lake platform management center selects an indicator space, transmits the indicator space to call the data development interface, and asynchronously returns a message indicating that the interface call was successful.

[0041] 2. Export the data from the space task-related tables and generate a jobInfo.json file.

[0042] Export the di_job_folder task grouping information, filter by task, and export the cascading grouping information.

[0043] Export the di_job task information and di_job_config task configuration information, one data entry per task. The development SQL for data related to workflow orchestration also needs to be exported. At the same time, export the data source information associated with the task (including the mapped databaseId and nifiControllerServiceId).

[0044] Export associated data source information (anonymized export)

[0045] Generate a jobInfo.json structure from the above table information, as shown below:

[0046] {

[0047] "dataType":"dataDev",

[0048] "data":{

[0049] "datasource":[{

[0050] "datasourceId":"",

[0051] "datasourceName":"",

[0052] "datasourceDriver":"",

[0053] "dbtype":"",

[0054] "databaseName":"",

[0055] "schema":"",

[0056] "poolsize":"",

[0057] "lakeArea":"",

[0058] "incrementColumn":[{

[0059] "tableName":"",

[0060] "increColumn":"",

[0061] "comments":"

[0062] }

[0063]

[0064]

[0065]

[0066]

[0067] 3. Export task workflow template

[0068] Export the process orchestration task NIFI process template file (for integrated tasks, only table data is imported, and manual publishing is required after import). Each task template is named with the task NIFI group id, generating a process XML file.

[0069] 4. Generate a zip file

[0070] The structure of the exported information files for steps 2 and 3 is attached. Figure 3 Compress the file to generate a zip file, see attached. Figure 4 .

[0071] 5. Upload to the minio server

[0072] Upload the zip file generated in step 4 to the minIo server.

[0073] 6. Push Redis export information

[0074] The exported results and the minIo upload results are pushed to Redis.

[0075] Importing to the customer platform:

[0076] 1. Management Center calls data development and import interface

[0077] The management center downloads the exported file based on the minIo file information obtained from Redis beforehand.

[0078] Select the downloaded task file and call the data development import interface.

[0079] Select the target space to import the indicator, and obtain the data source correspondence between the original indicator space and the target space.

[0080] Using the above information as parameters, call the data development import interface and asynchronously return a message indicating that the interface has been successfully connected.

[0081] 2. Select NIFI node

[0082] The NIFI node with the most resources is selected based on the algorithm that minimizes the number of tasks and components.

[0083] 3. Save the data source and create the controller.

[0084] Before saving data source and controller information in the target space, deduplication is performed. If the data source does not exist, the data source information is saved, the controller is created in the NIFI, and the data source ID and controller ID of the target space are saved.

[0085] 4. Import data table information

[0086] First, process the di_job_folder (task grouping information table), add target space and user information, generate a new UUID as the group ID, and then import it.

[0087] Then iterate through the imported tasks:

[0088] The data development SQL task adds space, NIFI instance, and user information, and replaces the group ID by directly importing it into the table.

[0089] The process orchestration task replaces the original controller ID in the NIFI process template with the controller ID of the new NIFI instance, then imports the NIFI node, records the new NIFI group ID, and records the mapping relationship between the original component ID and the new component ID.

[0090] The di_job table records the increase in space, nifi instance, user information, replaces the group ID, and updates the task nifi groupid for import.

[0091] The di_job_config table records user information, replaces the group ID, updates the task nifi group ID, replaces the data source ID in the integrated task configuration details with the primary key of the integrated development data source, replaces nifiControllerServiceId with the controller ID corresponding to nifi, and replaces the data source ID in the SQL task configuration with the primary key of the integrated development data source for import.

[0092] 5. Push Redis import information

[0093] The message indicating a successful import is pushed to Redis, which is then used by the management center to retrieve the import status.

[0094] The above steps involve a one-click export of already running indicator environment tasks, downloading the process tasks and configuration information as files from minIo. These are then imported into the target environment with a single click. After execution, indicator-related tasks can be invoked in the target environment. This truly achieves rapid cross-environment deployment of indicator space tasks, enabling one-click deployment of indicator tasks efficiently and quickly. The execution results are shown in the attached document. Figure 3 .

[0095] Example 2, based on Example 1, proposes a system for rapidly deploying visual large-screen metrics tasks. The system includes:

[0096] A data export module is used in the source data governance platform to call the data development export interface according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and associated data source information under the indicator space, generating a JSON format task information file.

[0097] The data export module exports the NIFI process template file for the process orchestration task, compresses the NIFI process template file into a zip file and uploads it to the minIo server. Finally, the export result information and the minIo upload result are pushed to the Redis server.

[0098] The JSON-formatted task information file jobInfo.json contains, but is not limited to, task grouping information, task information, task configuration information, and anonymized associated data source information.

[0099] A data import module is used to download a zip file based on the minIo file information stored in Redis in the target data governance platform, select the target indicator space, and call the import interface developed by the data development, passing in the downloaded zip file and the correspondence between the original indicator space and the target space as parameters to perform the import operation;

[0100] The data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of minimizing the number of tasks and components, saving or updating the data source and controller information of the target space, parsing the contents of the zip file, updating the task group, task, task configuration and NIFI process template information, and replacing the necessary ID information to adapt to the target environment. Finally, it pushes the information of successful import to the Redis server.

[0101] During the data import process, the module adds the target space, NIFI instance, and user information for data development SQL tasks, and replaces the group ID for import. For workflow orchestration tasks, it replaces the original controller ID in the NIFI workflow template with the controller ID of the new NIFI instance, and records the new NIFI group ID and the mapping relationship between the original component ID and the new component ID. At the same time, it updates the relevant information in the di_job and di_job_config tables, including space, NIFI instance, user information, group ID, NIFI group ID, and data source ID.

[0102] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for rapidly deploying visualization dashboard metrics tasks, characterized in that: The method includes an export step and an import step, wherein: Export steps: In the source data governance platform, perform the following operations: Select the indicator space to be exported, call the data development export interface, and asynchronously obtain the confirmation information that the interface has been successfully called. Export task grouping information, task information, task configuration information and their associated data source information under the indicator space, and organize the information into a JSON format task information file; Export the NIFI workflow template files for the workflow orchestration tasks. Each task template is named with the task's NIFI group ID and saved as an XML file. Compress all files into a single zip file and upload the zip file to the minIo server; The exported results and minIo upload results are pushed to the Redis server for later use. The following import steps are also included and performed in the target data governance platform: Retrieve information about the zip file stored in minIo from Redis and download the zip file; Select a target indicator space in the target data governance platform, and call the data development import interface, passing in the downloaded zip file and the correspondence between the original indicator space and the target space as parameters; Based on the principle of minimizing the number of tasks and components, the NIFI node with the best resources is automatically selected for subsequent operations. Save the data source information in the target space and create or update the NIFI controller information; Parse the contents of the zip file, including task grouping information, task information, task configuration information, and NIFI process template file, update this information to adapt to the target environment, such as replacing group ID, NIFI group id, and controller ID, and import the updated information into the target data governance platform; For process orchestration tasks, it is also necessary to replace the original controller ID in the NIFI process template with the controller ID of the new NIFI instance, and redeploy these processes on the NIFI node. The message indicating successful import is pushed to the Redis server so that the management center can query the import status.

2. The method for rapidly deploying visual large-screen indicator tasks according to claim 1, characterized in that: The JSON-formatted task information file jobInfo.json contains, but is not limited to: task grouping information, task information, task configuration information, and anonymized associated data source information.

3. The method for rapidly deploying visual large-screen indicator tasks according to claim 2, characterized in that: During the import process, updating task information includes, but is not limited to: updating the space to which the task belongs, the nifi instance, user information, replacing the group ID and nifi group id, and updating the data source ID and nifiControllerServiceId in the task configuration information when necessary.

4. The method for rapidly deploying visual large-screen indicator tasks according to claim 1, characterized in that: The method enables rapid migration and deployment of visualization dashboard metrics tasks across different data governance platform environments by providing one-click export and one-click import functions, ensuring task consistency and continuity.

5. A system for implementing a method for rapidly deploying visual large-screen indicator tasks according to any one of claims 1-4, characterized in that: The system includes: A data export module is used in the source data governance platform to call the data development export interface according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and associated data source information under the indicator space, generating a JSON format task information file. A data import module is used to download a zip file based on the minIo file information stored in Redis in the target data governance platform, select the target indicator space, and call the import interface developed by the data development team. The downloaded zip file and the correspondence between the original indicator space and the target space data source are passed in as parameters to perform the import operation.

6. A system for rapidly deploying visual large-screen metrics tasks according to claim 5, characterized in that: The data export module exports the NIFI workflow template file for the workflow orchestration task, compresses the NIFI workflow template file into a zip file and uploads it to the minIo server, and finally pushes the export result information and the minIo upload result to the Redis server.

7. A system for rapidly deploying visual large-screen metrics tasks according to claim 5, characterized in that: The JSON-formatted task information file jobInfo.json contains, but is not limited to, task grouping information, task information, task configuration information, and anonymized associated data source information.

8. A system for rapidly deploying visual large-screen indicator tasks according to claim 5, characterized in that: The data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of minimizing the number of tasks and components, saving or updating the data source and controller information of the target space, parsing the contents of the zip file, updating the task group, tasks, task configuration and NIFI process template information, and replacing the necessary ID information to adapt to the target environment. Finally, it pushes the information of successful import to the Redis server.

9. A system for rapidly deploying visual large-screen metrics tasks according to claim 5, characterized in that: During the import process, the data import module adds the target space, NIFI instance, and user information to the data development SQL task and replaces the group ID before importing. For workflow orchestration tasks, replace the original controller ID in the NIFI workflow template with the controller ID of the new NIFI instance, and record the new NIFI group ID and the mapping relationship between the original component ID and the new component ID; at the same time, update the relevant information in the di_job and di_job_config tables, including space, NIFI instance, user information, group ID, NIFI group ID, and data source ID.

Images