Method and system for realizing rapid deployment of visual large screen index task

Abstract

The invention relates to the technical field of computers, in particular to a method and a system for quickly deploying visual large screen index tasks, comprising an export step and an import step, in the export step, in a source data management platform, executing the following operations: selecting an index space to be exported, calling a data development export interface, and executing the export step and the import step; asynchronously acquiring confirmation information that the interface is successfully tuned; exporting task grouping information, task information, task configuration information and data source information associated with the task grouping information, the task information and the task configuration information in the index space, and organizing the information into a task information file in a json format; the method and the system for quickly deploying the visual large-screen index task have the beneficial effects that the requirement of a customer for visually displaying platform operation indexes is quickly and efficiently met through the implementation of exporting the index platform task template and importing the customer platform, the use function of the platform is improved, one-key deployment is really realized, and the operation efficiency of the customer is improved. And the use is fast.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for realizing rapid deployment of large-screen visualization indicator tasks. Background Art

[0002] With the advent of the big data era, data development and sharing have become mainstream, and data visualization has become increasingly important in all walks of life. In the big data era, the operational data of the data platform needs to be displayed on a large screen, and indicators such as the amount of data managed by each system, the number of visits, and the amount of exchanged data every day, every week, and every month need to be visualized.

[0003] After the platform is deployed, when you need to use the visualization function to display the business data of the governance platform, you often need to reconfigure the relevant indicator tasks to perform the visualization. This implementation often consumes a lot of manpower and has an error rate in re-implementation. In order to solve the complexity of the above reconfiguration, it is urgent to provide a set of visualization indicator task export and import functions to meet the needs of customers to use the visualization large screen to display operational data after the governance platform is deployed. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for realizing rapid deployment of large-screen visualization indicator tasks, so as to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for realizing rapid deployment of a visualization large screen indicator task, the method comprising an export step and an import step, wherein:

[0006] Export steps: In the source data governance platform, do the following:

[0007] Select the indicator space to be exported, call the data development export interface, and asynchronously obtain confirmation information of successful interface call;

[0008] Export the task grouping information, task information, task configuration information and their associated data source information under the indicator space, and organize the information into a task information file in json format;

[0009] Export the nifi process template file of the process orchestration task. Each task template is named after the task nifi group id and saved as an xml file.

[0010] Compress all files into a zip file and upload the zip file to the minio server;

[0011] Push the export result information and minio upload results to the redis server for subsequent use.

[0012] Preferably, the task information file jobInfo.json in json format includes but is not limited to: task grouping information, task information, task configuration information and associated data source information that has been desensitized.

[0013] Preferably, the following import step is also included and is performed in the target data governance platform:

[0014] Get the information of the zip file stored in minio from redis and download the zip file;

[0015] Select a target indicator space in the target data governance platform, call the import interface of data development, and pass in the downloaded zip file, the data source correspondence between the original indicator space and the target space as parameters;

[0016] Based on the principle of minimizing the number of tasks and components, the NIFI node with the best resources is automatically selected for subsequent operations;

[0017] Save the data source information in the target space and create or update the NiFi controller information;

[0018] Parse the contents of the zip file, including task group information, task information, task configuration information, and nifi process template files, and update this information to adapt to the target environment, such as replacing the group ID, nifi group ID, and controller ID, and import the updated information into the target data governance platform;

[0019] For process orchestration tasks, you also need to replace the original controller ID in the NiFi process template with the controller ID of the new NiFi instance, and redeploy these processes on the NiFi node;

[0020] The successful import information is pushed to the redis server so that the management center can query the import status.

[0021] Preferably, in the import step, updating of the task information includes but is not limited to: updating the space, nifi instance, user information to which the task belongs, replacing the group ID and nifi group ID, and updating the data source ID and nifiControllerServiceId in the task configuration information when necessary.

[0022] Preferably, the method provides one-click export and one-click import functions to achieve rapid migration and deployment of visualization large-screen indicator tasks between different data governance platform environments, ensuring the consistency and continuity of tasks.

[0023] A system for realizing rapid deployment of large-screen visualization indicator tasks, the system comprising:

[0024] A data export module is used to call the data development export interface in the source data governance platform according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and related data source information under the indicator space, and generate a task information file in json format;

[0025] A data import module is used to download the zip file in the target data governance platform according to the minio file information stored in redis, select the target indicator space, and call the import interface of data development, passing the downloaded zip file and the data source correspondence between the original indicator space and the target space as parameters to perform the import operation.

[0026] Preferably, the data export module exports the nifi process template file of the process orchestration task, compresses the nifi process template file into a zip file and uploads it to the minio server, and finally pushes the export result information and the minio upload result to the redis server.

[0027] Preferably, the task information file jobInfo.json in json format includes but is not limited to task grouping information, task information, task configuration information, and associated data source information after desensitization processing.

[0028] Preferably, the data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of the least number of tasks and components, saving or updating the data source and controller information of the target space, parsing the zip file content, updating the task grouping, tasks, task configuration and nifi process template information, and replacing the necessary ID information to adapt to the target environment, and finally pushing the successfully imported information to the redis server.

[0029] Preferably, during the import process, the data import module adds the target space, nifi instance, user information, and replaces the group ID for import for data development sql tasks; for process orchestration tasks, replaces the original controller ID in the nifi process template with the controller ID of the new nifi instance, and records the new nifi group id and the mapping relationship between the original component ID and the new component ID; at the same time, updates the relevant information in the di_job and di_job_config tables, including space, nifi instance, user information, group ID, nifi group id, and data source ID.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The method and system for realizing rapid deployment of large-screen visualization indicator tasks proposed in the present invention, by exporting indicator platform task templates and importing them into the implementation of the customer platform, quickly and efficiently solves the customer's requirements for visual display of platform operation indicators, improves the platform's usage functions, and truly achieves one-click deployment and rapid use. Through the import function, the project where the governance platform is deployed can visualize the platform indicators. It greatly improves work efficiency, reduces the complexity and error rate of manual operations, and ensures the customer's need to view the platform's operation status. It promotes the deployment of the governance platform, enhances the platform's functions, and improves the winning rate of bidding. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Deriving a flow chart for the indicator space of the present invention;

[0033] Figure 2 Importing a flow chart for the target space of the present invention;

[0034] Figure 3 Exporting a file structure diagram for the present invention;

[0035] Figure 4 Generate a zip image for the invention export file. DETAILED DESCRIPTION

[0036] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] For example, see Figures 1 to 4 The present invention provides a technical solution: a method for realizing rapid deployment of a visualization large-screen indicator task, the method comprising the following steps:

[0038] Export of indicator platforms:

[0039] 1. The management center calls the data development export interface

[0040] The data lake platform management center selects the indicator space, passes the indicator space to call the data development interface, and asynchronously returns the interface call success information.

[0041] 2. Export the data of the space task related table and generate the jobInfo.json structure

[0042] Export di_job_folder task grouping information, filter by task, and export cascade grouping information

[0043] Export di_job task information and di_job_config task configuration information. Each task has a piece of data. The data development SQL associated with the process orchestration also needs to be exported. At the same time, export the task-related data source information (including the mapped databaseId and nifiControllerServiceId)

[0044] Export associated data source information (desensitized export)

[0045] Generate the jobInfo.json structure from the above table information, as shown below:

[0046] {

[0047] "dataType":"dataDev",

[0048] "data":{

[0049] "datasource":[{

[0050] "datasourceId":"",

[0051] "datasourceName":"",

[0052] "datasourceDriver":"",

[0053] "dbtype":"",

[0054] "databaseName":"",

[0055] "schema":"",

[0056] "poolsize":"",

[0057] "lakeArea":"",

[0058] "incrementColumn":[{

[0059] "tableName":"",

[0060] "increColumn":"",

[0061] "comments":""

[0062] }

[0063]

[0064]

[0065]

[0066]

[0067] 3. Export task flow template

[0068] Export the process orchestration task nifi process template file (the integration task only imports table data and is manually published after import). Each task template uses the task nifi group id as the file name to generate a process xml file.

[0069] 4. Generate zip file

[0070] For the exported information file structure of steps 2 and 3, please see the attached Figure 3 , compress and generate a zip file, see attached Figure 4 .

[0071] 5. Upload minio server

[0072] Upload the zip file generated in step 4 to the minio server.

[0073] 6. Push redis export information

[0074] Push the export result information and minio upload results to redis.

[0075] Import of client platform:

[0076] 1. The management center calls the data development import interface

[0077] The management center downloads the export file based on the miniIo file information obtained from redis in advance.

[0078] Select the downloaded task file and call the import interface of data development.

[0079] Select the target space where the indicator needs to be imported, and obtain the data source correspondence between the original indicator space and the target space.

[0080] Use the above information as parameters to call the data development import interface, and asynchronously return the successful information of calling the interface.

[0081] 2. Select the NIFI node

[0082] According to the algorithm of minimizing the number of tasks and components, the NIFI node with the best resources is selected.

[0083] 3. Save the data source and create a controller

[0084] Before saving the data source and controller information in the target space, duplicate detection is performed. If the data source information does not exist, the data source information is saved, a controller is created in NiFi, and the data source ID and controller ID of the target space are saved.

[0085] 4. Import data table information

[0086] First, process di_job_folder (task group information table), add target space and user information, generate a new UUID as the group ID, and import it.

[0087] Then iterate through the task imports:

[0088] The data development SQL task adds space, nifi instance, user information, and replaces the group ID to import directly into the table.

[0089] The process orchestration task replaces the original controller ID in the nifi process template with the controller ID of the new nifi instance, then imports it into the NIFI node, records the new nifi group id, and records the mapping relationship between the original component ID and the new component ID;

[0090] The di_job table records the addition of space, nifi instance, user information, replaces the group id, and updates the task nifi groupid for import;

[0091] The di_job_config table records the addition of user information, the replacement of group id, the update of task nifi group id, and the replacement of the data source id in the integrated task configuration details with the primary key of the integrated development data source, the replacement of nifiControllerServiceId with the controller id corresponding to nifi, and the replacement of the data source ID in the sql task configuration with the primary key of the integrated development data source for import;

[0092] 5. Push Redis import information

[0093] The successful import information is pushed to redis so that the management center can obtain the import status.

[0094] The above steps are to export the indicator environment tasks that have been running normally with one click, and download the process tasks and configuration information as files from minio. Import them into the target environment space with one click. After the execution is completed, the indicator-related tasks can be called in the target environment space. It truly achieves the rapid deployment of indicator space tasks across environments, and achieves one-click deployment of indicator tasks, which is efficient and fast. The execution results can be seen in the attached Figure 3 .

[0095] Embodiment 2, based on embodiment 1, proposes a system for realizing rapid deployment of visualization large screen indicator tasks, the system comprising:

[0096] A data export module is used to call the data development export interface in the source data governance platform according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and related data source information under the indicator space, and generate a task information file in json format;

[0097] The data export module exports the nifi process template file of the process orchestration task, compresses the nifi process template file into a zip file and uploads it to the miniIo server, and finally pushes the export result information and the miniIo upload result to the redis server;

[0098] The task information file jobInfo.json in json format includes but is not limited to task grouping information, task information, task configuration information, and associated data source information after desensitization processing.

[0099] A data import module is used to download the zip file in the target data governance platform according to the minio file information stored in redis, select the target indicator space, and call the import interface of data development, passing the downloaded zip file and the data source correspondence between the original indicator space and the target space as parameters to perform the import operation;

[0100] The data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of the least number of tasks and components, saving or updating the data source and controller information of the target space, parsing the zip file content, updating the task grouping, tasks, task configuration and nifi process template information, and replacing the necessary ID information to adapt to the target environment, and finally pushing the successfully imported information to the redis server;

[0101] During the import process, the data import module adds the target space, nifi instance, and user information to the data development SQL task, and replaces the group ID for import; for the process orchestration task, replaces the original controller ID in the nifi process template with the controller ID of the new nifi instance, and records the new nifi group id and the mapping relationship between the original component ID and the new component ID; at the same time, updates the relevant information in the di_job and di_job_config tables, including space, nifi instance, user information, group ID, nifi group id, and data source ID.

[0102] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for realizing rapid deployment of large-screen visualization indicator tasks, characterized in that: The method comprises an export step and an import step, wherein: Export steps: In the source data governance platform, do the following: Select the indicator space to be exported, call the data development export interface, and asynchronously obtain confirmation information of successful interface call; Export the task grouping information, task information, task configuration information and their associated data source information under the indicator space, and organize the information into a task information file in json format; Export the nifi process template file of the process orchestration task. Each task template is named after the task nifi group id and saved as an xml file. Compress all files into a zip file and upload the zip file to the minio server; Push the export result information and minio upload results to the redis server for subsequent use.

2. A method for realizing rapid deployment of visualization large-screen indicator tasks according to claim 1, characterized in that: The task information file jobInfo.json in json format includes but is not limited to: task grouping information, task information, task configuration information, and desensitized associated data source information.

3. A method for realizing rapid deployment of large-screen visualization indicator tasks according to claim 1, characterized in that: The following import steps are also included and are performed in the target data governance platform: Get the information of the zip file stored in minio from redis and download the zip file; Select a target indicator space in the target data governance platform, call the import interface of data development, and pass in the downloaded zip file, the data source correspondence between the original indicator space and the target space as parameters; Based on the principle of minimizing the number of tasks and components, the NIFI node with the best resources is automatically selected for subsequent operations; Save the data source information in the target space and create or update the NiFi controller information; Parse the contents of the zip file, including task group information, task information, task configuration information, and nifi process template files, and update this information to adapt to the target environment, such as replacing the group ID, nifigroupid, and controller ID, and import the updated information into the target data governance platform; For process orchestration tasks, you also need to replace the original controller ID in the NiFi process template with the controller ID of the new NiFi instance, and redeploy these processes on the NiFi node; The successful import information is pushed to the redis server so that the management center can query the import status.

4. A method for realizing rapid deployment of large-screen visualization indicator tasks according to claim 3, characterized in that: In the import step, the task information is updated including but not limited to: updating the space, nifi instance, user information to which the task belongs, replacing the group ID and nifi group ID, and updating the data source ID and nifiControllerServiceId in the task configuration information when necessary.

5. A method for realizing rapid deployment of large-screen visualization indicator tasks according to claim 1, characterized in that: The method provides one-click export and one-click import functions to achieve rapid migration and deployment of visualization large-screen indicator tasks between different data governance platform environments, ensuring the consistency and continuity of tasks.

6. A system for realizing rapid deployment of large-screen visualization indicator tasks according to the method for realizing rapid deployment of large-screen visualization indicator tasks according to any one of claims 1 to 5, characterized in that: The system comprises: A data export module is used to call the data development export interface in the source data governance platform according to the indicator space selected by the user, asynchronously obtain the interface call success information, and export the task grouping information, task information, task configuration information and related data source information under the indicator space, and generate a task information file in json format; A data import module is used to download the zip file in the target data governance platform according to the minio file information stored in redis, select the target indicator space, and call the import interface of data development, passing the downloaded zip file and the data source correspondence between the original indicator space and the target space as parameters to perform the import operation.

7. A system for realizing rapid deployment of large-screen visualization indicator tasks according to claim 6, characterized in that: The data export module exports the nifi process template file of the process orchestration task, compresses the nifi process template file into a zip file and uploads it to the miniIo server, and finally pushes the export result information and the miniIo upload result to the redis server.

8. A system for realizing rapid deployment of large-screen visualization indicator tasks according to claim 6, characterized in that: The task information file jobInfo.json in json format includes but is not limited to task grouping information, task information, task configuration information, and associated data source information after desensitization processing.

9. A system for realizing rapid deployment of large-screen visualization indicator tasks according to claim 6, characterized in that: The data import module is also responsible for automatically selecting the NIFI node with the best resources based on the principle of the least number of tasks and components, saving or updating the data source and controller information of the target space, parsing the zip file content, updating the task grouping, task, task configuration and nifi process template information, and replacing the necessary ID information to adapt to the target environment, and finally pushing the successfully imported information to the redis server.

10. A system for realizing rapid deployment of large-screen visualization indicator tasks according to claim 6, characterized in that: During the import process, the data import module adds the target space, nifi instance, user information, and replaces the group ID for data development SQL tasks for import; For process orchestration tasks, replace the original controller ID in the nifi process template with the controller ID of the new nifi instance, and record the new nifi group id and the mapping relationship between the original component ID and the new component ID; at the same time, update the relevant information in the di_job and di_job_config tables, including space, nifi instance, user information, group ID, nifi group id and data source ID.

Images