A Visual Data Offloading Method and System Based on the Implementation of Batch Process Services

Abstract

The present invention discloses a visualization data unloading method and system based on batch process services, belonging to the technical field of data processing, and solving problems such as large workload in the prior art where data is unloaded directly on the server through shell scripts manually. The present invention configures a data unloading control table, a server information table, and an ftp server information table for data unloading in the database table; uses Java and sqluldr2 to build a common component for completing the data unloading function in the batch service application; registers the data in the data unloading control table for the process, so as to generate an unloading process for each table in the data unloading control table as a data unloading process; configures a data unloading process group based on the generated data unloading process, associates all the processes with the process group, and displays the execution of the data unloading process group and log viewing through the interface. The present invention is used for visualization data unloading.

Description

Technical Field

[0001] A visualized data unloading method and system based on batch process service are used for visualized data unloading, belonging to the technical field of data processing. Background Art

[0002] During the data migration process, the migrating party needs to complete data unloading and export the data in text format. The more common method is to manually write a large number of shell scripts on the database server, manually control the execution of the scripts, and manually observe the progress of the script execution through logs, processes, etc. This method requires developers to have certain basic development capabilities and server operation capabilities for shell script language, and the modification process needs to be manually modified and executed on the server, so there are the following technical problems:

[0003] 1. Manually unloading data directly on the server through shell scripts, which requires a lot of shell script writing;

[0004] 2. Manually unloading data on the server directly through shell scripts poses a security risk, that is, the developer knows the password of the server user and has high read and write permissions, and can modify the server content at will, thus causing unsafe server operation problems;

[0005] 3. Manually unloading data directly on the server through shell scripts does not allow you to intuitively see the progress of the data unloading workflow, and there is no operation process interface. Summary of the invention

[0006] In response to the above research problems, the purpose of the present invention is to provide a visual data unloading method and system based on batch process service, so as to solve the problems of the existing technology of manually unloading data directly on the server through shell scripts, which has a large workload, potential safety hazards and the inability to intuitively see the progress of the data unloading workflow.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A visual data unloading method based on batch process service implementation includes the following steps:

[0009] Step 1: Configure the data unloading control table, server information table and ftp server information table for data unloading in the database table;

[0010] Step 2: Based on the data unloading control table, server information table and FTP server information table, use Java and SQLULDR2 to build a common component that implements the data unloading function in the batch service application;

[0011] Step 3: Based on the results obtained in Step 1 and Step 2, perform process registration on the data in the data offloading control table, so as to generate an offloading process for each table in the data offloading control table as a data offloading process;

[0012] Step 4: Configure a data offloading process group based on the generated data offloading process, associate all processes with this process group, and display the execution of the data offloading process group and log viewing through the interface.

[0013] Furthermore, the specific steps of Step 1 are as follows:

[0014] Step 1.1: Configure a data offloading control table in the database. Among them, the data offloading control table is used to control the tables that need to perform data offloading work. The configuration fields in the data offloading control table include table name, custom file export name, character set, custom delimiter, row number splitting or file size splitting;

[0015] Step 1.2: Configure a server information table in the database. Among them, the configuration fields in the server information table include the user name, password, ip, and instance name of the user in the database. The user name and password of the user in the database include the user name and password of the user who has been granted read-only permissions. When it is the user name and password of the user who has been granted read-only permissions, then do not save the user name and password of the user in the actual system used database;

[0016] Step 1.3: Configure an ftp server information table in the database. Among them, the configuration fields in the ftp server information table include export path, ftp user name, ftp password, and ftp address.

[0017] Furthermore, the specific steps of Step 2 are as follows:

[0018] Step 2.1: Use the entity class corresponding to the data offloading control table as a parameter to pass into a data offloading control record, and splice it through the java string splicing method based on the server information table and the data offloading control record. After splicing, obtain the sqludr2 script statement. Among them, the sqludr2 script statement includes the user name, password, ip, instance name, sql statement, character set, delimiter, file export name, row number splitting, and log of the user in the database. Among them, the data offloading control record represents a certain row of data in the transmitted data offloading control table;

[0019] Step 2.2: Use the java APl to call the sqludr2 script statement generated in Step 2.1 to generate the data file corresponding to the parameters passed in Step 2.1 on the server, as well as the corresponding log file;

[0020] Step 2.3: After uninstalling the generated data file, use the java API to compress the uninstalled data file and call the gzip tool to compress it into the gz file format;

[0021] Step 2.4: Connect to the remote file ftp server based on the ftp server information table, and upload the compressed data file. If an exception occurs during the upload process, the common component can throw the exception information, which is captured by the upstream process and registered in the batch process table. The error information can be viewed through the generated log file. Otherwise, upload it to the ftp server.

[0022] Further, the specific steps of step 3 are as follows:

[0023] Step 3.1: Read all row data, server information table, and ftp information table in the configured data unloading control table. Use the common component to generate the data unloading process corresponding to each row record in the data unloading control table according to the row records, and register the data unloading process in the newly created batch process table. The fields of the batch process table include the name of the data unloading process, running batch, whether it can be repeated, start time, end time, running duration, running status information, and error stack;

[0024] Step 3.2: If an exception occurs during the data unloading process, view it through the corresponding data unloading process log, which is in the stack information in the batch process table;

[0025] Step 3.3: If a problem occurs during the unloading process of any data unloading process in the batch process table, decide whether the data unloading process can be repeated according to the parameter configuration in the batch process table. Repeating the operation will generate a new running log of the data unloading process, and the fields of the newly generated log include the name of the data unloading process, running batch, whether it can be repeated, start time, end time, running duration, running status information, and error stack.

[0026] Further, the specific steps of step 4 are as follows:

[0027] Step 4.1: Register all the data unloading processes in step 3.1 into the same data unloading process group, and this data unloading process group defines the data unloading process for data migration work;

[0028] Step 4.2: The data migration personnel select the process group on the interface in the server, click the start execution button, and the batch scheduling platform starts to execute the data unloading process under this process group, and observes the execution situation of each data unloading process in real time. At the same time, the running status of the data unloading process group will be recorded in the database. If an error occurs in a certain data unloading process, view it through the corresponding process log or repeat the execution of this data unloading process.

[0029] A visualization data offloading system implemented based on batch process services, including

[0030] Configuration module: Configure a data offloading control table, a server information table, and an ftp server information table for data offloading in the database table;

[0031] Construction module: Based on the data offloading control table, the server information table, and the ftp server information table, use Java and sqluldr2 to construct a common component that realizes the data offloading function in batch service applications;

[0032] Process registration module: Based on the results obtained in Step 1 and Step 2, perform process registration on the data in the data offloading control table, so as to generate a data offloading process for the offloading process of each table in the data offloading control table;

[0033] Process group offloading module: Configure a data offloading process group based on the generated data offloading process, associate all processes with this process group, and display the execution of the data offloading process group and log viewing through the interface.

[0034] Furthermore, the specific implementation steps of the configuration module are:

[0035] Step 1.1: Configure a data offloading control table in the database. Among them, the data offloading control table is used to control the tables that need to perform data offloading work. The configuration fields in the data offloading control table include table name, custom file export name, character set, custom delimiter, row number splitting, or file size splitting;

[0036] Step 1.2: Configure a server information table in the database. Among them, the configuration fields in the server information table include the user name, password, ip, and instance name of the user in the database. The user name and password of the user in the database include the user name and password of the user who has been granted read-only permissions. When it is the user name and password of the user who has been granted read-only permissions, the user name and password of the user used in the actual system do not need to be saved;

[0037] Step 1.3: Configure an ftp server information table in the database. Among them, the configuration fields in the ftp server information table include export path, ftp user name, ftp password, and ftp address.

[0038] Furthermore, the specific implementation steps of the construction module are:

[0039] Step 2.1: Use the entity class corresponding to the data offloading control table as a parameter to pass in a certain data offloading control record, and splice it based on the server information table and the data offloading control record in the way of Java string splicing. After splicing, the sqludr2 script statement is obtained. Among them, the sqludr2 script statement includes the user name, password, ip, instance name, sql statement, character set, delimiter, file export name, line number segmentation and log of the user in the database. Among them, the data offloading control record represents a certain row of data in the data offloading control table to be transmitted;

[0040] Step 2.2: Use the Java API to call the sqludr2 script statement generated in Step 2.1 to generate the data file corresponding to the parameters passed in Step 2.1 and the corresponding log file on the server;

[0041] Step 2.3: After unloading the generated data file, use the Java API to compress the unloaded data file, and call the gzip tool to compress it into the gz file format;

[0042] Step 2.4: Connect to the remote file ftp server based on the ftp server information table, upload the compressed data file. If an exception occurs during the upload process, the common component can throw the exception information, which is captured by the upstream process and registered in the batch process table. The error information can be viewed through the generated log file. Otherwise, upload it to the ftp server.

[0043] Furthermore, the specific implementation steps of the process registration module are as follows:

[0044] Step 3.1: Read all row data, server information table and ftp information table in the configured data offloading control table, and use the common component to generate the data offloading process corresponding to each row record according to the row records in the data offloading control table, and register the data offloading process in the newly created batch process table. Among them, the fields of the batch process table include the name of the data offloading process, the running batch, whether it can be executed repeatedly, the start time, the end time, the running time consumption, the running status information and the error stack;

[0045] Step 3.2: If an exception occurs during the data offloading process, view it through the corresponding data offloading process log, and this log is in the stack information in the batch process table;

[0046] Step 3.3: If there is a problem during the unloading process of any data unloading process in the batch process table, determine whether the data unloading process can be repeated according to the parameter configuration in the batch process table. Repeating the operation will generate a new operation log for the data unloading process. The fields of the regenerated log include the data unloading process name, operation batch, whether it can be repeated, start time, end time, operation duration, operation status information, and error stack.

[0047] Furthermore, the specific implementation steps of the process group unloading module are as follows:

[0048] Step 4.1: Register all the data unloading processes in Step 3.1 into the same data unloading process group, which defines the data unloading process for data migration work.

[0049] Step 4.2: The data migration personnel select the process group on the interface in the server, click the start execution button, and the batch scheduling platform starts to execute the data unloading processes under this process group, and observes the execution status of each data unloading process in real time. At the same time, the running status of the data unloading process group will be recorded in the database. If an error occurs in a certain data unloading process, view it through the corresponding process log or repeat the execution of this data unloading process.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] First, the present invention can realize the visualization of the data unloading process for data migration, that is, it can intuitively see the progress of the data unloading work process, and can unload the work process with an operation process interface.

[0052] Second, the present invention optimizes the manual control of the data unloading process by developers, and the system can automatically execute the unloading according to the process group.

[0053] Third, the present invention reduces the workload of developers for a large number of debugging and modifying shell scripts through parameterized configuration, that is, it reduces a large amount of workload.

[0054] Fourth, the present invention reduces the modification of shell scripts by developers and avoids the risk of direct operation on the server. Specific Embodiments

[0055] The following will further describe the present invention in combination with specific embodiments.

[0056] To improve the development efficiency and increase the security of the server, the data unloading function and the visualization process are realized by writing batch process services. The development engineer only needs to configure the key information such as the table name to be unloaded and the unloading method in the table to realize the data unloading process work.

[0057] A visualization data unloading method based on batch process service implementation, comprising the following steps:

[0058] Step 1: Configure a data unloading control table, a server information table, and an ftp information table for data unloading in the database table; the data unloading control table stores many tables that need to unload data, and each table represents a row record in the data unloading control table, but there is only one piece of data in the server information table and the ftp information table.

[0059] The specific steps are as follows:

[0060] Step 1.1: Configure a data unloading control table in the database. The table name of the data unloading control table can be app_unload_table_control. Among them, the data unloading control table is used to control the tables that need to perform data unloading work. The configuration fields in the data unloading control table include table name, custom file export name, character set, custom delimiter, row number splitting or file size splitting;

[0061] For example, if a developer adds a record to the app_unload_table_control table, this record is the table name that needs to unload data. For example, insert into app_unload_table_control values('loan_account', 'loan_account.txt', 'american_america.zhs16gbk', '~@~', 0, 0), where loan_account is the table name, loan_account.txt is the custom file export name, american_america.zhs16gbk is the character set, and ~@~ is the custom delimiter.

[0062] Step 1.2: Configure a server information table in the database. Among them, the configuration fields in the server information table include the username, password, ip, and instance name of the user in the database. The username and password of the user in the database include the username and password of the user who has been granted read-only permissions. When it is the username and password of the user who has been granted read-only permissions, then do not save the username and password of the user in the actual system database; such as inert into app_oracle_info values('vlog', 'vlog', '10.1.2.5', 'loan'), where the first vlog is the username, the second vlog is the password, 10.1.2.5 is the ip, and loan is the instance name.

[0063] Step 1.3: Configure the ftp server information table in the database. The configuration fields in the ftp server information table include the export path, ftp username, ftp password, and ftp address.

[0064] For example, insert into app_ftp_info values(' / home / data / 20990909 / ', 'ftpuser', 'ftpuser', '10.2.2.4'). Here, / home / data / 20990909 / is the export path, the first ftpuser is the ftp username, the second ftpuser is the ftp password, and 10.2.2.4 is the ftp address.

[0065] Step 2: Based on the data unloading control table, server information table, and ftp server information table, use Java and sqluldr2 to build a common component that realizes the data unloading function in the batch service application.

[0066] The specific steps are as follows:

[0067] Step 2.1: Use the entity class corresponding to the data unloading control table as a parameter to pass in a data unloading control record, and splice it based on the server information table and the data unloading control record through the Java string splicing method. After splicing, the sqludr2 script statement is obtained. The sqludr2 script statement includes the username, password, ip, instance name, sql statement (such as select t.*from loan_account t″), character set, delimiter, file export name, row segmentation, and log of the user in the database. The data unloading control record represents a certain row of data in the data unloading control table to be transmitted.

[0068] Step 2.2: Use the Java API to call the sqludr2 script statement generated in Step 2.1 to generate the data file corresponding to the parameters passed in Step 2.1 and the corresponding log file on the server.

[0069] Step 2.3: After unloading the generated data file, use the Java API to compress the unloaded data file and call the gzip tool to compress it into the gz file format.

[0070] Step 2.4: Connect to the remote file ftp server based on the ftp server information table, upload the compressed data file. If an exception occurs during the upload process, the common component can throw the exception information, which is captured by the upstream process and registered in the batch process table. The error information can be viewed through the generated log file. Otherwise, upload it to the ftp server.

[0071] Step 3: Based on the results obtained in Step 1 and Step 2, perform process registration on the data in the data offloading control table to generate a data offloading process for the offloading process of each table in the data offloading control table;

[0072] The specific steps are as follows:

[0073] Step 3.1: Read all row data, server information table, and ftp information table in the configured data offloading control table, and use the common component to generate the data offloading processes corresponding to each row record according to the row records in the data offloading control table, and register the data offloading processes into the newly created batch process table. The fields of the batch process table include the name of the data offloading process, running batch, whether it can be executed repeatedly, start time, end time, running duration, running status information, and error stack. Among them, to register the data offloading process into the newly created batch process table, the sys_batch_flowDao.insert method is used.

[0074] Step 3.2: If an exception occurs during the data offloading process, view it through the corresponding data offloading process log, and this log is in the stack information in the batch process table;

[0075] Step 3.3: If a problem occurs during the offloading process of any data offloading process in the batch process table, decide whether the data offloading process can be executed repeatedly according to the parameter configuration in the batch process table. Repeated execution will generate a new running log of the data offloading process, and the fields of the newly generated log include the name of the data offloading process, running batch, whether it can be executed repeatedly, start time, end time, running duration, running status information, and error stack.

[0076] Step 4: Configure the data offloading process group based on the generated data offloading process, associate all processes with this process group, and display the execution of the data offloading process group and log viewing through the interface.

[0077] The specific steps are as follows:

[0078] Step 4.1: Uniformly register each data offloading process in Step 3.1 into the same data offloading process group, and this data offloading process group defines the data offloading process for the data migration work;

[0079] Step 4.2: The data migration personnel select the process group on the interface, click the start execution button, and the batch scheduling platform starts to execute the data offloading processes under this process group, and observes the execution status of each data offloading process in real time. At the same time, the running status of the data offloading process group will be recorded in the database. If an error occurs in a certain data offloading process, view it through the corresponding process log or repeat the execution of this data offloading process.

[0080] In summary, the data offloading function of the present invention applied to data migration work solves the problems of large workload, insecurity, and lack of interface caused by the traditional method of directly unloading data on the server through shell scripts manually. By using the batch process service, a configurable data offloading solution is developed. Developers only need to fill in elements such as the table name to be unloaded, data offloading method, delimiter, and batch run batch in the data table to achieve the data offloading function and the interface process display function.

[0081] The above are only representative embodiments among the numerous specific application scopes of the present invention, and do not constitute any limitation to the protection scope of the present invention. Any technical solutions formed by transformation or equivalent replacement fall within the scope of the present invention's rights protection.

Claims

1. A visualization data offloading method implemented based on batch process services, characterized in that, It includes the following steps: Step 1: Configure a data unloading control table, a server information table, and an ftp server information table for data unloading in the database table; Step 2: Based on the data unloading control table, the server information table, and the ftp server information table, use Java and sqluldr2 to build a common component that realizes the data unloading function in the batch service application; Step 3: Based on the results obtained in Step 1 and Step 2, perform process registration on the data in the data unloading control table to generate an unloading process for each table in the data unloading control table as a data unloading process; Step 4: Configure a data unloading process group based on the generated data unloading process, associate all processes with this process group, and display the execution of the data unloading process group and log viewing through the interface; The specific steps of Step 2 are as follows: Step 2.1: Use the entity class corresponding to the data unloading control table as a parameter to pass in a data unloading control record, and splice it based on the server information table and the data unloading control record through the Java string splicing method. After splicing, an sqluldr2 script statement is obtained. Among them, the sqluldr2 script statement includes the username, password, ip, instance name, sql statement, character set, delimiter, file export name, row segmentation, and log of the user in the database. Among them, the data unloading control record represents a certain row of data in the data unloading control table to be transmitted; Step 2.2: Use the Java API to call the sqluldr2 script statement generated in Step 2.1 to generate the data file corresponding to the parameters passed in Step 2.1 and the corresponding log file on the server; Step 2.3: After unloading the generated data file, use the Java API to compress the unloaded data file, and call the gzip tool to compress it into the gz file format; Step 2.4: Connect to the remote file ftp server based on the ftp server information table, upload the compressed data file. If an exception occurs during the upload, the common component can throw the exception information, which is captured by the upstream process and registered in the batch process table, and the error information can be viewed through the generated log file. Otherwise, upload it to the ftp server.

2. The visualization data offloading method implemented based on batch process services according to claim 1, wherein, The specific steps of Step 1 are as follows: Step 1.1: Configure a data unloading control table in the database. Among them, the data unloading control table is used to control the table that needs to perform data unloading work. The configuration fields in the data unloading control table include the table name, custom file export name, character set, custom delimiter, row segmentation or file size segmentation; Step 1.2: Configure a server information table in the database. Among them, the configuration fields in the server information table include the username, password, ip, and instance name of the user in the database. The username and password of the user in the database include the username and password of the user who has been granted read-only permissions. When it is the username and password of the user who has been granted read-only permissions, the username and password of the user in the actual system used do not need to be saved; Step 1.3: Configure the ftp server information table in the database. The configuration fields in the ftp server information table include the export path, ftp username, ftp password, and ftp address.

3. The visualization data offloading method based on batch process service implementation according to claim 1, wherein The specific steps of step 3 are as follows: Step 3.1: Read all the row data, server information table, and ftp information table in the configured data unloading control table. Use the common component to generate the data unloading process corresponding to each row record in the data unloading control table according to the row records, and register the data unloading process in the newly created batch process table. The fields in the batch process table include the name of the data unloading process, running batch, whether it can be executed repeatedly, start time, end time, running duration, running status information, and error stack. Step 3.2: If an exception occurs during the data unloading process, view it through the corresponding data unloading process log, which is in the stack information in the batch process table. Step 3.3: If a problem occurs during the unloading process of any data unloading process in the batch process table, determine whether the data unloading process can be executed repeatedly according to the parameter configuration in the batch process table. Repeated execution will generate a new running log of the data unloading process. The fields of the newly generated log include the data unloading process name, running batch, whether it can be executed repeatedly, start time, end time, running duration, running status information, and error stack.

4. A visualization data offloading method implemented based on batch process services according to claim 3, characterized in that The specific steps of step 4 are as follows: Step 4.1: Uniformly register each data unloading process in step 3.1 into the same data unloading process group, which defines the data unloading process for data migration work. Step 4.2: The data migration personnel select the process group on the interface in the server, click the start execution button, and the batch scheduling platform starts to execute the data unloading process under this process group, and observes the execution situation of each data unloading process in real time. At the same time, the running status of the data unloading process group will be recorded in the database. If an error occurs in a certain data unloading process, view it through the corresponding process log or repeat the execution of the data unloading process.

5. A visualization data offloading system implemented based on batch process services, characterized in that, including Configuration module: Configure the data unloading control table, server information table, and ftp server information table for data unloading in the database table. Construction module: Based on the data unloading control table, server information table, and ftp server information table, use java and sqluldr2 to construct a common component to complete the data unloading function in the batch service application. Process registration module: Based on the results obtained in step 1 and step 2, perform process registration on the data in the data unloading control table to generate a data unloading process for the unloading process of each table in the data unloading control table. Process group unloading module: Configure the data unloading process group based on the generated data unloading process, associate all processes with this process group, and display the execution of the data unloading process group and log viewing through the interface. The specific implementation steps of the construction module are as follows: Step 2.1: Use the entity class corresponding to the data unloading control table as a parameter to pass in a certain data unloading control record, and splice it through the Java string splicing method based on the server information table and the data unloading control record. After splicing, an sqluldr2 script statement is obtained. Among them, the sqluldr2 script statement includes the user name, password, IP, instance name, SQL statement, character set, delimiter, file export name, line number segmentation, and log in the database. Among them, the data unloading control record represents a certain row of data in the data unloading control table to be transmitted; Step 2.2: Use the Java API to call the sqluldr2 script statement generated in Step 2.1 to generate the data file corresponding to the parameters passed in Step 2.1 and the corresponding log file on the server; Step 2.3: After unloading the generated data file, use the Java API to compress the unloaded data file, and call the gzip tool to compress it into the gz file format; Step 2.4: Connect to the remote file ftp server based on the ftp server information table. Upload the compressed data file. If an exception occurs during the upload, the public component can throw the exception information, which is captured by the upstream process and registered in the batch process table. The error information can be viewed through the generated log file. Otherwise, upload it to the ftp server.

6. The visualization data offloading system implemented based on batch process services according to claim 5, characterized in that The specific implementation steps of the configuration module are as follows: Step 1.1: Configure a data unloading control table in the database. Among them, the data unloading control table is used to control the table that needs to perform data unloading work. The configuration fields in the data unloading control table include the table name, custom file export name, character set, custom delimiter, line number segmentation or file size segmentation; Step 1.2: Configure a server information table in the database. Among them, the configuration fields in the server information table include the user name, password, IP, and instance name of the user in the database. The user name and password of the user in the database include the user name and password of the user who has been granted read-only permissions. When it is the user name and password of the user who has been granted read-only permissions, the user name and password of the user in the actual system used do not need to be saved; Step 1.3: Configure an ftp server information table in the database. Among them, the configuration fields in the ftp server information table include the export path, ftp user name, ftp password, and ftp address.

7. A visualization data offloading system implemented based on batch process services according to claim 6, characterized in that, The specific implementation steps of the process registration module are as follows: Step 3.1: Read all the row data, server information table, and ftp information table in the configured data unloading control table, and use the public component to generate the data unloading process corresponding to each row record according to the row records in the data unloading control table, and register the data unloading process in the newly created batch process table. Among them, the fields of the batch process table include the name of the data unloading process, the running batch, whether it can be executed repeatedly, the start time, the end time, the running time consumption, the running status information, and the error stack; Step 3.2: If an exception occurs during the data offloading process, view it through the corresponding data offloading process log, which is in the stack information in the batch process table; Step 3.3: If a problem occurs during the offloading process of any data offloading process in the batch process table, decide whether the data offloading process can be repeated according to the parameter configuration in the batch process table. Repeating the operation will generate a new operation log for the data offloading process. The fields of the regenerated log include the data offloading process name, run batch, whether it can be repeated, start time, end time, run duration, run status information, and error stack.

8. A visualization data offloading system implemented based on batch process services according to claim 7, characterized in that The specific implementation steps of the process group offloading module are as follows: Step 4.1: Register all the data offloading processes in Step 3.1 into the same data offloading process group, which defines the data offloading process for data migration work; Step 4.2: The data migration personnel select the process group on the interface in the server, click the start execution button, and the batch scheduling platform starts to execute the data offloading processes under this process group, and observes the execution status of each data offloading process in real time. At the same time, the running status of the data offloading process group will be recorded in the database. If an error occurs in a certain data offloading process, view it through the corresponding process log or repeat the data offloading process.