A method for creating data mining tasks based on the ZORRO system

Abstract

The present invention discloses a data mining task creation method based on the ZORRO system, including: filling in the ZORRO task configuration information in the ZORROWEB system, calling the configuration generation interface to generate the DLL file and XML file required for the data mining plan task service, starting the data mining plan task, verifying the plan task, setting the task execution plan, selecting the data mining plan task operation mode, allocating the number of plan task processing threads according to the total amount of data, allocating the data to be mined to each processing thread, and executing the data mining task. The present invention can meet the data synchronization of customized complex logic data mining application scenarios, solve the problem of data synchronization interruption caused by task stop and the problem of slow page query response speed, and greatly improve the user experience.

Description

Technical Field

[0001] The invention relates to data mining technology, and belongs to a data mining task creation method based on a ZORRO system. Background Art

[0002] At present, in various applications in the field of big data, there are various complex data situations, such as large data volume, complex data structure, and unreasonable data table design. These situations will lead to slow data query speed, difficulty in presenting page data, difficulty in establishing data models, and poor customer experience. Most of the various software or tools commonly used in the industry are used for full or incremental data synchronization. They cannot meet the requirements for processing complex data synchronization scenarios proposed by customers. This requires technicians to repeatedly write various planned tasks to adapt to various data synchronization application scenarios proposed by customers, resulting in increased development costs, longer project development cycles, and greater difficulty in operation and maintenance, which has a certain impact on project delivery. Summary of the invention

[0003] The purpose of the present invention is to provide a data mining task creation method based on the ZORRO system, including: filling in the ZORRO task configuration information in the ZORRO WEB system, calling the configuration generation interface to generate the DLL file and XML file required for the data mining plan task service, starting the data mining plan task, verifying the plan task, setting the task execution plan, selecting the data mining plan task operation mode, allocating the number of plan task processing threads according to the total amount of data, allocating the data to be mined for each processing thread, and executing the data mining task. The present invention can meet the data synchronization of customized complex logic data mining application scenarios, solve the problem of data synchronization interruption caused by task stop and the problem of slow page query response speed, and greatly improve the user experience.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] S1: Fill in the ZORRO task configuration information in the ZORRO WEB system according to the data mining task requirements.

[0006] S2: Call the ZORRO API configuration generation interface to generate the DLL file and XML file required for the data mining plan task service according to the ZORRO task configuration information.

[0007] S3: Detect that new ZORRO task configuration information is generated, and start executing the data mining plan task according to the configuration file requirements.

[0008] S4: Start the data mining plan task self-check thread, verify the parameter information including task configuration and planned execution time, and set the task execution plan.

[0009] S5: Execute the data mining plan task according to the preset data mining plan task operation mode.

[0010] S6: Query the total amount of data participating in this data mining plan task, and automatically allocate the number of processing threads of the data mining plan task according to the total amount of data.

[0011] S7: Add the processing thread to the thread scheduling queue, allocate the data to be mined to the processing thread, execute the corresponding processing thread task, and verify the processing result after the corresponding processing thread task is completed.

[0012] S8: Repeat step S7 until all processing threads have finished running, and recycle the system resources occupied by this data mining plan task.

[0013] Furthermore, the data mining plan task running modes include: extreme speed mode and normal mode. In the normal mode, the data mining task running information needs to be recorded in the database.

[0014] Furthermore, the method further comprises:

[0015] During the execution of the data mining plan task, the operation log and real-time log of the data mining plan task are generated, including: program error log, program exception log, program debugging log, program normal log. All log contents can be viewed in the ZORRO WEB system.

[0016] Furthermore, before the data mining task is executed, the ZORRO WEB system and ZORRO API system are deployed on the IIS server, the execution path of the DLL file and the XML file is created, and the ZORRO data mining scheduled task service is installed, and 1) the execution path of the DLL file and the XML file; 2) the log storage path of the data mining scheduled task is written into the configuration file.

[0017] The beneficial effects of the present invention are:

[0018] (1) It solves the problem of slow response speed of page function query, greatly improves the page response speed and enhances customer experience.

[0019] (2) Data synchronization that meets customized complex logic requirements makes up for the shortcomings of existing products on the market that only support full or incremental data synchronization.

[0020] (3) A plug-in-based data synchronization programming mechanism is implemented to meet the needs of dynamic plug-in updates and concurrent programming, so that the host and plug-ins can be developed simultaneously without interfering with each other, shortening the project development cycle.

[0021] (4) A breakpoint resume module was added to solve the problem of data synchronization interruption caused by task stop due to unexpected server shutdown or restart, and the need to restart data synchronization.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Create a method step diagram for data mining tasks based on the ZORRO system.

[0024] Figure 2 Create a method flow chart for data mining tasks based on the ZORRO system.

[0025] Figure 3 This is a visual diagram of ZORRO. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0027] Combination Figure 3 Before executing the data mining task, the user deploys the ZORRO WEB system and ZORRO API system on the IIS server, then installs the ZORRO data mining scheduled task service (ZORRO SERVICES), creates the ZORRO data mining scheduled task running BIN directory, including: DLL startup directory, XML configuration directory, Logs directory and Completed DLL directory, and stores the directory information in the database. In the configuration file of ZORRO SERVICES, point the execution path of the DLL file and XML file to the DLL startup directory, XML configuration directory, and Completed DLL directory, and point the log storage path of the data mining task to the Logs directory.

[0028] After completing the preparatory configuration, start ZORRO SERVICES and start creating data mining tasks. Figure 1 , Figure 2 The present invention provides a method for creating a data mining task based on the ZORRO system, comprising the following steps:

[0029] S1: Fill in the ZORRO task configuration information in the ZORRO WEB system according to the data mining task requirements.

[0030] S2: Call the ZORRO API configuration generation interface to generate the DLL file and XML file required for the data mining plan task service according to the ZORRO task configuration information, and return the execution result.

[0031] S3: ZORRO SERVICES detects that new ZORRO task configuration information has been generated and starts executing the data mining plan task according to the configuration file requirements.

[0032] S4: Start the data mining plan task self-check thread, verify the parameter information including task configuration and planned execution time, and set the task execution plan.

[0033] S5: Execute the data mining plan task according to the preset data mining plan task operation mode.

[0034] Preferably, the data mining plan task operation mode includes: an extreme speed mode and a normal mode. In the normal mode, the data mining operation information needs to be recorded in the database.

[0035] S6: Query the total amount of data participating in this data mining plan task, and automatically allocate the number of processing threads of the data mining plan task according to the total amount of data.

[0036] S7: Add the processing thread to the thread scheduling queue, allocate the data to be mined to the processing thread, execute the corresponding processing thread task, and verify the processing result after the corresponding processing thread task is completed.

[0037] S8: Repeat step S7 until all processing threads have finished running, and recycle the system resources occupied by this data mining plan task.

[0038] In addition, during the execution of data mining plan tasks, the operation logs and real-time logs of each data mining plan task processing thread will be continuously generated during the operation of ZORRO SERVICES, including: program error log, program exception log, program debug log, program normal log. All log contents can be viewed in the ZORRO WEB system.

[0039] The present invention mentions a data mining task creation method based on the ZORRO system, including: filling in the ZORRO task configuration information in the ZORRO WEB system, calling the configuration generation interface to generate the DLL file and XML file required for the data mining plan task service, starting the data mining plan task, verifying the plan task, setting the task execution plan, selecting the data mining plan task operation mode, allocating the number of plan task processing threads according to the total amount of data, allocating the data to be mined to each processing thread, and executing the data mining task. The present invention can meet the data synchronization of customized complex logic data mining application scenarios, solve the problem of data synchronization interruption caused by task stop and the problem of slow page query response speed, and greatly improve the user experience.

[0040] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0041] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A data mining task creation method based on the ZORRO system, characterized in that: The method comprises: S1: Fill in the ZORRO task configuration information in the ZORRO WEB system according to the data mining task requirements; S2: Call the ZORRO API configuration generation interface to generate the DLL file and XML file required for the data mining plan task service according to the ZORRO task configuration information; S3: Detect that new ZORRO task configuration information is generated, and start executing the data mining plan task according to the configuration file requirements; S4: Start the data mining plan task self-check thread, verify the parameter information including task configuration and planned execution time, and set the task execution plan; S5: executing the data mining plan task according to the pre-set data mining plan task operation mode; S6: Query the total amount of data participating in this data mining plan task, and automatically allocate the number of processing threads of the data mining plan task according to the total amount of data; S7: adding the processing thread to the thread scheduling queue, allocating the data to be mined to the processing thread, executing the corresponding processing thread task, and verifying the processing result after the corresponding processing thread task is completed; S8: Repeat step S7 until all processing threads have finished running, and reclaim the system resources occupied by this data mining plan task; The method further comprises: Before executing the data mining task, deploy the ZORRO WEB system and ZORRO API system on the IIS server, create the execution path of the DLL file and XML file, install the ZORRO data mining scheduled task service, and write 1) the execution path of the DLL file and XML file; 2) the log storage path of the data mining scheduled task into the configuration file.

2. The data mining task creation method based on the ZORRO system according to claim 1, characterized in that: The data mining plan task operation modes include: extreme speed mode and normal mode.

3. The data mining task creation method based on the ZORRO system as claimed in claim 2, characterized in that: The method also includes recording the data mining task running information into a database in a normal mode.

4. The data mining task creation method based on the ZORRO system according to claim 1, characterized in that: The method further comprises: During the execution of the data mining plan task, the operation log and real-time log of the data mining plan task are generated, including: program error log, program exception log, program debugging log, program normal log. All log contents can be viewed in the ZORRO WEB system.

Images