Scheduling method of unified scheduling platform based on big data backtracking task

Abstract

The invention discloses a scheduling method of a unified scheduling platform based on a big data backtracking task. The scheduling method comprises the following steps: S1, configuring basic attributes of a big data backtracking task package in a visual page of the unified scheduling platform; s2, newly adding detailed configuration of a backtracking task in a visual page of the unified scheduling platform according to basic attributes of related backtracking task requirements; s3, for the newly added backtracking task, converting the element information of different encryption modes in the sample into an element information file of the same encryption mode, retaining the element information of the original encryption mode, and adding an index column according to the sample file; s4, for the successfully verified backtracking task package, performing task merging in a mode of combining the package with additional parameters at regular time, and submitting task calculation; and S5, business personnel perform task basic information viewing, state updating and file operation on the submitted package task through the visual page. The backtracking task package can be flexibly configured, and result information display can be performed for the backtracking task.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a scheduling method for a unified scheduling platform based on big data backtracking tasks. Background Art

[0002] With the rapid development of the Internet and mobile Internet, in order to meet the needs of different users for backtesting of various big data risk control products, a related scheduling tool platform is designed and developed to help relevant users get test results more efficiently and accurately and perform data analysis on them. Therefore, developing a unified scheduling platform based on big data backtracking tasks has important technical and practical significance for meeting the needs of modern large-scale applications, improving system performance and user experience, and solving problems existing in the early stages of the system. In the early stages of the product, only a single sample of the user is processed and the sample element information is required to be processed in the same encryption method. The multi-data source mode of the sample is not supported, and the result file is only saved in a single data source. Notifications such as task success or failure and task information are only notified to relevant users via email. With the increase in backtracking task tests, business personnel need to process each batch of samples into element information with a unified encryption method, and need to submit N tasks for processing. After the task is calculated, it is also necessary to return the element information in the unified encryption method in the result to the element information in the original encryption method. In this case, not only the resources of the big data cluster are wasted, but also the time for users to submit tasks is wasted, and the cost and risk of localized processing results are increased. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a scheduling method based on a unified scheduling platform for big data backtracking tasks, which can flexibly configure backtracking task packages, flexibly select and submit backtracking tasks according to sample types, verify element information samples according to different encryption methods as unified element samples, merge successfully verified backtracking packages of the same type and submit them for task calculation, and display result information for backtracking tasks.

[0004] The technical solution of the present invention is: a scheduling method based on a unified scheduling platform for big data backtracking tasks, comprising the following steps:

[0005] S1. Based on the actual business situation, the technicians configure the basic attributes of the big data backtracking task package in the unified scheduling platform visualization page;

[0006] S2. Business personnel submit package tasks according to their needs, and add detailed configuration of the backtracking tasks in the unified scheduling platform visualization page according to the basic attributes of the relevant backtracking task requirements;

[0007] S3. For the newly added backtracking task, the element information of different encryption methods in the sample is converted into the element information file of the same encryption method and the element information of the original encryption method is retained. An index column is added according to the sample file to facilitate the subsequent return of the result file to be consistent with the encryption method and order of the original sample;

[0008] S4. For the successfully verified backtracking task package, regularly merge the package + additional parameters and submit the task calculation;

[0009] S5. Business personnel can use the visualization page to view basic task information, update status, and perform file operations on the submitted package tasks.

[0010] Furthermore, step S1 includes the following steps:

[0011] S11. Technical personnel can visually add, delete, modify and check the package attributes of the big data backtracking task according to the actual business situation. The package attributes include: dictionary name, dictionary key value, dictionary sorting, and dictionary remarks (additional parameter information that needs to be configured for this package).

[0012] Furthermore, step S2 includes the following steps:

[0013] S21. Business personnel can submit the backtracking task according to the usage of the business and the relevant backtracking task requirements in the same scheduling platform according to the package attributes set by the technicians in S11;

[0014] S22, default properties - export type can be selected as HIVE table, CSV file, HIVE table + CSV file format;

[0015] S23, default attribute - sample type can be selected as HVIE table name or uploaded file format;

[0016] When uploading files, you must comply with the platform's file name format requirements. If it does not comply with the relevant agreements, it will be prompted that the file is illegal and you need to upload the file again.

[0017] Furthermore, step S3 includes the following steps:

[0018] S31. After the package is added successfully, the sample file of the package is verified, and the element information of different encryption methods (MD5, SHA256, SM3, MD5) is converted into the element information of the same encryption method (MD5) and the element information of the original encryption method is retained;

[0019] S32. Add an index column identifier according to the sample file that is passed in, so that the calculation results returned later are consistent with the element encryption method and order of the original sample file;

[0020] S33, changing the status of the package task to verification success.

[0021] Furthermore, step S4 includes the following steps:

[0022] S41, taking the successfully verified task as a condition, the packages that need to be calculated and the additional parameter identifiers provided by the packages are grouped, the packages are merged and a backtracking task is created;

[0023] S42, submit the created backtracking task to the execution queue, and perform calculation once according to the task calculation parallelism set by the business personnel; if the previous batch task of a certain time has not been completed, put the batch task into the queue and wait for calculation;

[0024] S43, after the submitted backtracking task is calculated, the different sample files and result files in the batch task are split and processed, the element information and other field information of the original encryption method of the sample are retained, the element information verified as the unified encryption method in S31 is removed, and the files are sorted according to the index column identifier added in S32;

[0025] S44, according to the export type selected in S22 for different samples in the same backtracking task, saving the splitting result of S43 to a corresponding result file type;

[0026] S45. The total number of items obtained from the comprehensive search of different samples in the same backtracking task is saved in a relational database to facilitate subsequent data analysis by business personnel, and the system is notified that the calculation of the backtracking task for this batch is completed.

[0027] Furthermore, step S5 includes the following steps:

[0028] S51. For successfully executed backtracking tasks, business personnel can view the basic information of the newly added backtracking package: file name, package name, sample type, verification task name, backtracking task name, upload time, status, planned execution time, total number of original data records, number of valid data records, total number of comprehensive search records, HIVE verification result table name, HIVE result table name, failure description, extended field, creator, creation time, update time, etc.; and can operate the source file and result file of the backtracking task;

[0029] S52. For the failed backtracking tasks, the technical personnel shall troubleshoot the problem according to the backtracking task name in the basic information of the backtracking package in S51. If it is not a sample problem, the business personnel may modify the task status of the backtracking task to verification success and return to step S4.

[0030] Furthermore, a unified scheduling platform based on big data backtracking tasks includes flexible configuration of backtracking task packages, flexible selection of submitting backtracking tasks according to sample types, verification of element information samples according to different encryption methods as unified element samples, merging of successfully verified backtracking packages of the same type and submitting them for task calculation, and display of result information for backtracking tasks.

[0031] The flexible configuration of the backtracking task package is to perform addition, deletion, modification and query operations on the backtracking task package through visual operation rules.

[0032] The flexible selection of submitting the backtracking task according to the sample type is to select the backtracking task packages of different sample types and different result types through visual operation rules.

[0033] The purpose of verifying the element information samples according to the encryption method as elements of the unified encryption method is that, through the automated verification of the database collision operation, the element information of different encryption methods is converted into element information of the unified encryption method, which is convenient for subsequent backtracking task calculation operations.

[0034] The method of merging successfully verified backtracking packages of the same type and submitting them for task calculation is to merge and calculate the same backtracking task in the form of package + additional parameters, reduce cluster resource waste, and split the results of samples according to the same backtracking task of different samples.

[0035] The display of result information for the backtracking task is to display the result information of the backtracking task and provide operations on the source file and the result file, so as to facilitate subsequent data analysis.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. Flexibly configure and submit backtrace task packages, and submit backtrace tasks based on different sample files.

[0038] 2. Automated conversion operations convert element information with different encryption methods into element information with a unified encryption method to ensure the consistency of element information.

[0039] 3. To meet the needs of the same backtracking task, different samples can be merged to submit the backtracking task, and the backtracking results of the merged samples can be split according to the samples to reduce the waste of cluster resources and improve the work efficiency of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Shown is the calculation flow chart of the system of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and implementation examples of the present invention.

[0042] like Figure 1 As shown, the present invention comprises the following steps:

[0043] S1. Technical personnel can maintain the attributes of available backtracking task packages on the unified scheduling platform according to the actual business situation. The attributes include: dictionary name, dictionary key value, dictionary sort, and dictionary remarks (additional parameter information that needs to be configured for the package).

[0044] In the implementation of the present invention, new package attributes are configured visually, and the attributes of future packages can also be deleted, modified, and checked.

[0045] S2. Business personnel can select corresponding packages based on the usage of the business and the relevant backtracking task requirements in the same scheduling platform, and submit the backtracking task according to the package attributes set by the technical staff in S11.

[0046] In the implementation of the present invention, the visual selection requires detailed configuration of the newly added retrospective package.

[0047] A. Perform file verification based on the selected sample type. If the selected sample type is in the form of uploaded files, the uploaded file name needs to be verified. If it does not meet the agreed requirements, it will prompt that the file name is illegal and needs to be re-uploaded to submit the backtracking task.

[0048] S3. After obtaining the samples uploaded by the user, the unified scheduling platform verifies and processes the newly added traceback task samples.

[0049] In the implementation of the present invention, Figure 1 The sample verification process is shown in the following figure:

[0050] A. After the package is added successfully, obtain the sample data of different sample types uploaded by the business personnel, determine the encryption method of the elements in the sample (based on the sample column name), and convert the element information of different encryption methods into element information of the same encryption method and save the element information of the original encryption method, so that after the subsequent backtracking task calculation is successful, the result file is saved in the same encryption method as the original sample.

[0051] B. Then add index column identifiers to the sample so that after the subsequent backtracking task is successfully calculated, the order of the result file is consistent with the order of the original sample file.

[0052] C. Save the verified files to facilitate subsequent backtracking task calculations.

[0053] S4. Regularly merge the successfully verified backtracking tasks and submit them for task calculation.

[0054] In the implementation of the present invention, Figure 1The calculation process of the multi-sample backtracking task is shown in the figure.

[0055] A. Scheduled tasks (which can also be triggered manually on the unified scheduling platform) scan tasks with a successful verification status, use the successfully verified tasks as conditions, identify the packages that need to be calculated and the additional parameters provided by the packages as groups, merge the packages and create a batch of backtracking tasks.

[0056] B. Put the submitted tasks into the task queue, and submit them for calculation in sequence according to the task calculation parallelism set by the business personnel, maintaining the principle of first-in-first-out. If some urgent tasks need to be processed, the priority of the task batch can be manually adjusted to give priority to calculation. And the parallelism of task calculation can also be adjusted according to the usage of big data cluster resources.

[0057] C. After receiving the submitted task, Big Data will parse the task parameters and aggregate and deduplicate all the incoming sample files.

[0058] D. For the aggregated sample, calculate the relevant results according to the incoming package selection, and save the calculation results of the aggregated sample.

[0059] E. Traverse and split the result files according to the sample set passed in, and save the sample results corresponding to the sample to the corresponding export type according to the order of the original sample files according to the export type selected in S2. Store the result hit rate corresponding to the sample (the amount of data in each batch of sample files after splitting) in the relational database to facilitate the business personnel who submit the task to conduct subsequent data analysis.

[0060] S5. The visualization page performs subsequent operations on the backtracking task.

[0061] In the implementation of the present invention, Figure 1 As shown,

[0062] A. If the task execution fails, the technical staff will coordinate to investigate the cause of the error. If it is not due to sample factors, the task status will be changed to successful verification and the task will wait for the next batch of tasks to be scheduled. If it is due to human factors such as samples, the file needs to be uploaded again and returned to step S2.

[0063] B. If the task is successfully executed, the business personnel can view the basic information of the backtracking task on the visualization platform, including:

[0064] File name, package name, sample type, verification task name, backtracking task name, upload time, status, planned execution time, total number of original data records, number of valid data records, total number of comprehensive search records, HIVE verification result table name, HIVE result table name, failure description, extended field, creator, creation time, update time, etc. And operate on the source file and result file of the backtracking task.

[0065] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A scheduling method based on a unified scheduling platform for big data backtracking tasks, characterized in that: The following steps are involved: S1. Configure the basic properties of the big data backtracking task package in the unified scheduling platform visualization page; S2. Add detailed configuration of the backtracking task according to the basic attributes of the relevant backtracking task requirements in the unified scheduling platform visualization page; S3. For the newly added backtracking task, the element information of different encryption methods in the sample is converted into the element information file of the same encryption method and the element information of the original encryption method is retained. An index column is added according to the sample file to facilitate the subsequent return of the result file to be consistent with the encryption method and order of the original sample; S4. For the successfully verified backtracking task package, regularly merge the package + additional parameters and submit the task calculation; S5. Business personnel can view basic task information, update status, and perform file operations on submitted package tasks through the visualization page.

2. The scheduling method of a unified scheduling platform based on big data backtracking tasks according to claim 1 is characterized in that: Step S1 includes the following steps: S11. Package attributes of the visualized add, delete, modify and query big data backtracking task, wherein the package attributes include: dictionary name, dictionary key value, dictionary sorting, and dictionary remarks.

3. The scheduling method of a unified scheduling platform based on big data backtracking tasks according to claim 2 is characterized in that: Step S2 includes the following steps: S21. Business personnel submit the backtracking task according to the usage of the business and the relevant backtracking task requirements in the same scheduling platform according to the package attributes set by the technical personnel in S11; S22, default properties - export type can be selected as HIVE table, CSV file, HIVE table + CSV file format; S23, default attribute - sample type can be selected as HVIE table name or uploaded file format; When uploading files, you must comply with the platform's file name format requirements. If it does not comply with the relevant agreements, it will be prompted that the file is illegal and you need to upload the file again.

4. The scheduling method of a unified scheduling platform based on big data backtracking tasks according to claim 1 is characterized in that: Step S3 includes the following steps: S31. After the package is added successfully, the sample file of the package is verified, the element information of different encryption methods is converted into element information of the same encryption method and the element information of the original encryption method is retained; S32. Add an index column identifier according to the sample file that is passed in, so that the calculation results returned later are consistent with the element encryption method and order of the original sample file; S33, changing the status of the package task to verification success.

5. The scheduling method of a unified scheduling platform based on big data backtracking tasks according to claim 4 is characterized in that: Step S4 includes the following steps: S41, taking the successfully verified task as a condition, the packages that need to be calculated and the additional parameter identifiers provided by the packages are grouped, the packages are merged and a backtracking task is created; S42, submit the created backtracking task to the execution queue, and perform calculation once according to the task calculation parallelism set by the business personnel; if the previous batch task of a certain time has not been completed, put the batch task into the queue and wait for calculation; S43, after the submitted backtracking task is calculated, the different sample files and result files in the batch task are split and processed, the element information field information of the original encryption method of the sample is retained, the element information verified as the unified encryption method in S31 is removed, and sorted according to the index column identifier added in S32; S44, according to the export type selected in S22 for different samples in the same backtracking task, saving the splitting result of S43 to a corresponding result file type; S45. The total number of items obtained from the comprehensive search of different samples in the same backtracking task is saved in a relational database to facilitate subsequent data analysis by business personnel, and the system is notified that the calculation of the backtracking task for this batch is completed.

6. The scheduling method of a unified scheduling platform based on big data backtracking tasks according to claim 1 is characterized in that: Step S5 includes the following steps: S51. For successfully executed backtracking tasks, business personnel can view the basic information of the newly added backtracking package: file name, package name, sample type, verification task name, backtracking task name, upload time, status, planned execution time, total number of original data records, number of valid data records, total number of comprehensive retrieval records, HIVE verification result table name, HIVE result table name, failure description, extended field, creator, creation time, and update time; and can operate the source file and result file of the backtracking task; S52. For the failed backtracking tasks, the technical personnel shall troubleshoot the problem according to the backtracking task name in the basic information of the backtracking package in S51. If it is not a sample problem, the business personnel may modify the task status of the backtracking task to verification success and return to step S4.

Images