Method for solving scheduling task oriented to multiple portrait label jobs

Abstract

The invention discloses a method for solving a multi-portrait label job-oriented scheduling task, which belongs to the technical field of system development and comprises the following steps of: 1, acquiring data required by a user portrait and taking the data as a source end of offline data scheduling; 2, synchronizing fields into a hive data warehouse; 3, checking the number of source end data and the number of data synchronized into the portrait intermediate table, if the number of source end data is consistent with the number of data synchronized into the portrait intermediate table, starting a scheduling job of offline label data, calculating the offline label data of the user portrait, and if the number of source end data is not consistent with the number of data synchronized into the portrait intermediate table, deleting the synchronized data of the day and rescheduling; and 4, checking whether the calculated offline label data is normal, if not, sending a mail and enterprise WeChat to inform related personnel, and if yes, starting a scheduling task of aggregating the user label data. According to the method, the required label can be automatically generated, rescheduling and mail notification sending are carried out when the task goes wrong, and the reliability and stability of data are ensured.

Description

technical field [0001] The invention relates to the technical field of system development, in particular to a method for solving job scheduling tasks for multiple portrait labels. Background technique [0002] Due to the requirements of the user portrait system, hundreds of label scripts need to be developed. In order to reduce the coupling between each label data, each label needs to submit a spark task separately, and a scheduled job is required to refresh the new labels generated the previous day. . But many problems arise: [0003] 1. The generation of tags requires specialized personnel to write calculation scripts. With the increase of tags, the scale of scheduling tasks increases, and the human cost increases; [0004] 2. The dependencies between tasks are messy; [0005] 3. It is not convenient to check which task is currently being executed; [0006] 4. Problems cannot be quickly located; [0007] 5. It is not convenient to record the execution of historical sc...

AI Technical Summary

Problems solved by technology

[0003] 1. The generation of tags requires specialized personnel to write calculation scripts. With the increase of tags, the scale of scheduling tasks increases, and the human cost increases;

[0004] 2. The dependencies between tasks are messy;

[0005] 3. It is not convenient to check which task is currently being executed;

[0006] 4. Problems cannot be quickly located;

[0007] 5. It is not convenient to record the execution of historical scheduling tasks

Images