Big data platform scheduling warning method, device and computer-readable storage medium

Abstract

The present invention relates to a big data technology, and discloses a big data platform scheduling early warning method, including: acquiring and analyzing historical source table data and historical scheduling task data to obtain historical label data, monitoring the current source table and the current scheduling task to obtain current data, comparing the historical label data with the current data to obtain a comparison result, judging abnormal information data in the comparison result, extracting keywords in the abnormal information data, using the keywords to determine the error scenario, and using a preset method to issue an early warning. The present invention also proposes a big data platform scheduling early warning device, an electronic device, and a computer-readable storage medium. The present invention can determine the scheduling error scenario and send the error scenario to the user for early warning.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device, electronic device and computer-readable storage medium for scheduling and warning of a big data platform. Background Art

[0002] With the rise of big data and artificial intelligence, the demand for huge data is increasing. The existing big data platform can clean, analyze and summarize the data through the existing scheduling tools to generate business analysis reports. However, the existing scheduling tools cannot determine the scheduling error scenario and cannot send the error scenario to the user for early warning. Summary of the invention

[0003] The present invention provides a big data platform scheduling warning method, device, electronic device and computer-readable storage medium, the main purpose of which is to determine the scheduling error scenario and send the error scenario to the user for warning.

[0004] To achieve the above purpose, the present invention provides a big data platform scheduling early warning method, comprising:

[0005] Acquire historical source table data, analyze the historical source table data, and obtain historical source table label data;

[0006] Acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data;

[0007] Monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data;

[0008] Compare the current source table data with the historical source table label data to obtain source table data monitoring comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain scheduling task comparison results;

[0009] Determine the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, perform keyword extraction processing on the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for early warning.

[0010] Optionally, the acquiring historical source table data, analyzing the historical source table data, and obtaining historical source table label data includes:

[0011] The historical source table data is calculated using a maximum analysis algorithm to obtain a maximum source table data volume and a maximum source table data increment, and the maximum source table data volume and the maximum source table data increment are summarized to obtain the historical source table label data.

[0012] Optionally, the acquiring of historical scheduling task data, analyzing the historical scheduling task data, and obtaining historical scheduling task label data includes:

[0013] According to the task type of the historical scheduling task data, the historical scheduling task data is classified, counted and calculated to obtain the running start time, running end time, running duration and maximum resource consumption of each task type of the historical scheduling task data;

[0014] The running start time, running end time, running duration and maximum resource consumption of each task type are summarized to obtain the historical scheduling task label data.

[0015] Optionally, the monitoring of the current source table and the current scheduled task to obtain the current source table data and the current scheduled task data includes:

[0016] The JobMonitor program is used to monitor the current source table and the current scheduled task, and the current source table data including the current source table data volume and the current source table data increment and the current task scheduling data including the task type, the start time, end time, running duration and resource consumption of the task of this type are obtained.

[0017] Optionally, the determining abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes:

[0018] When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is judged as abnormal information data;

[0019] When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is judged as abnormal information data; and

[0020] When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is judged as abnormal information data.

[0021] Optionally, performing keyword extraction processing on the abnormal information data to obtain keywords, using the keywords to determine the error reporting scenario, and using a preset method to push the abnormal information data and the error reporting scenario for early warning includes:

[0022] Calculating the gravity value, relevance, and occurrence frequency of the characters in the abnormal information data to obtain the keywords of the error information;

[0023] The abnormal information data and the error scenario are sent to the user for early warning in a preset manner.

[0024] In order to solve the above problems, the present invention also provides a big data platform scheduling early warning device, the device comprising:

[0025] A label data acquisition module is used to acquire historical source table data, analyze the historical source table data, obtain historical source table label data, acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data;

[0026] A data monitoring and comparison module is used to monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data, compare the current source table data with the historical source table label data to obtain the source table data monitoring and comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain the scheduling task comparison results;

[0027] The comparison result judgment and early warning module is used to judge the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, perform keyword extraction processing on the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for early warning.

[0028] Optionally, the determining abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes:

[0029] When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is judged as abnormal information data;

[0030] When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is judged as abnormal information data; and

[0031] When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is judged as abnormal information data.

[0032] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0033] a memory storing at least one instruction; and

[0034] The processor executes the instructions stored in the memory to implement the above-mentioned big data platform scheduling and early warning method.

[0035] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned big data platform scheduling and warning method.

[0036] The embodiment of the present invention first obtains and analyzes historical source table data and historical scheduling task data to obtain historical source table label data and historical scheduling task label data, then monitors the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data, compares the historical source table label data with the current source table data to obtain the source table data monitoring and comparison results, compares the historical scheduling task label data with the current scheduling task data to obtain the scheduling task comparison results, judges the abnormal information data in the source table data monitoring and comparison results and the scheduling task comparison results, obtains keywords by performing keyword extraction processing on the abnormal information data, uses the keywords to determine the error scenario, and uses a preset method to push the abnormal information data and the error scenario for early warning. Therefore, the big data platform scheduling early warning method, device, and computer-readable storage medium proposed in the present invention can determine the scheduling error scenario and send the error scenario to the user for early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a process flow of a big data platform scheduling and early warning method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a module of a big data platform scheduling and early warning method provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the internal structure of an electronic device of a big data platform scheduling and early warning method provided by an embodiment of the present invention;

[0040] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] The present invention provides a big data platform scheduling early warning method. Figure 1 FIG. 1 is a flow chart of a method for scheduling and warning a big data platform according to an embodiment of the present invention. The method can be executed by a device, which can be implemented by software and / or hardware.

[0043] In this embodiment, the big data platform scheduling early warning method includes:

[0044] S1. Obtain historical source table data, analyze the historical source table data, and obtain historical source table label data.

[0045] The source table data described in the embodiment of the present invention includes the ODS (Open Data Services) layer table of the big data platform and the full table data volume of the fact table and dimension table processed by ODS, and the daily increment.

[0046] In detail, analyzing the historical source table data to obtain the historical source table label data includes: using a maximum analysis algorithm to calculate the historical source table data to obtain the maximum value of the source table data volume and the maximum value of the source table data increment, and summarizing the maximum value of the source table data volume and the maximum value of the source table data increment to obtain the historical source table label data.

[0047] Furthermore, the maximum value analysis algorithm may use the SQL MAX() function, for example:

[0048] There is a student score table A in the database that contains all subjects. To find the highest score in the Chinese subject in the student score table in the database, you can use the SQL MAX() function, such as: SELECT MAX(Chinese) FROM A.

[0049] S2. Obtain historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data.

[0050] The scheduling task data described in the embodiment of the present invention includes different types of task data of the big data platform, including MapReduce, Hive running time and running duration, consumed resources, etc.

[0051] In detail, an embodiment of the present invention classifies, counts and calculates the historical scheduling task data according to the task type of the historical scheduling task data to obtain the running start time, running end time, running duration and maximum resource consumption of each task type of the historical scheduling task data.

[0052] Furthermore, the embodiment of the present invention can use the SQLGROUP BY function to classify, count and calculate the historical scheduling task data, obtain the running start time and running end time of each task type, use the SQL MAX() function to calculate the maximum value of resource consumption, and use the SQLdatediff() function to calculate the running time.

[0053] In detail, the embodiment of the present invention summarizes the running start time, running end time, running duration and maximum resource consumption of each task type to obtain the historical scheduling task label data.

[0054] S3. Monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data.

[0055] In detail, the current source table includes the source table currently being used by the big data platform, and the current scheduled task includes the task currently being run by the big data platform.

[0056] The embodiment of the present invention utilizes the JobMonitor program to monitor the current source table and the current scheduled task, and obtains the current source table data and the current task scheduling data.

[0057] Preferably, the current source table data includes the current source table data volume and the current source table data increment, and the current task scheduling data includes the task type, the running start time, running end time, running duration and resource consumption of the task of this type.

[0058] Furthermore, the JobMonitor program refers to a program that monitors the status of running tasks, monitors the task pool, and monitors tasks waiting to be run.

[0059] S4. Compare the current source table data with the historical source table label data to obtain source table data monitoring comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain scheduling task comparison results.

[0060] In detail, the embodiment of the present invention can calculate the difference between the data volume and data increment of the current source table data and the maximum value of the corresponding data volume and data increment in the historical source table tag data to obtain the source table data monitoring comparison result;

[0061] Furthermore, the embodiment of the present invention can calculate the difference between the start and end time of the current scheduled task and the start and end time of the corresponding task in the historical scheduled task label data, and calculate the difference between the resource consumption of the current scheduled task and the maximum resource consumption corresponding to the historical scheduled task label data to obtain the scheduling task comparison result.

[0062] S5. Determine whether there is abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result.

[0063] Specifically, the determining of abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes:

[0064] When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is abnormal information data;

[0065] When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is abnormal information data;

[0066] When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is abnormal information data.

[0067] Further, if there is no abnormal data in the source table data monitoring comparison result and the scheduling task comparison result, return to S3; if there is abnormal data in the source table data monitoring comparison result and the scheduling task comparison result, execute S6.

[0068] S6. Extract and process the abnormal information data to obtain keywords, use the keywords to determine the error reporting scenario, and use a preset method to push the abnormal information data and the error reporting scenario for early warning.

[0069] In detail, the embodiment of the present invention uses the TF-IDF algorithm to calculate the gravity value, relevance, and occurrence frequency of the characters in the abnormal information data to obtain the keywords of the error information, including:

[0070] Using the character W in the abnormal information data i and W j The gravity value and correlation between the characters are used to obtain the dependency correlation between the characters:

[0071]

[0072] Among them, len(W i ,W j ) represents the character W i and W j The length of the dependency path between them, b is a hyperparameter;

[0073] Use the following formula to calculate the character W: i and W j The attraction between:

[0074]

[0075] Where tfidf(W) is the TF-IDF value of character W, and d is the i and W j The Euclidean distance between the word vectors of ;

[0076] Therefore, we get the character W i and W j The correlation between them is:

[0077] weight(W i,W j )=Dep(W i ,W j )*f grav (W i ,W j );

[0078] Furthermore, the frequency of occurrence of each character in the abnormal information data is calculated, and the character with the highest frequency is selected as a keyword in the abnormal information data.

[0079] For example: Calculate the character W in the abnormal information data i The frequency of occurrence is as follows:

[0080]

[0081] in is the character set of abnormal information data, W i , W j , W k is the character in the abnormal information data, and η is the damping coefficient which can be set manually.

[0082] Furthermore, the embodiment of the present invention uses the keyword to determine the error scenario. If the abnormal information data is 'Half an hour later than the task start time and has not started running', and the extracted keyword is 'late', then the scheduling error scenario at this time is determined to be a task delay scenario.

[0083] Furthermore, the embodiment of the present invention uses methods including email and telephone to send the abnormal information data and the error scenario to the user for early warning.

[0084] like Figure 2 The figure shows a functional module diagram of the big data platform scheduling and early warning device of the present invention.

[0085] The big data platform scheduling and early warning device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the big data platform scheduling and early warning device can include a tag data acquisition module 101, a data monitoring and comparison module 102, and a comparison result judgment and early warning module 103. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0086] In this embodiment, the functions of each module / unit are as follows:

[0087] The label data acquisition module 101 is used to acquire historical source table data, analyze the historical source table data, obtain historical source table label data, and acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data.

[0088] The source table data described in the embodiment of the present invention includes the ODS (Open Data Services) layer table of the big data platform and the full table data volume of the fact table and dimension table processed by ODS, and the daily increment.

[0089] In detail, analyzing the historical source table data to obtain the historical source table label data includes: using a maximum analysis algorithm to calculate the historical source table data to obtain the maximum value of the source table data volume and the maximum value of the source table data increment, and summarizing the maximum value of the source table data volume and the maximum value of the source table data increment to obtain the historical source table label data.

[0090] Furthermore, the maximum value analysis algorithm may use the SQL MAX() function, for example:

[0091] There is a student score table A in the database that contains all subjects. To find the highest score in the Chinese subject in the student score table in the database, you can use the SQL MAX() function, such as: SELECT MAX(Chinese) FROM A.

[0092] The scheduling task data described in the embodiment of the present invention includes different types of task data of the big data platform, including MapReduce, Hive running time and running duration, consumed resources, etc.

[0093] In detail, an embodiment of the present invention classifies, counts and calculates the historical scheduling task data according to the task type of the historical scheduling task data to obtain the running start time, running end time, running duration and maximum resource consumption of each task type of the historical scheduling task data.

[0094] Furthermore, the embodiment of the present invention can use the SQLGROUP BY function to classify, count and calculate the historical scheduling task data, obtain the running start time and running end time of each task type, use the SQL MAX() function to calculate the maximum value of resource consumption, and use the SQLdatediff() function to calculate the running time.

[0095] In detail, the embodiment of the present invention summarizes the running start time, running end time, running duration and maximum resource consumption of each task type to obtain the historical scheduling task label data.

[0096] The data monitoring and comparison module 102 is used to monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data, compare the current source table data with the historical source table label data to obtain the source table data monitoring and comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain the scheduling task comparison results.

[0097] In detail, the current source table includes the source table currently being used by the big data platform, and the current scheduled task includes the task currently being run by the big data platform.

[0098] The embodiment of the present invention utilizes the JobMonitor program to monitor the current source table and the current scheduled task, and obtains the current source table data and the current task scheduling data.

[0099] Preferably, the current source table data includes the current source table data volume and the current source table data increment, and the current task scheduling data includes the task type, the running start time, running end time, running duration and resource consumption of the task of this type.

[0100] Furthermore, the JobMonitor program refers to a program that monitors the status of running tasks, monitors the task pool, and monitors tasks waiting to be run.

[0101] In detail, the embodiment of the present invention can calculate the difference between the data volume and data increment of the current source table data and the maximum value of the corresponding data volume and data increment in the historical source table tag data to obtain the source table data monitoring comparison result;

[0102] Furthermore, the embodiment of the present invention can calculate the difference between the start and end time of the current scheduled task and the start and end time of the corresponding task in the historical scheduled task label data, and calculate the difference between the resource consumption of the current scheduled task and the maximum resource consumption corresponding to the historical scheduled task label data to obtain the scheduling task comparison result.

[0103] The comparison result judgment and warning module 103 is used to judge the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, extract and process the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for warning.

[0104] Specifically, the determining of abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes:

[0105] When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is abnormal information data;

[0106] When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is abnormal information data;

[0107] When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is abnormal information data.

[0108] Furthermore, if there is no abnormal data in the source table data monitoring comparison result and the scheduling task comparison result, the data monitoring comparison module is returned; if there is abnormal data in the source table data monitoring comparison result and the scheduling task comparison result, the keywords of the error information are extracted.

[0109] Furthermore, the embodiment of the present invention uses the TF-IDF algorithm to calculate the gravity value, relevance, and occurrence frequency of the characters in the abnormal information data to obtain the keywords of the error information, including:

[0110] Using the character W in the abnormal information data i and W j The gravity value and correlation between the characters are used to obtain the dependency correlation between the characters:

[0111]

[0112] Among them, len(W i ,W j ) represents the character W i and W j The length of the dependency path between them, b is a hyperparameter;

[0113] Use the following formula to calculate the character W: i and W j The attraction between:

[0114]

[0115] Where tfidf(W) is the TF-IDF value of character W, and d is the i and W j The Euclidean distance between the word vectors of ;

[0116] Therefore, we get the character W i and W j The correlation between them is:

[0117] weight(Wi ,W j )=Dep(W i ,W j )*f grav (W i ,W j );

[0118] Furthermore, the frequency of occurrence of each character in the abnormal information data is calculated, and the character with the highest frequency is selected as a keyword in the abnormal information data.

[0119] For example: Calculate the character W in the abnormal information data i The frequency of occurrence is as follows:

[0120]

[0121] in is the character set of abnormal information data, W i , W j , W k is the character in the abnormal information data, and η is the damping coefficient which can be set manually.

[0122] Furthermore, the embodiment of the present invention uses the keyword to determine the error scenario. If the abnormal information data is 'Half an hour later than the task start time and has not started running', and the extracted keyword is 'late', then the scheduling error scenario at this time is determined to be a task delay scenario.

[0123] Furthermore, the embodiment of the present invention uses methods including email and telephone to send the abnormal information data and the error scenario to the user for early warning.

[0124] like Figure 3 The figure is a schematic diagram of the structure of an electronic device for implementing the big data platform scheduling and early warning method of the present invention.

[0125] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a big data platform scheduling and warning program.

[0126] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the big data platform scheduling warning program, but also can be used to temporarily store data that has been output or is to be output.

[0127] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a big data platform scheduling and early warning program, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0128] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0129] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0131] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0132] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0133] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0134] The big data platform scheduling warning program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0135] Acquire historical source table data, analyze the historical source table data, and obtain historical source table label data;

[0136] Acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data;

[0137] Monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data;

[0138] Compare the current source table data with the historical source table label data to obtain source table data monitoring comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain scheduling task comparison results;

[0139] Determine the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, perform keyword extraction processing on the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for early warning.

[0140] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0141] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0142] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0143] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0145] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0146] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0147] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A big data platform scheduling early warning method, characterized in that: The method comprises: Acquire historical source table data, analyze the historical source table data, and obtain historical source table label data, wherein the historical source table data includes the open data service ODS layer table of the big data platform and the full table data volume and daily increment of the fact table and dimension table processed through the ODS layer table, calculate the historical source table data using a maximum value analysis algorithm, obtain the maximum data volume and the maximum data increment of the historical source table, and summarize the maximum data volume and the maximum data increment to obtain the historical source table label data; Acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data; Monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data; Compare the current source table data with the historical source table label data to obtain source table data monitoring comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain scheduling task comparison results; Determine the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, perform keyword extraction processing on the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for early warning.

2. The big data platform scheduling and early warning method according to claim 1, characterized in that: The acquiring of historical source table data, analyzing the historical source table data, and obtaining historical source table label data includes: The historical source table data is calculated using a maximum analysis algorithm to obtain a maximum source table data volume and a maximum source table data increment, and the maximum source table data volume and the maximum source table data increment are summarized to obtain the historical source table label data.

3. The big data platform scheduling and early warning method according to claim 1, characterized in that: The acquiring of historical scheduling task data, analyzing the historical scheduling task data, and obtaining historical scheduling task label data includes: According to the task type of the historical scheduling task data, the historical scheduling task data is classified, counted and calculated to obtain the running start time, running end time, running duration and maximum resource consumption of each task type of the historical scheduling task data; The running start time, running end time, running duration and maximum resource consumption of each task type are summarized to obtain the historical scheduling task label data.

4. The big data platform scheduling and early warning method according to any one of claims 1 to 3, characterized in that: The monitoring of the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data includes: The JobMonitor program is used to monitor the current source table and the current scheduled task, and the current source table data including the current source table data volume and the current source table data increment and the current task scheduling data including the task type, the start time, end time, running duration and resource consumption of this type of task are obtained.

5. The big data platform scheduling and early warning method according to claim 4, characterized in that: The determining of abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes: When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is judged as abnormal information data; When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is judged as abnormal information data; and When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is judged as abnormal information data.

6. The big data platform scheduling and early warning method according to claim 5, characterized in that: The performing keyword extraction processing on the abnormal information data to obtain keywords, using the keywords to determine the error reporting scenario, and using a preset method to push the abnormal information data and the error reporting scenario for early warning, includes: Calculating the gravity value, relevance, and occurrence frequency of the characters in the abnormal information data to obtain keywords of the abnormal information data; The abnormal information data and the error scenario are sent to the user for early warning in a preset manner.

7. A big data platform dispatching and early warning device, characterized in that: The device comprises: A label data acquisition module is used to acquire historical source table data, analyze the historical source table data, obtain historical source table label data, acquire historical scheduling task data, analyze the historical scheduling task data, and obtain historical scheduling task label data, wherein the historical source table data includes the open data service ODS layer table of the big data platform and the full table data volume and daily increment of the fact table and dimension table processed through the ODS layer table, and the historical source table data is calculated using the maximum value analysis algorithm to obtain the maximum data volume and the maximum data increment of the historical source table, and the maximum data volume and the maximum data increment are summarized to obtain the historical source table label data; A data monitoring and comparison module is used to monitor the current source table and the current scheduling task to obtain the current source table data and the current scheduling task data, compare the current source table data with the historical source table label data to obtain the source table data monitoring and comparison results, and compare the current scheduling task data with the historical scheduling task label data to obtain the scheduling task comparison results; The comparison result judgment and early warning module is used to judge the abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result, perform keyword extraction processing on the abnormal information data to obtain keywords, use the keywords to determine the error scenario, and use a preset method to push the abnormal information data and the error scenario for early warning.

8. The big data platform dispatching and early warning device according to claim 7, characterized in that: The determining of abnormal information data in the source table data monitoring comparison result and the scheduling task comparison result includes: When the data volume and data increment of the current source table data exceed the maximum values ​​of the corresponding data volume and data increment in the historical source table tag data, it is judged as abnormal information data; When the running start and end time of the current scheduled task are inconsistent with the running start and end time of the corresponding task in the historical scheduled task tag data, it is judged as abnormal information data; and When the resource consumption of the current scheduled task exceeds the corresponding maximum resource consumption in the historical scheduled task tag data, it is judged as abnormal information data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the big data platform scheduling and early warning method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the big data platform scheduling and early warning method as described in any one of claims 1 to 6 is implemented.

Images