Express transportation warehouse counting scheduling monitoring method, device and equipment and storage medium

Abstract

The invention relates to the technical field of data monitoring, in particular to an express number warehouse scheduling monitoring method, device and equipment and a storage medium, and the method comprises the following steps: creating a basic table based on a pre-designed structure of the basic table by using an SQL script; creating a shell script for realizing a monitoring and scheduling task, and setting a timed task; a shell script is executed regularly based on the timed task, when an abnormal condition occurs, abnormal information is acquired to confirm an abnormal type, an abnormal solution is acquired from the created basic table based on the confirmed abnormal type, and an abnormal solution instruction is generated according to the abnormal solution; the method disclosed by the invention is developed by using the shell + sql script language, has the advantages of high efficiency, light weight and the like, and can effectively reduce the development and maintenance threshold, improve the development efficiency, the development flexibility and the updating convenience so as to effectively monitor the business data index and the business data logic and reduce the possibility that the front-end application level lacks data support and the like.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring technology, and in particular to a method, device, equipment and storage medium for monitoring the scheduling of express data warehouses. Background Art

[0002] In modern enterprise operations, the scheduling and configuration of data warehouses plays a vital role; scheduling and configuration ensure that data can be updated and synchronized according to the established plan, so that data can continuously provide support for business decision-making and process optimization.

[0003] However, in the actual scheduling configuration process, various unforeseen abnormal situations are often encountered, which have a negative impact on the accuracy and timeliness of the data, such as:

[0004] 1. Changes in the data structure of the source system may cause data mismatch during data synchronization. Specifically, when the source system is upgraded or optimized, the original data structure may change. If the scheduling configuration is not updated in time, the data may not be correctly extracted and loaded into the data warehouse;

[0005] 2. Data in the source system is prone to input errors or delays. Human errors or system failures may cause errors in the data entry process, or the data may fail to arrive within the scheduled time, which directly affects the accuracy and real-time nature of the data in the data warehouse.

[0006] 3. When the data validation rules in the data warehouse are not set properly, it may cause wrong data to be received or correct data to be rejected incorrectly, thus affecting the data quality;

[0007] 4. Data warehouses usually rely on multiple data sources. If any data source is delayed or fails to provide data in a timely manner, the entire data processing process may be hindered.

[0008] Therefore, it is necessary to monitor the data scheduling configuration process of the data warehouse. It can be seen that the existing technology still needs to be improved and enhanced. Summary of the invention

[0009] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a fast data warehouse scheduling monitoring method, which is developed using shell+sql scripting language and has the advantages of high efficiency and light weight. It can effectively monitor business data indicators and business data logic, and reduce the possibility of problems such as lack of data support at the front-end application level.

[0010] The first aspect of the present invention provides a method for scheduling and monitoring an express data warehouse, comprising: designing the structure of a basic table, using an SQL script to create a basic table based on the designed structure of the basic table; creating a shell script for implementing monitoring and scheduling tasks, and setting a scheduled task; regularly executing the shell script based on the scheduled task to regularly monitor the express data warehouse; when an abnormal situation occurs, obtaining exception information to confirm the type of exception, and filling the obtained exception information into the created basic table; based on the confirmed exception type, obtaining an exception solution from the created basic table, and generating an exception resolution instruction according to the exception solution.

[0011] Optionally, in a first implementation method of the first aspect of the present invention, the structure of the basic table is designed, and a basic table is created based on the structure of the designed basic table using an SQL script, including: obtaining design information, the design information including task information, queue information, dependency information, trigger conditions and action information, and warning conditions and action information; designing the structure of the basic table based on the acquired design information, the basic table including a task table, a queue table, a dependency table, a trigger table, and a warning table; based on the structure of the designed basic table, writing corresponding SQL creation statements for each basic table to complete the creation of the basic table.

[0012] Optionally, in a second implementation method of the first aspect of the present invention, the creation of a shell script for implementing monitoring and scheduling tasks includes: writing a shell script for implementing monitoring and scheduling tasks based on the acquired design information; saving the written shell script in a preset file format to obtain a script file; granting execution permissions to the script file by executing a command to obtain an execution file; performing an execution test on the execution file, and deploying the execution file that passes the execution test to the express data warehouse scheduling monitoring system.

[0013] Optionally, in a third implementation of the first aspect of the present invention, the setting of the scheduled task includes: running a cron editor and opening a crontab configuration file corresponding to the cron editor; writing an execution plan for the periodic task in the crontab configuration file, the execution plan including an execution time and an execution path, and the execution path is an actual path of the execution file; saving the written crontab configuration file, and exiting the cron editor to complete the setting of the scheduled task.

[0014] Optionally, in a fourth implementation method of the first aspect of the present invention, the shell script is periodically executed based on the scheduled task to periodically monitor the express data warehouse, including: periodically executing the shell script based on the scheduled task, creating a monitoring log each time the shell script is executed to record the monitoring status corresponding to each execution; writing a monitoring script for monitoring the execution status of the shell script, the monitoring script including the name of the monitored shell script, the monitoring interval, a monitoring function for monitoring whether the shell script is running, and a restart function for restarting the shell script; when it is found that the shell script is not running normally, restarting the shell script based on the restart function.

[0015] Optionally, in a fifth implementation of the first aspect of the present invention, when an abnormal situation occurs, abnormal information is obtained to confirm the type of abnormality, and the acquired abnormal information is filled into the created basic table, including: judging whether there is an abnormal situation based on the monitoring log, and obtaining abnormal information when an abnormal situation occurs; performing preliminary classification of the abnormality type based on the error code in the abnormal information to obtain preliminary classification information; using regular expressions to extract key information from the preliminary classification information, and using a log analysis tool to perform statistical analysis on the extracted key information to obtain analysis information; inputting the analysis information into an abnormality recognition model pre-built based on a decision tree algorithm to obtain an abnormality type corresponding to the abnormal situation; and filling the acquired abnormal information and the abnormality type into the created basic table.

[0016] Optionally, in a sixth implementation manner of the first aspect of the present invention, based on the confirmed exception type, an exception solution is obtained from the created basic table, and an exception resolution instruction is generated according to the exception solution, including: in the created basic table, an identification index is created for the exception type identifier based on a binary search tree structure; based on the confirmed exception type and the created identification index, a string similarity algorithm is used to match the exception type identifier to extract the exception solution corresponding to the exception type identifier; exception resolution instructions are generated based on the extracted exception solution, and the generated exception resolution instructions are tested; and the exception resolution instructions that pass the instruction test are issued.

[0017] The second aspect of the present invention provides an express data warehouse scheduling and monitoring device, including: a first creation module, used to design the structure of a basic table, and use an SQL script to create a basic table based on the designed structure of the basic table; a second creation module, used to create a shell script for implementing monitoring and scheduling tasks, and set a scheduled task; an execution module, used to periodically execute the shell script based on the scheduled task to regularly monitor the express data warehouse; an acquisition module, used to obtain exception information to confirm the exception type when an abnormal situation occurs, and fill the acquired exception information into the created basic table; a generation module, used to obtain an exception solution from the created basic table based on the confirmed exception type, and generate an exception resolution instruction according to the exception solution.

[0018] Optionally, in a first implementation method of the second aspect of the present invention, the first creation module includes: a first acquisition unit, used to acquire design information, the design information including task information, queue information, dependency information, trigger conditions and action information, and warning conditions and action information; a design unit, used to design the structure of the basic table based on the acquired design information, the basic table including the task table, queue table, dependency table, trigger table and warning table; a first writing unit, used to write corresponding SQL creation statements for each basic table based on the designed structure of the basic table to complete the creation of the basic table.

[0019] Optionally, in a second implementation method of the second aspect of the present invention, the second creation module includes: a second writing unit, used to write a shell script for implementing monitoring and scheduling tasks based on the acquired design information; a first saving unit, used to save the written shell script in a preset file format to obtain a script file; a permission granting unit, used to grant execution permission to the script file by executing a command to obtain an execution file; a testing unit, used to perform an execution test on the execution file, and deploy the execution file that passes the execution test to the express data warehouse scheduling and monitoring system.

[0020] Optionally, in a third implementation of the second aspect of the present invention, the second creation module also includes: a running unit, used to run the cron editor and open the crontab configuration file corresponding to the cron editor; a third writing unit, used to write an execution plan for the periodic task in the crontab configuration file, the execution plan including the execution time and the execution path, and the execution path is the actual path of the execution file; a second saving unit, used to save the written crontab configuration file, and exit the cron editor to complete the setting of the scheduled task.

[0021] Optionally, in a fourth implementation method of the second aspect of the present invention, the execution module includes: an execution unit, used to periodically execute the shell script based on the scheduled task, and create a monitoring log each time the shell script is executed to record the monitoring status corresponding to each execution; a fourth writing unit, used to write a monitoring script for monitoring the execution status of the shell script, the monitoring script including the name of the monitored shell script, a monitoring interval, a monitoring function for monitoring whether the shell script is running, and a restart function for restarting the shell script; a restart unit, used to restart the shell script based on the restart function when it is found that the shell script is not running normally.

[0022] Optionally, in a fifth implementation of the second aspect of the present invention, the acquisition module includes: a second acquisition unit, used to determine whether there is an abnormal situation based on the monitoring log, and to obtain abnormal information when an abnormal situation occurs; a classification unit, used to perform preliminary classification of the abnormal type based on the error code in the abnormal information to obtain preliminary classification information; an analysis unit, used to extract key information from the preliminary classification information using regular expressions, and use a log analysis tool to perform statistical analysis on the extracted key information to obtain analysis information; an identification unit, used to input the analysis information into an abnormality identification model pre-built based on a decision tree algorithm to obtain an abnormality type corresponding to the abnormal situation; and a filling unit, used to fill the acquired abnormal information and the abnormality type into the created basic table.

[0023] Optionally, in a sixth implementation of the second aspect of the present invention, the generation module includes: a creation unit, used to create an identification index for the exception type identifier based on a binary search tree structure in the created basic table; a matching unit, used to match the exception type identifier based on the confirmed exception type and the created identification index, using a string similarity algorithm to extract the exception solution corresponding to the exception type identifier; a generation unit, used to generate exception resolution instructions based on the extracted exception solution, and perform instruction testing on the generated exception resolution instructions; and a publishing unit, used to publish the exception resolution instructions that pass the instruction test.

[0024] The third aspect of the present invention provides an express data warehouse scheduling monitoring device, which includes: a memory and at least one processor, wherein the memory stores instructions; at least one of the processors calls the instructions in the memory to enable the express data warehouse scheduling monitoring device to execute each step of the express data warehouse scheduling monitoring method described in any one of the above items.

[0025] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the various steps of any of the above-mentioned express data warehouse scheduling and monitoring methods are implemented.

[0026] In the technical solution of the present invention, a basic table is created based on the structure of a pre-designed basic table by using an SQL script; a shell script for realizing monitoring and scheduling tasks is created, and a scheduled task is set; the shell script is executed regularly based on the scheduled task, and when an abnormal situation occurs, the abnormal information is obtained to confirm the abnormal type, and the abnormal solution is obtained from the created basic table based on the confirmed abnormal type, and the abnormal solution is generated according to the abnormal solution. The method disclosed in the present application is developed using the shell+sql script language, and has the advantages of high efficiency and light weight, which can effectively reduce the development and maintenance threshold, improve development efficiency, development flexibility and convenience of updating, and enhance the reusability of the code to effectively monitor business data indicators and business data logic, and reduce the possibility of problems such as lack of data support at the front-end application level. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A first flow chart of a data processing method provided by an embodiment of the present invention;

[0028] Figure 2 A second flow chart of the data processing method provided by an embodiment of the present invention;

[0029] Figure 3 A third flow chart of the data processing method provided by an embodiment of the present invention;

[0030] Figure 4 A fourth flow chart of the data processing method provided by an embodiment of the present invention;

[0031] Figure 5 A fifth flow chart of the data processing method provided by the embodiment of the present invention;

[0032] Figure 6 A sixth flow chart of the data processing method provided by an embodiment of the present invention;

[0033] Figure 7 A seventh flow chart of the data processing method provided by the embodiment of the present invention;

[0034] Figure 8 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0035] Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention provides a method, device, equipment and storage medium for monitoring the scheduling of express data warehouses. In the present invention, the terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0038] 101. Design the structure of the basic table and use SQL script to create the basic table based on the designed structure of the basic table;

[0039] In this embodiment, by designing the structure of the basic table and creating the basic table, the data of the express data warehouse can be automatically monitored, reducing manual intervention and improving data processing efficiency.

[0040] 102. Create shell scripts for monitoring and scheduling tasks, and set up scheduled tasks;

[0041] In this embodiment, the use of shell scripts in combination with scheduled tasks can ensure that the monitoring tasks are regularly executed at predetermined time intervals, thereby ensuring the continuity and timeliness of monitoring.

[0042] 103. Regularly execute the shell script based on the scheduled task to regularly monitor the express data warehouse;

[0043] In this embodiment, by regularly executing the shell script, abnormal problems occurring during the dispatch configuration of the express data warehouse can be discovered in a timely manner, so as to achieve rapid resolution of the abnormal problems and improve the stability and reliability of the data transmission process.

[0044] 104. When an abnormal situation occurs, obtain the abnormal information to confirm the abnormal type, and fill the acquired abnormal information into the created basic table;

[0045] In this embodiment, when an abnormal situation occurs during the monitoring process, the system can automatically obtain the abnormal information and record it in the basic table to facilitate subsequent abnormal analysis and processing. The recorded abnormal information can be used as data support to help managers make decision-making analysis to optimize the operating status of the express data warehouse.

[0046] 105. Based on the confirmed exception type, obtain the exception solution from the created basic table, and generate the exception resolution instruction according to the exception solution;

[0047] In this embodiment, according to the confirmed exception type, the system can automatically match the corresponding solution and generate exception resolution instructions to achieve rapid response and resolution of the exception problem.

[0048] The present application discloses a method for dispatching and monitoring an express data warehouse. By using an SQL script, a basic table is created based on the structure of a pre-designed basic table; a shell script for realizing monitoring and scheduling tasks is created, and a scheduled task is set; the shell script is periodically executed based on the scheduled task, and when an abnormal situation occurs, the abnormal information is obtained to confirm the abnormal type, and the abnormal solution is obtained from the created basic table based on the confirmed abnormal type, and an abnormal solution instruction is generated according to the abnormal solution; the method disclosed in the present application is developed using the shell+sql script language, and has the advantages of high efficiency and light weight, which can effectively reduce the threshold of development and maintenance, improve development efficiency, development flexibility and convenience of update, and enhance the reusability of the code, so as to effectively monitor business data indicators and business data logic, and reduce the possibility of problems such as lack of data support at the front-end application level.

[0049] See also Figure 2 The second embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0050] 201. Acquire design information, where the design information includes task information, queue information, dependency information, trigger conditions and action information, and warning conditions and action information;

[0051] In this embodiment, task information refers to a detailed description of a specific task that needs to be completed in a system or project. The detailed description must clearly define its objectives, execution steps, and expected results. Queue information involves the order in which tasks are arranged and how they are processed in the system. Queue information determines the order in which tasks are executed, ensuring that tasks are processed according to established priorities and schedules, which helps optimize resource allocation and improve overall efficiency. Dependency information describes the interdependencies between tasks. Some tasks may not start until other tasks are completed, or the execution results of some tasks may affect the execution of other tasks. Dependency information helps the system reasonably arrange the execution order of tasks to avoid execution failures due to task conflicts. Trigger conditions And action information refers to actions that are automatically performed under specific conditions. The specific conditions may be time, events or other specific triggering factors. When the specific conditions are met, the system will automatically perform corresponding actions, such as sending notifications, starting tasks or updating data, etc., which helps to achieve automated processing and improve the response speed and efficiency of the system; warning conditions and action information refer to the warning mechanism that the system will automatically trigger when certain abnormalities or potential problems occur during the operation of the system. The warning conditions may be system performance degradation, insufficient resources or other abnormal indicators. When the warning conditions are met, the system will perform corresponding warning actions, such as sending alarms, suspending tasks or launching emergency plans, etc., which helps to discover and deal with problems in a timely manner and ensure the stable operation of the system.

[0052] 202. Design a structure of a basic table based on the acquired design information, wherein the basic table includes a task table, a queue table, a dependency table, a trigger table, and an early warning table;

[0053] In this embodiment, the task table records all task information to be executed, including key attributes such as task ID, task type, priority, expected start time and end time, which helps to grasp the overall arrangement of tasks; the queue table is used to manage the execution order of tasks. When a task is completed, the next task will be automatically taken out of the queue for execution, ensuring that the tasks can be executed according to the established priority and order; the dependency table records the dependencies between tasks. For example, some tasks may need to wait for other tasks to be completed before they can be started. By maintaining the dependencies, the correctness and integrity of the data processing flow are ensured; the trigger table defines the trigger actions under specific conditions. For example, if a task times out and is not completed, the corresponding remedial measures can be triggered by the rules in the trigger table, such as rescheduling the task or sending an alarm notification; the warning table is used to store preset warning conditions and their corresponding processing strategies. When the monitored data reaches or exceeds the preset threshold (such as excessive CPU usage), actions will be taken according to the rules in the warning table, such as automatically adjusting resource allocation or sending an alarm email to the administrator.

[0054] 203. Based on the designed structure of the basic table, write the corresponding SQL creation statement for each basic table to complete the creation of the basic table.

[0055] See also Figure 3 The third embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0056] 301. Based on the obtained design information, write shell scripts for implementing monitoring and scheduling tasks;

[0057] In this embodiment, the basic framework of the written shell script may include a script name, a function for executing a monitoring task, a function for executing a scheduling task, and a main function.

[0058] 302. Save the written shell script in a preset file format to obtain a script file;

[0059] In this embodiment, the shell script can be saved as a "monitor_and_schedule.sh" file to obtain a script file.

[0060] 303. Grant execution permission to the script file by executing the command to obtain the execution file;

[0061] In this embodiment, the execution permission is granted to the script file through the "chmod+x" execution command to obtain the execution file.

[0062] 304. Perform execution test on the execution file, and deploy the execution file that passes the execution test to the express data warehouse scheduling monitoring system;

[0063] In this embodiment, the operation of the execution file is tested through the ". / monitor_and_schedule.sh" command. When the execution file works as expected and no errors occur, it indicates that the written shell script can correctly perform monitoring and scheduling tasks, and the shell script can be deployed to the express warehouse scheduling monitoring system.

[0064] See also Figure 4 The fourth embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0065] 401. Run the cron editor and open the crontab configuration file corresponding to the cron editor;

[0066] 402. Write an execution plan for a periodic task in a crontab configuration file, wherein the execution plan includes an execution time and an execution path, and the execution path is an actual path of the execution file;

[0067] In this embodiment, the format of the execution plan of the written periodic task can be "***** / path / to / command", where the five asterisks represent minutes, hours, days, months, and days of the week, respectively, and " / path / to / command`" is the complete path of the command or script to be executed regularly.

[0068] 403. Save the written crontab configuration file and exit the cron editor to complete the setting of the scheduled task.

[0069] See also Figure 5 The fifth embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0070] 501. Periodically executing the shell script based on the scheduled task, and creating a monitoring log each time the shell script is executed to record the monitoring situation corresponding to each execution;

[0071] In this embodiment, by periodically executing shell scripts through scheduled tasks, continuous monitoring of the system or application can be achieved without manual intervention, thereby improving work efficiency; a monitoring log is created each time a shell script is executed, which can record the monitoring situation in real time, making it easier to track and analyze problems; the monitoring log records the monitoring situation of each execution in detail, which helps to quickly locate and solve problems that arise, and can ensure the continuity and reliability of the monitoring process, reducing the risk of forgetting to manually execute monitoring tasks.

[0072] 502. Write a monitoring script for monitoring the execution of the shell script, wherein the monitoring script includes the name of the monitored shell script, a monitoring interval, a monitoring function for monitoring whether the shell script is running, and a restart function for restarting the shell script;

[0073] 503. When it is found that the shell script does not run normally, restart the shell script based on the restart function;

[0074] In this embodiment, by writing a monitoring script, continuous monitoring of the shell script can be achieved without manual intervention, thereby improving work efficiency; the monitoring script regularly checks the running status of the shell script, and once an abnormality is found, the script is immediately restarted, reducing system failure time and ensuring that the shell script can be quickly restored when a problem occurs, thereby improving the stability and reliability of the entire system.

[0075] See also Figure 6 The sixth embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0076] 601. Determine whether there is an abnormal situation based on the monitoring log, and when an abnormal situation occurs, obtain abnormal information;

[0077] In this embodiment, the abnormal conditions include task execution time being too long, task execution failure, task queue mismatch, task data file not arriving, task not triggering a job, checking whether the task is valid, the task is configured with invalid dependencies, confirming whether there is a problem with the configuration, task delay and non-execution, and no valid job in the task queue.

[0078] 602. Perform preliminary classification of the exception type based on the error code in the exception information to obtain preliminary classification information;

[0079] In this embodiment, a preliminary classification of the exception types is performed using pre-set rules to classify the error codes into different exception categories.

[0080] 603. Use regular expressions to extract key information from the preliminary classification information, and use log analysis tools to perform statistical analysis on the extracted key information to obtain analysis information;

[0081] In this embodiment, regular expressions are used to extract key information from the information after preliminary classification. Regular expressions are a powerful text processing tool that can identify and extract character strings that conform to specific patterns. Splunk log analysis tools are then used to perform statistical analysis on the extracted key information, including counting, frequency analysis, trend analysis, etc., to obtain more in-depth abnormal information.

[0082] 604. Input the analysis information into a pre-built abnormality recognition model based on a decision tree algorithm to obtain an abnormality type corresponding to the abnormal situation;

[0083] In this embodiment, the information obtained from the statistical analysis is input into a pre-built anomaly recognition model based on a decision tree algorithm; the decision tree is a commonly used machine learning algorithm that can perform classification based on the characteristics of the input data.

[0084] 605. Fill the acquired exception information and the exception type into the created basic table;

[0085] In this embodiment, the identified exception information and the corresponding exception type are recorded in a basic table, which may be a database table or any other form of data storage structure for storing and managing exception data.

[0086] See also Figure 7 The seventh embodiment of the express warehouse scheduling monitoring method in the embodiment of the present invention includes:

[0087] 701. In the created basic table, create an identification index for the abnormal type identification based on a binary search tree structure;

[0088] In this embodiment, by creating an identification index for the abnormal type identification based on a binary search tree structure in the basic table, the efficiency of data retrieval can be significantly improved; a binary search tree (BST) is an ordered tree structure that allows fast search, insertion, and deletion operations; in the scenario of abnormal type identification, using BST as an index structure can achieve fast search, efficient insertion and deletion, search orderliness, and dynamic adjustment.

[0089] 702. Based on the confirmed exception type and the created identification index, a string similarity algorithm is used to match the exception type identification to extract the exception solution corresponding to the exception type identification.

[0090] In this embodiment, the Levenshtein distance algorithm can be used to match the abnormality type identification, and the matching can be achieved through the string similarity algorithm, which can quickly identify and match known abnormality types, improve the efficiency of problem diagnosis, reduce manual intervention, and realize the extraction of automated solutions.

[0091] 703. Generate an exception resolution instruction based on the extracted exception resolution solution, and perform an instruction test on the generated exception resolution instruction;

[0092] In this embodiment, a test environment similar to the production environment is selected to ensure that all necessary software and hardware resources are ready; then a set of data sets containing known abnormal situations is created or selected, and these data sets should be able to trigger specific problems mentioned in the abnormal solutions; according to the extracted abnormal solutions, all necessary parameters and conditions are set to ensure that the instructions can be executed correctly; run the abnormal solution instructions and test them with the prepared test data; record all outputs and results during the test and check whether the test results meet expectations. If the test results meet expectations, the abnormal solution instructions can be issued.

[0093] 704. Issue an exception resolution instruction that passes the instruction test.

[0094] The above describes the method for dispatching and monitoring the express warehouse in the embodiment of the present invention. The following describes the device for dispatching and monitoring the express warehouse in the embodiment of the present invention. Figure 8 In one embodiment of the present invention, an express warehouse scheduling monitoring device includes:

[0095] The first creation module 801 is used to design the structure of the basic table, and use the SQL script to create the basic table based on the designed structure of the basic table; the second creation module 802 is used to create a shell script for implementing monitoring and scheduling tasks, and set a scheduled task; the execution module 803 is used to regularly execute the shell script based on the scheduled task to regularly monitor the express data warehouse; the acquisition module 804 is used to obtain exception information to confirm the exception type when an exception occurs, and fill the obtained exception information into the created basic table; the generation module 805 is used to obtain the exception solution from the created basic table based on the confirmed exception type, and generate exception resolution instructions according to the exception solution.

[0096] In this embodiment, the first creation module 801 includes: a first acquisition unit 8011, used to acquire design information, the design information includes task information, queue information, dependency information, trigger conditions and action information, and warning conditions and action information; a design unit 8012, used to design the structure of the basic table based on the acquired design information, the basic table includes a task table, a queue table, a dependency table, a trigger table and a warning table; a first writing unit 8013, used to write corresponding SQL creation statements for each basic table based on the designed structure of the basic table to complete the creation of the basic table.

[0097] In this embodiment, the second creation module 802 includes: a second writing unit 8021, which is used to write a shell script for implementing monitoring and scheduling tasks based on the acquired design information; a first saving unit 8022, which is used to save the written shell script in a preset file format to obtain a script file; a permission granting unit 8023, which is used to grant execution permission to the script file by executing a command to obtain an execution file; a testing unit 8024, which is used to perform an execution test on the execution file, and deploy the execution file that passes the execution test to the express data warehouse scheduling monitoring system.

[0098] In this embodiment, the second creation module 802 also includes: a running unit 8025, which is used to run the cron editor and open the crontab configuration file corresponding to the cron editor; a third writing unit 8026, which is used to write an execution plan for the periodic task in the crontab configuration file, and the execution plan includes an execution time and an execution path, and the execution path is the actual path of the execution file; a second saving unit 8027, which is used to save the written crontab configuration file and exit the cron editor to complete the setting of the scheduled task.

[0099] In this embodiment, the execution module 803 includes: an execution unit 8031, which is used to periodically execute the shell script based on the scheduled task, and create a monitoring log each time the shell script is executed to record the monitoring situation corresponding to each execution; a fourth writing unit 8032, which is used to write a monitoring script for monitoring the execution of the shell script, and the monitoring script includes the name of the monitored shell script, the monitoring interval time, a monitoring function for monitoring whether the shell script is running, and a restart function for restarting the shell script; a restart unit 8033, which is used to restart the shell script based on the restart function when it is found that the shell script is not running normally.

[0100] In this embodiment, the acquisition module 804 includes: a second acquisition unit 8041, which is used to determine whether there is an abnormal situation based on the monitoring log, and when an abnormal situation occurs, obtain the abnormal information; a classification unit 8042, which is used to perform preliminary classification of the abnormal type based on the error code in the abnormal information to obtain preliminary classification information; an analysis unit 8043, which is used to extract key information from the preliminary classification information using regular expressions, and use a log analysis tool to perform statistical analysis on the extracted key information to obtain analysis information; an identification unit 8044, which is used to input the analysis information into an abnormality recognition model pre-built based on a decision tree algorithm to obtain an abnormality type corresponding to the abnormal situation; a filling unit 8045, which is used to fill the acquired abnormal information and the abnormality type into the created basic table.

[0101] In this embodiment, the generation module 805 includes: a creation unit 8051, which is used to create an identification index for the exception type identifier in the created basic table based on a binary search tree structure; a matching unit 8052, which is used to match the exception type identifier based on the confirmed exception type and the created identification index, using a string similarity algorithm to extract the exception solution corresponding to the exception type identifier; a generation unit 8053, which is used to generate an exception solution instruction based on the extracted exception solution, and perform an instruction test on the generated exception solution instruction; a publishing unit 8054, which is used to publish the exception solution instruction that passes the instruction test.

[0102] Based on the same idea as the method in the above embodiment, the device provided by the present application can implement the method in the above embodiment.

[0103] above Figure 8 The express data warehouse scheduling monitoring device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The express data warehouse scheduling monitoring device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0104] Fig. 9 : is a structural diagram of a fast data warehouse scheduling monitoring device provided by an embodiment of the present invention. The fast data warehouse scheduling monitoring device 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 910 (for example, one or more processors) and a memory 920, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 933 or data 932. Among them, the memory 920 and the storage medium 930 can be short-term storage or persistent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the fast data warehouse scheduling monitoring device 900. Furthermore, the processor 910 can be configured to communicate with the storage medium 930, and execute a series of instruction operations in the storage medium 930 on the fast data warehouse scheduling monitoring device 900 to implement the steps of the fast data warehouse scheduling monitoring method provided by the above-mentioned method embodiments.

[0105] The express warehouse scheduling monitoring device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input and output interfaces 960, and / or one or more operating systems 931, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It can be understood by those skilled in the art that Fig. 9 The structure of the express data warehouse scheduling monitoring equipment shown does not constitute a limitation of the express data warehouse scheduling monitoring equipment, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0106] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the express data warehouse scheduling monitoring method.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0109] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for monitoring the dispatching of express warehouses, characterized in that: include: Design the structure of the basic table, and use SQL scripts to create the basic table based on the designed structure of the basic table; Create shell scripts for monitoring and scheduling tasks, and set up scheduled tasks; The shell script is periodically executed based on the scheduled task to regularly monitor the express data warehouse; When an abnormal situation occurs, obtain the abnormal information to confirm the abnormal type, and fill the acquired abnormal information into the created basic table; Based on the confirmed exception type, an exception solution is obtained from the created basic table, and an exception resolution instruction is generated according to the exception solution.

2. The method for dispatching and monitoring express warehouses according to claim 1, characterized in that: The structure of the designed basic table is designed, and a basic table is created based on the structure of the designed basic table using an SQL script, including: Acquire design information, wherein the design information includes task information, queue information, dependency information, trigger conditions and action information, and warning conditions and action information; Designing a structure of a basic table based on the acquired design information, wherein the basic table includes a task table, a queue table, a dependency table, a trigger table, and an early warning table; Based on the designed structure of the basic table, write the corresponding SQL creation statement for each basic table to complete the creation of the basic table.

3. The method for dispatching and monitoring express warehouses according to claim 2 is characterized in that: The shell script created to implement monitoring and scheduling tasks includes: Based on the obtained design information, write shell scripts for implementing monitoring and scheduling tasks; Use the preset file format to save the written shell script to obtain the script file; Grant execution permission to the script file by executing the command to obtain the execution file; Perform execution tests on the execution files, and deploy the execution files that pass the tests to the express data warehouse scheduling and monitoring system.

4. The method for dispatching and monitoring express warehouses according to claim 3 is characterized in that: Described and set up scheduled tasks, including: Run the cron editor and open the crontab configuration file corresponding to the cron editor; Write an execution plan for periodic tasks in the crontab configuration file. The execution plan includes the execution time and execution path. The execution path is the actual path of the execution file. Save the written crontab configuration file and exit the cron editor to complete the setting of the scheduled task.

5. The method for dispatching and monitoring express warehouses according to claim 1, characterized in that: The periodic execution of the shell script based on the scheduled task to periodically monitor the express data warehouse includes: Periodically executing the shell script based on the scheduled task, and creating a monitoring log each time the shell script is executed to record the monitoring situation corresponding to each execution; Writing a monitoring script for monitoring the execution of the shell script, wherein the monitoring script includes the name of the monitored shell script, a monitoring interval, a monitoring function for monitoring whether the shell script is running, and a restart function for restarting the shell script; When it is found that the shell script is not running normally, the shell script is restarted based on the restart function.

6. The method for dispatching and monitoring express warehouses according to claim 5, characterized in that: When an abnormal situation occurs, the abnormal information is obtained to confirm the abnormal type, and the acquired abnormal information is filled into the created basic table, including: Determine whether there is an abnormal situation based on the monitoring log, and obtain abnormal information when an abnormal situation occurs; Perform preliminary classification of the exception type based on the error code in the exception information to obtain preliminary classification information; Use regular expressions to extract key information from the preliminary classification information, and use log analysis tools to perform statistical analysis on the extracted key information to obtain analysis information; Inputting the analysis information into a pre-built abnormality recognition model based on a decision tree algorithm to obtain an abnormality type corresponding to the abnormal situation; The acquired exception information and the exception type are filled into the created basic table.

7. The method for dispatching and monitoring express warehouses according to claim 1, characterized in that: The method of obtaining an exception solution from the created basic table based on the confirmed exception type and generating an exception resolution instruction according to the exception solution includes: In the created basic table, an identification index is created for the abnormal type identification based on a binary search tree structure; Based on the confirmed exception type and the created identification index, a string similarity algorithm is used to match the exception type identification to extract the exception solution corresponding to the exception type identification; Generate an exception resolution instruction based on the extracted exception resolution solution, and perform instruction testing on the generated exception resolution instruction; Issue exception resolution instructions that pass the instruction test.

8. A fast-transport warehouse dispatch monitoring device, characterized in that: include: The first creation module is used to design the structure of the basic table and use the SQL script to create the basic table based on the designed structure of the basic table; The second creation module is used to create shell scripts for implementing monitoring and scheduling tasks, and setting up scheduled tasks; An execution module, used for periodically executing the shell script based on the scheduled task to regularly monitor the express data warehouse; The acquisition module is used to acquire the abnormal information to confirm the abnormal type when an abnormal situation occurs, and fill the acquired abnormal information into the created basic table; The generation module is used to obtain an exception solution from the created basic table based on the confirmed exception type, and generate an exception resolution instruction according to the exception solution.

9. A fast-transport warehouse dispatch monitoring device, characterized in that: The express data warehouse scheduling monitoring device comprises: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors calls the instructions in the memory to enable the express data warehouse scheduling monitoring device to execute each step of the express data warehouse scheduling monitoring method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the express data warehouse scheduling and monitoring method as described in any one of claims 1 to 7 are implemented.

Images