Defect task processing method, device, equipment and storage medium

By automatically identifying and processing defect tasks, using logs, regular expressions and scripts, the cumbersome problem of task recognition and processing in the data platform is solved, efficient defect task management is achieved, and labor and time costs are reduced.

CN114418019BActive Publication Date: 2024-11-05PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210081558.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-24
Publication Date
2024-11-05
Estimated Expiration
2042-01-24

AI Technical Summary

Technical Problem

In the prior art, the process of institutions identifying and handling defective tasks in data platforms is cumbersome, and they rely on manual query and communication, resulting in resource tightness and high labor costs.

Method used

Through automated identification of defects, logs, regular expressions and scripts are used to identify invalid, partitioned unscanned and data tables over-threshold tasks, generate personalized processing suggestions and notify processors, reducing labor and time costs.

Benefits of technology

The identification and processing process of defect tasks is simplified, the identification accuracy is ensured, labor and time costs are reduced, and the stability of the data platform is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114418019B_ABST
    Figure CN114418019B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of big data technology, and discloses a defect task processing method, device, equipment and storage medium for reducing labor and time costs. It includes: identifying defect tasks through preset log recognition methods, regular expressions and scripts, and automatically identifying defect tasks. This method replaces manual operations, can automatically identify defect tasks in various applications, generate corresponding task information for each defect task, including personalized processing suggestions, save the task information of the defect task to a preset defect list table, and notify the corresponding processing personnel according to the task information, so that the processing personnel can rectify the defect task according to the processing suggestions, simplifying the identification and processing process of defect tasks, ensuring the accuracy of the identification results, and reducing labor and time costs. In addition, the present invention also relates to distributed technology, and the big data platform is a distributed computing platform that can run applications of different organizations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device, equipment and storage medium for processing defective tasks. Background Art

[0002] In order to solve the problems of inconsistent data sources for multiple institutions, poor data timeliness, and difficulty in supporting institutional business data needs, the data platform provides a "data front-end service platform". This service provides institutions with three core supports: platform, system, and data, and has become an important part of helping institutions transform into digital ones. With the launch and promotion of this service, the number of tasks of institutions continues to grow. Affected by the limitations of information technology of institutional personnel, the performance of institutional tasks cannot be guaranteed. With limited cluster resources, the growing number of tasks and tasks that cannot guarantee performance may lead to tight institutional resources and continued deterioration. Therefore, in order to ensure the stability of the service platform, it is necessary to identify and resolve task defects in the institution.

[0003] In related technologies, it is necessary to manually query defective tasks in an organization and propose targeted rectification suggestions for different defective tasks. At the same time, it is necessary to communicate with the person in charge of the defective task in the organization. The process is cumbersome. The method of manually counting the daily increase and decrease of defective tasks is complicated and cannot guarantee the quality. Continuous manpower investment is required during the service period, and the labor and time costs are huge. Summary of the invention

[0004] The present invention provides a defect task processing method, device, equipment and storage medium, which are used to identify defect tasks in an automated manner, generate personalized processing suggestions, and notify corresponding processing personnel to handle the defect tasks. The identification process is simple, and the labor and time costs are reduced while ensuring the identification accuracy.

[0005] To achieve the above-mentioned purpose, the first aspect of the present invention provides a method for processing defective tasks, comprising: identifying invalid tasks of each application by identifying logs in each application; identifying tasks for which partition tables of each application are not partition scanned and tasks for which views are not directly used according to preset regular expressions; identifying tasks for which data table sizes of each application exceed preset thresholds according to preset scripts; obtaining relevant information of the defective task according to the task type of the defective task; wherein different types of defective tasks obtain different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect and task resource consumption; for each defective task, generating task information of the defective task according to the relevant information of the defective task and preset configuration information; wherein the task information at least includes: defect type, defective task name, resource consumption information and processing suggestions; saving the task information of the defective task in a preset defect list table; sending the task information of each defective task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defective task through the task information.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the step of identifying defective tasks of each application by identifying logs in each application includes: obtaining a first data table that has not been accessed within a preset historical time period from the audit log of each application, and determining the related tasks of the first data table as the invalid tasks; according to a preset script, the step of identifying tasks whose data table size exceeds a preset threshold of each application includes: obtaining the data volume of each data table in each application through a preset script, and determining a second data table that exceeds the preset threshold according to the data volume; and determining the related tasks of the second data table as tasks whose data table size exceeds the preset threshold.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of identifying the tasks for which partition scanning has not been performed on the partition table through a preset regular expression includes: identifying the partition table added in each application and the tasks related to the partition table through a preset regular expression; identifying the SQL script of the tasks related to the partition table through a preset regular expression, and determining, through the filtering conditions in the SQL script, that the tasks for which partition fields of the partition table have not been filtered and / or the partitions of the day have not been filtered are the tasks for which partition scanning has not been performed on the partition table.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of identifying the tasks that are not directly used by the view through a preset regular expression includes: identifying the SQL scripts of the tasks in the various applications through a preset regular expression; and determining through the SQL script that the tasks that exist in both the headquarters view and the data table are the tasks that are not directly used by the view before they are implemented.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the task information of the defect task also includes the information of the handler of the defect task; the step of sending the task information of each defect task in the defect list table to the corresponding handler includes: generating a new defect upload request according to the task information of the defect task; the new defect upload request includes the information of the handler of the defect task; sending the new defect upload request to the central system, and through the central system, sending the new defect upload request to the handler of the defect task according to the handler information.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the method further includes: obtaining submitted defect tasks in the defect list table; wherein the submitted defect tasks include: defect tasks whose task information has been submitted among all defect tasks that have not been identified within a preset time period; generating a defect upload closing request based on the submitted defect tasks, sending the defect upload closing request to a central system, and controlling the closure of the submitted defect tasks through the central system.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, the method further includes: receiving first feedback information sent by the processing personnel regarding the defect task; obtaining second feedback information for each of the defect tasks collected in the central system; determining a target defect task from the defect tasks of each identified application based on the first feedback information and the second feedback information; saving the target defect task to a pre-set whitelist table, and deleting the target defect task in the defect list table.

[0012] The second aspect of the present invention provides a device for processing defective tasks, including: a first identification module, used to identify defective tasks of each application by identifying logs in the respective applications; a second identification module, used to identify tasks for which partition tables of the respective applications have not been partition scanned and tasks for which views have not been directly used according to a preset regular expression; a third identification module, used to identify tasks for which the size of a data table of the respective applications exceeds a preset threshold according to a preset script; an acquisition module, used to acquire relevant information of the defective task according to the task type of the defective task; wherein different types of defective tasks acquire different relevant information; the relevant information It at least includes: task title, defect level, defect type, task name involved in the defect and task resource consumption; a generation module, used to generate task information of the defect task for each defect task according to the relevant information of the defect task and preset configuration information; wherein the task information at least includes: defect type, defect task name, resource consumption information and processing suggestions; a saving module, used to save the task information of the defect task into a preset defect list table; a sending module, used to send the task information of each defect task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defect task through the task information.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the first identification module is specifically used for: a first identification unit, used to obtain, from the audit logs of the respective applications, a first data table that has not been accessed within a preset historical time period, and determine the related tasks of the first data table as the invalid tasks; a second identification unit, used to identify, through a preset regular expression, tasks for which partition scans of the partition table have not been performed and tasks for which the view has not been directly used; a third identification unit, used to obtain, through a preset script, the data volume of each data table in the respective applications, and determine, based on the data volume, a second data table that exceeds the preset threshold; and determine the related tasks of the second data table as tasks for which the data table size exceeds the preset threshold.

[0014] Optionally, in a second implementation method of the second aspect of the present invention, the second identification module is specifically used to: identify the partition tables added in the various applications and the tasks related to the partition tables through a preset regular expression; identify the SQL scripts of the tasks related to the partition tables through a preset regular expression, and determine through the filtering conditions in the SQL scripts that the partition fields of the partition table and / or the tasks that have not filtered the partition fields of the partition table and / or the partitions of the day are tasks that have not performed partition scanning on the partition table.

[0015] Optionally, in a third implementation method of the second aspect of the present invention, the second identification module is specifically used to: identify the SQL scripts of the tasks in the various applications through a preset regular expression; and determine through the SQL script that the tasks for which both the headquarters view and the data table exist are tasks that are directly used without the view being implemented.

[0016] Optionally, in a fourth implementation method of the second aspect of the present invention, the task information of the defect task also includes the information of the person who handles the defect task; the sending module is specifically used to: generate a new defect upload request according to the task information of the defect task; the new defect upload request includes the information of the person who handles the defect task; send the new defect upload request to the central system, and through the central system, send the new defect upload request to the person who handles the defect task according to the information of the person who handles the defect task.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the device also includes a task closing module, which is used to: obtain submitted defect tasks in the defect list table; wherein the submitted defect tasks include: defect tasks whose task information has been submitted among all defect tasks that have not been identified within a preset time period; generate a defect upload closing request based on the submitted defect tasks, send the defect upload closing request to the central system, and control the closure of the submitted defect tasks through the central system.

[0018] Optionally, in a sixth implementation method of the second aspect of the present invention, the device also includes a task deletion module, which is used to: receive first feedback information sent by the processing personnel regarding the defect task; obtain second feedback information for each of the defect tasks collected in the central system; determine a target defect task from the defect tasks of each identified application based on the first feedback information and the second feedback information; save the target defect task to a pre-set whitelist table, and delete the target defect task in the defect list table.

[0019] The third aspect of the present invention provides a defect task processing device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the defect task processing device executes the above-mentioned defect task processing method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the above-mentioned defect task processing method.

[0021] In the technical solution provided by the present invention, by identifying the logs in each application, the invalid tasks of each application are identified; according to the preset regular expression, the tasks of the partition table of each application that have not been partition scanned and the tasks of the view that have not been directly used are identified; according to the preset script, the tasks of the data table size of each application that exceeds the preset threshold are identified; according to the task type of the defective task, the relevant information of the defective task is obtained; wherein the relevant information obtained for different types of defective tasks is different; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect and task resource consumption; for each defective task, the task information of the defective task is generated according to the relevant information of the defective task and the preset configuration information; wherein the task information at least includes: defect type, defective task name, resource consumption information and processing suggestions; the task information of the defective task is saved in a preset defect list table; the task information of each defective task in the defect list table is sent to the corresponding processing personnel, so that the processing personnel can rectify the defective task through the task information. In the embodiment of the present invention, defect tasks are identified through pre-set log recognition methods, regular expressions, and scripts to achieve the purpose of automatically identifying defect tasks. This method replaces manual operations, can automatically identify defect tasks, and generate corresponding task information for each defect task, including personalized processing suggestions. At the same time, the corresponding processing personnel are notified to process the defect task according to the processing suggestions, which simplifies the identification and processing process of defect tasks, ensures the accuracy of the identification results, and reduces labor and time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an embodiment of a method for processing defective tasks in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of another embodiment of a method for processing defective tasks in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of an embodiment of a device for processing defective tasks in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of another embodiment of a device for processing defective tasks in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of an embodiment of a defect task processing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present invention provide a defect task processing method, device, equipment and storage medium, which are used to identify defect tasks in an automated manner, generate personalized processing suggestions, and notify corresponding processing personnel to handle the defect tasks. The identification process is simple, and the labor and time costs are reduced while ensuring the identification accuracy.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0030] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0031] In related technologies, in order to ensure the stability of the service platform, it is necessary to identify and resolve task defects in the organization. Currently, the following tasks need to be done every day:

[0032] 1) Manually query the "Task Performance Index" table cx_system_yarn_jobinfo_new, and use this table to filter out tasks that occupy more than 3 hours of CPU time;

[0033] 2) For the above tasks, manually query the SQL (structured query language) script related to the task, and manually analyze whether the performance of the SQL script can be optimized (for example, whether it is an invalid task, whether it is a full table scan task, whether it is a repeated landing task);

[0034] 3) For tasks that do have defects, find the task leader and notify him to make relevant rectifications. Different rectification opinions need to be put forward for different tasks, and the communication process is time-consuming and labor-intensive;

[0035] 4) Collect statistics and summarize the information of new defects, rectified defects, whitelist defects, etc. of multiple institutions on the same day.

[0036] This governance work is very necessary. The current practice is that the query and analysis processes are repeated every day, the communication process of connecting hundreds of colleagues from multiple institutions and tens of thousands of tasks is cumbersome, the daily manual statistics of the increase and decrease of defects are complicated and cannot ensure quality and quantity, the manpower consumption is huge, and continuous manpower investment is required every day during the project. Therefore, the present invention proposes a defect task processing method, device, equipment and storage medium to solve the above problems.

[0037] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a method for processing defective tasks in an embodiment of the present invention includes:

[0038] 101. Identify invalid tasks of each application by identifying logs in each application;

[0039] 102. According to a preset regular expression, identify tasks for which partition tables of each application are not subjected to partition scanning and tasks for which views are not directly used;

[0040] 103, according to a preset script, identifying tasks in which the data table size of each application exceeds a preset threshold;

[0041] Since there are many types of defective tasks, this embodiment mainly identifies invalid tasks, tasks where partition tables are not scanned for partitions, tasks where views are not directly used, and tasks where the size of a data table exceeds a preset threshold; therefore, it is usually pre-set according to the type of defective task, such as invalid tasks can be identified by searching logs, tasks where partition tables are not scanned for partitions can be identified by searching for partition table related data, tasks where attempts are not directly used can be identified by searching for views and data tables in tasks, and tasks where the size of a data table exceeds a preset threshold can be identified by searching for data table directories. The above-mentioned defective tasks are usually pre-defined defective tasks, and four types of defective tasks can usually be pre-defined based on the experience of manually handling defective tasks; invalid tasks can be represented as BNH_SYSTEM_TASK_TABLE_NOVISIT; tasks where partition tables are not scanned for partitions can be represented as BNH_SYSTEM_TASK_PARTITION_TABLE_FULL_SCAN; tasks where views are not directly used can be represented as BNH_SYSTEM_TASK_USE_VIEW_DIRECTLY; and tasks where the size of a data table exceeds a preset threshold can be represented as BNH_SYSTEM_TASK_TARG_TABLE_SIZE. The preset threshold can be set according to actual needs, such as 500G.

[0042] Specifically, different types of defective tasks can be identified through different preset identification methods. For example, logs in various applications can be identified by identifying log information, and invalid tasks can be identified through the information in the logs. Preset regular expressions can also be used, where regular expressions are usually used to retrieve and replace texts that conform to a certain pattern (rule), to identify various data tables, partition tables, etc., to identify tasks that have not been partitioned and tasks that have not been directly used by views. Preset scripts can also be used to identify script information to identify information in various data tables and identify tasks whose data table sizes exceed preset thresholds. Or different types of defective tasks can be identified through different judgment criteria. The above-mentioned defective identification tasks are automatically identified through terminal devices or servers and other devices, mainly querying task performance data tables or logs and other information in various applications run by the data front-end service platform (that is, the big data platform) to identify defective tasks in various applications.

[0043] 104. According to the task type of the defect task, obtain relevant information of the defect task; wherein, different types of defect tasks obtain different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect, and task resource consumption;

[0044] 105, for each defect task, generating task information of the defect task according to relevant information of the defect task and preset configuration information; wherein the task information at least includes: defect type, defect task name, resource consumption information and processing suggestions;

[0045] Since the above defect tasks include different types of defects, it is necessary to generate task information corresponding to different defect tasks. Specifically, corresponding task information can be generated for different defect tasks according to preset configuration information.

[0046] In actual implementation, if the above-mentioned defective task is an invalid task, it is necessary to obtain relevant information of the invalid task based on the information of the identified invalid task (such as log information, corresponding application, etc.), and then generate the title, defect level, defect type, task name involved in the defect and task resource consumption situation of the invalid task based on the task title, defect level, defect type, task name involved in the defect, task resource consumption situation, etc., and processing suggestions for the invalid task, including suggestions for taking tasks of related platforms offline, etc. Specific processing suggestions can be generated in a targeted manner based on the actual relevant information of the invalid task.

[0047] If the above-mentioned defective task is a task in which partition scanning is not performed on the partition table, it is necessary to obtain relevant information of the task in which the partition table is not scanned for partitions based on the information of the task in which the partition table is not scanned for partitions (such as data table information, partition table information, script information, etc.), and then generate the title, defect level, defect type, task name involved in the defect and task resource consumption of the task in which the partition table is not scanned for partitions, defect level, defect type, defective task name, task name involved in the defect and task resource consumption based on the task title, defect level, defect type, defective task name, task name involved in the defect and task resource consumption, and processing suggestions for the task in which the partition table is not scanned for partitions, wherein the processing suggestions include the partition table, partition field, general writing method of partition screening corresponding to the defective task, etc.

[0048] If the above-mentioned defective task is a task that is directly used before the view is implemented, it is necessary to obtain relevant information of the task that is directly used before the view is implemented based on the information of the task that is directly used before the view is implemented (such as data table information, script information, etc.), and then generate the title, defect level, defect type, task name involved in the defect and task resource consumption of the task that is directly used before the view is implemented based on the task title, defect level, defect type, task name involved in the defect and task resource consumption in the relevant information, and the generated rectification suggestion shall include the view that needs to be rectified due to the defect.

[0049] If the above defective task is a task whose data table size exceeds the preset threshold, it is necessary to obtain relevant information of the task whose data table size exceeds the preset threshold based on the information of the task whose data table size exceeds the preset threshold (such as the data volume information of the data table, etc.), and then generate the title, defect level, defect type, task name involved in the defect, and task resource consumption of the task whose data table size exceeds the preset threshold based on the task title, defect level, defect type, task name involved in the defect, and task resource consumption in the relevant information. The generated rectification suggestion includes the data table involved in the defect that needs to be rectified and recommends retaining only necessary data and deleting redundant data, etc. The above data table usually refers to the organization table inserted into each organization, which can also be called the insertion organization table.

[0050] 106. Save the task information of the defect task to the preset defect list table;

[0051] In order to facilitate the subsequent analysis and summary of the rectification status of the defect tasks in each application and the rectification effect of each type of defect, as well as to facilitate the query of the submission of each defect task, the task information of the above defect tasks can be saved in the defect list table.

[0052] 107. Send the task information of each defect task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defect task according to the task information.

[0053] The task information of the above-mentioned defective task can be sent to the central system, and the central system will send different task information to the corresponding processing personnel according to the processing person information in the task information. For example, the task information of the defective task is sent to the mailbox or mobile phone of the processing personnel by email or text message. After receiving the task information, the processing personnel will rectify the defective task according to the processing suggestions and other task information included therein. At the same time, the processing status of the defective task or other problems can also be fed back to the central system. In addition, the processor can also determine whether the defective task is correctly identified based on the task information. If the identification is correct, the defective task will be rectified and the processing information will be fed back, which can be automatic feedback or manual feedback by the processor. If the identification is wrong, or the processor believes that no rectification is required, the processing information can be directly fed back, and the processing information is used to indicate that the task does not need to be rectified.

[0054] In the technical solution provided by the present invention, by identifying the logs in each application, the invalid tasks of each application are identified; according to the preset regular expression, the tasks of the partition table of each application that have not been partition scanned and the tasks of the view that have not been directly used are identified; according to the preset script, the tasks of the data table size of each application that exceeds the preset threshold are identified; according to the task type of the defective task, the relevant information of the defective task is obtained; wherein different types of defective tasks obtain different relevant information; the relevant information includes at least: task title, defect level, defect type, task name involved in the defect and task resource consumption; for each defective task, the task information of the defective task is generated according to the relevant information of the defective task and the preset configuration information; wherein the task information includes at least: defect type, defective task name, resource consumption information and processing suggestions; the task information of the defective task is saved in a preset defect list table; the task information of each defective task in the defect list table is sent to the corresponding processing personnel, so that the processing personnel can rectify the defective task through the task information. In the embodiment of the present invention, defect tasks are identified through pre-set log recognition methods, regular expressions, and scripts to achieve the purpose of automatically identifying defect tasks. This method replaces manual operations, can automatically identify defect tasks, and generate corresponding task information for each defect task, including personalized processing suggestions. At the same time, the corresponding processing personnel are notified to process the defect task according to the processing suggestions, which simplifies the identification and processing process of defect tasks, ensures the accuracy of the identification results, and reduces labor and time costs.

[0055] See also Figure 2 Another embodiment of the method for processing defective tasks in the embodiment of the present invention includes:

[0056] 201. Obtaining a first data table that has not been accessed within a preset historical time period from the audit logs of each application program, and determining a task related to the first data table as an invalid task;

[0057] Specifically, a first data table that has not been accessed in the past 30 days is screened out from the audit log through the HDFS (Hadoop Distributed File System) system, a component in the big data platform, and a task related to the first data table is determined to be the invalid task.

[0058] 202. Identify the partition tables added in each application and the tasks related to the partition tables through a preset regular expression; identify the SQL scripts of the tasks related to the partition tables through a preset regular expression, and determine, through the screening conditions in the SQL script, that the tasks of the partition fields of the partition tables that are not screened and / or the partitions of the day that are not screened are the tasks for which the partition tables are not scanned for partitions;

[0059] The above regular expression can also be called a regular expression, which is a concept in computer science. Regular expressions are usually used to retrieve and replace text that conforms to a certain pattern (rule). Specifically, it can be used by each data table and script information to identify tasks that are not scanned by partitions in partition tables and tasks that are not directly used by views.

[0060] Specifically, the daily incremental partition table in each application can be identified through a preset regular expression, and the identified partition table can be immediately inserted into the preset daily incremental partition table list. For tasks related to the "daily incremental partition table list", the SQL script of the relevant task can be identified through a preset regular expression, specifically through the where filtering conditions in the SQL script, judging whether the partition field of the table is filtered, and judging whether the partition of the day is filtered. If not, the relevant task is determined as a task in which the partition table is not scanned.

[0061] 203. Identify the SQL scripts of tasks in various applications through preset regular expressions; determine through SQL scripts that tasks that have both headquarters views and data tables are tasks that are directly used without views being implemented;

[0062] Usually, in order to save performance, the organization needs to implement the headquarters view separately first, and then use the implemented data table to associate with other data tables of the organization for use. The headquarters view cannot be directly used to associate with the organization's data table for use. Therefore, the preset regular expression can be used to identify whether there are headquarters views and organization tables in the tasks of each application. If so, the task is determined as a task that directly uses the above view without implementation.

[0063] 204. Obtain the data size of each data table in each application through a preset script, determine the second data table that exceeds the preset threshold according to the data size; and determine the related task of the second data table as the task whose data table size exceeds the preset threshold.

[0064] The above preset thresholds can be set according to actual needs. Usually, after the headquarters data is divided into multiple institutions, the data volume is already very limited. If there are still data tables with a data volume exceeding the preset threshold (such as 500G), it can generally be determined that there is an overuse of storage resources. Then, a shell script (a script program written for shell) is used to scan the HDFS institution data directory to obtain the directory size of all second data tables in the institution library, which is the data volume size of the data table. For tasks with data tables exceeding 500G, it is determined that the above tasks have data table sizes exceeding the preset threshold.

[0065] 205. According to the task type of the defect task, obtain relevant information of the defect task; wherein different types of defect tasks obtain different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect, and task resource consumption;

[0066] 206. For each defect task, generate task information of the defect task according to relevant information of the defect task and preset configuration information; wherein the task information at least includes: defect type, defect task name, resource consumption information and processing suggestions;

[0067] 207. Save the task information of the defect task into the preset defect list table;

[0068] 208. Generate a new defect upload request according to the task information of the defect task; the new defect upload request includes information of a person who handles the defect task;

[0069] The above can generate a "new defect" post request based on all the information in the defect task information, including defect type, defect task name, resource consumption, processing suggestions and processing personnel information; or generate a "new defect" post request based on part of the information in the defect task information, which can be pre-configured according to actual needs.

[0070] 209. Send the newly added defect upload request to the central system, and the central system sends the newly added defect upload request to the handler of the defect task according to the handler information, so that the handler can rectify the defect task according to the task information.

[0071] Specifically, for each defect task in the above defect list table (the task information usually includes: title, information of the person handling the task, defect level, defect description and rectification suggestion), submit a "new defect" post request to the central system (or Shenbing system), submit the new defect to the Shenbing system, and after receiving the defect task, the system will push the defect task to the corresponding person handling the task by email according to the information of the person handling the task, so that the person handling the defect can rectify the defect task according to the task information.

[0072] In the above method, defect tasks can be automatically identified, and personalized suggestions can be reviewed for each row based on the identified defect tasks, and finally notified to specific personnel, greatly reducing manpower consumption in all aspects. In addition, the automated identification process improves statistical accuracy and problem traceability, and reduces repetitive work such as rework due to statistical errors and omissions. Moreover, the defect list table can be used to record rectification tasks and rectification timeliness, analyze and summarize the rectification status of each organization and the rectification effect of each type of defect, and help plan the further rectification direction of the cluster based on the overall data situation.

[0073] In addition, the above method further comprises the following steps:

[0074] Step 1, obtaining the submitted defect tasks in the defect list table; wherein the submitted defect tasks include: defect tasks whose task information has been submitted among all defect tasks that have not been identified within a preset time period;

[0075] Step 2: Generate a defect upload request based on the submitted defect task, send the defect upload request to the central system, and control the closure of the submitted defect task through the central system.

[0076] The above-mentioned submitted defect tasks usually refer to submitted defects that are not in the defect list scanned on the same day. The submitted defect tasks can be closed, so the submitted defect tasks can be sent to the corresponding processing personnel to close the tasks through the processing personnel. Specifically, a "close defect" post request can be submitted to the central system to close the submitted defect tasks; or after identifying the submitted defect tasks, the tasks can be directly and automatically closed.

[0077] The above method further comprises the following steps:

[0078] Step 3, receiving first feedback information sent by the processing personnel regarding the defect task;

[0079] Step 4, obtaining the second feedback information for each defect task collected in the central system;

[0080] Step 5, determining a target defect task from the identified defect tasks of each application program according to the first feedback information and the second feedback information;

[0081] Step 6: Save the target defect task to a preset whitelist table, and delete the target defect task in the defect list table.

[0082] The above-mentioned target defect task can be understood as a task that does not require rectification. If the feedback information of the handler indicates that the defect task does not require rectification, or the defect feedback information collected by the online defect panel indicates a defect task that does not require rectification, then the task can be determined as a target defect task and saved to the whitelist table. In order to avoid unnecessary resource occupation, the target defect task in the defect list can be deleted. Specifically, the defect tasks in the whitelist can be removed from the full defect list scanned daily.

[0083] Finally, the above method also includes: based on the above defect list table, recording defect tasks and the rectification time limit of defect tasks, and counting the rectification status of each organization and the rectification effect of each type of defect.

[0084] For the analysis of rectification situations that cannot be summarized or completed by manual operations (such as statistics on historical rectification situations, cumulative rectification numbers, and comparison of rectification situations of various institutions, etc.), the rectification tasks and rectification time limits can be recorded through the defect list table, and the rectification situation of each institution and the rectification effect of each type of defect can be analyzed and summarized, and the further rectification direction of the cluster can be planned based on the overall data situation. The system can also be expanded and reused according to the needs of the matter. For example, the naming specifications of tasks that are implemented in advance for standardized management or the launch of a certain function can also be identified, notified, and rectified through this system, and the system's reusability and scalability can be explored.

[0085] The above describes the method for processing defective tasks in the embodiment of the present invention. The following describes the device for processing defective tasks in the embodiment of the present invention. Figure 3 , an embodiment of a device for processing defective tasks in an embodiment of the present invention includes:

[0086] A first identification module 301 is used to identify invalid tasks of each application by identifying logs in each application;

[0087] The second identification module 302 is used to identify the tasks of each application program that have not been partition-scanned and the tasks that have not been directly used without the view being implemented, according to a preset regular expression;

[0088] The third identification module 303 is used to identify the tasks of each application program whose data table size exceeds a preset threshold according to a preset script;

[0089] The acquisition module 304 is used to acquire relevant information of the defect task according to the task type of the defect task; wherein different types of defect tasks acquire different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect, and task resource consumption;

[0090] The generation module 305 is used to generate task information of the defect task according to the defect task; wherein the task information at least includes: defect type, defect task name, resource consumption information and processing suggestions;

[0091] A saving module 306 is used to save the task information of the defect task into a preset defect list table;

[0092] The sending module 307 is used to send the task information of each defect task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defect task according to the task information.

[0093] In the technical solution provided by the present invention, by identifying the logs in each application, the invalid tasks of each application are identified; according to the preset regular expression, the tasks of the partition table of each application that have not been partition scanned and the tasks of the view that have not been directly used are identified; according to the preset script, the tasks of the data table size of each application that exceeds the preset threshold are identified; according to the task type of the defective task, the relevant information of the defective task is obtained; wherein different types of defective tasks obtain different relevant information; the relevant information includes at least: task title, defect level, defect type, task name involved in the defect and task resource consumption; for each defective task, the task information of the defective task is generated according to the relevant information of the defective task and the preset configuration information; wherein the task information includes at least: defect type, defective task name, resource consumption information and processing suggestions; the task information of the defective task is saved in a preset defect list table; the task information of each defective task in the defect list table is sent to the corresponding processing personnel, so that the processing personnel can rectify the defective task through the task information. In the embodiment of the present invention, defect tasks are identified through pre-set log recognition methods, regular expressions, and scripts to achieve the purpose of automatically identifying defect tasks. This method replaces manual operations, can automatically identify defect tasks, and generate corresponding task information for each defect task, including personalized processing suggestions. At the same time, the corresponding processing personnel are notified to process the defect task according to the processing suggestions, which simplifies the identification and processing process of defect tasks, ensures the accuracy of the identification results, and reduces labor and time costs.

[0094] See also Figure 4 Another embodiment of the device for processing defective tasks in the embodiment of the present invention includes:

[0095] The first identification module 301 is specifically used to obtain the first data table that has not been accessed within a preset historical time period from the audit log of each application, and determine the related tasks of the first data table as invalid tasks; the third identification module 303 is specifically used to obtain the data volume of each data table in each application through a preset script, and determine the second data table that exceeds the preset threshold according to the data volume; determine the related tasks of the second data table as the task whose data table size exceeds the preset threshold.

[0096] The second identification module 302 is specifically used to: identify the partition tables added in each application and the tasks related to the partition tables through a preset regular expression; identify the SQL scripts of the tasks related to the partition tables through a preset regular expression, and determine, through the filtering conditions in the SQL script, the tasks of the unfiltered partition fields of the partition table and / or the unfiltered partitions of the day as the tasks for which the partition table has not been partition scanned.

[0097] The second identification module 302 is specifically used to: identify the SQL scripts of tasks in each application program through a preset regular expression; and determine through the SQL scripts that tasks that have both headquarters views and data tables are tasks that are directly used without views being implemented.

[0098] The task information of the defect task also includes the information of the person who handles the defect task; the sending module 307 is specifically used to: generate a new defect upload request based on the task information of the defect task; the new defect upload request includes the information of the person who handles the defect task; send the new defect upload request to the central system, and through the central system, send the new defect upload request to the person who handles the defect task based on the information of the person who handles the defect.

[0099] The device also includes a task closing module 308, which is used to: obtain submitted defect tasks in the defect list table; wherein the submitted defect tasks include: defect tasks whose task information has been submitted among all defect tasks that have not been identified within a preset time period; generate a defect upload closing request based on the submitted defect tasks, send the defect upload closing request to the central system, and control the closure of the submitted defect tasks through the central system.

[0100] The device also includes a task deletion module 309, which is used to: receive first feedback information sent by the processing personnel regarding the defect task; obtain second feedback information for each defect task collected in the central system; determine the target defect task from the defect tasks of each application identified based on the first feedback information and the second feedback information; save the target defect task to a pre-set whitelist table, and delete the target defect task in the defect list table.

[0101] In the above scheme, defect tasks can be automatically identified, and personalized suggestions can be reviewed for each row based on the identified defect tasks, and finally notified to specific personnel, greatly reducing manpower consumption in all aspects. In addition, the automated identification procedure improves statistical accuracy and problem traceability, and reduces repetitive work such as rework due to statistical errors and omissions. Moreover, the defect list table can be used to record rectification tasks and rectification time limits, analyze and summarize the rectification status of each organization and the rectification effect of each type of defect, and help plan the further rectification direction of the cluster based on the overall data situation. Moreover, the system can also be expanded and reused according to the needs of the matter. For example, matters such as task naming specifications that are implemented in advance for standardized management or for the launch of a certain function can also be identified, notified, and rectified through this system, and the system's reusability and scalability can be explored.

[0102] above Figure 3 and Figure 4 The apparatus for processing defective tasks in the embodiment of the present invention is described in detail from a modular perspective. The following describes the apparatus for processing defective tasks in the embodiment of the present invention in detail from a hardware processing perspective.

[0103] Figure 5 1 is a schematic diagram of the structure of a defect task processing device provided by an embodiment of the present invention. The defect task processing device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the defect task processing device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the defect task processing 500.

[0104] The defect task processing device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the defect task processing device shown does not constitute a limitation of the defect task processing device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0105] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the method for processing defective tasks.

[0106] The present invention also provides a defect task processing device, which includes a memory and a processor. The memory stores instructions, and when the instructions are executed by the processor, the processor executes the steps of the defect task processing method in the above-mentioned embodiments.

[0107] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0108] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing defective tasks, characterized in that: The method is applied to a big data platform, and the method comprises: Identifying invalid tasks of each application by identifying logs in the respective application; According to a preset regular expression, identify the tasks for which partition tables of the respective applications are not subjected to partition scanning and the tasks for which views are not directly used; According to a preset script, identifying tasks where the data table size of each application exceeds a preset threshold; According to the task type of the defect task, obtain relevant information of the defect task; wherein different types of defect tasks obtain different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect and task resource consumption; For each of the defective tasks, generating task information of the defective task according to the relevant information of the defective task and the preset configuration information; wherein the task information at least includes: defect type, defective task name, resource consumption information and processing suggestions; Saving the task information of the defect task into a preset defect list table; Sending the task information of each defect task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defect task according to the task information; The sending of the task information of each defect task in the defect list table to a corresponding processing personnel so that the processing personnel can rectify the defect task according to the task information also includes: According to the task information, determining whether the defect task is identified correctly; If the identification is correct, the defective task is rectified and the processing information is fed back; If an error is identified, processing information is directly fed back, wherein the processing information is used to indicate that the defective task does not need to be rectified.

2. The method for processing defective tasks according to claim 1, characterized in that: The step of identifying defective tasks of each application by identifying logs in the application includes: Acquire a first data table that has not been accessed within a preset historical time period from the audit logs of each application program, and determine a task related to the first data table as the invalid task; According to a preset script, the step of identifying tasks in which the data table size of each application exceeds a preset threshold comprises: The data size of each data table in each application is obtained through a preset script, and a second data table that exceeds the preset threshold is determined based on the data size; and a related task of the second data table is determined as a task whose data table size exceeds the preset threshold.

3. The method for processing defective tasks according to claim 2, characterized in that: The step of identifying the tasks for which the partition table is not subjected to partition scanning by using a preset regular expression comprises: Identify the partition tables added in the respective applications and the tasks related to the partition tables by using a preset regular expression; The SQL script of the task related to the partition table is identified by a preset regular expression, and the tasks that do not filter the partition fields of the partition table and / or do not filter the partitions of the day are determined as tasks that do not perform partition scanning on the partition table through the screening conditions in the SQL script.

4. The method for processing defective tasks according to claim 2, characterized in that: The step of identifying the task that is not directly used by the view by using a preset regular expression includes: Identify the SQL scripts of the tasks in each application by using a preset regular expression; The tasks that determine through the SQL script that both the headquarters view and the data table exist are tasks that are directly used without the view being implemented.

5. The method for processing defective tasks according to claim 1, characterized in that: The task information of the defect task also includes information of the person who handles the defect task; The step of sending the task information of each defect task in the defect list table to the corresponding processing personnel includes: Generate a new defect upload request according to the task information of the defect task; the new defect upload request includes the information of the person who handles the defect task; The newly added defect upload request is sent to a central system, and the central system sends the newly added defect upload request to a handler of the defect task according to the handler information.

6. The method for processing defective tasks according to claim 1, characterized in that: The method further comprises: Obtaining the submitted defect tasks in the defect list table; wherein the submitted defect tasks include: defect tasks whose task information has been submitted among all defect tasks that have not been identified within a preset time period; A defect closing upload request is generated according to the submitted defect task, and the defect closing upload request is sent to a central system, and the submitted defect task is closed by controlling the central system.

7. The method for processing defective tasks according to claim 1, characterized in that: The method further comprises: Receiving first feedback information sent by the processing personnel regarding the defect task; Acquire second feedback information collected in the central system for each of the defective tasks; Determining a target defective task from the identified defective tasks of each application program according to the first feedback information and the second feedback information; The target defect task is saved in a preset whitelist table, and the target defect task in the defect list table is deleted.

8. A defect task processing device, characterized in that: The device comprises: A first identification module, configured to identify defective tasks of each application by identifying logs in each application; A second identification module is used to identify, according to a preset regular expression, tasks for which partition tables of the respective applications have not been subjected to partition scanning and tasks for which views have not been directly used; A third identification module is used to identify tasks where the size of the data table of each application exceeds a preset threshold according to a preset script; An acquisition module, used for acquiring relevant information of the defective task according to the task type of the defective task; wherein different types of defective tasks acquire different relevant information; the relevant information at least includes: task title, defect level, defect type, task name involved in the defect and task resource consumption; A generating module, for generating task information of each defect task according to relevant information of the defect task and preset configuration information; wherein the task information at least includes: defect type, defect task name, resource consumption information and processing suggestions; A saving module, used to save the task information of the defect task into a preset defect list table; A sending module, used to send the task information of each defect task in the defect list table to the corresponding processing personnel, so that the processing personnel can rectify the defect task according to the task information; The sending module is also used to determine whether the defective task is correctly identified based on the task information; if the identification is correct, the defective task is rectified and processing information is fed back; if the identification is incorrect, the processing information is directly fed back, wherein the processing information is used to indicate that the defective task does not need to be rectified.

9. A defect task processing device, characterized in that: The defect task processing device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instruction in the memory to enable the defective task processing device to execute the defective task processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the defect task processing method as described in any one of claims 1-7 is implemented.