Abnormality capture method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides an exception capture method and device, a storage medium and electronic equipment, and the method comprises the steps: inputting to-be-processed data into a pre-written data analysis program; the data analysis program comprises an exception capture framework and a plurality of data processing logics; the exception capture framework covers a group of exception capture logics in the data processing process; the exception capture framework is obtained by extracting and packaging each exception capture logic in advance; each data processing logic is used for processing specific data; under the condition that a group of abnormal information appears in the process of processing the to-be-processed data by the data analysis program, calling an exception capture logic corresponding to each piece of abnormal information in the group of abnormal information from an exception capture framework; and each piece of abnormal information is captured through the abnormal capture logic corresponding to each piece of abnormal information, and the problem that the storage occupation of the abnormal information cannot be reduced while stable operation of a data analysis program cannot be ensured in related technologies is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to an exception capturing method, device, storage medium and electronic device. Background Art

[0002] With the development of science and technology, the scale of data that needs to be processed in the field of data processing is getting larger and larger. In addition to traditional stand-alone applications, data analysis programs using distributed architecture are becoming more and more common and gradually becoming mainstream. The data processing program in the data analysis program refers to a software program or script used to receive, convert, analyze, clean, summarize or otherwise manipulate data. Data processing programs are usually designed to automate data processing processes to improve efficiency and accuracy and reduce human errors. Since the raw data usually undergoes multiple steps such as collection, cleaning, and preprocessing during data processing, data that does not meet the format requirements of the current processing logic will occasionally appear. For such incorrect data, exceptions should be thrown and captured and processed before being presented to the operation and maintenance personnel to facilitate problem analysis and subsequent disposal.

[0003] Currently, there are four exception handling schemes. The first exception handling scheme is not to capture the exception and directly stop the program when an exception occurs. This method is only suitable for initial code debugging or when you are very confident in the input data format and processing program code. The actual real-time processing program usually requires that the program cannot exit due to exceptions in a small amount of data.

[0004] The second exception handling solution is to use the exception information printing file that comes with the distributed framework. However, the exception information printing file is on different servers, which makes operation and maintenance difficult, and when there is a lot of exception data, it will take up a lot of storage space.

[0005] The third exception handling solution is to use a collection program to collect the exception information files of these distributed frameworks to an additional log processing program. However, this solution mainly solves the operation and maintenance problem, but does not solve the problem of occupying storage space, and the added collection program will also consume a lot of system performance.

[0006] The fourth exception handling solution is to print the exception information to the HDFS system (Hadoop (an open source distributed computing framework) distributed file system) of the distributed architecture. The exception information can be viewed through the API interface provided by the HDFS system. However, this solution also has the problem of occupying storage space.

[0007] The above solutions all have the problem of being unable to effectively solve the storage occupancy problem of abnormal information while ensuring the stable operation of the data analysis program. Summary of the invention

[0008] The embodiments of the present application provide an exception capture method, device, storage medium and electronic device to at least solve the problem in the related art that it is impossible to ensure the stable operation of the data analysis program while reducing the storage occupancy of the exception information.

[0009] According to one aspect of an embodiment of the present application, an exception capture method is provided, comprising: obtaining data to be processed, and inputting the data to be processed into a pre-written data analysis program so as to process the data to be processed through the data analysis program; the data analysis program comprises an exception capture framework and multiple data processing logics; the exception capture framework covers a group of exception capture logics in the data processing process; each exception capture logic in the group of exception capture logics is used to capture specific exception information; the exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the multiple data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in the group of exception capture logics; in the case where a group of exception information appears in the process of the data analysis program processing the data to be processed, calling the exception capture logic corresponding to each exception information in the group of exception information from the exception capture framework; the exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information; and each exception information is captured by the exception capture logic corresponding to each exception information.

[0010] According to another aspect of the embodiment of the present application, an exception capture device is provided, comprising: a data analysis module, used to obtain data to be processed, and input the data to be processed into a pre-written data analysis program, so as to process the data to be processed through the data analysis program; the data analysis program comprises an exception capture framework and multiple data processing logics; the exception capture framework covers a group of exception capture logics in the data processing process; each exception capture logic in the group of exception capture logics is used to capture specific exception information; the exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the multiple data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in the group of exception capture logics; a logic calling module, used to call the exception capture logic corresponding to each exception information in the group of exception information from the exception capture framework when a group of exception information appears in the process of the data analysis program processing the data to be processed; the exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information; the exception capture module is used to capture each exception information through the exception capture logic corresponding to each exception information.

[0011] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned exception capturing method when running.

[0012] According to another aspect of an embodiment of the present application, there is also provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the exception capturing method through the computer program.

[0013] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps in any of the above-mentioned exception capture method embodiments when executed by a processor.

[0014] Through the embodiments of the present application, an exception capture framework that may be encountered by all operators of the big data analysis framework during the data processing process is introduced into the data analysis program. The exception capture framework covers a set of exception capture logics, and each exception capture logic captures and processes specific exception information to ensure that any exception information encountered by the operators of the big data analysis framework during the data processing process can be captured and processed in a timely manner, avoiding program crashes or interruptions caused by exception information, and ensuring the stable operation of the data analysis program; in addition, the exception capture framework can monitor the exception information of the data analysis program during operation in real time. When exception information appears, matching exception capture logic is found from the predefined exception capture framework to capture it. Through the exception capture logic, the exception information can be effectively processed instead of simply storing the entire exception information, which greatly reduces the storage occupancy of the exception information, allowing the data analysis program to run more efficiently, solving the problem in the related technology that it is impossible to ensure the stable operation of the data analysis program while reducing the storage occupancy of the exception information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a hardware structure block diagram of an optional exception capture method of an embodiment of the present application;

[0016] Figure 2 is a flowchart of an optional exception capture method according to an embodiment of the present application;

[0017] Figure 3 is an optional flow chart of constructing a data analysis program and applying a data analysis program according to an embodiment of the present application;

[0018] Figure 4 is a flowchart of an optional abnormality reporting according to an embodiment of the present application;

[0019] Figure 5 is a structural block diagram of an optional abnormality capture device according to an embodiment of the present application;

[0020] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In data analysis programs, when using various operators of the big data analysis framework (such as map, foreach, mapPartition, flatMap, reduce, etc.) to process data, various exceptions may be encountered. If these exceptions are not handled, they will not only interrupt the data processing process, but also lead to a large amount of accumulation of exceptions, which will take up a lot of storage space. In order to solve the above problems, the present application introduces an exception capture framework in the data analysis program that covers all operators of the big data analysis framework that may encounter during the data processing process. The exception capture framework covers a set of exception capture logics, and each exception capture logic captures and processes specific exception information to ensure that any exception information encountered by the operators of the big data analysis framework during the data processing process can be captured and processed in a timely manner, avoiding program crashes or interruptions caused by exception information, and ensuring the stable operation of the data analysis program; in addition, the exception capture framework can monitor the exception information of the data analysis program during its operation in real time. When exception information occurs, the matching exception capture logic is found from the predefined exception capture framework to capture it. Through the exception capture logic, the exception information can be effectively processed, such as recording key information, generating exception reports, etc., instead of simply storing the entire exception information, which greatly reduces the storage occupancy of the exception information and enables the data analysis program to run more efficiently.

[0024] According to one aspect of an embodiment of the present application, a method for catching an exception is provided. Optionally, in this embodiment, the above-mentioned method for catching an exception can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0025] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI, Bluetooth. The terminal device 102 may not be limited to a PC, a mobile phone, a tablet computer, or other devices.

[0026] The exception capture method of the embodiment of the present application can be executed by the terminal device 102, or can be executed by the server 104 and the terminal device 102 together. Taking the exception capture method of the embodiment executed by the server 104 as an example, Figure 2 is a flow chart of an optional exception capture method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:

[0027] Step S202, obtaining the data to be processed, and inputting the data to be processed into a pre-written data analysis program, so as to process the data to be processed through the data analysis program; the data analysis program includes an exception capture framework and multiple data processing logics; the exception capture framework covers a group of exception capture logics in the data processing process; each exception capture logic in a group of exception capture logics is used to capture specific exception information; the exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the multiple data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in a group of exception capture logics.

[0028] A data analysis program is a complete software program or script designed to receive, transform, analyze, clean, summarize, or otherwise manipulate large data sets. A data analysis program is a prepackaged exception capture framework that is further programmed with specific data processing logic. A data analysis program can automate data processing processes, improve efficiency and accuracy, and reduce human errors.

[0029] Data processing logic is another important component of data analysis programs. It is not a physical entity that actually exists, but refers to the code structure that the logic used to receive, transform, analyze, clean, summarize or otherwise manipulate data in the data analysis program relies on. Data processing logic is used to process specific data. Each data processing logic operates on a specific type or class of data to achieve functions such as data transformation, analysis, cleaning, and aggregation. Each data processing logic corresponds to at least one exception capture logic in a set of exception capture logics to ensure that exceptions thrown during data processing can be captured and processed in a timely manner.

[0030] The exception capture framework is an encapsulated code structure that covers a set of exception capture logics that may be encountered by all operators (such as map, foreach, mapPartition, flatMap, reduce) of the big data analysis framework during data processing. The embodiment of the present application extracts and encapsulates the full set of exception capture logics for capturing and processing exceptions to form an exception capture framework. When writing a data analysis program to call various operators to write processing logic, it is only necessary to call the exception capture framework provided by the embodiment of the present application to capture the exception when the data processing logic throws an exception, thereby realizing the capture and processing of the exception without the need for developers to perform additional operations. Each exception capture logic in a set of exception capture logics is used to capture and process the exception information thrown during the data processing process. The exception capture framework extracts and encapsulates each exception capture logic, allowing developers to handle exceptions more conveniently without having to repeatedly write exception handling code in each data processing logic.

[0031] Exception capture logic is a component of the exception capture framework. It is not an actual physical entity, but a code structure that the logic relies on. Specifically, it refers to the logic design or code structure used to monitor, identify, capture and process the exception information generated during the program operation in the data analysis program or any software system. Each exception capture logic captures and processes one or a specific type of exception to ensure the stability and reliability of the data analysis process. For example, the exception capture logic can monitor the running status of the program in real time. When the program encounters an exception during execution, it can quickly identify the type and source of the exception; it can also capture exception information, including the type of exception, error information, the code location where the exception occurred (such as stack trace), the system status when the exception occurred, etc.; it can also process the captured exception information according to the preset processing strategy, for example, including recording the exception information to a log file, generating an exception report for developers to analyze, triggering a specific exception handling process (such as a retry mechanism, rollback operation, etc.), or displaying a friendly error prompt to the user.

[0032] Optionally, Figure 3 A flowchart of constructing a data analysis program and applying the data analysis program in one embodiment is shown in FIG. Figure 3As shown, the server extracts and encapsulates the exception capture logic that may be encountered in the data processing process of all operators (such as map, foreach, mapPartition, flatMap, reduce) of various big data analysis frameworks in advance, obtains a constructed exception capture framework, and sets a calling interface for the exception capture framework. The server selects a suitable big data analysis framework, such as Spark, Flink, Storm, etc., based on the data processing requirements. On the basis of the selected big data analysis framework, the server calls the pre-built exception capture framework through the calling interface of the exception capture framework, further writes specific data processing logic on the basis of the exception capture framework, integrates the written data processing logic with the exception capture framework, and forms a complete data analysis program. In the actual application process, the server obtains the data to be processed, and inputs the data to be processed into the constructed data analysis program, and starts to process the data to be processed through the data processing logic in the data analysis program.

[0033] In some embodiments, this embodiment provides an example of extracting and encapsulating exception catching logic. For example, the following code is a data processing logic that extracts the value of the time field from data in RDD (Resilient Distributed Dataset) [JSONObject] format and converts it into milliseconds:

[0034]

[0035]

[0036] The functions completed by the data processing logic include traversing the data one by one, extracting the content of the time field and converting it into milliseconds and writing it back to the JSONObject data, and catching and processing exceptions. It can be seen that it is relatively cumbersome for developers to complete the functions that need to be completed, and they need to write a lot of code that is irrelevant to the actual data processing logic. In addition, the exception catching function has the defect of occupying a lot of storage space.

[0037] The embodiment of the present application extracts and encapsulates the code for traversing data and capturing and processing exceptions. When developers write data processing logic, they can inherit the data processing class DefaultDataProcessor provided in the embodiment of the present application and implement the process() method therein. The example is as follows:

[0038]

[0039] Comparative analysis shows that the exception capture logic obtained by the embodiment of the present application occupies less storage space, has low performance loss, and guarantees the efficiency of the data analysis program. In the process of writing the data analysis program, after the developer calls the encapsulation method, the data analysis program will not be interrupted by abnormal data, and the developer can focus on the writing of the actual data processing logic, without paying attention to the traversal of RDD (elastic distributed data set) data, the abnormal capture and processing when the data format is incorrect, and those repeated and cumbersome functions are handed over to the parent class of the abnormal capture logic extracted and encapsulated by the present embodiment for unified processing. In addition, the embodiment of the present application extracts and encapsulates each abnormal capture logic in advance, and the obtained abnormal capture framework makes the developer's code not involve the data structure provided by the big data framework, such as RDD (elastic distributed data set) and DataFrame (data frame), etc. If the big data framework needs to be replaced (such as Spark replaced with Flink), it is only necessary to replace the parent class with the parent class provided by the corresponding big data framework in the present application embodiment. Because the data processing logic uniformly calls the abnormal capture framework provided by the embodiment of the present application, the abnormal capture code will have only one copy, which is convenient for writing and adjusting the abnormal capture code, reducing the difficulty of development and maintenance.

[0040] Step S204, when a group of exception information appears in the process of processing the data to be processed by the data analysis program, the exception capture logic corresponding to each exception information in the group of exception information is called from the exception capture framework; the exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information.

[0041] Among them, abnormal information refers to the error information or warning information generated during the data processing process due to data format errors, data missing, calculation errors, etc.

[0042] Optionally, during the data processing process of the data analysis program, the server collects a set of exception information generated in real time through the monitoring mechanism of the exception capture framework, and matches the corresponding exception capture logic from the predefined exception capture logic framework according to the type and specific content of each exception information in the set of exception information. The server calls the matched exception capture logic through the exception capture framework to capture and process the corresponding exception information, for example, record the exception information, generate an exception report, trigger the exception handling process, and other operations.

[0043] It should be noted that: during the data processing of the data analysis program, although the server has collected a set of exception information in real time through the monitoring mechanism of the exception capture framework, it is still necessary to further capture and process these exception information through the exception capture logic. The main reasons are as follows: simply collecting exception information is not enough to ensure the stable operation of the program. Capturing and processing exception information through exception capture logic can prevent program crashes or data corruption caused by exception information, thereby enhancing the robustness and stability of the program. Secondly, when processing exception information, the exception capture logic usually records the context environment and stack information of the exception. This information is very important debugging information for developers. By analyzing this information, developers can locate the problem more quickly and take appropriate measures to solve it.

[0044] Step S206: Capture each piece of exception information through the exception capture logic corresponding to each piece of exception information.

[0045] Through the above steps S202 to S206, an exception capture framework that may be encountered by all operators of the big data analysis framework during the data processing process is introduced into the data analysis program. The exception capture framework covers a set of exception capture logics, and each exception capture logic captures and processes specific exception information to ensure that any exception information encountered by the operators of the big data analysis framework during the data processing process can be captured and processed in a timely manner, avoiding program crashes or interruptions caused by exception information, and ensuring the stable operation of the data analysis program; in addition, the exception capture framework can monitor the exception information of the data analysis program during operation in real time. When exception information appears, matching exception capture logic is found from the predefined exception capture framework to capture it. Through the exception capture logic, the exception information can be effectively processed instead of simply storing the entire exception information, which greatly reduces the storage occupancy of the exception information, allowing the data analysis program to run more efficiently, solving the problem in the related technology that the stable operation of the data analysis program cannot be guaranteed while reducing the storage occupancy of the exception information.

[0046] In an exemplary embodiment, each exception capture logic is further used to process the exception information captured by each exception capture logic. The data analysis program also includes an exception handling program, which is used to locate the exception problem corresponding to the exception information.

[0047] Among them, the exception handler is an important part of the data analysis program. It refers to a piece of code or logic that helps developers or maintenance personnel locate the specific code segment or operation that caused the exception based on the exception information (such as error code, error message, location of the exception, etc.) in the design of the data analysis program. The exception handler can easily display the summarized and extracted exception information on the web, and the maintenance personnel can easily locate the data processing logic that throws the exception and the reason for the exception, which is convenient for subsequent disposal.

[0048] In some embodiments, after capturing each exception information, the above exception capturing method further includes:

[0049] Each exception information is classified to obtain multiple exception categories; each exception category in the multiple exception categories includes at least one exception information; when there is a target exception category in the multiple exception categories that meets the preset exception reporting conditions, the exception information in the target exception category is reported to the exception handling program; the exception reporting conditions are conditions set based on specified parameters that allow exception reporting.

[0050] Among them, the exception category refers to multiple categories or groups obtained by classifying the exception information according to the preset classification indicators in the exception handling process. Each exception category contains at least one exception information with the same or similar classification indicators. For example, the classification indicator can be the exception type, that is, the exception information of the same exception type is classified into the same category of exception categories, or it can be the data processing logic corresponding to the exception information, that is, the exception information with the same data processing logic is classified into the same category of exception categories.

[0051] The target anomaly category refers to a specific category among multiple anomaly categories that meets the preset anomaly reporting conditions.

[0052] Specified parameters refer to the parameters used to determine whether the anomaly category meets the anomaly reporting conditions, such as the number of anomalies, anomaly type, and anomaly impact range.

[0053] Exception reporting conditions refer to the conditions set based on the specified parameters that allow exception reporting. When an exception category meets the exception reporting conditions, the exception information in this category will be reported to the exception handler for further processing.

[0054] Optionally, after capturing each abnormal information, the server classifies each abnormal information according to a preset classification index (such as the abnormal type of the abnormal information, the corresponding data processing logic), and obtains multiple abnormal categories, each of which contains at least one abnormal information with the same or similar abnormal type. The server monitors each abnormal category, records the preset specified parameters, and determines whether there is a target abnormal category that meets the abnormal reporting conditions according to the preset abnormal reporting conditions. If there is a target abnormal category, the server extracts all abnormal information from the target abnormal category, including the abnormal type, abnormal description, data processing logic identifier where the abnormality occurred, related data content, etc., organizes the extracted abnormal information into a unified format or data structure, and reports the organized abnormal information to the exception handling program.

[0055] Through this embodiment, each exception information is classified to obtain multiple exception categories. By setting the exception reporting conditions based on the specified parameters, it can be ensured that only the exception information in the target exception category that meets the exception reporting conditions is reported to the exception handling program. There is no need to print all the exception information to local files or HDFS systems, thereby greatly reducing the storage resource occupancy. While ensuring that the data analysis program is not terminated due to throwing an exception and occupying a small storage space and system performance, the exception is captured, aggregated and reported to a separate exception handling program, ensuring that the operation and maintenance personnel can easily check the exception information, locate the data processing logic that throws the exception and the reason for the exception, and facilitate subsequent processing.

[0056] In an exemplary embodiment, the above exception capture method further includes:

[0057] Determine a parameter value corresponding to a specified parameter for each of a plurality of exception categories; the specified parameter includes at least one of the following: waiting time, the amount of exception information; determine, as a target exception category, an exception category whose parameter value corresponding to the specified parameter is greater than or equal to a preset parameter threshold value among the plurality of exception categories; and an exception reporting condition is that the parameter value corresponding to the specified parameter is greater than or equal to the preset parameter threshold value.

[0058] In this embodiment, the specified parameters include the waiting time or the number of abnormal information.

[0059] Waiting time refers to the length of time from the time when the last exception information in each exception category was generated to the time when the last exception information was reported.

[0060] The number of abnormal information refers to the number of abnormal information accumulated in each abnormal category within a preset time period.

[0061] The preset parameter threshold refers to the parameter value threshold used to determine whether the anomaly category meets the anomaly reporting conditions. When the specified parameter value of the anomaly category is greater than or equal to the preset parameter threshold, the anomaly category is regarded as the target anomaly category.

[0062] Optionally, Figure 4 FIG. 1 is a flowchart of an abnormality reporting in an embodiment, such as Figure 4 As shown, after determining multiple exception categories, the server calculates the parameter value corresponding to the specified parameter for each exception category, specifically including calculating the waiting time of each exception category, or counting the number of exception information in each exception category. The server compares the parameter value of each exception category with the preset parameter threshold. If the parameter value of a certain exception category is greater than or equal to the preset parameter threshold, the exception category is determined as the target exception category.

[0063] Through this embodiment, the exception category whose parameter value corresponding to the specified parameter (waiting time, number of exception information) is greater than or equal to the preset parameter threshold is determined as the target exception category, which can avoid indiscriminate processing of all exception information, thereby saving processing resources and improving processing efficiency. In addition, only the exception information in the target exception category is reported to the exception handling program, which can achieve effective management and control of the exception information, help reduce the storage occupancy of the exception information in the system, and avoid the problem of system performance degradation or storage resource exhaustion due to excessive exception information.

[0064] In an exemplary embodiment, reporting the exception information in the target exception category to the exception handling program includes:

[0065] According to the abnormal information in the target abnormal category, the reporting information is determined; the reporting information is compressed to obtain compressed data, and the compressed data is reported to the abnormality handling program.

[0066] The reporting information refers to the information that is prepared to be reported to the exception handling program after screening and sorting the exception information in the target exception category. The reporting information usually contains key data such as the key features of the exception, the time of occurrence, and the corresponding data processing logic, which is used to help the exception handling program quickly locate and solve the problem.

[0067] Compressed data refers to the data obtained by compressing the reported information. Compression is usually performed to reduce the storage space and transmission time of data, while ensuring the integrity and security of data during transmission. Since the data processing logic of big data analysis jobs usually processes data in a unified format, the exceptions thrown are usually of the same type or a few types. Therefore, data compression can reduce the storage space of exception information.

[0068] Optionally, the server screens out key data according to the exception information in the target exception category, organizes the data to form structured reporting information, compresses the reporting information to obtain compressed data, and reports the compressed data to the exception handling program.

[0069] Through this embodiment, key abnormal information is filtered out from the abnormal information in the target abnormal category as the reporting information, which can accurately locate the abnormal information that poses a threat to the stable operation of the data analysis program, avoid reporting all abnormal information, and thus reduce unnecessary data transmission and storage occupancy; secondly, the reported information is compressed, which not only significantly reduces the storage space requirements of the compressed data, but also reduces the transmission cost and time.

[0070] In an exemplary embodiment, determining reporting information according to the abnormal information in the target abnormal category includes:

[0071] The abnormal information in the target abnormal category is determined as the reporting information.

[0072] Through this embodiment, the abnormal information in the target abnormal category is determined as the reporting information, rather than the abnormal information in all abnormal categories as the reporting information, which can avoid storing a large amount of irrelevant abnormal information, thereby effectively reducing the storage occupancy of the abnormal information.

[0073] In an exemplary embodiment, determining reporting information according to the abnormal information in the target abnormal category includes:

[0074] A preset number of key abnormality information is extracted from the target abnormality category, and the preset number of key abnormality information is used as reporting information.

[0075] Among them, key exception information refers to the exception information extracted from the target exception category according to the preset screening criteria, which has an important impact on the running stability of the data analysis program, data quality or business logic, and needs to be given priority attention and processing. The preset screening criteria can be the frequency of the occurrence of the exception, the type of exception, the degree of threat to the data analysis program, etc. For example, in the case where the target exception category contains multiple exception information of the same exception type or the same data processing logic, only one exception information of the same exception type or the same data processing logic can be extracted from the target exception category as a key exception information. For example, when the target exception category includes exception information of configuration errors or reading a large amount of incorrectly formatted data, one of the exception information can be extracted, or the exception information of configuration errors or reading a large amount of incorrectly formatted data can be discarded.

[0076] Through this embodiment, key exception information is extracted from the target exception category, which can more accurately locate the core problems that affect the stable operation of the data analysis program, and help developers or operation and maintenance personnel to quickly identify and solve problems; extract and report a preset number of key exception information, discard other identical or similar exception information in the same exception category, and discard artificially configured or unimportant exception information in the same exception category, which can not only provide detailed exception features through a small amount (i.e., a preset number) of key exception information, but also remove redundant exception information, reduce the number of times the exception log is sent, avoid printing too many logs and occupying too much content, thereby significantly reducing the storage occupancy of the exception information. Compared with the solution of writing all exception logs to the hdfs system or writing spark-executor-log (Spark driver log), the probability of the data analysis program terminating due to insufficient storage space is greatly reduced.

[0077] In an exemplary embodiment, determining reporting information according to the abnormal information in the target abnormal category includes:

[0078] A preset number of key exception information is extracted from the target exception category, and the key exception content of each key exception information in the preset number of key exception information is extracted, and the key exception content of each key exception information is determined as the reporting information; the key exception content of each key exception information includes the data processing logic, exception type and exception content corresponding to each key exception information.

[0079] Among them, key exception content refers to the specific information contained in each key exception information, including the data processing logic, exception type and exception content corresponding to the exception information. The data processing logic describes the data processing steps or processes that the program is executing when the exception occurs; the exception type refers to the specific classification of the exception, such as data exception, calculation exception, resource exception, etc.; the exception content describes the specific situation or error information of the exception.

[0080] Through this embodiment, on the basis that a small amount (i.e., a preset number) of key exception information can provide detailed exception features, key exception content in the key exception information is further extracted, and redundant features in the exception information that are not related to the exception are filtered out, so as to ensure that the reported exception information is more accurate and targeted, which helps developers or operation and maintenance personnel to locate problems more quickly and reduce misjudgments or delays caused by information redundancy. After filtering out redundant features, the storage amount of exception information is significantly reduced.

[0081] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware server, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0083] According to another aspect of the embodiment of the present application, a kind of abnormal capture device is also provided, and this device is used to realize the abnormal capture method provided in the above-mentioned embodiment, and has been described and will not be repeated. As used below, the term "module" can realize the combination of software and / or hardware of predetermined function. Although the device described in the following embodiment is preferably realized with software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0084] Figure 5 is a structural block diagram of an optional abnormality capture device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0085] The data analysis module 501 is used to obtain the data to be processed and input the data to be processed into a pre-written data analysis program so as to process the data to be processed through the data analysis program; the data analysis program includes an exception capture framework and multiple data processing logics; the exception capture framework covers a group of exception capture logics in the data processing process; each exception capture logic in a group of exception capture logics is used to capture specific exception information; the exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the multiple data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in a group of exception capture logics;

[0086] The logic calling module 502 is used to call the exception capture logic corresponding to each exception information in the set of exception information from the exception capture framework when a set of exception information appears in the process of processing the to-be-processed data by the data analysis program; the exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information;

[0087] The exception capture module 503 is used to capture each exception information through the exception capture logic corresponding to each exception information.

[0088] In an exemplary embodiment, each exception capture logic is also used to process the exception information captured by each exception capture logic. The data analysis program also includes an exception handling program, which is used to locate the exception problem corresponding to the exception information. The exception capture module 503 is also used to classify each exception information to obtain multiple exception categories; each exception category in the multiple exception categories includes at least one exception information; when there is a target exception category in the multiple exception categories that meets the preset exception reporting conditions, the exception information in the target exception category is reported to the exception handling program; the exception reporting condition is a condition set based on specified parameters that allows exception reporting.

[0089] In an exemplary embodiment, the exception capture module 503 is also used to determine the parameter value corresponding to the specified parameter for each exception category in multiple exception categories; the specified parameter includes at least one of the following: waiting time, the number of exception information; the exception category in the multiple exception categories whose parameter value corresponding to the specified parameter is greater than or equal to the preset parameter threshold is determined as the target exception category; the exception reporting condition is that the parameter value corresponding to the specified parameter is greater than or equal to the preset parameter threshold.

[0090] In an exemplary embodiment, the exception capture module 503 is further used to determine reporting information according to the exception information in the target exception category; compress the reporting information to obtain compressed data, and report the compressed data to the exception handling program.

[0091] In an exemplary embodiment, the exception capturing module 503 is further configured to determine the exception information in the target exception category as reporting information.

[0092] In an exemplary embodiment, the exception capturing module 503 is further configured to extract a preset amount of key exception information from the target exception category, and use the preset amount of key exception information as reporting information.

[0093] In an exemplary embodiment, the exception capture module 503 is also used to extract a preset number of key exception information from the target exception category, extract the key exception content of each key exception information in the preset number of key exception information, and determine the key exception content of each key exception information as reporting information; the key exception content of each key exception information includes the data processing logic, exception type and exception content corresponding to each key exception information.

[0094] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0095] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-mentioned exception capture methods in the embodiments of the present application.

[0096] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0097] Optionally, in this embodiment, the storage medium is configured to store a program code for executing any of the above exception capturing methods.

[0098] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0099] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0100] According to another aspect of the embodiments of the present application, a computer program product or a computer program is also provided. Optionally, in this embodiment, the computer program product or the computer program includes a computer instruction, and the processor of the computer device reads the computer instruction from the computer readable storage medium, and the processor executes the computer instruction, so that the computer device performs the steps in any of the above method embodiments.

[0101] Optionally, in this embodiment, the computer program product or computer program may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0102] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0103] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned exception capture method is also provided. The electronic device may be a smart device, a server, a terminal, or a combination thereof.

[0104] Figure 6 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604 and the memory 606 communicate with each other through the communication bus 608, wherein,

[0105] Memory 606, used to store computer programs;

[0106] The processor 602 is used to implement the program code of any of the above-mentioned exception catching methods when executing the computer program stored in the memory 606.

[0107] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface is used for communication between the above electronic device and other devices.

[0108] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0109] As an example, the memory 606 may include but is not limited to each module in the abnormality capture device. In addition, it may also include but is not limited to other module units in the abnormality capture device, which will not be repeated in this example.

[0110] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0111] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0112] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only. The device for implementing the above-mentioned exception capture method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations shown.

[0113] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0114] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0115] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0116] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0117] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0118] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0119] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically separately, or at least two units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An exception capture method, characterized in that: include: Acquire the data to be processed, and input the data to be processed into a pre-written data analysis program, so as to process the data to be processed by the data analysis program; the data analysis program includes an exception capture framework and multiple data processing logics; The exception capture framework covers a set of exception capture logic in the data processing process; Each exception capture logic in the set of exception capture logics is used to capture specific exception information; The exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the plurality of data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in the set of exception capture logics; In the case where a group of exception information appears in the process of the data analysis program processing the data to be processed, calling the exception capture logic corresponding to each exception information in the group of exception information from the exception capture framework; The exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information; Each of the exception information is captured by the exception capture logic corresponding to each of the exception information.

2. The method according to claim 1, characterized in that Each of the exception capture logics is further used to process the exception information captured by each of the exception capture logics, the data analysis program further includes an exception handling program, and the exception handling program is used to locate the exception problem corresponding to the exception information, and the method further includes: Classifying each abnormal information to obtain a plurality of abnormal categories; each abnormal category in the plurality of abnormal categories includes at least one abnormal information; When there is a target exception category among the multiple exception categories that meets the preset exception reporting conditions, the exception information in the target exception category is reported to the exception handling program; the exception reporting conditions are conditions set based on specified parameters that allow exception reporting.

3. The method according to claim 2, characterized in that The method further comprises: Determine a parameter value corresponding to the specified parameter for each of the plurality of exception categories; the specified parameter includes at least one of the following: waiting time, the number of exception information; Among the multiple exception categories, the exception category whose parameter value corresponding to the specified parameter is greater than or equal to the preset parameter threshold is determined as the target exception category; the exception reporting condition is that the parameter value corresponding to the specified parameter is greater than or equal to the preset parameter threshold.

4. The method according to claim 2, characterized in that: The step of reporting the abnormal information in the target abnormal category to the abnormality handling program includes: Determine reporting information according to the abnormal information in the target abnormal category; The reported information is compressed to obtain compressed data, and the compressed data is reported to the exception handling program.

5. The method according to claim 4, characterized in that The determining of reporting information according to the abnormal information in the target abnormal category includes: The abnormal information in the target abnormal category is determined as the reporting information.

6. The method according to claim 4, characterized in that The determining of reporting information according to the abnormal information in the target abnormal category includes: A preset number of key abnormality information is extracted from the target abnormality category, and the preset number of key abnormality information is used as the reporting information.

7. The method according to claim 4, characterized in that The determining of reporting information according to the abnormal information in the target abnormal category includes: A preset number of key exception information is extracted from the target exception category, key exception content of each key exception information in the preset number of key exception information is extracted, and the key exception content of each key exception information is determined as the reporting information; the key exception content of each key exception information includes the data processing logic, exception type and exception content corresponding to each key exception information.

8. An abnormality capturing device, characterized in that: include: A data analysis module, used for acquiring data to be processed, and inputting the data to be processed into a pre-written data analysis program, so as to process the data to be processed by the data analysis program; the data analysis program includes an exception capture framework and multiple data processing logics; The exception capture framework covers a set of exception capture logic in the data processing process; Each exception capture logic in the set of exception capture logics is used to capture specific exception information; The exception capture framework is obtained by extracting and encapsulating each exception capture logic in advance; each data processing logic in the plurality of data processing logics is used to process specific data; each data processing logic corresponds to at least one exception capture logic in the set of exception capture logics; A logic calling module, configured to call, from the exception capturing framework, an exception capturing logic corresponding to each piece of exception information in the set of exception information when a set of exception information appears in the process of the data analysis program processing the data to be processed; The exception capture logic corresponding to each exception information refers to the exception capture logic used to capture each exception information; The exception capture module is used to capture each exception information through the exception capture logic corresponding to each exception information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any one of the methods of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images