Data anomaly processing method, device, electronic device and storage medium

Abstract

The present invention provides a data exception processing method, device, electronic device and storage medium, wherein the method comprises: when an exception occurs in a data stream processed by a current processing node, capturing the exception data; determining whether to re-throw the exception data; when determining not to re-throw the exception data, obtaining the target exception processing configuration information corresponding to the current processing node; and processing the exception data based on the target exception processing configuration information. The data exception processing method, device, electronic device and storage medium provided by the present invention realize the unified processing of exception data and simplify the complexity of data exception processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data exception processing method, device, electronic equipment and storage medium. Background Art

[0002] When processing data on big data processing platforms such as Flink and Spark, data processing exceptions may occur due to input data format issues. In order not to affect the normal process of data processing, it is usually necessary to process the abnormal data.

[0003] In the related art, data verification is added to the data stream to distinguish abnormal data from normal data through data verification, and then the normal data is processed, and the abnormal data is diverted or discarded.

[0004] However, in the above-mentioned related technologies, data verification needs to be performed at each processing node, and the data processing process is relatively complicated, thereby increasing the complexity of data exception processing. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a data exception processing method, device, electronic device and storage medium.

[0006] The present invention provides a method for processing data anomalies, comprising:

[0007] When an exception occurs in the data stream processed by the current processing node, the abnormal data is captured;

[0008] Determine whether to re-throw the exception data;

[0009] When it is determined not to re-throw the exception data, obtaining target exception processing configuration information corresponding to the current processing node;

[0010] The exception data is processed based on the target exception processing configuration information.

[0011] According to a data exception processing method provided by the present invention, the processing of the exception data based on the target exception processing configuration information includes:

[0012] Packaging and marking the abnormal data to obtain an abnormal data packet;

[0013] The abnormal data packet is processed based on the target abnormal processing configuration information.

[0014] According to a data exception processing method provided by the present invention, the abnormal data is packaged and marked to obtain an abnormal data packet, including:

[0015] The source information of the abnormal data, the abnormal location information of the abnormal data, and the abnormal data are packaged and marked to obtain the abnormal data packet.

[0016] According to a data exception processing method provided by the present invention, when an exception occurs in a data stream processed by a current processing node, before capturing the abnormal data, the method further includes:

[0017] Acquire the exception handling configuration information of each processing node input by the user; wherein the exception handling configuration information includes any one of the following: discard, log and divert.

[0018] According to a data exception processing method provided by the present invention, the processing of the abnormal data packet based on the target exception processing configuration information includes:

[0019] When the target exception processing configuration information is the discard, the abnormal data packet is discarded.

[0020] According to a data exception processing method provided by the present invention, the processing of the abnormal data packet based on the target exception processing configuration information includes:

[0021] When the target exception handling configuration information is the log record, the exception data packet is written into the exception log.

[0022] According to a data exception processing method provided by the present invention, the processing of the abnormal data packet based on the target exception processing configuration information includes:

[0023] When the target exception processing configuration information is the diversion and the current processing node supports diversion, the abnormal data packet is diverted.

[0024] According to a data exception processing method provided by the present invention, the processing of the abnormal data packet based on the target exception processing configuration information includes:

[0025] When the target exception processing configuration information is the diversion and the current processing node does not support diversion, copying the data stream to obtain a first data stream and a second data stream;

[0026] Screening normal data in the first data stream and outputting the normal data;

[0027] The abnormal data packets are screened in the second data stream, and the abnormal data packets are diverted.

[0028] The present invention also provides a data anomaly processing device, comprising:

[0029] A capture unit, used to capture abnormal data when it is determined that the data flow of the current processing node is abnormal;

[0030] A determination unit, used to determine whether to re-throw the abnormal data;

[0031] A first acquisition unit, configured to acquire target exception processing configuration information corresponding to the current processing node when it is determined not to re-throw the exception data;

[0032] A processing unit is used to process the exception data based on the target exception processing configuration information.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned data exception processing methods are implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data exception processing methods are implemented.

[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data exception processing methods are implemented.

[0036] The present invention provides a data exception processing method, device, electronic device and storage medium. When it is determined not to re-throw the exception data, the target exception processing configuration information corresponding to the current processing node is obtained, and the exception data is processed based on the target exception processing configuration information, thereby realizing unified processing of the exception data and simplifying the complexity of data exception processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is one of the flow charts of the data anomaly processing method provided by the present invention;

[0039] Figure 2 This is the second flow chart of the data anomaly processing method provided by the present invention;

[0040] Figure 3 This is the third flow chart of the data anomaly processing method provided by the present invention;

[0041] Figure 4 It is one of the structural schematic diagrams of the processing node provided by the present invention;

[0042] Figure 5 This is the second structural diagram of the processing node provided by the present invention;

[0043] Figure 6 This is the fourth flow chart of the data anomaly processing method provided by the present invention;

[0044] Figure 7 This is the fifth flow chart of the data anomaly processing method provided by the present invention;

[0045] Figure 8 It is a structural schematic diagram of the data anomaly processing device provided by the present invention;

[0046] Fig. 9 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Combine the following Figure 1-Figure 7 The data exception processing method of the present invention is described.

[0049] Figure 1 This is one of the flow charts of the data anomaly processing method provided by the present invention, such as Figure 1 As shown, the data anomaly processing method includes the following steps:

[0050] Step 101: When an exception occurs in a data stream processed by a current processing node, the exception data is captured.

[0051] For example, when the current processing node receives a data stream, it processes the data stream. During the data processing, if it is determined that an abnormality occurs in the processed data stream, the abnormal data is captured.

[0052] It should be noted that in a big data processing platform, a data stream may be processed sequentially by at least one processing node; therefore, before data processing, an exception handling module needs to be embedded in each processing node, and the exception handling module is used to perform exception handling for each abnormal data.

[0053] It should be noted that the specific method of embedding the exception handling module in the processing node needs to be determined based on the implementation type of the data processing process; when the data processing process is defined by a process engine, the process engine usually configures and packages the operator (that is, the processing node) and runs it on the big data processing platform, so an exception handling module can be added when packaging the operator; in addition, the configuration of the exception handling module can also uniformly adopt the configuration mode of the process engine, that is, adding an exception handling module to the configurable process engine.

[0054] It should be noted that when the data processing flow is not defined by the process engine but implemented directly through code, it is necessary to intercept the underlying implementation of the operator through aspect-oriented programming (AOP), and then weave the exception handling module into the underlying implementation flow of the operator; or, by modifying the underlying code of the big data processing platform, add the exception handling module to the call processing logic.

[0055] In addition, for the mode implemented directly through code, because the processing nodes such as map, filter, and flatmap are all processed through the transformation entry (transform) of the data flow, the data exception handling function can be implemented by intercepting this transformation entry and embedding the exception handling module into the call of the transformation entry.

[0056] Step 102: Determine whether to re-throw the abnormal data.

[0057] For example, when the exception data of the current processing node is captured, the user's configuration on whether to re-throw the current processing node is obtained. When the user's configuration is not to re-throw, it means that the user hopes that the exception data of the current processing node will be handled through the corresponding exception handling method; when the user's configuration is to re-throw, it means that the corresponding exception may be triggered based on external accidental factors and no exception handling is required. At this time, the exception data of the current processing node can be directly re-thrown to trigger the big data processing platform to retry the exception data.

[0058] Step 103: When it is determined not to re-throw the exception data, obtain target exception processing configuration information corresponding to the current processing node.

[0059] For example, when the user's configuration is not to re-throw, the target exception handling configuration information corresponding to the current processing node is obtained. The target exception handling configuration information is the exception handling method configured by the user according to needs. The exception handling method can be one of discarding, diverting or logging. The specific exception handling method needs to be based on the user's configuration of the exception handling module of the current processing node.

[0060] Step 104: Process the abnormal data based on the target abnormality processing configuration information.

[0061] For example, when the target exception processing configuration information corresponding to the current processing node is obtained, the exception data is processed based on the target exception processing configuration information; for example, if the target exception processing configuration information is discard, the exception data of the current processing node is discarded.

[0062] A data exception processing method provided by the present invention obtains target exception processing configuration information corresponding to the current processing node when it is determined not to re-throw the exception data, processes the exception data based on the target exception processing configuration information, realizes unified processing of the exception data, and simplifies the complexity of data exception processing.

[0063] Optionally, Figure 2 This is the second flow chart of the data anomaly processing method provided by the present invention, such as Figure 2 As shown, Figure 1 Step 104 in the embodiment can be specifically implemented by the following steps:

[0064] Step 1041: Packaging and marking the abnormal data to obtain an abnormal data packet.

[0065] Optionally, source information of the abnormal data, abnormal location information of the abnormal data, and the abnormal data are packaged and marked to obtain the abnormal data packet.

[0066] For example, when abnormal data is captured, the source information of the abnormal data and the abnormal location information of the abnormal data are obtained, and the source information and the abnormal location information of the abnormal data are marked, and then the abnormal data itself, the marked source information and the abnormal location information are packaged to obtain an abnormal data packet. It should be noted that the packaging here can be achieved by corresponding operations such as packaging and encapsulation.

[0067] Step 1042: Process the abnormal data packet based on the target abnormal processing configuration information.

[0068] For example, after obtaining the abnormal data packet, the abnormal data packet is subjected to abnormal processing based on the target abnormal processing configuration information.

[0069] The data exception processing method provided by the present invention, when capturing abnormal data, packages the source information, abnormal location information and the abnormal data itself into an abnormal data packet, so as to facilitate the abnormal processing of the abnormal data.

[0070] Optionally, Figure 3 This is the third flow chart of the data exception processing method provided by the present invention, such as Figure 3As shown, in Figure 1 Before step 101 in the above, the data anomaly processing method further includes the following steps:

[0071] Step 105: Obtain the exception handling configuration information of each processing node input by the user.

[0072] The exception handling configuration information includes any one of the following: discarding, logging, and diverting.

[0073] For example, after the exception handling module is embedded in each processing node, the user can configure the exception handling module of each processing node. The configuration of the exception handling module includes whether to re-throw and the configuration of the exception handling when not re-throwing. Regarding the configuration of whether to re-throw, the user can select the re-throw option or the non-re-throw option according to the needs; regarding the configuration of the exception handling when not re-throwing, the user can select the diversion option, the discard option or the log option according to the needs. After the user has configured each processing node, the exception handling configuration information of each processing node input by the user can be obtained.

[0074] It should be noted that the configuration of the exception handling module of each processing node can be the same or different; for example, the big data processing platform includes three processing nodes, and the configuration of the exception handling modules of the three processing nodes can be not re-throwing and diverting; or the exception handling module of the first processing node can be configured to re-throw, the exception handling module of the second processing node can be configured to not re-throw and discard, and the exception handling module of the third processing node can be configured to not re-throw and divert. Each processing node can be configured separately based on demand, and the present invention is not limited to this.

[0075] Figure 4 is one of the structural diagrams of the processing node provided by the present invention, such as Figure 4 As shown, it includes the current processing node, the previous processing node and the next processing node. Each processing node represents a data processing module, which is generally implemented through a standard interface provided by the big data processing platform. The current processing node receives the data stream sent by the previous processing node and processes the data stream based on the processing logic.

[0076] Figure 5 This is the second structural diagram of the processing node provided by the present invention, such as Figure 5 As shown, the present invention embeds an exception handling module in each processing node, for example, Figure 5In the process, by taking over the interface call point of the current processing node, the functions of exception capture and exception handling are packaged in the outer layer of the current processing node. The function of packaging exception handling is to embed the exception handling module; in the process of data processing on the data stream based on the processing logic, if it is determined that the data stream has an exception, the exception data is captured, and the data exception handling process is executed through the exception handling module; if it is determined that the data stream has no exception, the processed data stream is sent to the next processing node.

[0077] It should be noted that the data exception handling method of the present invention can package processing nodes such as map, filter and flatMap. The processing node is the data direct processing node. Packaging the processing node is to embed the exception handling module in the processing node, and the user can configure the exception handling module.

[0078] The data exception processing method provided by the present invention configures the exception processing module of each processing node, so as to facilitate unified processing of abnormal data.

[0079] Further, Figure 6 FIG. 4 is a flow chart of the data anomaly processing method provided by the present invention. Figure 6 As shown, Figure 1 Step 104 in the embodiment can be specifically implemented by the following steps:

[0080] Step 1043: When the target exception processing configuration information is discard, discard the abnormal data packet.

[0081] For example, when the acquired target exception processing configuration information is discard, the abnormal data packet of the current processing node may be directly discarded.

[0082] Alternatively, if Figure 6 As shown, Figure 1 Step 104 in the embodiment can also be implemented by the following steps:

[0083] Step 1044: When the target exception handling configuration information is the log record, write the exception data packet into the exception log.

[0084] For example, when the acquired target exception processing configuration information is log recording, the exception data packet of the current processing node is written into the exception log, so that the user can view the exception data in the exception data packet in the exception log.

[0085] Alternatively, if Figure 6 As shown, Figure 1 Step 104 in the embodiment can also be implemented by the following steps:

[0086] Step 1045: When the target exception processing configuration information is the diversion and the current processing node supports diversion, diversion processing is performed on the abnormal data packet.

[0087] For example, when the target exception processing configuration information obtained is diversion, if the current processing node supports diversion, for example, the Flink big data processing platform supports diversion processing, the abnormal data packet is directly diverted; the specific diversion processing process includes writing the abnormal data packet into a preset file, or writing the abnormal data packet into a preset database, or outputting alarm information carrying the abnormal data packet, etc.

[0088] It should be noted that after writing the abnormal data packet into the preset file or into the preset database, if the device storing the preset file or the device running the preset database fails, it is necessary to restart the device to solve the failure.

[0089] It should be noted that the diversion processing flow may also be other processing flows, for example, sending the abnormal data packet to a certain device, etc., and the present invention is not limited to this.

[0090] The data exception processing method provided by the present invention can realize separate processing of the exception data based on the target exception processing configuration information when the exception data is captured, that is, the exception data is separated and processed uniformly, and different exception processing methods such as diversion, logging, discarding and re-throwing can be supported, which reflects the diversification of exception processing methods; at the same time, it will not interfere with the normal processing of data without exceptions.

[0091] Further, Figure 7 FIG. 5 is a flowchart of the data exception processing method provided by the present invention. Figure 7 As shown, Figure 1 Step 104 in the embodiment can also be implemented by the following steps:

[0092] Step 1046: When the target exception processing configuration information is the diversion and the current processing node does not support diversion, copy the data stream to obtain a first data stream and a second data stream.

[0093] For example, when the target exception handling configuration information obtained is diversion, if the current processing node does not support diversion, for example, the Spark big data processing platform does not support diversion processing, the data stream is copied to obtain two identical data streams, namely the first data stream and the second data stream.

[0094] Step 1047: Filter normal data in the first data stream and output the normal data.

[0095] For example, when the same first data stream and second data stream are obtained, the abnormal data is deleted in the first data stream based on the above-mentioned marking of the abnormal data, and the remaining data in the first data stream is used as normal data, that is, the normal data is filtered out, and then the normal data is output to the main process. The specific main process can be the next processing node, or it can be output to other processes after the processing is completed. The present invention is not limited to this.

[0096] Step 1048: Screen the abnormal data packets in the second data stream, and perform diversion processing on the abnormal data packets.

[0097] For example, when the same first data stream and second data stream are obtained, abnormal data packets are filtered out in the second data stream based on the above-mentioned marking of abnormal data, and normal data in the second data stream is deleted, and then the filtered abnormal data packets are diverted; the specific diversion processing process includes writing the abnormal data packets into a preset file, or writing the abnormal data packets into a preset database, or outputting alarm information carrying the abnormal data packets, etc.

[0098] The data exception processing method provided by the present invention can realize unified exception processing based on filtering and diversion when the current processing node does not support diversion.

[0099] The following describes the data exception processing method of the present invention based on the process engine implementation of the Flink big data processing platform as an example:

[0100] The process engine of the Flink big data processing platform implements the overall big data processing process by parsing operators. When processing operators, it decides whether to run the exception handling process based on whether the user has configured an exception handling module. If the user has configured an exception handling module, the exception data will be diverted from the data stream, and the exception handling module will perform unified exception handling on the exception data; exception handling can include re-throwing, discarding, logging, or diversion.

[0101] The data exception processing device provided by the present invention is described below. The data exception processing device described below and the data exception processing method described above can be referred to each other.

[0102] Figure 8 Schematic diagram of the structure of the data anomaly processing device provided by the present invention. Figure 8 As shown, the data anomaly processing device includes a capturing unit 801, a determining unit 802, a first acquiring unit 803 and a processing unit 804; wherein:

[0103] A capture unit 801 is used to capture abnormal data when it is determined that the data flow of the current processing node is abnormal;

[0104] A determination unit 802 is used to determine whether to re-throw the abnormal data;

[0105] A first acquisition unit 803 is used to acquire target exception processing configuration information corresponding to the current processing node when it is determined not to re-throw the exception data;

[0106] The processing unit 804 is configured to process the exception data based on the target exception processing configuration information.

[0107] A data exception processing device provided by the present invention obtains target exception processing configuration information corresponding to the current processing node when determining not to re-throw the exception data, and processes the exception data based on the target exception processing configuration information, thereby achieving unified processing of the exception data and simplifying the complexity of data exception processing.

[0108] Based on any of the above embodiments, the processing unit 804 is specifically configured to:

[0109] Packaging and marking the abnormal data to obtain an abnormal data packet;

[0110] The abnormal data packet is processed based on the target abnormal processing configuration information.

[0111] Based on any of the foregoing embodiments, the processing unit 804 is further specifically configured to:

[0112] The source information of the abnormal data, the abnormal location information of the abnormal data, and the abnormal data are packaged and marked to obtain the abnormal data packet.

[0113] Based on any of the above embodiments, the device further includes a second acquisition unit;

[0114] The second acquisition unit is used to acquire the exception handling configuration information of each processing node input by the user; wherein the exception handling configuration information includes any one of the following: discarding, logging and diverting.

[0115] Based on any of the foregoing embodiments, the processing unit 804 is further specifically configured to:

[0116] When the target exception processing configuration information is the discard, the abnormal data packet is discarded.

[0117] Based on any of the foregoing embodiments, the processing unit 804 is further specifically configured to:

[0118] When the target exception handling configuration information is the log record, the exception data packet is written into the exception log.

[0119] Based on any of the foregoing embodiments, the processing unit 804 is further specifically configured to:

[0120] When the target exception processing configuration information is the diversion and the current processing node supports diversion, the abnormal data packet is diverted.

[0121] Based on any of the foregoing embodiments, the processing unit 804 is further specifically configured to:

[0122] When the target exception processing configuration information is the diversion and the current processing node does not support diversion, copying the data stream to obtain a first data stream and a second data stream;

[0123] Screening normal data in the first data stream and outputting the normal data;

[0124] The abnormal data packets are screened in the second data stream, and the abnormal data packets are diverted.

[0125] Fig. 9 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the data exception processing method, which includes: when an exception occurs in the data stream processed by the current processing node, capturing the exception data;

[0126] Determine whether to re-throw the exception data;

[0127] When it is determined not to re-throw the exception data, obtaining target exception processing configuration information corresponding to the current processing node;

[0128] The exception data is processed based on the target exception processing configuration information.

[0129] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0130] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data exception processing method provided by the above methods, the method comprising: when an exception occurs in a data stream processed by a current processing node, capturing abnormal data;

[0131] Determine whether to re-throw the exception data;

[0132] When it is determined not to re-throw the exception data, obtaining target exception processing configuration information corresponding to the current processing node;

[0133] The exception data is processed based on the target exception processing configuration information.

[0134] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor to execute the data exception processing method provided by the above methods, the method comprising: when an exception occurs in a data stream processed by a current processing node, capturing abnormal data;

[0135] Determine whether to re-throw the exception data;

[0136] When it is determined not to re-throw the exception data, obtaining target exception processing configuration information corresponding to the current processing node;

[0137] The exception data is processed based on the target exception processing configuration information.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing data anomalies, characterized in that: include: When an exception occurs in the data stream processed by the current processing node, the abnormal data is captured; Determine whether to re-throw the exception data; When it is determined not to re-throw the exception data, obtaining target exception processing configuration information corresponding to the current processing node; Processing the abnormal data based on the target abnormality processing configuration information; The processing of the abnormal data based on the target abnormality processing configuration information includes: Packaging and marking the abnormal data to obtain an abnormal data packet; The abnormal data packet is processed based on the target abnormal processing configuration information.

2. The data anomaly processing method according to claim 1, characterized in that: The step of packaging and marking the abnormal data to obtain an abnormal data packet includes: The source information of the abnormal data, the abnormal location information of the abnormal data, and the abnormal data are packaged and marked to obtain the abnormal data packet.

3. The data anomaly processing method according to claim 1, characterized in that: When an exception occurs in the data stream processed by the current processing node, before capturing the abnormal data, the method further includes: Acquire the exception handling configuration information of each processing node input by the user; wherein the exception handling configuration information includes any one of the following: discard, log and divert.

4. The data anomaly processing method according to claim 3, characterized in that: The processing of the abnormal data packet based on the target abnormal processing configuration information includes: When the target exception processing configuration information is the discard, the abnormal data packet is discarded.

5. The data anomaly processing method according to claim 3, characterized in that: The processing of the abnormal data packet based on the target abnormal processing configuration information includes: When the target exception handling configuration information is the log record, the exception data packet is written into the exception log.

6. The data anomaly processing method according to claim 3, characterized in that: The processing of the abnormal data packet based on the target abnormal processing configuration information includes: When the target exception processing configuration information is the diversion and the current processing node supports diversion, the abnormal data packet is diverted.

7. The data anomaly processing method according to claim 3, characterized in that: The processing of the abnormal data packet based on the target abnormal processing configuration information includes: When the target exception processing configuration information is the diversion and the current processing node does not support diversion, copying the data stream to obtain a first data stream and a second data stream; Screening normal data in the first data stream and outputting the normal data; The abnormal data packets are screened in the second data stream, and the abnormal data packets are diverted.

8. A data anomaly processing device, characterized in that: include: A capture unit, used to capture abnormal data when it is determined that the data flow of the current processing node is abnormal; A determination unit, used to determine whether to re-throw the abnormal data; A first acquisition unit, configured to acquire target exception processing configuration information corresponding to the current processing node when it is determined not to re-throw the exception data; A processing unit, configured to process the abnormal data based on the target abnormality processing configuration information; The processing unit is specifically used for: Packaging and marking the abnormal data to obtain an abnormal data packet; The abnormal data packet is processed based on the target abnormal processing configuration information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data exception processing method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data exception processing method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data exception processing method according to any one of claims 1 to 7 are implemented.

Images