Data processing method, device, electronic device and storage medium

Abstract

The present application provides a data processing method, device, electronic device and storage medium, which determine normal data and abnormal data through a configured abnormal data judgment engine, add the normal data to a normal data table for subsequent processing, and store the abnormal data in an abnormal data table corresponding to the abnormal type, and store abnormal data of different abnormal types in different abnormal data tables respectively, thereby realizing accurate and effective classification monitoring and management of abnormal data in the data, and ensuring the accuracy of subsequent data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of electronic technology, the amount of data that needs to be processed by electronic devices such as computers and servers in people's daily lives is increasing, and the functions realized based on the processed data are becoming more and more abundant.

[0003] In the prior art, due to factors such as errors in the data values ​​provided by the data source, non-compliance with the data format, or data transmission delays, the data received by the electronic device acting as the data processor may not be accurate enough, affecting subsequent processing.

[0004] Therefore, in the process of data processing by electronic devices, how to ensure that normal data and abnormal data in the data stream to be processed can be processed more accurately and separately is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] The present application provides a data processing method, device, electronic device and storage medium to separately process normal data and abnormal data in a data stream to be processed.

[0006] In a first aspect, the present application provides a data processing method, comprising: obtaining a data stream to be processed, the data stream including multiple data items; determining normal data and abnormal data among the multiple data items, as well as the abnormal type of the abnormal data; storing the normal data in a normal data table, and processing the normal data in the normal data table according to preset processing rules of the data stream; storing the abnormal data in an abnormal data table corresponding to the abnormal type, and displaying information of the abnormal data table for each abnormal type through a display page.

[0007] In an embodiment of the first aspect of the present application, determining the normal data and abnormal data in the multiple data, as well as the abnormal type of the abnormal data, includes: configuring an abnormal data judgment engine based on at least one abnormal type of data in the data stream and an abnormal judgment rule for each abnormal type; and determining the normal data and abnormal data in the multiple data, as well as the abnormal type of the abnormal data through the abnormal data judgment engine.

[0008] In an embodiment of the first aspect of the present application, it also includes: using a watermark engine to process the data in the data stream; wherein, when the watermark engine processes the data, a different waiting time is set for each data in the data stream. After processing a piece of data, if the data is not received within the waiting time for the next piece of data, other data in the data stream will not be processed, and other data will be processed after the waiting time for the next piece of data.

[0009] In an embodiment of the first aspect of the present application, the abnormal data judgment engine is implemented based on a Kafka consumer program written in Flink; the abnormal data judgment engine stores the abnormal judgment rules through a state storage ValueStated unit.

[0010] In an embodiment of the first aspect of the present application, the abnormality type includes one or more of the following: data format abnormality, data value abnormality, data frequency abnormality and data delay abnormality.

[0011] In an embodiment of the first aspect of the present application, the information of the exception data table includes one or more of the following: the number of exception data in the exception data table, the proportion of exception data of the exception type in the data stream, and the repair method of the exception type.

[0012] In an embodiment of the first aspect of the present application, determining the abnormal type of the abnormal data includes: determining a label of the abnormal data through the abnormal data judgment engine, wherein the label is used to indicate the abnormal type; before storing the abnormal data in the abnormal data table corresponding to the abnormal type, it also includes: determining the abnormal type of the abnormal data according to the label of the abnormal data.

[0013] The second aspect of the present application provides a data processing device, which can be used to execute the data processing method provided in the first aspect of the present application, and the device includes: an acquisition module, used to obtain a data stream to be processed, wherein the data stream includes multiple data; a judgment module, used to determine normal data and abnormal data among the multiple data, and the abnormal type of the abnormal data; a normal data processing module, used to store the normal data in a normal data table, and process the normal data in the normal data table according to the preset processing rules of the data stream; an abnormal data processing module, used to store the abnormal data in an abnormal data table corresponding to the abnormal type, and display information of the abnormal data table for each abnormal type through a display page.

[0014] The third aspect of the present application provides an electronic device, comprising: a processor and a memory; wherein a computer program is stored in the memory, and when the processor executes the computer program, the processor can be used to execute the data processing method as described in any one of the first aspects of the present application.

[0015] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, it can be used to execute the data processing method as described in any one of the first aspects of the present application.

[0016] In summary, the data processing method, device, electronic device and storage medium provided by the present application determine normal data and abnormal data through the configured abnormal data judgment engine, add the normal data to the normal data table for subsequent processing, and store the abnormal data in the abnormal data table corresponding to the abnormal type, and store abnormal data of different abnormal types through different abnormal data tables, thereby realizing accurate and effective classification monitoring and management of abnormal data in the data, and ensuring the accuracy of subsequent data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A flowchart of an embodiment of a data processing method provided by this application;

[0019] Figure 2 A flowchart of another embodiment of the data processing method provided by this application;

[0020] Figure 3 A flowchart of another embodiment of the data processing method provided by this application;

[0021] Figure 4 This is a schematic diagram of the principle of water mark engine processing data;

[0022] Figure 5 A schematic structural diagram of an embodiment of a data processing device provided by this application;

[0023] Figure 6 This is a schematic structural diagram of an embodiment of an electronic device provided in this application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] This application is applied in the scenario of data processing, and servers and other devices can be used to process the acquired data. In some specific implementations, Kafka is a high-throughput, persistent, distributed publish-subscribe message queue system. When the data to be processed is high-precision positioning data, the server can process the high-precision positioning data in the form of a data stream based on Kafka. However, due to the influence of factors such as errors in the data values ​​provided by the data source, data formats that do not comply with the rules, or data transmission delays, the data received by the server as the data processor may not be accurate enough, affecting the accuracy of the server's subsequent consumption data, processing data, and storage data, resulting in abnormalities in subsequent data processing.

[0027] Therefore, the present application also provides a data processing method, which determines normal data and abnormal data through a configured abnormal data judgment engine, and stores the abnormal data in an abnormal data table corresponding to the abnormal type, thereby achieving accurate and effective classification monitoring and management of normal data and abnormal data in the data, and ensuring the accuracy of subsequent data processing. The technical solution of the present application is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0028] Figure 1 This is a flow chart of an embodiment of the data processing method provided by this application, as shown in FIG. Figure 1 As shown, the data processing method provided in this embodiment includes:

[0029] S101: Obtain a data stream to be processed.

[0030] In some embodiments, the execution subject of the data processing method provided in this embodiment can be any electronic device that processes data, such as a computer, server, workstation, etc. In the embodiments of this application, the data processing method is described using the electronic device as the execution subject, rather than limiting the execution subject.

[0031] Before the electronic device processes the data, it first obtains the data to be processed through S101. When this embodiment is applied to a system such as Kafka, the data obtained by the electronic device is streaming data, and the streaming data stream includes multiple data items, for example, multiple data items can be multiple high-precision positioning data items. The electronic device obtains the data stream to be processed by consuming from Kafka, and the data stream includes multiple high-precision positioning data items.

[0032] S102: Determine normal data and abnormal data among the plurality of pieces of data acquired in S101, and determine the abnormality type of the abnormal data.

[0033] After the electronic device obtains multiple pieces of data, it does not process the data immediately, but judges whether the data is abnormal in S102, thereby determining normal data and abnormal data in the multiple pieces of data in the data stream, so that the normal data and abnormal data can be processed separately later.

[0034] In some embodiments, the abnormal data provided by the embodiments of the present application may include one or more of the following abnormal types: abnormal data format, abnormal data value, abnormal data frequency, abnormal data delay, etc.

[0035] In some embodiments, the electronic device provided in this embodiment can pre-configure an abnormal data judgment engine using a Kafka consumer program written based on Flink. The electronic device can then use the configured abnormal data judgment engine to perform an abnormality judgment on each piece of data in the data stream, ultimately determining normal data and abnormal data among the multiple pieces of data in the data stream, as well as the abnormality type of the abnormal data.

[0036] In one specific implementation, the electronic device can also store exception judgment rules based on Flink's state storage ValueStated unit, allowing the abnormal data judgment engine to compare the data to be judged with the exception judgment rules stored in the ValueStated unit to determine whether the data to be judged is normal data or abnormal data. Therefore, to configure the abnormal data judgment engine, the operator of the electronic device can pre-enter at least one abnormality type for the data in the data stream and the abnormality judgment rule for each abnormality type, so that the electronic device can store these abnormality types and judgment rules in the ValueStated unit.

[0037] Furthermore, when the abnormal data judgment engine determines that data is abnormal, the engine also determines the abnormal data's anomaly type based on the comparison results with the anomaly judgment rules and adds a tag indicating the abnormality type to the abnormal data, effectively identifying the abnormal data. This allows subsequent processing of the data to determine whether it is abnormal data and its data type based on the tag, thereby performing different operations on abnormal data of different types.

[0038] S103: adding the normal data determined in S102 to the normal data table, and processing the normal data in the normal data table according to the preset processing rules of the data flow.

[0039] S104: Add the abnormal data determined in S102 to the abnormal data table. In particular, a plurality of abnormal data tables are provided in the embodiment of the present application, and each abnormal data table can be used to store abnormal data corresponding to an abnormal type. Then, in S104, before storing the abnormal data, the electronic device can determine the abnormal type of the abnormal data based on the label of the abnormal data, and then store the abnormal data in the abnormal data table corresponding to the abnormal type. In a specific implementation, the electronic device can store the abnormal data in the data table specifically by means of data output (sink).

[0040] For example, when the abnormal data types include data format abnormality, data value abnormality, data frequency abnormality, and data delay abnormality, the electronic device may be configured with four abnormal data tables to store the four types of abnormal data, respectively. Their corresponding relationships are as follows: abnormal data table 1 stores abnormal data with format abnormality, abnormal data table 2 stores abnormal data with value abnormality, abnormal data table 3 stores abnormal data with frequency abnormality, abnormal data table 4 stores abnormal data with delay abnormality, etc. When the electronic device determines that the abnormal data to be stored has a storage format abnormality, it may store the abnormal data in abnormal data table 1, and so on.

[0041] In some embodiments, the electronic device may further display information about the abnormal data table for each abnormal type on a display page, wherein the displayed information includes one or more of the following: the number of abnormal data in the abnormal data table, the proportion of abnormal data of that type in the multiple data, the abnormal type of the abnormal data, and a method for repairing the abnormal type.

[0042] In some embodiments, the electronic device can display the information of the abnormal data table in all multiple data in the entire data stream through a display page after all multiple data in the entire data stream are processed, so as to display the overall statistics and analysis of the data to the operator of the electronic device; or, the electronic device can also display information of all abnormal types in real time on the display page while processing multiple data in the data stream, thereby displaying real-time abnormal information to the operator, enabling the operator to monitor and manage the data stream being processed.

[0043] In some embodiments, the electronic device can also display methods and suggestions for repairing each type of exception on the display page, thereby more effectively prompting the operator of the electronic device the first-level repair method for the current exception and improving the efficiency of processing exception data.

[0044] In summary, the data processing method provided in this embodiment uses a configured abnormal data judgment engine to determine normal data and abnormal data, adding the normal data to a normal data table for subsequent processing, while storing the abnormal data in an abnormal data table corresponding to the abnormal type and not processing the abnormal data in subsequent processing. Furthermore, the abnormal data table stores abnormal data of different abnormal types separately, enabling accurate and effective classification, monitoring, and management of abnormal data in the data, ensuring the accuracy of subsequent data processing.

[0045] Figure 2 This is a flow chart of another embodiment of the data processing method provided by this application, as shown in FIG. Figure 2 The data processing method shown shows Figure 1 A specific implementation of the data processing method shown is as follows: Figure 2As shown, after the electronic device as the execution subject obtains high-precision positioning data through Kafka, it can first parse the data to be processed through methods such as Flink connector Kafka to obtain data headers, timestamps, and other information. Subsequently, the electronic device configures the abnormal data judgment engine based on the abnormality type and the abnormality judgment rules for each abnormality type. The configured abnormal data judgment engine is used to determine whether the data is abnormal, and the normal data is added to the normal data table through the sink method for subsequent processing. The abnormal data is added to the corresponding abnormal data table through the sink method according to its abnormality type.

[0046] In some embodiments, the abnormal data determined by the abnormal data determination engine is recorded as the first-level abnormal data, and then its abnormal type is determined according to the label of the first-level abnormal data, and stored in different abnormal data tables by diversion. The abnormal data of one abnormal type stored in each abnormal data table is called the second-level abnormal data. Figure 2 In the example shown, the abnormal data tables and their corresponding abnormal types include: abnormal data table 1 - abnormal data with abnormal storage format, abnormal data table 2 - abnormal data with abnormal storage value, abnormal data table 3 - abnormal data with abnormal storage frequency, and abnormal data table 4 - abnormal data with abnormal storage delay. The above abnormal types are used as examples in the figure. It can be understood that other abnormalities can also be included in actual applications, and the total number n of abnormal types and abnormal data tables is not limited. Therefore, the data processing method provided by this embodiment can perform secondary abnormal data diversion processing on abnormal data, thereby better realizing accurate and effective classification monitoring and management of abnormal data in the data, and ensuring the accuracy of subsequent data processing.

[0047] Figure 3 This is a flow chart of another embodiment of the data processing method provided by this application, such as Figure 3 The data processing method shown shows Figure 1 In the data processing method shown, a specific implementation method of configuring an abnormal data judgment engine and using the engine to judge whether the data is abnormal is as follows: Figure 3As shown, to statistically analyze anomaly data in the acquired data stream, a Kafka consumer program can be written using Flink. The program then determines whether each data item in the data stream is an anomaly based on the specific content and the anomaly judgment rules stored in the Value Stated. Specifically, Flink's state storage, ValueStated, can be used to store historical data needed for specific judgment criteria that require comparisons with multiple data items to reach a conclusion. In this process, a corresponding judgment rule engine can be encapsulated within the anomaly judgment engine, enabling dynamic control and modification of anomaly judgment rules such as the judgment criteria columns, specific judgment timestamp values, and the normal range of judgment indicators using a modifiable external configuration file. Finally, normal data processed by the anomaly judgment engine is added to the normal data table, while abnormal data is added to different abnormal data tables based on the anomaly type.

[0048] In some embodiments, as Figure 3 In the method shown, after the electronic device determines normal data and abnormal data based on the abnormal data judgment engine, it can also update information such as the current number of normal data and abnormal data to the abnormal data judgment engine, so that the abnormal data judgment engine can judge the abnormal types involving historical data such as the frequency of occurrence of normal data and abnormal data.

[0049] Furthermore, an embodiment of the present application also provides a data processing method, which can be applied to an electronic device to process multiple data in a data stream. Specifically, when the electronic device as the execution subject processes multiple data in the data stream, it uses a water mark engine to process the multiple data. Among them, when the water mark engine processes the data, it sets a different waiting time for each data in the data stream. After processing a piece of data, if the data is not received within the waiting time for the next data, it does not process other data in the data stream, and processes other data after the waiting time for the next data. Specifically, when the electronic device processes the data, in addition to the abnormality of the data itself, there are also abnormal data related to the data transmission time. When the electronic device uses the time window set in the ideal state to process these abnormal data, abnormal situations may occur in the processing process, so it is necessary to process such abnormal data separately.

[0050] In some embodiments, a water mark is a concept used to represent input completeness associated with the time of an action that processes data. For an action with a processing time of X, a water mark refers to all data before the data has been obtained. Therefore, when the window provided by the water mark ends, it can be considered that all data smaller than the water mark has entered the window, the data in the window can be processed, and subsequent data can continue to be received through the water mark of the next window.

[0051] Figure 4 This is a schematic diagram of the principle of water mark engine processing data, such as Figure 4 As shown, when the electronic device is processing the data stream, the multiple data in the acquired data stream are processed according to Figure 4 When the order of the data is arranged in the order in, it can be seen that the order of multiple data has changed. The electronic device needs to provide different waiting times for different data to ensure that all data are effectively processed. For example, the electronic device sets a waiting time W1 for the data with sequence numbers 1-4. Before W1, the received data 4 and 3 are added to the window corresponding to the waiting time W1. After the data with sequence number 1 is received, the window of waiting time W1 ends, and it is considered that the data 1-4 in W1 have been received. Therefore, the data in the window corresponding to the waiting time W1 can be processed subsequently. And continue to wait for the reception of data 5-7 in the window corresponding to the next waiting time W2, and so on. Therefore, when using the water mark engine to process data, different waiting times can be provided for different data, so that the electronic device has received all the data before the data at the end of the window corresponding to the waiting time, thereby ensuring that the electronic device can more accurately and effectively process abnormal data caused by abnormal data transmission time.

[0052] In the aforementioned embodiments, the data processing methods provided by the embodiments of the present application are described. In order to implement the various functions of the data processing methods provided by the embodiments of the present application, the electronic device serving as the execution subject may include a hardware structure and / or a software module, and implement the aforementioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a particular one of the aforementioned functions is implemented in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0053] For example, Figure 5 This is a structural diagram of an embodiment of a data processing device provided by this application, as shown in FIG. Figure 5The data processing device shown includes: an acquisition module 1001, a judgment module 1002, a normal data processing module 1003, and an abnormal data processing module 1004. Among them, the acquisition module 1001 is used to obtain a data stream to be processed, and the data stream includes multiple data; the judgment module 1002 is used to determine normal data and abnormal data in the multiple data, as well as the abnormal type of the abnormal data; the normal data processing module 1003 is used to store the normal data in a normal data table and process the normal data in the normal data table according to the preset processing rules of the data stream; the abnormal data processing module 1004 is used to store the abnormal data in an abnormal data table corresponding to the abnormal type and display information of the abnormal data table for each abnormal type through a display page. For another example, the data processing device 100 may also include a watermark engine processing module for processing data in the data stream, etc. The specific principles and implementation methods of the above steps performed by each module in the data processing device can be referred to the description of the data processing method in the previous embodiment of this application, and will not be repeated here.

[0054] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. It can be a separate processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above-mentioned determined module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0055] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0056] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0057] Figure 6 This is a structural diagram of an embodiment of an electronic device provided by this application, such as Figure 6The electronic device 200 provided by the present application includes: a processor 2001 and a memory 2002; wherein the memory 2002 stores a computer program. When the processor 2001 executes the computer program, the processor 2001 can be used to perform any data processing method as described in the aforementioned embodiments of the present application. In addition, the processor 2001 can obtain data streams, etc. through the communication interface 2003. Figure 6 Each unit in the system can be connected via a bus.

[0058] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it can be used to execute any data processing method as described in the aforementioned embodiments of the present application.

[0059] An embodiment of the present application also provides a chip for executing instructions, wherein the chip is used to execute a data processing method as described in any of the aforementioned embodiments of the present application.

[0060] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a data stream to be processed, wherein the data stream includes multiple pieces of data; determining normal data and abnormal data among the plurality of pieces of data, and abnormal types of the abnormal data; Storing the normal data in a normal data table, and processing the normal data in the normal data table according to a preset processing rule of the data stream; Storing the abnormal data in an abnormal data table corresponding to the abnormal type, and displaying information of the abnormal data table for each abnormal type on a display page; Also includes: A watermark engine is used to process the data in the data stream to determine abnormal data related to the data transmission time; wherein, when the watermark engine processes the data, a different waiting time is set for each data in the data stream. After processing a piece of data, if the data is not received within the waiting time for the next data, the other data in the data stream will not be processed, and the other data will be processed after the waiting time for the next data.

2. The method according to claim 1, characterized in that The determining of normal data and abnormal data in the plurality of pieces of data, and the abnormality type of the abnormal data, includes: Configuring an abnormal data judgment engine according to at least one abnormal type of data in the data stream and an abnormality judgment rule for each abnormal type; The abnormal data judgment engine determines normal data and abnormal data in the plurality of data, as well as abnormal types of the abnormal data.

3. The method according to claim 1, characterized in that The abnormal data judgment engine is implemented based on the Kafka consumer program written in Flink; The abnormal data judgment engine stores the abnormality judgment rule through a state storage ValueStated unit.

4. The method according to claim 1, wherein The exception types include one or more of the following: Abnormal data format, abnormal data value, abnormal data frequency and abnormal data delay.

5. The method according to claim 1, wherein The information in the abnormal data table includes one or more of the following: The number of abnormal data in the abnormal data table, the proportion of abnormal data of the abnormal type in the data stream, and the repair method of the abnormal type.

6. The method according to claim 2, characterized in that The determining the abnormal type of the abnormal data includes: determining a label of the abnormal data by the abnormal data judgment engine, wherein the label is used to indicate the abnormal type; Before storing the abnormal data in the abnormal data table corresponding to the abnormal type, the method further includes: determining the abnormal type of the abnormal data according to the label of the abnormal data.

7. A data processing device, characterized in that: include: An acquisition module, configured to acquire a data stream to be processed, wherein the data stream includes a plurality of data items; a judgment module, configured to determine normal data and abnormal data among the plurality of pieces of data, and an abnormality type of the abnormal data; A normal data processing module, configured to store the normal data into a normal data table and process the normal data in the normal data table according to a preset processing rule of the data stream; An exception data processing module, configured to store the exception data into an exception data table corresponding to the exception type, and display information of the exception data table for each exception type through a display page; A watermark engine processing module is used to use a watermark engine to process the data in the data stream to determine abnormal data related to the data transmission time; wherein, when the watermark engine processes the data, a different waiting time is set for each data in the data stream. After processing a piece of data, if the data is not received within the waiting time for the next data, the other data in the data stream will not be processed, and the other data will be processed after the waiting time for the next data.

8. An electronic device, characterized in that: include: A processor and a memory; wherein a computer program is stored in the memory, and when the processor executes the computer program, the processor can be used to execute the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, it can be used to perform the data processing method according to any one of claims 1 to 6.

Images