Real-time streaming data quality monitoring method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and information security, is applied to the field of intelligent government affairs, and relates to a real-time streaming data quality monitoring method, which comprises the following steps: reading data streams input and output in a message queue, and analyzing the data streams to obtain a target data set; detecting whether the target data set is abnormal or not; if the target data set is abnormal, determining an abnormal type and an abnormal degree of the target data set; and triggering a corresponding preset alarm mechanism based on the exception type and the exception degree. The invention further provides a real-time streaming data quality monitoring device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the target data set can be stored in a block chain. According to the invention, the timeliness of real-time streaming data quality detection is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and information security technology, and in particular to a real-time streaming data quality monitoring method, apparatus, computer equipment and storage medium. Background Art

[0002] In the rapid development of big data, real-time data and offline data each have their own characteristics and application scenarios, but real-time data has significant advantages over offline data in some aspects. For example, real-time data can process and respond immediately when data is generated, with extremely low latency, usually reaching milliseconds or seconds. In contrast, offline data needs to be processed and analyzed in batches after data generation, so there will be higher latency. Real-time data is also more suitable for scenarios with high real-time requirements, such as financial transactions, real-time transaction large screens, etc.

[0003] However, the existing real-time data assurance process relies on Flume to synchronize data to HDFS and then perform verification. The entire process requires the participation of many platforms, which is time-consuming and complicated, and it is difficult to meet the immediacy of real-time data verification. Therefore, a method for real-time streaming data quality detection that can meet the immediacy is urgently needed. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a real-time streaming data quality monitoring method, apparatus, computer equipment and storage medium to ensure the timeliness of real-time streaming data quality detection.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a real-time streaming data quality monitoring method, which adopts the following technical solution:

[0006] Read the input and output data streams in the message queue, and parse the data streams to obtain the target data set;

[0007] Detecting whether the target data set has anomalies;

[0008] If the target data set has an abnormality, determining the type and degree of the abnormality of the target data set;

[0009] A corresponding preset alarm mechanism is triggered based on the abnormality type and the abnormality degree.

[0010] Furthermore, the step of parsing the data stream to obtain the target data set includes:

[0011] Traversing all key-value pairs in the data stream;

[0012] A flat field set is generated based on the key-value pairs to obtain the target data set.

[0013] Furthermore, the step of reading the input and output data streams in the message queue includes:

[0014] Determine a data reading method carried by a preset scene, wherein the data reading method includes real-time reading or timed reading;

[0015] Determine a TOPIC tag corresponding to the scenario in the message queue as a target tag;

[0016] The input and output data streams corresponding to the target tag are read based on the data reading method.

[0017] Furthermore, the step of detecting whether the target data set has an abnormality includes:

[0018] Determine whether the numerical data of the target data set exceeds a preset data threshold;

[0019] If the numerical data of the target data set exceeds a preset data threshold, it is determined that an abnormality exists in the target data set.

[0020] Furthermore, the step of triggering a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree includes:

[0021] Inputting the abnormality type and the abnormality degree into a pre-trained alarm model to obtain an alarm level output by the alarm model;

[0022] Determine a corresponding alarm mode according to the alarm level as a target alarm mode;

[0023] An alarm signal is sent to a designated person through the target alarm method.

[0024] Furthermore, the step of detecting whether the target data set has an abnormality includes:

[0025] Retrieving the latest data set in the database, and comparing the latest data set with the target data set to see whether they are consistent;

[0026] If the latest data set is inconsistent with the target data set, it is determined that an abnormality exists in the target data set.

[0027] Furthermore, the step of comparing the latest data set with the target data set to see whether they are consistent includes:

[0028] Calling a preset MD5 hash algorithm, and performing hash processing on the latest data set and the target data set respectively by using the MD5 hash algorithm to obtain a latest hash value and a target hash value respectively;

[0029] Compare the latest hash value with the target hash value to see if they are consistent;

[0030] If the latest hash value is consistent with the target hash value, it is determined that the latest data set is consistent with the target data set;

[0031] If the latest hash value and the target hash value are inconsistent, it is determined that the latest data set is inconsistent with the target data set.

[0032] In order to solve the above technical problems, the embodiment of the present application also provides a real-time streaming data quality monitoring device, which adopts the following technical solution:

[0033] A parsing module, used for reading the input and output data streams in the message queue, and parsing the data streams to obtain a target data set;

[0034] A detection module, used to detect whether the target data set has anomalies;

[0035] A determination module, configured to determine the type and degree of abnormality of the target data set if the target data set is abnormal;

[0036] The alarm module is used to trigger a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0038] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned real-time streaming data quality monitoring method when executing the computer-readable instructions.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0040] A computer-readable storage medium stores computer-readable instructions, which implement the steps of the above-mentioned real-time streaming data quality monitoring method when executed by a processor.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0042] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0045] Figure 2 A flowchart of an embodiment of a real-time streaming data quality monitoring method according to the present application;

[0046] Figure 3 is a structural diagram of an embodiment of a real-time streaming data quality monitoring device according to the present application;

[0047] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0052] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0053] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0054] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0055] It should be noted that the real-time streaming data quality monitoring method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the real-time streaming data quality monitoring device is generally arranged in the server / terminal device.

[0056] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0057] Continue to refer Figure 2 , shows a flow chart of an embodiment of a real-time streaming data quality monitoring method according to the present application. The real-time streaming data quality monitoring method comprises the following steps:

[0058] Step S201, reading the input and output data streams in the message queue, and parsing the data streams to obtain a target data set.

[0059] In this embodiment, the real-time streaming data quality monitoring method is executed on the electronic device (eg Figure 1 shown Server / Terminal Equipment ) can read the data stream input and output in the message queue through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0060] In this embodiment, the message queue (KAFKA) is used as a middleware to efficiently transmit messages between different systems or application components to achieve decoupling. This application reads the input and output data streams in the message queue, which is a non-intrusive real-time stream data quality detection to avoid interference with data transmission.

[0061] Step S202: Detect whether there is any anomaly in the target data set.

[0062] In this embodiment, by performing anomaly detection on the target data set, problems can be discovered and solved in a timely manner.

[0063] Step S203: if the target data set has an abnormality, determine the abnormality type and abnormality degree of the target data set.

[0064] In this embodiment, when an abnormality is identified in the target data set, the abnormality type of the identified target data set is determined, and the abnormality degree of the target data set is further detected, so as to facilitate subsequent alarm.

[0065] Step S204: dynamically triggering a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree.

[0066] In this embodiment, the type and degree of anomaly determine the alarm mechanism. This mechanism can ensure that when data anomalies occur, the system can quickly identify and take corresponding actions. According to the type and degree of anomalies, different levels of alarm mechanisms are designed. It can include email notifications, SMS alarms, system pop-ups, sound alarms, and other methods. The automated abnormal alarm mechanism can greatly improve the efficiency and timeliness of data quality monitoring.

[0067] This application can also include multiple user roles with different permissions, including administrator, development role, read-only role, etc. Administrators can control permissions for projects, project users, and data resources; development roles can customize and configure monitoring rules and publish them; read-only roles can be users of other projects, making it easier for other project users to quickly learn to configure monitoring rules.

[0068] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0069] In some optional implementations of this embodiment, step S201, i.e., the step of reading the input and output data streams in the message queue, includes:

[0070] Determine a data reading method carried by a preset scene, wherein the data reading method includes real-time reading or timed reading;

[0071] Determine a TOPIC tag corresponding to the scenario in the message queue as a target tag;

[0072] The input and output data streams corresponding to the target tag are read based on the data reading method.

[0073] In this embodiment, the input and output data streams corresponding to the TOPIC tag in the message queue are read in real time or on a scheduled basis in a non-intrusive business scenario manner according to different scenarios. By dynamically selecting the data reading method according to the scenario: real-time or scheduled, the system can flexibly respond to different business needs. For example, for scenarios that require immediate response, such as financial transactions, a real-time reading method can be used; while for scenarios of data aggregation and analysis, such as reports, a scheduled reading method can be used. By selecting the data reading method according to the preset scenario and using specific TOPIC tags in the message queue to manage and process data streams, the flexibility, scalability, performance, and reliability of the system can be significantly improved, while reducing the complexity and maintenance costs of the system.

[0074] In some optional implementations of this embodiment, step S201, i.e., the step of parsing the data stream to obtain the target data set, includes:

[0075] Traversing all key-value pairs in the data stream;

[0076] A flat field set is generated based on the key-value pairs to obtain the target data set.

[0077] In this embodiment, the JSON format data in the TOPIC tag is parsed into a flat field set. The flat field set converts the nested data structure into a flat form, so that each field directly corresponds to a value without having to access it through multiple layers of nesting. Because it is no longer necessary to traverse multiple layers of nested structures to find a specific field, it can effectively reduce the computational complexity of data access and significantly improve the efficiency of data processing.

[0078] In some optional implementations of this embodiment, step S202, i.e., the step of detecting whether the target data set has an abnormality, includes:

[0079] Determine whether the numerical data of the target data set exceeds a preset data threshold;

[0080] If the numerical data of the target data set exceeds a preset data threshold, it is determined that an abnormality exists in the target data set.

[0081] In this embodiment, it is ensured that the message queue TOPIC data accurately reflects the actual data situation and avoids abnormal data exceeding the threshold. By setting the data threshold and monitoring the abnormal value, it is ensured that the target data set does not contain a large amount of erroneous data, thereby improving the overall data quality.

[0082] In addition, in some optional implementations of this embodiment, step S202, i.e., the step of detecting whether the target data set has an abnormality, includes:

[0083] Detecting whether the target data set has missing values ​​or blank values;

[0084] If the missing value or blank value exists, it is determined that the target data set has an abnormality.

[0085] In this embodiment, the data of the data resource TOPIC is checked to see if it is complete, to avoid missing values ​​and blank values. The presence of missing values ​​or blank values ​​will destroy the integrity of the data and affect the accuracy and reliability of data analysis. By detecting and determining whether the target data set has missing values ​​or blank values, it is determined that the target data set is abnormal, and an alarm is issued in the subsequent steps. It can ensure that the data used in the subsequent analysis of the corrected target data set is complete and valid, thereby avoiding errors in the analysis results.

[0086] In some optional implementations of this embodiment, step S202, i.e., the step of detecting whether the target data set has an abnormality, includes:

[0087] Retrieving the latest data set in the database, and comparing the latest data set with the target data set to see whether they are consistent;

[0088] If the latest data set is inconsistent with the target data set, it is determined that an abnormality exists in the target data set.

[0089] In this embodiment, the monitoring data is updated in a timely manner to avoid the untimely update of the accident data. By comparing the latest data set with the target data set, it can be ensured that the target data set is the latest and most accurate. This is particularly important for application scenarios that require real-time or near real-time data, such as financial market analysis.

[0090] In addition, the present application can also obtain the corresponding data of each system or different data sources according to the TOPIC tag of the target data set, and compare whether the target data set is consistent with the corresponding data of each system or different data sources. If not, it is determined that there is an abnormality in the target data set and an alarm is issued to avoid conflicts caused by inconsistencies between data.

[0091] Specifically, the step of comparing the latest data set with the target data set to see whether they are consistent includes:

[0092] Calling a preset MD5 hash algorithm, and performing hash processing on the latest data set and the target data set respectively by using the MD5 hash algorithm to obtain a latest hash value and a target hash value respectively;

[0093] Compare the latest hash value with the target hash value to see if they are consistent;

[0094] If the latest hash value is consistent with the target hash value, it is determined that the latest data set is consistent with the target data set;

[0095] If the latest hash value and the target hash value are inconsistent, it is determined that the latest data set is inconsistent with the target data set.

[0096] In this embodiment, when comparing the latest hash value and the target hash value, if the two are inconsistent, it can be immediately determined that the latest data set has changed. The MD5 hash algorithm can generate a fixed-length hash value that is unique to the original data. Therefore, by comparing the hash values, it is very efficient to verify whether the two data sets are completely consistent without having to compare the entire data set byte by byte.

[0097] In some optional implementations of this embodiment, step S204, i.e., the step of triggering a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree, includes:

[0098] Inputting the abnormality type and the abnormality degree into a pre-trained alarm model to obtain an alarm level output by the alarm model;

[0099] Determine a corresponding alarm mode according to the alarm level as a target alarm mode;

[0100] An alarm signal is sent to a designated person through the target alarm method.

[0101] In this embodiment, the alarm model can adopt a neural network model. Through the pre-trained alarm model, it can perform a comprehensive analysis based on the abnormality type and degree of abnormality, and output an accurate alarm level. This accuracy helps to avoid false alarms and missed alarms, and ensure the effectiveness and pertinence of the alarm signal. Once the abnormality is detected and the alarm level is determined, the system can immediately send an alarm signal to the designated person according to the preset alarm method. This rapid response mechanism helps to take timely measures to solve the problem and prevent the problem from further deteriorating. Different alarm levels correspond to different alarm methods, such as text messages, emails, telephone calls, instant messages, etc. By determining the target alarm method according to the alarm level, personalized customization of the alarm can be achieved to meet the needs of different scenarios and personnel.

[0102] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0103] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target data set, the above-mentioned target data set can also be stored in a node of a blockchain.

[0104] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0105] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0106] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application belongs to the field of smart government affairs, and this solution can promote the construction of smart cities.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0108] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0109] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a real-time streaming data quality monitoring device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] like Figure 3 As shown, the real-time streaming data quality monitoring device 300 described in this embodiment includes: a parsing module 301 , a detection module 302 , a determination module 303 and an alarm module 304 .

[0111] in:

[0112] The parsing module 301 is used to read the input and output data streams in the message queue, and parse the data streams to obtain the target data set;

[0113] The detection module 302 is used to detect whether there is an abnormality in the target data set;

[0114] The determination module 303 is used to determine the abnormality type and abnormality degree of the target data set when the target data set has an abnormality;

[0115] The alarm module 304 is used to trigger a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree;

[0116] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0117] In some optional implementations of this embodiment, the parsing module 301 includes a first determination submodule, a second determination submodule and a reading submodule. Wherein: the first determination submodule is used to determine the data reading method carried by the preset scene, wherein the data reading method includes real-time reading or timed reading; the second determination submodule is used to determine the TOPIC tag corresponding to the scene in the message queue as the target tag; the reading submodule is used to read the input and output data streams corresponding to the target tag based on the data reading method.

[0118] In some optional implementations of this embodiment, the parsing module 301 further includes a traversal submodule and a generation submodule, wherein: the traversal submodule is used to traverse all key-value pairs in the data stream; the generation submodule is used to generate a flat field set based on the key-value pairs to obtain the target data set.

[0119] In some optional implementations of this embodiment, the detection module 302 includes a determination submodule and a first abnormality submodule. The determination submodule is used to determine whether the numerical data of the target data set exceeds a preset data threshold; the abnormality submodule is used to determine that the target data set is abnormal when the numerical data of the target data set exceeds the preset data threshold.

[0120] The detection module 302 also includes a comparison submodule and a second abnormality submodule. The comparison submodule is used to retrieve the latest data set in the database and compare the latest data set with the target data set to see if they are consistent; the second abnormality submodule is used to determine that the target data set has an abnormality when the latest data set is inconsistent with the target data set.

[0121] The detection module 302 further includes a missing determination submodule and a third abnormality submodule. The missing determination submodule is used to determine whether there are missing values ​​or blank values ​​in the target data set; the third abnormality submodule is used to determine whether there are abnormalities in the target data set when there are missing values ​​or blank values.

[0122] Specifically, the comparison submodule includes a calling unit, a comparison unit, a first determination unit, and a second determination unit. Among them: the calling unit is used to call a preset MD5 hash algorithm, and hash the latest data set and the target data set respectively through the MD5 hash algorithm to obtain the latest hash value and the target hash value respectively; the comparison unit is used to compare whether the latest hash value and the target hash value are consistent; the first determination unit is used to determine that the latest data set is consistent with the target data set when the latest hash value and the target hash value are consistent; the second determination unit is used to determine that the latest data set is inconsistent with the target data set when the latest hash value and the target hash value are inconsistent.

[0123] In some optional implementations of this embodiment, the alarm module 304 includes an input submodule, an alarm submodule, and a sending submodule. Among them: the input submodule is used to input the abnormality type and the abnormality degree into the pre-trained alarm model to obtain the alarm level output by the alarm model; the alarm submodule is used to determine the corresponding alarm mode according to the alarm level as the target alarm mode; the sending submodule is used to send an alarm signal to a designated person through the target alarm mode.

[0124] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0125] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0126] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0127] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0128] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the real-time streaming data quality monitoring method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0129] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the real-time streaming data quality monitoring method.

[0130] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0131] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0132] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the real-time streaming data quality monitoring method as described above.

[0133] The present application reads the data streams input and output in the message queue, and parses the data stream to obtain the target data set; detects whether the target data set has an anomaly; if the target data set has an anomaly, determines the anomaly type and degree of the target data set; triggers the corresponding preset alarm mechanism based on the anomaly type and degree of the anomaly. By parsing the data stream, the data stream becomes a target data set that is more suitable for subsequent processing, thereby improving the data processing speed of subsequent steps. By detecting the target data set, it is possible to accurately identify anomalies in the target data set, and instantly capture dynamic changes in the data, thereby achieving rapid processing and anomaly detection of the data stream. This immediacy and efficiency ensure the timeliness of data processing, so that problems can be quickly discovered and responded to.

[0134] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A real-time streaming data quality monitoring method, characterized in that: The steps include: Read the input and output data streams in the message queue, and parse the data streams to obtain the target data set; Detecting whether the target data set has anomalies; If the target data set has an abnormality, determining the type and degree of the abnormality of the target data set; A corresponding preset alarm mechanism is triggered based on the abnormality type and the abnormality degree.

2. The real-time streaming data quality monitoring method according to claim 1, characterized in that: The step of parsing the data stream to obtain a target data set includes: Traversing all key-value pairs in the data stream; A flat field set is generated based on the key-value pairs to obtain the target data set.

3. The real-time streaming data quality monitoring method according to claim 1, characterized in that: The step of reading the data stream input and output in the message queue includes: Determine a data reading method carried by a preset scene, wherein the data reading method includes real-time reading or timed reading; Determine a TOPIC tag corresponding to the scenario in the message queue as a target tag; The input and output data streams corresponding to the target tag are read based on the data reading method.

4. The real-time streaming data quality monitoring method according to claim 1, characterized in that: The step of detecting whether the target data set has an anomaly comprises: Determine whether the numerical data of the target data set exceeds a preset data threshold; If the numerical data of the target data set exceeds a preset data threshold, it is determined that an abnormality exists in the target data set.

5. The real-time streaming data quality monitoring method according to claim 1, characterized in that: The step of triggering a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree includes: Inputting the abnormality type and the abnormality degree into a pre-trained alarm model to obtain an alarm level output by the alarm model; Determine a corresponding alarm mode according to the alarm level as a target alarm mode; An alarm signal is sent to a designated person through the target alarm method.

6. The real-time streaming data quality monitoring method according to claim 1, characterized in that: The step of detecting whether the target data set has an anomaly comprises: Retrieving the latest data set in the database, and comparing the latest data set with the target data set to see whether they are consistent; If the latest data set is inconsistent with the target data set, it is determined that an abnormality exists in the target data set.

7. The real-time streaming data quality monitoring method according to claim 6, characterized in that: The step of comparing whether the latest data set is consistent with the target data set comprises: Calling a preset MD5 hash algorithm, and performing hash processing on the latest data set and the target data set respectively by using the MD5 hash algorithm to obtain a latest hash value and a target hash value respectively; Compare the latest hash value with the target hash value to see if they are consistent; If the latest hash value is consistent with the target hash value, it is determined that the latest data set is consistent with the target data set; If the latest hash value and the target hash value are inconsistent, it is determined that the latest data set is inconsistent with the target data set.

8. A real-time streaming data quality monitoring device, characterized in that: include: A parsing module, used for reading the input and output data streams in the message queue, and parsing the data streams to obtain a target data set; A detection module, used to detect whether the target data set has anomalies; A determination module, configured to determine the abnormality type and degree of the target data set when the target data set has an abnormality; The alarm module is used to trigger a corresponding preset alarm mechanism based on the abnormality type and the abnormality degree.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the real-time streaming data quality monitoring method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the real-time streaming data quality monitoring method according to any one of claims 1 to 7 are implemented.

Images