Log data quality monitoring method and device, equipment and storage medium

Abstract

The invention discloses a log data quality monitoring method and device, equipment and a storage medium, and the method comprises the steps: building a log data contract library after detecting that a data collection process is started, and recording data collection information in real time; after it is detected that the data transmission process or the data processing process is started, the conformity degree of the current data format and the standard data format in the contract library is monitored in real time, and the data triggering time and the data volume change result are recorded; after it is detected that the data storage process is started, a data integrity monitoring result is generated according to the storage data volume and the output data volume of the data processing process; after it is detected that the data query process is started, query response time is recorded, and a data consistency monitoring result is generated according to a query result and the storage data. According to the technical scheme, the reliability of the log data quality monitoring result can be guaranteed, and the service accuracy of a log system is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for monitoring log data quality. Background Technology

[0002] Currently, as enterprises increasingly rely on data-driven decision-making and artificial intelligence (AI) model training, data quality issues have become a key factor affecting the reliability of log systems and the accuracy of business operations. In big data architectures, log data often flows between multiple heterogeneous systems, including but not limited to: acquisition systems (such as Kafka, Flume), real-time processing systems (such as Flink, Spark, Airflow), storage systems (such as Hive, Iceberg, ClickHouse, Doris), and query systems (such as Presto, Trino).

[0003] During the aforementioned process, log data may frequently experience structural changes (field drift), data delays, and inconsistent data volumes. Existing log monitoring systems struggle to detect and locate these issues, necessitating an effective method for monitoring log data quality across different stages. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for monitoring log data quality, offering an effective way to monitor log data quality throughout the entire process. This ensures the reliability of log data quality monitoring results and improves the business accuracy of the log system.

[0005] According to one aspect of the present invention, a method for monitoring log data quality is provided, the method comprising:

[0006] Once the data acquisition process is detected to have started, a log data contract library is established, and data acquisition information is recorded in real time; the contract library defines a standard data format.

[0007] Once the data transmission or data processing flow is detected to have started, the system monitors in real time the degree of conformity between the current data format and the standard data format in the contract library, and records the data trigger time and data volume change results.

[0008] Once the data storage process is detected to have started, a data integrity monitoring result is generated based on the amount of stored data and the amount of output data from the data processing process.

[0009] Once the data query process is detected to have started, the query response time is recorded, and a data consistency monitoring result is generated based on the query results and the stored data.

[0010] Optionally, the standard data format includes: standard field type, field order, and null value setting identifier;

[0011] The real-time recorded data acquisition information includes:

[0012] It records the amount of data collected, the data type, and the data collection time in real time, and generates a unique identifier and data offset for each data entry.

[0013] Optionally, after detecting the start of a data transmission or data processing flow, record the data volume change results, including:

[0014] If the current process is a data transmission process, then record the change between the amount of data collected and the amount of data output by the data transmission process;

[0015] If the current process is a data processing process, then record the change between the output data volume of the data processing process and the output data volume of the data transmission process.

[0016] Optionally, the method includes:

[0017] Based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, and the preset weight ratios, a log data quality score is generated.

[0018] Optionally, based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, and the preset weight ratios, a log data quality score is generated, including:

[0019] A log data quality score is generated based on the amount of data collected, the amount of data stored, the number of fields that were successfully verified under the contract, the number of fields that participated in the contract verification, the data storage time, the data collection time, and the preset weight ratio.

[0020] Optionally, after generating the log data quality score, the following may also be included:

[0021] If the log data quality score is less than a preset score threshold, the cause of the log data quality anomaly is determined based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process.

[0022] Optionally, the method further includes:

[0023] Determine the absolute value of the difference between the amount of stored data and the amount of collected data. If the ratio between the absolute value of the difference and the amount of collected data is greater than or equal to a preset ratio, then trigger the alarm information corresponding to the log data.

[0024] According to another aspect of the present invention, a log data quality monitoring device is provided, the device comprising:

[0025] The data acquisition module is used to establish a log data contract library after detecting the start of the data acquisition process and to record data acquisition information in real time; the contract library defines a standard data format.

[0026] The transmission processing module is used to detect the start of the data transmission process or data processing process, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the data trigger time and data volume change results.

[0027] The data storage module is used to detect the start of the data storage process and generate data integrity monitoring results based on the amount of stored data and the amount of output data from the data processing process.

[0028] The data query module is used to detect when the data query process is started, record the query response time, and generate data consistency monitoring results based on the query results and stored data.

[0029] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0030] At least one processor; and

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the log data quality monitoring method according to any embodiment of the present invention.

[0033] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log data quality monitoring method according to any embodiment of the present invention.

[0034] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the log data quality monitoring method according to any embodiment of the present invention.

[0035] The technical solution provided by this invention provides a cross-stage, full-process log data quality monitoring method. This method involves establishing a log data contract library and recording data collection information in real time after detecting the start of a data acquisition process; monitoring the conformity of the current data format with the standard data format in the contract library in real time after detecting the start of a data transmission or data processing process, and recording the data trigger time and data volume changes; generating data integrity monitoring results based on the stored data volume and the output data volume of the data processing process after detecting the start of a data query process; and recording the query response time and generating data consistency monitoring results based on the query results and stored data after detecting the start of a data query process. This approach ensures the reliability of log data quality monitoring results and improves the business accuracy of the log system.

[0036] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a log data quality monitoring method provided by an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of another log data quality monitoring method provided according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of a log data quality monitoring device according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the log data quality monitoring method of this invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] Figure 1 This is a flowchart illustrating a log data quality monitoring method provided in an embodiment of the present invention. This embodiment is applicable to situations involving cross-stage monitoring of log data quality. The method can be executed by a log data quality monitoring device, which can be implemented in hardware and / or software and configured in an electronic device. Figure 1 As shown, the method includes:

[0045] Step 110: After the data acquisition process is detected to have started, a log data contract library is established, and data acquisition information is recorded in real time; the contract library defines a standard data format.

[0046] In this embodiment, after the data acquisition process is detected to have started, a standard data format corresponding to the log data can be set, and then this standard data format can be uploaded to the database to obtain the log data contract library, while recording the actual data acquisition information.

[0047] Step 120: After the data transmission process or data processing process is detected to have started, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the data trigger time and data volume change results.

[0048] In this embodiment, specifically, after the start of a data transmission process or a data processing process is detected, the current data format in the current process can be verified in real time using the standard data format in the contract library to determine the degree of conformity between the current data format and the standard data format. The data volume in each process is compared before and after, illegal data is isolated or repaired, and the data trigger time (e.g., data acquisition completion time, data transmission end time, data processing start time, data processing end time, data arrival time in the memory, etc.) and the data volume change results are recorded.

[0049] The advantage of this setup is that by recording the data trigger time in each process, delays in the data flow process can be tracked. By using the contract library to validate the current data format, invalid data, lost data, or duplicate data in the data flow process can be identified.

[0050] Step 130: After the data storage process is detected to have started, generate data integrity monitoring results based on the amount of stored data and the amount of output data from the data processing process.

[0051] In this step, the amount of stored data can be compared with the amount of output data from the data processing flow to obtain the data integrity monitoring results.

[0052] Step 140: After the data query process is detected to have started, record the query response time and generate data consistency monitoring results based on the query results and stored data.

[0053] In this step, the query results can be compared with the stored data to obtain the data consistency monitoring results.

[0054] The technical solution provided by this invention provides a cross-stage, full-process log data quality monitoring method. This method involves establishing a log data contract library and recording data collection information in real time after detecting the start of a data acquisition process; monitoring the conformity of the current data format with the standard data format in the contract library in real time after detecting the start of a data transmission or data processing process, and recording the data trigger time and data volume changes; generating data integrity monitoring results based on the stored data volume and the output data volume of the data processing process after detecting the start of a data query process; and recording the query response time and generating data consistency monitoring results based on the query results and stored data after detecting the start of a data query process. This approach ensures the reliability of log data quality monitoring results and improves the business accuracy of the log system.

[0055] Figure 2 A flowchart of another log data quality monitoring method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0056] Step 210: After the data acquisition process is detected to have started, establish a log data contract library and record the amount of data collected, the data type, and the data acquisition time in real time, and generate a unique identifier and data offset for each data entry.

[0057] In this embodiment, the contract library defines a standard data format, which includes: standard field types (e.g., string, int, float, bool, etc.), field order (e.g., fixed column order), and null value setting flags (e.g., whether NULL is allowed).

[0058] In this step, after the data acquisition process is detected to have started, the amount of data collected (including the number of data entries and the number of bytes), the data type of the collected data (e.g., JSON, event, binary, etc.), and the data acquisition time (accurate to milliseconds) can be recorded in real time, and a unique identifier and data offset corresponding to each data entry can be generated.

[0059] Step 220: After the data transmission process is detected to have started, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the changes between the data trigger time, the amount of data collected, and the amount of data output by the data transmission process.

[0060] In this embodiment, after the data transmission process is detected to have started, the standard data format in the contract library can be used to verify the field type, field nullability, etc. of the data in the current process. At the same time, the changes between the amount of data collected and the amount of data output by the data transmission process, the time when the data leaves the transmission process, the data order, etc. are recorded.

[0061] In this step, optionally, if the field type of the data in the current process does not conform to the standard data format or violates the nullability rule, an alarm message can be triggered; if the output data volume of the data transmission process is less than the collected data volume, it can be determined that data is lost; if the data order does not conform to the standard order defined in the contract library, the data can be marked as out of order; if the data dwell time in the data transmission process is greater than a preset threshold, the data can be marked as delayed.

[0062] Step 230: After the data processing flow is detected to have started, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the changes between the data trigger time, the amount of data output by the data processing flow, and the amount of data output by the data transmission flow.

[0063] In this step, the data in the current process can be validated using the standard data format in the contract library. This includes verifying whether the field types match the predefined types in the contract library, whether the field order matches the predefined order, the offset results, and the actual null value rate of nullable fields. It can also record the data processing time (including start and end times), and the changes between the data output volume of the data processing flow and the data output volume of the data transmission flow.

[0064] In one specific embodiment, the processing engine corresponding to the data processing flow can be Spark or Flink engine. When performing data cleaning and transformation operations, the reasons for data loss or addition can be analyzed based on the data volume change results.

[0065] Step 240: After the data storage process is detected to have started, generate data integrity monitoring results based on the amount of stored data and the amount of output data from the data processing process.

[0066] Step 250: After the data query process is detected to have started, record the query response time and generate data consistency monitoring results based on the query results and stored data.

[0067] Step 260: Generate a log data quality score based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, as well as the preset weight ratio.

[0068] In one embodiment of this example, a log data quality score is generated based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, as well as a preset weight ratio. This includes generating the log data quality score based on the amount of data collected, the amount of data stored, the number of fields successfully verified under the contract, the number of fields participating in the contract verification, the data storage time, the data acquisition time, and a preset weight ratio. The specific formula is as follows:

[0069]

[0070] Where, Count stc Indicates the amount of data collected, Count store V represents the amount of data stored. valid V represents the number of fields that successfully validated the contract. total This indicates the number of fields participating in contract verification; t5 represents the data storage time (i.e., the time when the data arrives at the storage device); t1 represents the data acquisition time; w1, w2, and w3 are preset weight ratios; e -λ This is an exponentially decaying term that changes over time.

[0071] In this embodiment, optionally, after generating the log data quality score, the method further includes: if the log data quality score is less than a preset score threshold, then determining the cause of the quality anomaly in the log data based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, respectively.

[0072] In a specific embodiment, a Bayesian network model can be used to locate the causes of log data quality anomalies based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process. For example, the data volume change results, field contract violation records, latency results corresponding to each process, and historical data anomaly pattern library can be input into the network model. Then, the network model outputs the causes of log data quality anomalies, including data volume anomalies, contract anomalies, or latency anomalies.

[0073] In this embodiment, an alarm notification method based on the monitoring results of each process is also proposed, mainly involving time monitoring and data volume monitoring dimensions:

[0074] In one implementation, assuming the data acquisition time is T1, the data transmission time is T2, the data processing start time is T3, the data processing end time is T4, and the data arrival time in the memory is T5, the delay between T4 and T3 can be calculated. If the delay value is less than 30 seconds, the data transmission delay is considered normal. If the delay value is greater than or equal to 30 seconds, the corresponding alarm information can be triggered. Furthermore, the delay between T5 and T1 can also be calculated. If the delay value is less than 40 seconds, the overall data flow delay is considered normal. If the delay value is greater than or equal to 40 seconds, the corresponding alarm information can be triggered.

[0075] In another implementation, assuming the number of data collected is A, the number of data outputs from the transmission process is B, the number of data outputs from the processing process is C, and the number of data stored is D, the absolute value of the difference between the amount of stored data D and the amount of collected data A can also be determined. If the ratio between the absolute value of the difference and the amount of collected data is less than a preset threshold (e.g., |DA| / A < 1%), then the data deviation is determined to be normal. If it is greater than or equal to the preset ratio, then the alarm information corresponding to the log data is triggered.

[0076] The technical solution provided by this invention, upon detecting the start of the data acquisition process, establishes a log data contract library, records the amount of acquired data, data type, and data acquisition time in real time, generates a unique identifier and data offset for each data entry, monitors the conformity of the current data format with the standard data format in the contract library in real time after detecting the start of the data transmission process, and records the changes between the data trigger time, the amount of acquired data, and the amount of data output by the data transmission process, and after detecting the start of the data processing process, monitors the conformity of the current data format with the standard data format in the contract library in real time, and records the changes between the data trigger time, the amount of data output by the data processing process, and the amount of data output by the data transmission process, and after detecting the start of the data storage process, generates a data integrity monitoring result, and after detecting the start of the data query process, records the query response time and generates a data consistency monitoring result. Based on the monitoring results and weight ratios corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, a log data quality score is generated. This provides a cross-stage, full-process log data quality monitoring method, which can ensure the reliability of log data quality monitoring results and improve the business accuracy of the log system.

[0077] Figure 3 This is a schematic diagram of a log data quality monitoring device provided in an embodiment of the present invention. The device is applied in electronic devices, such as... Figure 3 As shown, the device includes: a data acquisition module 310, a transmission and processing module 320, a data storage module 330, and a data query module 340.

[0078] The data acquisition module 310 is used to establish a log data contract library after detecting the start of the data acquisition process and to record data acquisition information in real time; the contract library defines a standard data format.

[0079] The transmission processing module 320 is used to detect the start of the data transmission process or data processing process, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the data trigger time and data volume change results.

[0080] The data storage module 330 is used to generate data integrity monitoring results based on the amount of stored data and the amount of output data of the data processing process after the data storage process is started.

[0081] The data query module 340 is used to detect the start of the data query process, record the query response time, and generate data consistency monitoring results based on the query results and stored data.

[0082] The technical solution provided by this invention provides a cross-stage, full-process log data quality monitoring method. This method involves establishing a log data contract library and recording data collection information in real time after detecting the start of a data acquisition process; monitoring the conformity of the current data format with the standard data format in the contract library in real time after detecting the start of a data transmission or data processing process, and recording the data trigger time and data volume changes; generating data integrity monitoring results based on the stored data volume and the output data volume of the data processing process after detecting the start of a data query process; and recording the query response time and generating data consistency monitoring results based on the query results and stored data after detecting the start of a data query process. This approach ensures the reliability of log data quality monitoring results and improves the business accuracy of the log system.

[0083] Based on the above embodiments, the standard data format includes: standard field type, field order, and null value setting identifier.

[0084] The data acquisition module 310 includes:

[0085] The data acquisition and recording unit is used to record the amount of data acquired, the data type, and the data acquisition time in real time, and to generate a unique identifier and data offset for each data entry.

[0086] The transmission processing module 320 includes:

[0087] The transmission result recording unit is used to record the change between the amount of data collected and the amount of data output by the data transmission process if the current process is a data transmission process.

[0088] The processing result recording unit is used to record the change between the output data volume of the data processing process and the output data volume of the data transmission process if the current process is a data processing process.

[0089] The device further includes:

[0090] The quality score generation module is used to generate log data quality scores based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, as well as preset weight ratios.

[0091] The root cause analysis module is used to determine the cause of the quality anomaly in the log data based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process if the log data quality score is less than a preset score threshold.

[0092] The alarm triggering module is used to determine the absolute value of the difference between the stored data volume and the collected data volume. If the ratio between the absolute value of the difference and the collected data volume is greater than or equal to a preset ratio, then the alarm information corresponding to the log data is triggered.

[0093] The quality score generation module includes:

[0094] The parameter calculation unit is used to generate a log data quality score based on the amount of data collected, the amount of data stored, the number of fields that were successfully verified under the contract, the number of fields that participated in the contract verification, the data storage time, the data collection time, and the preset weight ratio.

[0095] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in the embodiments of the present invention can be found in the methods provided in all the foregoing embodiments of the present invention.

[0096] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0097] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as log data quality monitoring methods.

[0100] In some embodiments, the log data quality monitoring method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log data quality monitoring method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the log data quality monitoring method by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0106] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0107] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for monitoring log data quality, characterized in that, The method includes: Once the data acquisition process is detected to have started, a log data contract library is established, and data acquisition information is recorded in real time; the contract library defines a standard data format. Once the data transmission or data processing flow is detected to have started, the system monitors in real time the degree of conformity between the current data format and the standard data format in the contract library, and records the data trigger time and data volume change results. Once the data storage process is detected to have started, a data integrity monitoring result is generated based on the amount of stored data and the amount of output data from the data processing process. Once the data query process is detected to have started, the query response time is recorded, and a data consistency monitoring result is generated based on the query results and the stored data.

2. The method according to claim 1, characterized in that, The standard data format includes: standard field type, field order, and null value setting identifier; The real-time recorded data acquisition information includes: It records the amount of data collected, the data type, and the data collection time in real time, and generates a unique identifier and data offset for each data entry.

3. The method according to claim 2, characterized in that, After detecting the initiation of a data transmission or data processing flow, record the changes in data volume, including: If the current process is a data transmission process, then record the change between the amount of data collected and the amount of data output by the data transmission process; If the current process is a data processing process, then record the change between the output data volume of the data processing process and the output data volume of the data transmission process.

4. The method according to claim 1, characterized in that, The method includes: Based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, and the preset weight ratios, a log data quality score is generated.

5. The method according to claim 4, characterized in that, Based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process, and according to the preset weight ratios, a log data quality score is generated, including: A log data quality score is generated based on the amount of data collected, the amount of data stored, the number of fields that were successfully verified under the contract, the number of fields that participated in the contract verification, the data storage time, the data collection time, and the preset weight ratio.

6. The method according to claim 4, characterized in that, After generating the log data quality score, the following is also included: If the log data quality score is less than a preset score threshold, the cause of the log data quality anomaly is determined based on the monitoring results corresponding to the data acquisition process, data transmission process, data processing process, and data storage process.

7. The method according to claim 2, characterized in that, The method further includes: Determine the absolute value of the difference between the amount of stored data and the amount of collected data. If the ratio between the absolute value of the difference and the amount of collected data is greater than or equal to a preset ratio, then trigger the alarm information corresponding to the log data.

8. A log data quality monitoring device, characterized in that, The device includes: The data acquisition module is used to establish a log data contract library after detecting the start of the data acquisition process and to record data acquisition information in real time; the contract library defines a standard data format. The transmission processing module is used to detect the start of the data transmission process or data processing process, monitor in real time the degree of conformity between the current data format and the standard data format in the contract library, and record the data trigger time and data volume change results. The data storage module is used to detect the start of the data storage process and generate data integrity monitoring results based on the amount of stored data and the amount of output data from the data processing process. The data query module is used to detect when the data query process is started, record the query response time, and generate data consistency monitoring results based on the query results and stored data.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log data quality monitoring method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log data quality monitoring method according to any one of claims 1-7.

Images