Data quality detection method and system based on hierarchical storage of disks and memories

Abstract

The invention relates to the technical field of data quality detection, in particular to a data quality detection method and system based on hierarchical storage of a disk and a memory, and the method comprises the steps: determining a target data table and a quality detection rule based on a user instruction to generate a data quality detection task; determining execution time and execution nodes through a preset scheduling engine; reading data from the target data table in batches through the execution node, and writing the read data into a disk file in batches; loading the data from the disk file in batches to a memory for quality detection to obtain a quality detection result of each batch of data, and recording problem data; and aggregating the quality detection results of each batch of data to generate a quality detection report. Therefore, a user can flexibly configure the quality detection rule to perform quality detection on the target data table according to actual requirements, and meanwhile, the scheduling engine and the execution node are utilized to perform multi-batch detection in a hierarchical manner through the disk and the memory, so that the scale of data quality detection can be greatly improved.

Description

Technical Field

[0001] This invention relates to the field of data quality inspection technology, and specifically to a data quality inspection method and system based on hierarchical storage of disks and memory. Background Technology

[0002] Data quality inspection is the process of checking the quality of data, such as whether the data is accurate, complete, and formatted correctly. This allows companies to fully understand the reliability of their data, thereby helping them make correct decisions, manage their business, and avoid data security risks.

[0003] With the rapid development of information technology, data quality inspection faces challenges such as larger scale, more numerous rules, more complex dimensions, and more stringent requirements. Under these massive data quality inspection scenarios and stringent requirements, traditional data quality inspection methods and systems exhibit significant shortcomings in practice. First, the quality inspection standards and rules are singular and fixed. Quality inspection systems often have built-in fixed quality inspection standards and rules, resulting in monotonous scenarios and rigid modes. They cannot be adapted or expanded to meet actual business objectives.

[0004] Secondly, the quality inspection dimensions are too simple, making it difficult to flexibly conduct multi-dimensional and multi-weighted data quality inspections for different inspection scenarios.

[0005] Third, the testing scale is limited. Due to limitations in hardware resources and other conditions, there are often problems such as memory overflow and slow computing in big data quality testing scenarios, making it impossible to complete the testing process safely and stably.

[0006] Therefore, how to conduct data quality inspection in a flexible, efficient, and secure manner has become an urgent technical problem to be solved. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a data quality detection method and system based on disk and memory hierarchical storage, so as to overcome the problems of single rules, simple dimensions and limited scale in the current data quality detection.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, this application provides a data quality detection method based on tiered storage of disk and memory, including: The configuration information is determined based on user instructions to generate a data quality inspection task, wherein the configuration information is used to determine the target data table and the quality inspection rules; The execution time and execution node of the data quality inspection task are determined by a preset scheduling engine. During the execution time, the execution node reads data from the target data table in batches and writes the read data to the disk file in batches. Data is loaded from disk files into memory in batches, and each batch of data is subjected to quality inspection based on quality inspection rules to obtain the quality inspection results of each batch of data and record the problematic data. The quality inspection results of each batch of data are aggregated to generate a quality inspection report.

[0009] Furthermore, in some embodiments of this application, the configuration information includes target data information and quality inspection information; The target data information includes: the type of the target database, the target data source, and the target data table; The quality inspection information includes: quality inspection standards, quality inspection rules corresponding to the quality inspection standards, inspection fields, the score ratio of each quality inspection standard, and the score ratio of each quality inspection rule.

[0010] Furthermore, in some embodiments of this application, the step of determining configuration information based on user instructions to generate a data quality detection task includes: The system provides users with multi-level pre-stored configuration information and configuration customization tools through a preset visual interface; among them, each quality inspection standard in the pre-stored configuration information corresponds to at least one quality inspection rule; It receives system pre-stored configuration information or user-defined configuration information selected by the user to generate a data quality inspection task.

[0011] Furthermore, in some embodiments of this application, it also includes: The execution mode of the data quality inspection task is determined based on user instructions; The execution modes include a single-run mode based on manual triggering and a periodic cyclic execution mode based on preset periodic information.

[0012] Furthermore, in some embodiments of this application, the step of performing quality inspection on each batch of data based on quality inspection rules to obtain the quality inspection results of each batch of data and recording problematic data includes: Based on the quality inspection rules under each quality inspection standard, the current batch of data is subjected to quality inspection to obtain the quality inspection results corresponding to each quality inspection rule for the current batch of data, and problematic data that does not meet the quality inspection rules are written to the remote file storage system in real time.

[0013] Furthermore, in some embodiments of this application, the process of calculating the quality inspection results of each batch of data also includes: The system reads and records read errors and execution errors of each batch of data, and calculates the risk of system memory overflow in real time; wherein, the risk of system memory overflow is used to optimize the amount of data in each batch when data is split into batches.

[0014] Furthermore, in some embodiments of this application, the aggregation of quality inspection results from each batch of data to generate a quality inspection report includes: Based on the quality inspection results and score ratios of the quality inspection rules under each quality inspection standard, calculate the score for each quality inspection standard. The final quality inspection score is calculated based on the score and score percentage of each quality inspection standard. A quality inspection report is generated based on the final quality inspection score.

[0015] Secondly, this application provides a data quality detection system based on tiered storage of disks and memory, comprising: A configuration module is used to determine configuration information based on user instructions in order to generate a data quality detection task, wherein the configuration information is used to determine the target data table and quality detection rules; The scheduling module determines the execution time and execution node of the data quality detection task through a preset scheduling engine; The calculation module is used to read data from the target data table in batches at the execution time through the execution node, and write the read data to the disk file in batches; load data from the disk file into memory in batches, and perform quality inspection on each batch of data based on quality inspection rules to obtain the quality inspection results of each batch of data and record the problematic data; aggregate the quality inspection results of each batch of data to generate a quality inspection report.

[0016] This invention relates to the field of data quality inspection technology, specifically to a data quality inspection method and system based on hierarchical storage of disk and memory. The method includes: determining configuration information based on user instructions to generate a data quality inspection task, wherein the configuration information is used to determine the target data table and quality inspection rules; determining the execution time and execution nodes of the data quality inspection task through a preset scheduling engine; reading data from the target data table in batches at the execution time through the execution nodes, and writing the read data to disk files in batches; loading data from the disk files into memory in batches, and performing quality inspection on each batch of data based on the quality inspection rules to obtain the quality inspection results of each batch of data, and recording problematic data; and aggregating the quality inspection results of each batch of data to generate a quality inspection report. In this way, users can flexibly configure quality inspection rules to perform quality inspection on the target data table according to actual needs, and by utilizing the scheduling engine and execution nodes to perform multi-batch inspections through hierarchical storage of disk and memory, the scale of data quality inspection can be greatly improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the data quality detection method based on hierarchical storage of disk and memory provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data quality detection system based on hierarchical storage of disk and memory provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] Figure 1 This is a flowchart illustrating the data quality detection method based on tiered storage of disk and memory provided in this embodiment of the invention. Please refer to [link / reference]. Figure 1 This embodiment may include the following steps: S101. Based on user instructions, determine configuration information to generate a data quality inspection task, wherein the configuration information is used to determine the target data table and quality inspection rules.

[0021] Specifically, users can select or customize target data tables and quality inspection rules in the preset system (i.e., a data quality inspection system based on disk and memory hierarchical storage), and then the system generates data quality inspection tasks based on the target data tables and quality inspection rules.

[0022] S102. Determine the execution time and execution node of the data quality detection task through the preset scheduling engine.

[0023] S103. During execution, the execution node reads data from the target data table in batches and writes the read data to the disk file in batches.

[0024] S104. Load data from disk files into memory in batches, perform quality checks on each batch of data based on quality inspection rules, obtain the quality inspection results of each batch of data, and record the problematic data.

[0025] Specifically, in this application, the execution node performs data quality detection tasks in batches, connecting to the corresponding database, reading data from the target data table in the database, and performing quality detection in memory. For example, after extracting a batch of data from the target data table, once the quality detection of the previous batch of data is completed in memory, that batch of data is loaded into memory for quality detection. It is understandable that due to the limitations of disk read, memory loading, and actual detection speed, multiple batches of data can be stored on disk and sequentially loaded into memory after the quality detection of the previous batch is completed in memory, to ensure that system memory overflow issues are not encountered.

[0026] S105. Aggregate the quality inspection results of each batch of data and generate a quality inspection report.

[0027] In this way, users can flexibly configure quality inspection rules to perform quality inspection on target data tables according to actual needs. At the same time, by utilizing the scheduling engine and execution nodes, and performing inspections in multiple batches through disk and memory hierarchical methods, the scale of data quality inspection can be greatly improved.

[0028] It is understandable that in the above-mentioned batch-by-batch quality inspection, the quality inspection results are the quality inspection results of each batch of data. In practical applications, the quality inspection results of each batch of data can be aggregated in real time. After the quality inspection of all batches of data is completed, the final quality inspection result is obtained, and then a quality inspection report is generated.

[0029] Furthermore, in some embodiments of this application, the configuration information includes target data information and quality inspection information; the target data information includes: the type of the target database, the target data source, and the target data table; the quality inspection information includes: quality inspection standards, quality inspection rules corresponding to the quality inspection standards, inspection fields, the scoring percentage of each quality inspection standard, and the scoring percentage of each quality inspection rule. By dynamically configuring and defining the quality inspection standards and their subordinate quality inspection rules, different business scenarios can be flexibly adapted, avoiding redundant development.

[0030] In practical applications, the system can pre-store some configuration information and provide configuration information customization tools. When users have data quality detection needs, the system can provide users with multi-level pre-stored configuration information and configuration information customization tools through a preset visual interface. Among them, each quality detection standard in the pre-stored configuration information corresponds to at least one quality detection rule.

[0031] Specifically, quality inspection standards, such as completeness, accuracy, standardization, and uniqueness, can be pre-defined in the system based on industry standards and business needs for data quality inspection. Simultaneously, quality inspection rules and specific calculation formulas can be configured for each quality inspection standard, such as continuous and discrete value range verification rules (i.e., inspection rules), precise regular expression matching, and fuzzy regular expression matching. For example, value range verification rules can be set under the accuracy standard to verify the accuracy of data values. Inspection fields can be bound, and the scoring weights of each standard and each rule under each standard can be configured (used to calculate the score of each standard with corresponding weights and to determine the final score).

[0032] It should be noted that when users customize configuration information through the configuration information customization tool, they can edit it temporarily when there is a need for data quality testing, or they can customize it in advance according to their own business characteristics so that it can be directly selected in subsequent testing. Understandably, when customizing configuration information, users can include customizing the name of the quality testing standard based on business characteristics, customizing the quality testing rules under the standard based on business needs, and setting calculation formulas.

[0033] Understandably, for a data quality inspection task, users can configure multiple quality inspection standards simultaneously and flexibly set the scoring ratio of each standard based on the characteristics of the data or the inspection business scenario to calculate the score of the data quality inspection task; and under the same quality inspection standard, users can configure multiple quality inspection rules and flexibly set the scoring ratio of each rule to calculate the score of each standard.

[0034] In addition, in this application, the target data information in the configuration information mentioned above includes the type of target database, the target data source, and the target data table.

[0035] Specifically, users can configure target data information through a preset interface, such as specifying the database type (e.g., MySQL, PostgreSQL), selecting the target data source from the system's registered data sources, and choosing the target database name and target data table as the data body for subsequent quality inspection (in some embodiments, the aforementioned inspection fields can also be configured during this process, rather than during the quality inspection information configuration). In practical applications, the system can support the inspection of various structured databases, as well as single-table configuration and batch configuration of multiple tables under the same source. This allows for compatibility with mainstream relational databases, enabling flexible selection of different data source instances, databases, tables, and fields through unified connection configuration.

[0036] Furthermore, in some embodiments of this application, the method further includes: determining the execution mode of the data quality detection task based on user instructions; wherein the execution mode includes a single-run mode based on manual triggering and a periodic cyclic execution mode based on preset periodic information.

[0037] Specifically, after setting the above configuration information, users can set the task running mode for the data quality detection task (or set the default mode in advance in the system), and finally submit the above configuration information and running mode information to the scheduling engine for scheduling management.

[0038] In this way, by using a pre-set scheduling engine, the execution time and execution nodes of the data quality detection task are planned and determined. The data quality detection task is executed at the determined execution nodes at the specified execution time, and the execution logs are recorded and stored.

[0039] Specifically, the execution node first reads the data quality detection configuration, then reads data from the target data table in batches and writes it to disk files in batches; then, it loads data from the disk files into memory in batches, completes the detection in memory and records problematic data, thereby avoiding the impact of unpredictable factors such as the management behavior of the target data source and network anomalies on the quality detection process, and avoiding direct writing of data to memory, effectively controlling memory peaks and avoiding system memory overflow, thus supporting large-scale data quality detection; in particular, the calculation results of each batch are recorded in real time, and problematic data are recorded to remote files in real time. After all data detection is completed, the phased calculation results are aggregated according to the scoring ratio of each standard and rule to obtain the final score.

[0040] It should be noted that during the above detection process, local algorithm optimizations can be performed for different quality detection standards and rules. For example, during the uniqueness detection process, when aggregating data to disk, the data can first be sorted by the uniqueness field using merge sort before subsequent processing. This ensures that problematic data is centrally distributed in the uniqueness detection calculation, effectively reducing computational memory costs and balancing computational time costs with improved computational efficiency.

[0041] The data quality inspection method based on tiered disk and memory storage provided in this application is suitable for data quality inspection scenarios with multiple data sources, massive amounts of inspection data, limited inspection resources, and complex quality inspection dimensions. Driven by the divide-and-conquer approach, the quality inspection calculation process proceeds sequentially from bottom to top, iteratively calculating. While ensuring memory safety, it can efficiently and accurately integrate the inspection results. Simultaneously, during the calculation process, each batch of problematic data is stored in real time on a remote file server, avoiding the occupation of the host machine's memory and disk, thus ensuring the stability and security of the computing cluster under large-scale inspection tasks.

[0042] The following is a complete implementation process to introduce the data quality detection method based on tiered disk and memory storage provided in this application: The first step, during the system initialization phase (i.e., the data quality inspection system based on tiered disk and memory storage), involves the administrator filling in registration database information via a visual console, such as host address, port, database name, username, and encrypted password, to verify the connection's validity. Upon successful verification, the data source is persisted to the database. Subsequent users can directly select the data source, data table, and other information from a dropdown list when creating data quality inspection tasks, thus determining the data to be inspected.

[0043] The second step is for users to generate all the information required for the data quality inspection task through a visual interface, including the quality inspection standards and the corresponding quality inspection rules for each standard. At the same time, users can set the scoring ratio of each rule in its respective standard, bind the inspection fields to the rules, and save the configuration information to the system's configuration library for subsequent use by the scheduling engine.

[0044] For example: fill in the task name, select the data source registered by the administrator (the system automatically loads the database and all tables under it for the user to select), select the target table, select or set the scheduling strategy to manual or automatic (for automatic execution, the scheduling cycle can be further set), check the quality inspection standards and rules, set the inspection fields, and set the scoring ratio of each standard and rule, etc.

[0045] The third step is for the scheduling engine to schedule execution nodes to perform data quality detection tasks.

[0046] Specifically, the aforementioned configuration information is submitted to the scheduling engine, which distributes it to the execution nodes for execution. This process includes: first, reading the quality detection configuration; then, reading raw data from the target data table according to a preset batch size and writing it to a local disk file; next, loading data from the disk file into memory in batches, and performing verification and scoring calculations item by item according to the rules under the configured standards. During execution, multiple tasks can be processed in parallel to improve overall detection efficiency.

[0047] In this process, after each batch of calculations and tests are completed, the system can accumulate local scores in real time according to the standard-rule weight structure; and write records that do not conform to the rules (such as null values ​​and out-of-bounds values) and data into a remote problem data storage file in a structured manner; after all batches are processed, the final quality score is aggregated and a quality report is generated and output (of course, in some other embodiments of this application, the final result can also be calculated after all batches of data have been calculated).

[0048] In addition, the data quality inspection system based on disk and memory hierarchical storage provided in this application also provides a preset quality inspection task management module, which is used to add, edit, delete, stop, and enable the parts of the task that need to be executed before or during the execution of the task.

[0049] During the aforementioned execution process, the system captures data reading anomalies, rule execution anomalies, and other risks such as memory overflows in a layered manner (i.e., for each batch of data), and records key anomaly information in detail. The system also generates task execution logs and displays the task execution status, results, and a final report to the user based on user commands.

[0050] Based on the same inventive concept, this application also provides a data quality detection system based on hierarchical storage of disk and memory. Figure 2 This is a schematic diagram of the data quality detection system based on tiered storage of disk and memory provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the system includes: Configuration module 11 is used to determine configuration information based on user instructions in order to generate a data quality inspection task. The configuration information is used to determine the target data table and quality inspection rules.

[0051] The scheduling module 12 determines the execution time and execution node of the data quality detection task through a preset scheduling engine.

[0052] The calculation module 13 is used to read data from the target data table in batches through the execution node at the execution time, and write the read data to the disk file in batches; load data from the disk file into memory in batches, and perform quality inspection on each batch of data based on quality inspection rules to obtain the quality inspection results of each batch of data and record the problematic data; aggregate the quality inspection results of each batch of data to generate a quality inspection report.

[0053] In some embodiments of this application, the above configuration module may include: a data source management module, a quality inspection standard management module, a task configuration module, and a task management module.

[0054] The data source management module is responsible for the full lifecycle management of interface configurations, supports local storage, and provides configuration verification and dynamic loading functions to ensure that configuration information is accurate and takes effect in real time. It also provides a visual interface and API to facilitate users in managing the target data information in the configuration information corresponding to data quality detection tasks.

[0055] The quality inspection standards management module is responsible for defining, maintaining, and managing the versions of quality inspection standards. It supports built-in standard templates and custom extensions, provides standard structure verification and dynamic loading capabilities, ensuring that quality dimension configurations are flexible, unified, and reusable; at the same time, it exposes standard management capabilities to external users through a visual interface and API.

[0056] The task configuration module is responsible for configuring the quality inspection task, allowing users to select and configure the configuration information mentioned in the above method embodiments, and supports efficient maintenance by users through a visual interface and API.

[0057] Additionally, a task management module can be set up to be responsible for the creation, binding, and full lifecycle management of data quality inspection tasks on the system side, including combining the above configuration information into complete task units. It also provides functions for task status tracking, execution log viewing, result preview, and problem data download; enabling one-stop configuration and maintenance of tasks through a visual interface and API.

[0058] The aforementioned scheduling and computation modules can be integrated into a computation engine module, responsible for determining execution nodes and execution times, as well as batch execution and scoring calculation of quality detection rules and data. It adopts a disk-memory hierarchical caching architecture, reading target data in batches and loading it into memory in segments for rule verification; it supports parallel computation of multiple rules, resource usage monitoring, and anomaly isolation, balancing large-scale data processing performance with system stability.

[0059] In addition, in some embodiments of this application, a task scheduling module can be set up, which is dedicated to determining, managing and triggering the execution mode of data quality detection tasks. It supports manual immediate execution and periodic execution (in this case, the task mode information in the above configuration information can be set separately in this module instead of in the above module), provides scheduling capabilities such as task start and stop, and does not require service restart when task configuration changes. The scheduling status is synchronized in real time to ensure that tasks run on demand, reliably and efficiently.

[0060] Regarding the system in the above embodiments, other specific ways in which the various modules perform operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0061] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0062] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0063] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0064] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0065] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0066] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0067] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0068] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0069] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data quality detection method based on tiered storage of disk and memory, characterized in that, include: The configuration information is determined based on user instructions to generate a data quality inspection task, wherein the configuration information is used to determine the target data table and the quality inspection rules; The execution time and execution node of the data quality inspection task are determined by a preset scheduling engine. During the execution time, the execution node reads data from the target data table in batches and writes the read data to the disk file in batches. Data is loaded from disk files into memory in batches, and each batch of data is subjected to quality inspection based on quality inspection rules to obtain the quality inspection results of each batch of data and record the problematic data. The quality inspection results of each batch of data are aggregated to generate a quality inspection report.

2. The data quality detection method based on tiered storage of disk and memory according to claim 1, characterized in that, The configuration information includes target data information and quality inspection information; The target data information includes: the type of the target database, the target data source, and the target data table; The quality inspection information includes: quality inspection standards, quality inspection rules corresponding to the quality inspection standards, inspection fields, the score ratio of each quality inspection standard, and the score ratio of each quality inspection rule.

3. The data quality detection method based on tiered storage of disk and memory according to claim 2, characterized in that, The step of determining configuration information based on user instructions to generate a data quality detection task includes: The system provides users with multi-level pre-stored configuration information and configuration customization tools through a preset visual interface; among them, each quality inspection standard in the pre-stored configuration information corresponds to at least one quality inspection rule; It receives system pre-stored configuration information or user-defined configuration information selected by the user to generate a data quality inspection task.

4. The data quality detection method based on tiered storage of disk and memory according to claim 3, characterized in that, Also includes: The execution mode of the data quality inspection task is determined based on user instructions; The execution modes include a single-run mode based on manual triggering and a periodic cyclic execution mode based on preset periodic information.

5. The data quality detection method based on tiered storage of disk and memory according to claim 4, characterized in that, The process involves performing quality checks on each batch of data based on quality inspection rules, obtaining the quality inspection results for each batch, and recording problematic data, including: Based on the quality inspection rules under each quality inspection standard, the current batch of data is subjected to quality inspection to obtain the quality inspection results corresponding to each quality inspection rule for the current batch of data, and problematic data that does not meet the quality inspection rules are written to the remote file storage system in real time.

6. The data quality detection method based on tiered storage of disk and memory according to claim 5, characterized in that, The process of calculating the quality inspection results for each batch of data also includes: The system reads and records read errors and execution errors of each batch of data, and calculates the risk of system memory overflow in real time; wherein, the risk of system memory overflow is used to optimize the amount of data in each batch when data is split into batches.

7. The data quality detection method based on tiered storage of disk and memory according to claim 6, characterized in that, The quality inspection results of the aggregated data from each batch are used to generate a quality inspection report, including: Based on the quality inspection results and score ratios of the quality inspection rules under each quality inspection standard, calculate the score for each quality inspection standard. The final quality inspection score is calculated based on the score and score percentage of each quality inspection standard. A quality inspection report is generated based on the final quality inspection score.

8. A data quality detection system based on tiered storage of disk and memory, characterized in that, include: A configuration module is used to determine configuration information based on user instructions in order to generate a data quality detection task, wherein the configuration information is used to determine the target data table and quality detection rules; The scheduling module determines the execution time and execution node of the data quality detection task through a preset scheduling engine; The calculation module is used to read data from the target data table in batches at the execution time through the execution node, and write the read data to the disk file in batches; load data from the disk file into memory in batches, and perform quality inspection on each batch of data based on quality inspection rules to obtain the quality inspection results of each batch of data and record the problematic data; aggregate the quality inspection results of each batch of data to generate a quality inspection report.

Images