Data quality detection method, apparatus and device

Abstract

Embodiments of the present specification provide a data quality detection method, device and equipment. The scheme comprises: obtaining an asset type identifier and source data information of a data asset to be detected; the source data information is used to describe the data source of the data asset; determining asset detection parameters based on the source data information; determining at least one quality detection rule corresponding to the data asset according to the asset type identifier; creating a rule detector corresponding to each quality detection rule; taking the asset detection parameters as input, executing each rule detector to obtain the quality detection result of the data asset.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data quality detection method. This specification also relates to a data quality detection apparatus, a computing device. Background Technology

[0002] As enterprises deepen their digital transformation, data assets have become a crucial component of their core competitiveness. Various business systems continuously generate massive amounts of data, and enterprises construct data assets to semantically encapsulate this data, facilitating understanding and use by business personnel. The quality of data assets directly impacts the accuracy and reliability of subsequent data analysis and decision support; therefore, quality inspection of data assets is a critical step in the data governance process.

[0003] With the continuous growth in the types and quantities of data assets, how to efficiently and flexibly conduct quality inspections on various types of data assets has become a major concern. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data quality inspection method, apparatus, and device, providing a data quality inspection solution that can adapt to various asset types and improves inspection efficiency and scalability.

[0005] According to a first aspect of one or more embodiments of this specification, a data quality detection method is provided, comprising: Obtain the asset type identifier and source data information of the data asset to be detected; the source data information is used to describe the data source of the data asset. Determine asset detection parameters based on the source data information; Determine at least one quality inspection rule corresponding to the data asset based on the asset type identifier; Create rule detectors corresponding to each of the aforementioned quality inspection rules; Using the asset detection parameters as input, each of the rule detectors is executed to obtain the quality detection results of the data assets.

[0006] According to a second aspect of one or more embodiments of this specification, a data quality detection apparatus is provided, comprising: The information acquisition module is used to acquire the asset type identifier and source data information of the data asset to be detected; the source data information is used to describe the data source of the data asset. The parameter conversion module is used to determine asset detection parameters based on the source data information; The detection rule determination module is used to determine at least one quality detection rule corresponding to the data asset based on the asset type identifier; The rule detector creation module is used to create rule detectors corresponding to each of the aforementioned quality detection rules. The quality inspection module is used to take the asset inspection parameters as input, execute each of the rule detectors, and obtain the quality inspection results of the data assets.

[0007] According to a third aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer instructions, implements the steps of the data quality detection method.

[0008] One embodiment of this specification can achieve at least the following beneficial effects: by acquiring asset type identifiers and source data information, the source data information is uniformly converted into standardized parameters. At the same time, the corresponding detection rules are determined based on the asset type identifiers, and corresponding rule detectors are created. Finally, the standardized parameters are input into the detectors to perform detection. Thus, the asset model and detection logic are decoupled: the asset type identifier determines "what to detect," the rule detector is responsible for "how to detect," and the standardized parameters encapsulate the "detection object." Therefore, the detection parameters can be independent of the data source format, the detection rules can be independent of the asset type configuration, and the detector can be dynamically expanded. Thus, when conducting quality assessment of data assets, it can adapt to diverse asset types and detection needs, and improve the flexibility and scalability of asset quality detection. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This diagram illustrates an application scenario of a data quality detection method provided in the embodiments of this specification. Figure 2 A schematic flowchart illustrating a data quality detection method provided in an embodiment of this specification; Figure 3 This is a schematic diagram of a data quality inspection process provided in an embodiment of this specification; Figure 4 A schematic diagram of a data asset quality inspection process provided in the embodiments of this specification; Figure 5 A schematic diagram of a hierarchical scheduling scheme for data asset quality inspection provided in the embodiments of this specification; Figure 6 A schematic diagram of a data asset quality inspection process provided in the embodiments of this specification; Figure 7 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a data quality detection device; Figure 8 A structural block diagram of a computing device provided according to an embodiment of this specification is shown. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0012] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0014] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0015] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0016] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0017] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0018] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0020] The following explains the terms and concepts used in one or more embodiments of this specification.

[0021] Data assets refer to semantic data models oriented towards business analysis. They are used to structurally encapsulate elements such as entities, behaviors, and attributes in business activities to facilitate subsequent analysis and use. Data assets themselves do not directly store data, but rather associate underlying source data through asset model information.

[0022] Data assets can include contextualized assets. Contextualized assets refer to assets with contextual semantics, mainly including events, entities and entity attributes, and groups. Data assets may also include some analytical assets (such as user flow and touchpoints). An event is the smallest indivisible unit of result generated by user behavior within a business domain. For example, a user completing a search generates a search event, and completing an add-to-cart event generates an add-to-cart event. Entities are the main subjects of business scenario analysis, such as payment users of a payment platform or registered users of a short video platform. Entity attributes are the perspectives used to analyze data, describing certain characteristics of things or phenomena or the attributes of entities in the business, such as user industry preferences. Groups are collections of entities with certain common characteristics, such as the monthly active user group of a payment platform.

[0023] Data assets can be aggregated and managed within contextualized datasets. A contextualized dataset is the smallest unit of granularity for the aggregation and consumption of contextualized assets; it is an aggregation of one or more contextualized assets. Contextualized datasets can also maintain relationships between assets (such as relationships between events, relationships between entities, etc.).

[0024] In data-driven business scenarios, enterprises typically encapsulate business data into data assets that can be analyzed and used. Different types of data assets (such as events, tags, and demographics) carry different business semantics, and their quality directly affects the accuracy of subsequent data analysis and decision-making.

[0025] Current data quality diagnosis and detection are typically based on table-level granularity, with the configuration of detection rules and the expansion of detection capabilities limited by the table structure itself. Specifically, users prepare data based on physical data tables, and related processing systems execute data reading, transformation, or loading operations upon receiving the data preparation request. Quality detection usually accompanies these data reading, transformation, or loading operations. For data assets, the creation phase lacks a pre-assessment of whether the underlying data meets processing requirements; after asset creation, structural changes or changes in data volume of the underlying data may render the asset unusable. Furthermore, in existing practices, data quality detection methods are difficult to customize on demand and isolate scenarios for different types of data use cases. When business needs change or the types of data assets increase, the flexibility and scalability of the detection solution face challenges.

[0026] To address the shortcomings of related technologies, the embodiments in this specification first acquire the asset type identifier of the data asset to be tested and source data information describing the data source. The source data information is then converted into standardized asset testing parameters. Simultaneously, at least one corresponding quality testing rule is determined based on the asset type identifier, and rule detectors are created corresponding to each quality testing rule. Finally, the asset testing parameters are input into the rule detectors and executed to obtain the quality testing results. This decouples the asset's type attributes from its underlying data source, allowing quality testing rules to be independently configured and extended based on the asset type without concern for differences in specific data sources. Furthermore, by uniformly converting the source data information into standardized asset testing parameters, heterogeneous data source information from different asset types can be input into the rule detectors in a unified format, further strengthening the decoupling between the testing logic and the asset model. By creating a corresponding rule detector for each quality testing rule, the separation of testing rules and execution units is achieved, ensuring the reusability of the testing logic and providing a foundation for the dynamic expansion of subsequent testing capabilities. The synergistic effect of the above features, due to the decoupling of detection rules from asset types, the uniformity of detection parameter formats, and the dynamic creation of detectors, enables the system to flexibly adapt to the detection needs of different asset types and support the on-demand configuration and expansion of detection rules. This effectively improves the determinism, flexibility, and maintainability of asset quality detection, ensuring the stability of assets during the preparation stage and subsequent use.

[0027] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram illustrating an application scenario of a data quality detection method provided in the embodiments of this specification.

[0029] In some embodiments of this specification, the data quality detection method obtains the asset type identifier and source data information of the data asset, converts the source data information into standardized detection parameters, and dynamically creates corresponding rule detectors based on the asset type identifier to perform detection. This decouples the asset model from the detection logic, flexibly supports the quality assessment needs of different asset types, and ensures the quality stability of the asset in both pre-detection (such as detection before asset release) and in-process inspection (such as detection after asset release).

[0030] The data quality detection methods provided in the embodiments of this specification can be applied to products such as data asset management platforms, data governance systems, data analysis tools, or data middleware platforms, serving as data quality control components for these products. These products can run on corresponding computing devices. Exemplarily, the computing device may include a personal computer, workstation, server, or a server cluster consisting of multiple servers, a cloud computing service center, or terminal devices such as laptops and tablets; this specification does not specifically limit its application to these devices.

[0031] In practical applications, all steps in the data quality detection method provided in the embodiments of this specification can be executed entirely on the server side, such as being deployed on an enterprise's private or public cloud platform to provide SaaS services to users; or they can be executed entirely on the client device, such as integrating localized quality detection capabilities into a local data analysis tool; or they can be partially executed on the server side and partially on the client device, which can be flexibly configured according to the actual scenario. For example, pre-event detection (such as real-time detection during asset creation) can be quickly completed on the server side and results can be returned, while in-event inspection (such as periodic batch detection) can be executed through server-side scheduling tasks, and the detection results can be notified to users through the client interface or message push.

[0032] like Figure 1 As shown, as a specific example of user interaction, users (such as data analysts) can access the data asset management platform deployed on server 20 through a graphical user interface on client device 10 (such as a personal computer or laptop).

[0033] For example, during the asset creation phase, users can select "Create Event Asset" on the platform interface, fill in information such as the asset name, associated data table, and data filtering conditions, and then trigger the "Submit for Detection" control. The system immediately enters the detection process, and the graphical user interface can dynamically display the detection progress bar, showing statuses such as "Verifying data source availability," "Verifying field type," and "Statistical data volume." After the detection is completed, the graphical user interface can display a "Detection passed" or "Detection failed" result label. If the detection fails, the failed detection items can be further listed (e.g., "Data volume exceeds threshold: current 2.5 billion rows, limit 2 billion rows"), along with repair suggestions (e.g., "It is recommended to reduce the data volume using the event row and column pruning function"). Users can adjust the configuration according to the suggestions and re-detect until it passes.

[0034] For example, the platform can perform periodic inspections on listed assets. When an asset fails the inspection, the system can automatically send a notification message to the user's linked account. After receiving the notification, the user can view the details of the anomaly and repair or reconfigure the asset based on the system's suggestions.

[0035] Based on the solutions provided in the embodiments of this specification, users can clearly understand the asset quality status and take corresponding measures in a timely manner throughout the entire process of data asset preparation and use, thereby ensuring the stability and reliability of data assets in the preparation and use stages.

[0036] This specification provides a data quality detection method in its embodiments. This application also relates to a data quality detection device and a computing device, which will be described in detail in the following embodiments.

[0037] Figure 2 This is a flowchart illustrating a data quality detection method provided in an embodiment of this specification.

[0038] From a programming perspective, the entity executing the process can be a program hosted on a server or terminal. It can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0039] like Figure 2 As shown, the process may include the following steps: Step 202: Obtain the asset type identifier and source data information of the data asset to be detected; the source data information is used to describe the data source of the data asset.

[0040] In some embodiments, the data quality inspection may specifically be aimed at inspecting the quality of data assets. Here, data assets refer to semantic data models oriented towards business analysis, such as event assets, tag assets, and audience assets. Data assets themselves do not directly store data, but rather associate underlying source data through asset model information. Source data refers to the physical storage data associated with the data assets, including data tables, files, views, etc.

[0041] Obtaining the asset type identifier and source data information of the data asset to be inspected can be performed in response to an asset creation request, asset editing request, or an inspection request triggered by a preset cycle. The asset type identifier identifies the business type of the data asset, such as event type, tag type, or population type. Source data information refers to the set of information used to locate and define the source data corresponding to the data asset, linking the asset to its underlying data so that inspection can be performed on the correct data range. In some embodiments, source data information includes at least source data source information and source data range information, where the source data source information may be a source table identifier, and the source data range information may include at least one of data filtering conditions, partition fields, and the number of partitions. Source data information serves as a bridge connecting the data asset and the underlying physical data; the quality inspection operation actually operates on the source data pointed to by the source data information.

[0042] In some embodiments, the source data information of a data asset can also be referred to as asset model information. An asset model refers to the specific implementation description of a data asset at the technical level, including physical table information, field information, data filtering conditions, etc., behind the asset. Asset model information is a set of metadata extracted from the asset model for performing quality checks. The source data information of a data asset can be metadata required for quality checks of the data asset, and may include table structure and business configuration information such as filtering conditions. In practical applications, the source data information of an asset refers to a set of information describing the technical implementation details of the data asset, including but not limited to at least one of the following: the source table identifier corresponding to the asset, the list and types of field fields in the source table, data filtering conditions, partition field identifiers, the number of partitions, and estimated data volume. Different types of data assets may have different asset model information structures, but all contain the basic metadata required for performing data quality checks.

[0043] For example, taking the "user click event" asset of e-commerce platform A as an example, the asset type is identified as "event". Its source data information can include: the source table identifier "click_log", used to locate the data table storing click logs; the data filtering condition "dt>= '2025-01-01'", used to limit the detection to data from this year onwards; the partition field identifier "dt", used to indicate that data is stored by date partitions; and the number of partitions "365", used to indicate that there are 365 partitions in total. Through the above source data information, the system can clearly define the specific data range that needs to be detected.

[0044] By acquiring the asset type identifier and source data information of data assets, the asset layer and data layer are separated. The asset type identifier is used for subsequent matching and detection rules, determining "what to detect"; the source data information is used for subsequent generation of standardized detection parameters, determining "which range of data to detect". The combination of the two enables quality detection to perform accurate and efficient quality assessment of the source data behind the assets, while avoiding coupling the asset model with the detection logic, which is beneficial to improving the scalability of the system.

[0045] Step 204: Determine asset detection parameters based on the source data information.

[0046] In some embodiments, after obtaining the source data information of the data asset, asset detection parameters are determined based on the source data information. The asset detection parameters are a set of parameters used to pass source data-related information to the detection logic. This parameter set may include metadata required for performing quality detection, such as source table identifiers, filtering conditions, partition information, and field lists. By determining the asset detection parameters, the subsequent quality detection logic does not need to be aware of the specific differences in the asset model, and the detection module can perform detection operations based on a unified parameter structure.

[0047] In some embodiments, determining asset detection parameters based on source data information can be achieved by converting the source data information into standardized asset detection parameters. Standardized asset detection parameters refer to a set of parameters formed by encapsulating heterogeneous data from different source data information according to a preset unified data structure and format. Different types of assets (such as events, tags, and groups) typically have different source data information structures. For example, the source data information for event-type assets may include table names, filtering conditions, partition fields, etc.; the source data information for tag-type assets may include table names, primary key fields, grouping fields, etc.; and the source data information for group-type assets may include table names, entity primary key fields, etc. In a specific example, a data platform's group asset is used to represent "highly active user groups," and its source table stores a list of user IDs that meet specific activity conditions. The source data information may include the table name `high_active_users`, the entity primary key field `user_id`, etc. By converting the aforementioned heterogeneous source data information into standardized asset detection parameters in a unified format, subsequent quality detection logic does not need to be aware of the specific differences in the asset model, and the detection module can perform detection operations based on parameters in a unified format.

[0048] In other embodiments, determining asset detection parameters based on source data information can also be implemented in a non-standardized manner. For example, a unified parameter acquisition interface can be implemented for each asset type using the adapter pattern or interface abstraction. The detector directly obtains the required source data information through this interface without explicitly converting heterogeneous data into a unified standardized parameter format. Specifically, a unified detection parameter interface is defined for each asset type, and the detector obtains the required detection parameters through this interface; adapters for different asset types implement their own data conversion logic, mapping heterogeneous source data information to a parameter format that the detector can consume. Based on the adapter pattern, when adding a new asset type, only the corresponding adapter implementation class needs to be added, without modifying the detector code.

[0049] Regardless of the method used, as long as source data information of different asset types can be transmitted to the detector in a way that is identifiable by the detection logic, the decoupling of the asset model and the detection logic can be achieved. For ease of description, the following embodiments in this specification are mainly described using a standardized conversion method. However, those skilled in the art should understand that other methods that can achieve the same function also fall within the protection scope of this application.

[0050] In some embodiments, the asset detection parameters include at least one of the following: source table identifier, data filtering conditions, partition field identifier, number of partitions, and source table field list. The source table identifier is used to locate a specific data table; in the embodiments of this specification, the asset detection parameters serve as a bridge connecting source data information and detection logic. Therefore, the detection logic can recognize only the asset detection parameters without needing to concern itself with the specific structure of the source data information corresponding to different types of data assets. The data filtering conditions are used to limit the data range to be detected; in some embodiments, the data filtering conditions can be used for rule detectors of types such as data volume detection, data integrity detection, and uniqueness detection. The partition field identifier is used to indicate the partition columns of the data table; in some embodiments, the partition field identifier can be applied to rule detectors of types such as data volume detection and partition detection. The number of partitions is used to indicate the total number of partitions; in some embodiments, the partition field identifier can be applied to rule detectors of types such as data volume detection (calculating the total data volume by combining partition fields). The source table field list is used to provide information such as field names and field types; in some embodiments, the partition field identifier can be applied to rule detectors of types such as data volume detection (calculating the total data volume by combining partition fields).

[0051] For example, for event-type assets, the source data information includes the table name `click_log`, the filtering condition `dt>='2025-01-01'`, and the partition field `dt`, etc.; for tag-type assets, the source data information includes the table name `user_tag`, the primary key field `user_id`, and the grouping field `tag_type`, etc.; for audience-type assets, the source data information includes the table name `high_active_users` and the entity primary key field `user_id`, etc. After conversion, both are encapsulated into standardized asset detection parameters containing fields such as `dsGuid` (source table identifier), `filter` (filtering condition), and `partitionColumnName` (partition field). In another embodiment, the above two different types of source data information each implement a unified parameter acquisition interface. The detector obtains the required data through interface calls, which also decouples the asset model from the detection logic.

[0052] Through the above transformation, the source data information generated by different types of assets is unified into standardized asset detection parameters. The detection module can perform subsequent quality detection operations based on the same parameter structure, which is conducive to improving the reusability of detection logic and the scalability of the system.

[0053] In some embodiments, asset detection parameters may further include semantic information at the asset level. Specifically, asset detection parameters may also include at least one of the following: unique asset identifier, asset name, asset type, dataset identifier to which the asset belongs, and tenant identifier to which the asset belongs.

[0054] There is a hierarchical relationship between assets, datasets, and tenants. A dataset is a container for assets; a dataset can contain one or more data assets. A contextualized dataset is the smallest unit of aggregation and consumption of contextualized assets; it is an aggregation of one or more contextualized assets. Contextualized datasets can maintain relationships between assets, such as relationships between events or between entities. A tenant is a unit of configuration isolation and management; an asset belongs to a specific tenant. In some embodiments, the hierarchical relationship of assets can be represented as: Tenant – Dataset – Asset – Source Table. Here, the tenant is at the highest level, serving as a business isolation unit; the dataset is the asset aggregation unit; the asset is a business semantic unit; and the source table is a physical storage unit.

[0055] The asset-level semantic information included in the asset detection parameters can be used in various scenarios. During the rule filtering phase, when determining quality detection rules, the system can recursively merge detection rules applicable to the asset from multi-level configurations based on the asset's tenant, dataset, and asset type. Tenant identifiers and dataset identifiers can be used to locate the corresponding level of rule configuration, thereby determining the detection items that should be performed on the asset.

[0056] During the detection execution phase, the rule detector primarily relies on metadata such as source table identifier, filtering conditions, partition fields, and number of partitions in the asset detection parameters to locate data and execute verification logic. Asset identifiers, tenant identifiers, and dataset identifiers do not participate in data operations at this stage, but they can be used for log output, issue report association, and to aggregate detection results into specific asset records for easy traceability during subsequent inspections and anomaly handling.

[0057] In the result recording and traceability stage, for inspection scenarios, the unique asset identifier can be used to uniquely identify each inspection result record, while the tenant identifier and dataset identifier can be used to statistically analyze asset quality status, send notifications, or perform tiered disposal by tenant or dataset dimension.

[0058] By incorporating semantic information at the asset level into asset detection parameters, precise rule matching can be achieved before detection, and complete recording of results and closed-loop management of the asset dimension can be achieved after detection.

[0059] In some embodiments, the hierarchical identification information (including the dataset identifier to which the asset belongs and the tenant identifier to which the asset belongs) can be read from the metadata of the data asset or parsed from the asset creation request or asset editing request. The method of obtaining the hierarchical identifier can be flexibly selected for different types of requests or different asset management scenarios, and this specification does not specifically limit this method.

[0060] Step 206: Determine at least one quality inspection rule corresponding to the data asset based on the asset type identifier.

[0061] In some embodiments, after obtaining the asset type identifier of the data asset, at least one quality inspection rule corresponding to the data asset is determined based on the asset type identifier. The asset type identifier is used to identify the business type of the data asset, such as event type, tag type, or audience type. Different types of assets typically correspond to different sets of quality inspection rules to suit their respective data characteristics and quality requirements.

[0062] In some embodiments, a quality inspection rule set corresponding to the asset type identifier can be read from a preset rule base based on the asset type identifier. The quality inspection rule set contains at least one quality inspection rule. Each quality inspection rule can correspond to one inspection item, such as column count inspection, data volume inspection, field type inspection, table availability inspection, data integrity inspection, data uniqueness inspection, etc.

[0063] For example, for data assets identified as "events," the corresponding quality inspection rule set can include column count inspection rules, data volume inspection rules, field type inspection rules, and table availability inspection rules. Specifically, the column count inspection rule can be configured with an upper limit threshold for the number of columns (e.g., 100 columns), the data volume inspection rule can be configured with an upper limit threshold for the number of data rows (e.g., 2 billion rows), and the field type inspection rule can be configured with a whitelist of valid field types (e.g., int, string, date, double, etc.).

[0064] For data assets identified as "tags," the corresponding quality inspection rule set can include column count inspection rules, data volume inspection rules, field type inspection rules, and table availability inspection rules, as well as data integrity inspection rules and data uniqueness inspection rules. Data integrity inspection rules are used to check whether key fields contain null values, and data uniqueness inspection rules are used to check whether primary key fields contain duplicate values.

[0065] In some embodiments, the quality inspection rule includes at least one of the following: rule type identifier, inspection dimension identifier, threshold parameter, rule description template, and repair suggestion template.

[0066] The rule type identifier is a unique identifier used to identify the type of a quality inspection rule. Different rule types correspond to different detection logics, such as column count detection (COLUMN_NUM_DETECTION), data volume detection (TOTAL_ROW_NUM_DETECTION), and field type detection (DATA_VALUE_TYPE_DETECTION). The rule type identifier is used to match the corresponding rule detector in the rule engine.

[0067] The detection dimension identifier is used to identify the detection dimension to which a quality detection rule belongs. Detection dimensions include, but are not limited to: data size dimension (DATA_SIZE), data quality dimension (DATA_QUALITY), and data value dimension (DATA_VALUE). The detection dimension identifier is used for rule classification, statistical analysis, and detection report generation.

[0068] Threshold parameters are specific values ​​or conditions used in quality control rules to determine whether data is acceptable. Different types of rules correspond to different types of threshold parameters. For example, column count detection rules include an upper limit threshold for the number of columns; data volume detection rules include an upper limit threshold for the number of data rows; field type detection rules include a whitelist of valid field types; and uniqueness detection rules include a boolean value indicating whether to check uniqueness.

[0069] A rule description template is a text template used to generate rule descriptions and may contain placeholders (such as %s). When outputting the detection results, the system fills the placeholders with the specific detection values ​​(such as the actual number of columns or the actual number of data rows) to generate complete rule description information.

[0070] A repair suggestion template is a text template used to generate repair suggestions, which are displayed to the user when the test fails. The repair suggestion template can include specific operational instructions, such as "The ability to prune event rows and columns can reduce unnecessary data volume."

[0071] Different quality inspection rules may include different combinations of fields. For example, a column count inspection rule may include a maximum column count threshold, but may not include a partition count threshold from a data volume inspection rule; a field type inspection rule may include a whitelist of valid field types, but may not include numeric thresholds. Therefore, in the embodiments of this specification, the quality inspection rules include at least one of the above fields, rather than being limited to including all fields, to adapt to the personalized needs of different inspection types.

[0072] In one specific embodiment, the configuration of quality inspection rules for event type assets can be as follows: { "assetType": "EVENT", "ruleConfigs": [ { "detectionRuleType": "COLUMN_NUM_DETECTION", "detectionType": "DATA_SIZE", "columnLimitNum": 100, "ruleDescTemplate": "Number of columns not exceeding 100", "suggestionTemplate": "The ability to prune event rows and columns can reduce the amount of unnecessary data." }, { "detectionRuleType": "TOTAL_ROW_NUM_DETECTION", "detectionType": "DATA_SIZE", "limitNum": 2000000000, "ruleDescTemplate": "Sets the limit for the number of event data rows within the accelerated partition range to not exceed %s", "suggestionTemplate": "The ability to prune event rows and columns can reduce the amount of unnecessary data." } ] }

[0073] Among them, detectionRuleType is the rule type identifier, used to match the corresponding rule detector in the rule engine; detectionType is the detection dimension identifier, used for rule classification; columnLimitNum and limitNum are threshold parameters; ruleDescTemplate is the rule description template; and suggestionTemplate is the repair suggestion template.

[0074] In another specific embodiment, the configuration of quality inspection rules for tag-type assets can be as follows: { "assetType": "DIMENSION", "detectionRuleScope": "OVERALL", "ruleConfigs": [ { "detectionRuleType": "DATA_VALUE_TYPE_DETECTION", "detectionType": "DATA_VALUE", "legitimateDataValueTypeList": [ "int", "string", "date", "double", "boolean", "bigint", "tinyint", "smallint", "float", "timestamp", "datetime" ], "resultDescTemplate": "The following field type does not meet the requirements: %s", "ruleDescTemplate": "Column value data type is limited to: %s", "suggestionTemplate": "Please replace with an asset table that meets the requirements" }, { "checkAvailability": true, "detectionRuleType": "TABLE_AVAILABILITY_DETECTION", "detectionType": "DATA_QUALITY", "resultDescTemplate": "Source table: %s has been deleted", "ruleDescTemplate": "The source table is in a normal state (it cannot be in a deleted, offline, or other similar state)", "suggestionTemplate": "Please replace with an asset table that meets the requirements" }, { "checkIntegrity": true, "detectionRuleType": "DATA_INTEGRITY_DETECTION", "detectionType": "DATA_QUALITY", "resultDescTemplate": "%s: NULL value exists, %s: meets the requirements", "ruleDescTemplate": "Entity fields are not allowed to have null values", "suggestionTemplate": "Please replace with an asset table that meets the requirements" }, { "detectionRuleType": "TOTAL_ROW_NUM_DETECTION", "detectionType": "DATA_SIZE", "limitNum": 2000000000, "speedUpLimitNum": 2000000000, "samplingLimitNum": 150000000000, "resultDescTemplate": "Summary within the acceleration range: %s", "ruleDescTemplate": "The number of rows in the most recent partition does not exceed %s", "suggestionTemplate": "Please replace with an asset table that meets the requirements" }, { "columnLimitNum": 150, "detectionRuleType": "COLUMN_NUM_DETECTION", "detectionType": "DATA_SIZE", "resultDescTemplate": "Number of columns: %s columns", "ruleDescTemplate": "Number of columns not exceeding 150", "suggestionTemplate": "Please replace with an asset table that meets the requirements" }, { "checkUniqueness": true, "detectionRuleType": "DATA_UNIQUENESS_DETECTION", "detectionType": "DATA_QUALITY", "resultDescTemplate": "%s: Duplicate value exists, example: %s", "ruleDescTemplate": "For tag-type assets, duplicate values ​​are not allowed in the entity primary key field". "suggestionTemplate": "Please replace with an asset table that meets the requirements"} ] }

[0075] Compared to event-type assets, tag-type assets have additional quality control rules for data integrity (DATA_INTEGRITY_DETECTION) and data uniqueness (DATA_UNIQUENESS_DETECTION). Integrity checks for null values ​​in the primary key fields of the tag asset, while uniqueness checks for duplicate records in the primary key fields. In the above configuration, `checkIntegrity` and `checkUniqueness` are threshold parameters specific to the respective control rules, used to enable integrity and uniqueness checks, respectively. `resultDescTemplate` and `suggestionTemplate` are used to generate a description of the control results and remediation suggestions, respectively.

[0076] Based on the embodiments in this specification, during the quality inspection process, the rule engine first reads the corresponding quality inspection rule set from the preset rule base according to the asset type identifier. Each quality inspection rule includes a rule type identifier, an inspection dimension identifier, threshold parameters, a rule description template, and a repair suggestion template. Then, the rule engine can dynamically create corresponding rule detector instances according to the rule type identifier through the factory pattern. For example, when the rule type identifier is "COLUMN_NUM_DETECTION", a column count detector is created; when the rule type identifier is "TOTAL_ROW_NUM_DETECTION", a data volume detector is created. Furthermore, when the rule detector is executed, it can obtain information such as the source table identifier, filtering conditions, and field list from the standardized inspection parameters, and verify the source data according to the threshold parameters in the rule (such as the upper limit of the number of columns and the upper limit of the data volume). Finally, after the inspection is completed, the rule engine can generate inspection description information (such as "actual number of columns: 101 columns, exceeding the threshold of 100 columns") according to the rule description template and the actual inspection value, and generate repair guidance according to the repair suggestion template.

[0077] It should be noted that there is no necessary sequential order between the step "determining quality inspection rules based on asset type identifier" and the aforementioned step "converting source data information into standardized inspection parameters". In some embodiments, the quality inspection rules can be determined first and then the source data information can be converted; in other embodiments, the source data information can be converted first and then the quality inspection rules can be determined; the two steps can also be executed in parallel. As long as the two steps are completed before the rule detector is executed, it is acceptable. Those skilled in the art can choose any order according to actual implementation needs, and all of these are within the scope of protection of this application.

[0078] Step 208: Create rule detectors corresponding to each of the aforementioned quality inspection rules.

[0079] In some embodiments, after determining at least one quality inspection rule corresponding to the data asset, a rule detector is created corresponding to each quality inspection rule. A rule detector is an executable unit used to perform a single inspection logic; each rule detector corresponds to a type of inspection, such as column count inspection, data volume inspection, field type inspection, etc. Rule detectors can be dynamically created based on the rule type identifier in the quality inspection rule.

[0080] There is a correspondence between quality inspection rules and rule detectors. Quality inspection rules are static, configurable inspection standards stored in data formats (such as JSON), containing information such as rule type identifiers and threshold parameters. Rule detectors are dynamically created executable units based on quality inspection rules, responsible for executing the specific inspection logic of those rules. For example, for a quality inspection rule with the rule type identifier "COLUMN_NUM_DETECTION", a column count detector object can be created; for a quality inspection rule with the rule type identifier "DATA_VOLUME_DETECTION", a data volume detector object can be created.

[0081] Creating rule detectors corresponding to each quality inspection rule can include obtaining the rule type identifier of each quality inspection rule, and then creating a rule detector instance corresponding to the rule type identifier. There are various ways to implement the creation of rule detectors, such as using the factory pattern, reflection, or conditional statements. The key is to create the corresponding rule detector based on the quality inspection rule; the specific implementation method can be chosen according to actual needs. The entity executing the rule detector creation can be the rule engine, scheduler, or other modules; it is not limited to a specific entity.

[0082] Once created, the rule detector is in a ready state, waiting to receive detection parameters and execute the detection. Creating a rule detector and executing it are two independent steps: the former prepares the detection capability, while the latter actually executes the detection. Separating the creation and execution of rule detectors improves the reusability and scalability of detection capabilities. When adding a new detection type, only the corresponding rule detector class needs to be added; there is no need to modify the calling logic of the detection process.

[0083] For example, for the "user click event" asset of e-commerce platform A, the quality inspection rules determined based on its asset type identifier "Event" include column count inspection rules, data volume inspection rules, field type inspection rules, and table availability inspection rules. The system creates rule detectors corresponding to these rules: column count detector, data volume detector, field type detector, and table availability detector. Each detector is in a ready state, waiting for subsequent input inspection parameters before executing the corresponding inspection operation.

[0084] In some embodiments, creating a rule detector corresponding to each of the quality detection rules specifically includes: obtaining the rule type identifier of each of the quality detection rules, and creating a rule detector instance corresponding to the rule type identifier through a rule engine based on the rule type identifier.

[0085] The rule engine is a software module used to load, parse, and manage quality inspection rules. It is responsible for matching the corresponding set of inspection rules based on the asset type identifier, instantiating rule detectors based on the asset inspection parameters, and scheduling the rule detectors to execute inspection tasks. The rule type identifier identifies the type of quality inspection rule. For example, the rule type identifier for column count inspection could be "COLUMN_NUM_DETECTION," and the rule type identifier for data volume inspection could be "DATA_VOLUME_DETECTION." Creating a corresponding rule detector instance based on the rule type identifier refers to the process of generating an executable unit corresponding to that inspection type. For example, when the rule type identifier is "COLUMN_NUM_DETECTION," a column count detector object can be created; when the rule type identifier is "DATA_VOLUME_DETECTION," a data volume detector object can be created. The created rule detector is in a ready state, waiting to receive inspection parameters and execute the inspection. In practical applications, the asset inspection parameters can be used as input parameters to instantiate a rule detector corresponding to the at least one quality inspection rule through the rule engine.

[0086] In some embodiments, the configuration and execution of detection rules are separated. Quality detection rules are configurable, storable, and modifiable detection standards, typically stored in a data format (such as JSON). Rule detectors are execution units dynamically created based on the quality detection rules, responsible for executing the specific detection logic. With this setup, adding a new rule type only requires adding the corresponding detector class and the corresponding rule to the configuration; adjusting the detection threshold only requires modifying the configuration of the quality detection rules, without modifying the detector code.

[0087] Based on some embodiments in this specification, the rule engine creates corresponding rule detector instances according to rule type identifiers, enabling on-demand creation and flexible scheduling of detection capabilities. The rule engine, as a unified management module, is responsible for rule loading, detector instantiation, and scheduling, ensuring that the configuration of detection rules and the execution logic of the detectors are independent. When a new detection type needs to be added, only the corresponding rule detector class needs to be added, and rules of that type need to be added to the rule configuration; when the detection threshold needs to be adjusted, only the rule configuration needs to be modified, without changing the detector code. This design improves the scalability and maintainability of the detection system and reduces the system modification costs caused by rule changes or type expansions.

[0088] Step 210: Using the asset detection parameters as input, execute each of the rule detectors to obtain the quality detection results of the data assets.

[0089] In some embodiments, after creating rule detectors corresponding to each quality inspection rule, the standardized asset inspection parameters generated in step 204 are used as input to execute each rule detector and obtain the quality inspection results of the data assets.

[0090] In some embodiments, executing each of the rule detectors specifically includes: executing multiple rule detectors among the at least one rule detector in parallel.

[0091] The rule detector receives standardized asset detection parameters as input, performs quality checks on the source data indicated by the parameters according to its built-in detection logic, and outputs detection sub-results. These sub-results may include information such as whether the detection item passed or failed, as well as specific detection values ​​and descriptive information. By integrating the detection sub-results output by each rule detector, the overall data asset quality detection result can be obtained.

[0092] For example, for the "user click event" asset of e-commerce platform A, standardized asset detection parameters are generated in step 204, including a dsGuid field with the value "click_log", a filter field with the value "dt>= '2025-01-01'", and a partitionColumnName field with the value "dt". In step 206, column count detection rules, data volume detection rules, field type detection rules, and table availability detection rules are determined. In step 208, column count detectors, data volume detectors, field type detectors, and table availability detectors are created respectively. In step 210, the standardized asset detection parameters are input to the above detectors, and each detector executes its corresponding detection logic: the column count detector obtains the number of columns in the "click_log" table and compares it with a threshold; the data volume detector counts the number of data rows that meet the filtering conditions and compares it with a threshold; the field type detector checks whether each field type is within the whitelist; and the table availability detector confirms that the "click_log" table exists and is in a normal state. Each detector outputs its own sub-results. The system integrates these sub-results to generate a quality inspection result that includes the pass / fail status of each test item, specific values, and suggested information.

[0093] Figure 3 This is a schematic diagram of a data quality inspection process provided in an embodiment of this specification.

[0094] like Figure 3 As shown, in some embodiments, the data assets to be detected, representing different business semantics, may include types such as events, tags, and groups. These data assets can be provided to the asset detection business processing layer, which performs a unified transformation of the model information of various data assets to generate asset detection parameters. Specifically, this transformation process encapsulates heterogeneous model information of different asset types (such as source table identifiers, filtering conditions, and partitioning information for event assets, and source table identifiers, primary key fields, and grouping fields for tag assets) according to a preset unified data structure and format, forming standardized asset detection parameters. In addition, the processing layer carries an asset type identifier for the transformed asset detection parameters. This asset type identifier is used to identify the business type (e.g., event, tag, or group) to which the current detection object belongs in the subsequent detection rule matching stage.

[0095] After the asset detection parameters are generated, the system can determine the set of quality detection rules applicable to the current asset based on the asset type identifier and the preset multi-level configuration system (such as global configuration, tenant configuration, dataset configuration, physical table configuration, asset configuration, etc.), and perform detection on the asset detection parameters according to the set of rules. Detection dimensions can include, but are not limited to, data source dimension detection (such as table availability detection), data value dimension detection (such as field type detection), data quality dimension detection (such as data integrity detection, data uniqueness detection), and data scale dimension detection (such as column count detection, data volume detection). Other detection types can also be extended according to actual needs.

[0096] Through the above process, the asset inspection business processing layer decouples the asset model from the inspection logic. Heterogeneous information of different asset types is unified into standardized asset inspection parameters. Inspection rules are matched according to asset type identifiers, and the inspection process is executed according to dimension classification. This realizes a unified processing framework for asset quality inspection, which is conducive to improving the reusability of inspection logic and the scalability of the system.

[0097] In some embodiments, when the rule detector encounters an anomaly during the execution of the detection operation (e.g., database connection timeout, SQL execution error, source table non-existence, etc.), the system captures the anomaly and records the anomaly information; the detection item marked as an anomaly is included as an anomaly item in the corresponding detection sub-result and incorporated into the final quality detection result of the data asset. Optionally, the system can decide whether to trigger a retry mechanism based on the anomaly type. For example, for recoverable anomalies such as database connection timeout, the detection item is marked as an anomaly after failing to retries a preset number of times. For detection failures caused by unrecoverable anomalies such as source table non-existence, the detection item is marked as failing. The specific implementation of the above anomaly handling mechanism can be flexibly configured according to the actual operating environment of the system, and the embodiments in this specification do not specifically limit it.

[0098] In some embodiments, each configuration level manages configuration items for detection rules, and its configuration logic follows the "configuration on demand" principle. The system's execution logic is as follows: when a detection item does not have a corresponding rule configured in the currently available configuration level for an asset, that detection item will not be executed in the asset's quality detection process. For example, if data integrity detection is not configured in the global, tenant, dataset, physical table, and asset configuration levels, the system will skip that detection item during detection and mark it as "not configured" in the detection results. For a particular asset type, if that type of asset typically does not require a specific detection item, not configuring that detection item is normal configuration behavior and does not constitute a system anomaly or functional deficiency. When the business determines that an asset requires a specific detection item, the configuration personnel only need to complete the configuration of that detection item in one or more corresponding configuration levels to enable the corresponding detection capability. This configuration mechanism can avoid unnecessary detection interruptions or false alarms due to missing configurations and allows users to flexibly decide to enable or disable a detection item according to business needs, thus adapting to the diverse detection needs of different asset types in different business scenarios.

[0099] While one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only possible execution order. The order of some steps may be adjusted according to actual needs, or some steps may be omitted. When the claims involve method steps, changes in the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0100] Figure 2 The proposed method acquires the asset type identifier and source data information of data assets. The asset type identifier is used to match the corresponding quality inspection rules, and the source data information is converted into a unified parameter format, eliminating the need for the inspection logic to be aware of the structural differences between different asset types. Based on this, corresponding rule detectors are dynamically created according to the quality inspection rules, and standardized parameters are input for execution, forming a complete chain of "rule configuration—detector creation—parameter input—detection execution." This design allows for the reuse of existing inspection execution processes when adding new asset types. It only requires converting the new asset model information into standardized asset inspection parameters and configuring the corresponding quality inspection rules at the business layer, without modifying the underlying logic of the inspection engine, thereby reducing the risk and cost of system expansion.

[0101] Furthermore, the configuration of detection rules can be flexibly adjusted at different levels, such as multi-tenancy and multi-dataset, balancing the stringency, execution efficiency, and resource consumption of detection according to actual needs. In summary, this solution effectively improves the scalability, flexibility, and execution efficiency of data quality detection, reduces system modifications caused by changes in asset type or rule adjustments, and ensures the quality stability of assets during the preparation and use phases.

[0102] based on Figure 2 In addition to the method described herein, this specification also provides some improved implementation methods, which will be described below.

[0103] In one or more embodiments of this specification, the determination of quality detection rules can also be combined with a multi-level configuration system. The step of determining at least one quality detection rule corresponding to the data asset based on the asset type identifier specifically includes: obtaining the quality detection rule corresponding to the data asset from a preset plurality of configuration levels; the plurality of configuration levels includes at least two of the following: a global configuration level, a tenant configuration level, a dataset configuration level, a physical table configuration level, and an asset configuration level with hierarchical relationships.

[0104] The global configuration level contains the system's default detection dimensions, items, and rules, applicable to all tenants and scenarios. The tenant configuration level overrides or extends the global configuration, customizing rules for specific tenants, with configurations isolated between tenants. In a multi-tenant architecture, a tenant refers to an independent business entity sharing the same system but with isolated data; a tenant can correspond to an enterprise, a department, or an independent business unit. The dataset configuration level, at the tenant level, allows for customized configurations of different datasets, with datasets isolated from each other. The physical table configuration level, at the dataset level, allows setting independent rules for specific physical tables. The asset configuration level, at the dataset level, allows setting independent rules for specific assets.

[0105] For example, a data platform serves multiple enterprises. Enterprise A, as an independent tenant, can be configured with quality inspection rules applicable to enterprise A; Enterprise B, as another independent tenant, can be configured with different quality inspection rules than enterprise A. Tenant-level configuration isolation ensures that the inspection rules of each enterprise do not affect each other. It should be noted that a tenant is different from a user. A user refers to an individual operator using the system, and a tenant can contain multiple users. The configuration of quality inspection rules is isolated on a tenant-by-tenant basis.

[0106] During configuration loading, the system can start from the asset configuration level. If a certain detection item is not configured at that level, it recursively searches upwards (dataset configuration level, tenant configuration level, global configuration level) until the corresponding configuration is found. If multiple levels have configurations for the same detection item, they are merged according to the rule that lower levels take precedence over higher levels. Configurations at each level can be stored independently and support dynamic updates. Configurations are merged according to priority during loading to ensure flexibility in the merging results. Furthermore, detection rules support dynamic adjustment and can be updated in real time based on business changes or data characteristics.

[0107] Based on some embodiments in this specification, by setting up a multi-level configuration system with hierarchical relationships, the system can flexibly support the configuration of detection rules at different granularities, such as global, tenant, dataset, physical table, and asset levels. The configurations at different levels are isolated from each other, making the impact of rule adjustments controllable. For example, adjusting tenant-level rules will not affect other tenants, and adjusting asset-level rules will not affect other assets under the same dataset. This design is beneficial for meeting the personalized detection needs of different business units in a multi-tenant environment, while maintaining the overall manageability of the system configuration.

[0108] In one or more embodiments of this specification, obtaining the quality detection rules corresponding to the data asset from a plurality of preset configuration levels includes: in the plurality of configuration levels, the lower configuration level inherits the quality detection rules of the upper configuration level.

[0109] Referring to the description of the multi-level configuration system above, the configuration levels, from top to bottom, include the global configuration level, tenant configuration level, dataset configuration level, physical table configuration level, and asset configuration level. Each level can inherit the configuration of the level above it and allows overriding or adding configuration items.

[0110] There is a hierarchical relationship between detection dimensions, detection items, and detection rules. Detection dimensions are macro-level classifications of data quality detection, such as data size, data quality, and data value. Detection items are specific detection targets within a particular detection dimension; for example, the data size dimension includes column count detection items and data volume detection items, while the data quality dimension includes table availability detection items, data integrity detection items, and data uniqueness detection items. Detection rules are the concrete content of a detection item instantiated, including the corresponding threshold parameters, rule description templates, and remediation suggestion templates. Within the same configuration level, one detection item corresponds to one detection rule.

[0111] For example, for event-type assets, the set of detection items may include column count detection, table availability detection, field type detection, and data volume detection; for tag-type assets, the set of detection items may also include integrity detection and uniqueness detection; for population-type assets, the set of detection items may be similar to that of tag-type assets, and may also include integrity detection (e.g., checking if the entity's primary key field has null values) and uniqueness detection (e.g., checking if the entity's primary key field has duplicate values) to ensure the integrity and uniqueness of population data. Each detection item can be configured with different threshold parameters at different configuration levels. Lower configuration levels can inherit the thresholds of upper configuration levels and allow overriding as needed.

[0112] The inheritance and overriding mechanism of configurations at each level allows the system to recursively search from lower to higher levels when loading configurations, merging configurations from all levels and forming the final detection rule set according to the rule that lower levels take precedence over higher levels. Since each detection is independent, the system can execute the detection operations of each detection item in parallel after the configuration is merged, thereby improving the overall detection efficiency.

[0113] Based on some embodiments in this specification, lower-level configuration layers inherit the quality detection rules from upper-level configuration layers, ensuring configuration continuity and eliminating the need to repeatedly configure the same detection items and thresholds at each level. Furthermore, lower layers can override upper-level configurations, providing flexible customization options for specific tenants, datasets, physical tables, or assets. This mechanism combining inheritance and overriding helps meet the personalized detection needs of different business scenarios while maintaining the consistency of default system rules, reducing repetitive configuration workload.

[0114] In one or more embodiments of this specification, under the same configuration level, one detection item corresponds to one detection rule; the step of obtaining the quality detection rule corresponding to the data asset from multiple preset configuration levels includes: when there are multiple quality detection rules for the same detection item in the multiple configuration levels, the quality detection rule of the lower level is determined as the quality detection rule corresponding to the detection item.

[0115] In practical applications, the same detection item (such as column count detection) may have different threshold parameters configured at different configuration levels. For example, the global configuration level might set the maximum number of columns to 100, the tenant configuration level to 120, and the asset configuration level to 150. When a detection item exists in multiple configuration levels, the system merges them according to the principle of prioritizing rules relative to lower levels: rules at the asset configuration level take precedence over those at the physical table configuration level, which in turn takes precedence over those at the dataset configuration level, which takes precedence over those at the tenant configuration level, and finally, those at the global configuration level. If a configuration item is not defined at a lower level, it inherits the configuration from the upper level.

[0116] Taking column count detection for event-type assets as an example, assuming the global configuration level sets a maximum of 100 columns, the tenant configuration level has no column count detection settings, the dataset configuration level sets a maximum of 120 columns, and the asset configuration level has no settings, the system searches upwards from the asset configuration level. If the asset configuration level has no settings, it inherits the 120 columns from the dataset configuration level. If the tenant configuration level has no settings, it continues to inherit the 100 columns from the global configuration level. However, since the 120 columns from the dataset configuration level already cover the global configuration, the final detection rule is that the number of columns should not exceed 120.

[0117] The above configuration merging process reflects the priority rule of "lower layers taking precedence over higher layers" in a multi-level configuration system. This rule, together with the configuration inheritance relationship, constitutes a dynamic assembly mechanism for detection rules, enabling the system to flexibly adjust detection strategies according to actual business needs, while maintaining the universality, scalability, and customization capabilities of the configuration. It can balance the rigor, efficiency, and resource consumption of asset quality detection as needed.

[0118] Based on some embodiments in this specification, by prioritizing rules at lower levels, the system can automatically handle conflicts between multi-level configurations, ensuring that the configuration rules closest to the asset itself take effect. This design allows users to precisely customize rules for specific assets without affecting other levels of configuration. Furthermore, the mechanism of automatically inheriting upper-level configurations when lower levels are undefined ensures the integrity of detection rules, eliminating the need to repeatedly configure all detection items at each level. The combination of multi-level configurations and lower-level priority rules facilitates refined management of detection rules, adapting to varying detection needs at different granularities (from global to individual asset).

[0119] In the process of data asset management, when preparing data assets, users typically entrust downstream processing systems to determine whether the size of the source table data, the table structure type, and whether the asset can be processed normally are met during the execution phase. Downstream processing systems often only verify the source table during the actual execution of the processing task. Users only learn of problems such as excessive source table data volume or incompatible table structure after the processing task fails due to timeouts or other reasons. By this time, a considerable amount of time may have passed between the user creating the asset and receiving the processing result.

[0120] Due to the lack of a pre-assessment mechanism, users lack reference quality standards when preparing assets, making it impossible to determine in advance whether the configured assets can be used normally. The time required for asset preparation is uncertain, and the stability of asset use is difficult to guarantee. If an asset fails due to quality issues after entering the downstream processing stage, users not only have to wait a long time to discover that the asset cannot be used normally, increasing time costs, but may also incur additional computing and storage resource consumption due to the materialization process.

[0121] Based on the above, some embodiments of this specification provide a data quality detection method that can trigger quality detection in real time during the asset creation or editing stage, so that users can obtain clear quality assessment results during the asset preparation process, thereby improving the efficiency of the asset preparation process and the certainty of asset availability.

[0122] Specifically, obtaining the asset type identifier and source data information of the data asset to be detected includes: in response to an asset creation request or an asset editing request, obtaining the asset type identifier and source data information of the data asset.

[0123] When a user creates a new asset or edits an existing asset in the data asset management platform, the system receives a corresponding request, which serves as a condition for triggering quality checks. For example, after a user fills in the relevant information for an event asset (asset name, source table identifier, filter conditions, partition fields, etc.) and clicks the submit control, the system responds to the asset creation request, obtaining the asset type identifier (such as "event") and source data information (such as source table identifier, filter conditions, partition fields, etc.). By triggering quality checks in real time during the asset creation or editing stage, quality assessment can be completed before the asset is put into use.

[0124] Referring to the description of step 202 above, the asset type identifier is used to identify the business type of the data asset, and the source data information is used to locate and limit the source data corresponding to the asset. This information is obtained upon receiving a create or edit request, providing input for subsequent testing processes.

[0125] Based on some embodiments of this specification, by triggering quality checks during the asset creation or editing stage, users can know the asset quality status without waiting for the asset to enter the subsequent processing stage. This is beneficial for discovering potential problems in the early stages of asset preparation and reducing ineffective waiting and resource consumption caused by the lack of preliminary assessment.

[0126] In one or more embodiments of this specification, after obtaining the quality inspection result of the data asset, the method further includes: sending the quality inspection result to the requesting party; the quality inspection result includes whether the quality inspection passed or failed.

[0127] The requester refers to the party issuing the asset creation or editing request, such as a user terminal. The quality inspection result can include information on whether each inspection item passed, as well as a judgment on whether the overall inspection passed. After the inspection is completed, the system returns the quality inspection result to the requester, for example, through interface display or message push. When the inspection result is unsuccessful, the quality inspection result can also include repair suggestions. Repair suggestions can be generated based on the repair suggestion template in the quality inspection rules. For example, when the number of columns exceeds a threshold, a prompt can be made to "reduce unnecessary data volume through event row and column pruning capabilities."

[0128] In some embodiments, the quality inspection results can serve as the basis for determining whether to release a data asset. If the quality inspection result indicates that the quality inspection has passed, the data asset is released so that it can be used by downstream systems; if the quality inspection result indicates that the quality inspection has failed, the data asset is not released, and the user can adjust and resubmit for inspection based on the repair suggestions.

[0129] Based on some embodiments of this specification, by returning quality inspection results containing pass / fail status and repair suggestions to the requesting party, users can promptly understand the asset quality status and obtain clear directions for improvement. Using the inspection results as the basis for publication helps control the quality of newly added assets from the source, preventing non-compliant assets from entering subsequent processing stages.

[0130] In one or more embodiments of this specification, the step of executing each of the rule detectors to obtain the quality inspection results of the data assets specifically includes: executing the detection operation of each of the rule detectors; obtaining the execution results of each of the rule detectors; and generating the quality inspection results of the data assets based on the execution results.

[0131] Referring to the description of step 210 above, each rule detector receives standardized asset detection parameters as input and performs specific verification operations according to its built-in detection logic. For example, the column count detector performs column count statistics on the source table, and the data volume detector performs row count statistics queries. Each rule detector outputs a corresponding execution result, including whether the detection item passed, the specific detection value, descriptive information, etc. The system collects the execution results of each rule detector, integrates and processes them, and generates a quality detection result that includes the results of each detection item and the overall detection status.

[0132] In practical applications, the aforementioned detection execution process for obtaining the quality detection results of the data assets by executing each rule detector can be independent of the triggering time, including scenarios such as pre-detection and in-process inspection.

[0133] Based on some embodiments in this specification, by decomposing the detection operation into three stages—execution, result acquisition, and result generation—it is beneficial to achieve modularization of the detection process. Each rule detector executes independently and outputs results. The system then generates a unified quality inspection result based on these results, making the detection process clear and controllable, and facilitating subsequent traceability and analysis of the inspection results.

[0134] In one or more embodiments of this specification, obtaining the execution results of each rule detector specifically includes: obtaining the execution results of each rule detector when the execution time of the detection operation does not exceed a preset time threshold; generating a detection record for the data asset when the execution time of the detection operation exceeds the preset time threshold, the detection record including an asynchronous execution identifier and first status information, the first status information indicating that detection is in progress; and obtaining the execution results of each rule detector based on the asynchronous execution identifier in the detection record.

[0135] During the detection operation, the system can monitor the execution time. When the execution time does not exceed a preset time threshold (e.g., 10 seconds), the detection operation is completed within an acceptable time, and the system directly obtains the execution results of each rule detector. When the execution time exceeds the preset time threshold, the system switches the current detection operation to asynchronous execution mode and generates a detection record for the data asset. The detection record contains an asynchronous execution identifier and first status information, whereby the first status information indicates that the detection is in progress. The asynchronous execution identifier is an identifier used to uniquely identify an asynchronous detection task, and the system stores this identifier in association with the current detection task.

[0136] In one specific embodiment, when executing the rule detector, the system first performs a detection operation (such as executing SQL to count the number of data rows in the source table) and records the execution time. When the execution time does not exceed a preset time threshold, the system directly obtains the detection results, assembles a detection report, and returns the quality detection results to the requester. When the execution time exceeds the preset time threshold, the system switches the current detection operation to asynchronous execution mode, generates a unique asynchronous execution identifier, returns the asynchronous execution identifier to the requester, and records the detection status as "detecting". The background detection task continues to execute, unaffected by the requester's waiting. The timed scheduling platform periodically starts asynchronous result acquisition tasks, acquires all asynchronous queries with a status of "detecting", and polls to check the execution status of each asynchronous query. When it is detected that an asynchronous query has been completed, the detection result is acquired, recorded, and the status is updated to "detection completed". Optionally, the system pushes the detection results to the requester through a callback method (such as HTTP callback, message queue, or instant messaging message). If the execution result of the data asset is obtained, the first status information in the detection record is updated to the second status information, which indicates that the detection is complete.

[0137] In some embodiments, when the amount of data in the source table corresponding to a data asset exceeds a preset threshold, the rule detector can also be executed asynchronously. After obtaining the source data information of the data asset to be detected, the system obtains an estimate of the amount of data in the source table corresponding to that asset. When the amount of data is less than or equal to the preset threshold, a synchronous detection mode is used, performing the detection in real time and returning the results; when the amount of data is greater than the preset threshold, an asynchronous detection mode is used, the system returns a prompt message to the user, the rule detector is executed asynchronously in the background, and the detection results are pushed to the user through a preset callback interface after the detection is completed.

[0138] Based on some embodiments in this specification, by monitoring the execution time of detection operations and dynamically selecting synchronous or asynchronous execution modes, results can be returned quickly in a short time, avoiding long waiting times for the requester when detection operations are time-consuming. The asynchronous execution identifier serves as the unique credential for the detection task, decoupling detection initiation from result acquisition, allowing background detection to continue independently of the requester. This dynamic strategy helps improve user experience while ensuring detection efficiency and adapts to varying detection needs across different data scales.

[0139] Figure 4 This is a schematic diagram of a data asset quality inspection process provided in an embodiment of this specification.

[0140] like Figure 4 As shown, in some embodiments, a data quality inspection process is triggered in response to a user creating or editing a data asset (e.g., creating or editing an event-type asset). Taking event-type data assets as an example, the data quality inspection process may include several optional inspection steps such as data source access inspection, data format verification, data quality inspection, and data size inspection.

[0141] In the data source access detection stage, the data source legality can be verified first, checking whether the tenant of the source table associated with the asset to be detected is consistent with the current operating tenant; then the data source legality can be verified, checking whether the storage engine of the source table conforms to the preset rules; after completing the above verification, the verification results of this stage are assembled.

[0142] In the data format verification stage, data format detection can be performed first to verify whether the data types of each column in the source table conform to preset rules; then data standard detection can be performed to verify whether the units of special fields conform to preset rules; after completing the above detection, the verification results of this stage can be assembled.

[0143] In the data quality inspection stage, a structured query statement (SQL) can be assembled first to check whether there are null values ​​in the entity fields of the data to be processed, and the returned query results can be received to perform data integrity verification. Then, SQL can be assembled to check whether there are outliers in special fields of the data to be processed, and the returned query results can be received to perform data accuracy verification. After completing the above verification, the verification results of this stage can be assembled.

[0144] In the data scale detection phase, the first step is to verify the number of columns in the source table to see if it exceeds a preset limit. Then, branching is performed based on the source table type and whether filters are configured. If the source table is a physical table and no filters are configured, the data volume of the specified partition is directly obtained, and the returned query results are received. If the source table is a view or a physical table with configured filters, SQL is assembled to query the data volume of the target partition, and the returned query results are received. Based on the obtained data volume, the row data volume is verified to meet preset rules. After completing the above verifications, the verification results for this phase are assembled.

[0145] After each of the above-mentioned inspection steps is completed, if the quality inspection passes, the verification results of each step are combined to generate an asset quality verification result. The system records this asset quality inspection result and returns it to the user.

[0146] Based on some embodiments of this specification, the above-described quality inspection process decomposes asset quality assessment into multiple independent stages. The results of each stage are assembled and verified separately, then combined uniformly. This helps to pinpoint specific problem types during the inspection process, facilitating targeted repairs by users based on feedback. The assembly of verification results from each stage and the recording of the final result provide a structured data foundation for subsequent asset quality inspections, anomaly tracing, and quality trend analysis.

[0147] Furthermore, in the actual use of data assets, even if the asset passes quality testing at creation, its underlying source data may still change subsequently. For example, structural changes to the source table (such as adding or deleting fields, or modifying field types), or a sudden increase in the amount of data in the source table, may cause the asset to fail to meet the original quality requirements, resulting in a situation where the asset suddenly becomes unusable and affecting the stability of downstream businesses. In addition, the resources occupied by the asset may also fluctuate abnormally due to changes in the amount of data, further exacerbating the uncertainty of asset availability.

[0148] In light of the above, relying solely on manual inspection or the stability of the source data itself to ensure asset quality is not only costly but also prone to overlooking problems. Therefore, an automated mechanism is needed to periodically review the quality of deployed assets, promptly identify anomalies caused by changes in underlying data, and take appropriate measures to notify relevant personnel or handle abnormal assets, ensuring the continuous stability of assets during use. Based on this, some embodiments in this specification provide periodic inspection schemes for published data assets, along with corresponding result recording and tiered handling mechanisms.

[0149] In one or more embodiments of this specification, multiple listed assets can be periodically inspected. The acquisition of the asset type identifier and source data information of the data asset to be inspected specifically includes: acquiring the asset type identifier and source data information of at least one published data asset according to a preset period.

[0150] Preset cycles can include fixed cycles, such as daily, hourly, or weekly, or cycles with non-fixed time intervals but executed according to a plan. Published data assets (also known as data assets already on the shelf) refer to data assets that have completed and passed pre-release quality testing and are ready for normal use by downstream business systems; in contrast, unpublished data assets refer to assets that are under creation, editing, or have not passed quality testing. By periodically inspecting published data assets, their quality status can be continuously monitored, ensuring the stability of the assets during use.

[0151] Based on some embodiments of this specification, by acquiring information on published data assets in batches according to a preset cycle, it is possible to periodically review the quality of existing assets, promptly identify potential problems caused by changes in underlying data, and provide a data foundation for subsequent anomaly handling.

[0152] In one or more embodiments of this specification, the acquisition of the asset type identifier and source data information of the data asset to be detected specifically includes: dividing the at least one published data asset according to the tenant granularity to obtain multiple tenant groups; and performing the asset acquisition operation corresponding to each tenant group in parallel.

[0153] A tenant is the basic unit of business isolation; assets of different tenants do not interfere with each other, making them suitable for parallel processing. During parallel execution, a first concurrency level can be used to control the number of groups executing simultaneously. The first concurrency level can be less than or equal to the number of groups. By configuring the concurrency parameter, the degree of parallelism can be dynamically adjusted according to system resource conditions, balancing detection efficiency and resource consumption. For example, if there are 10 tenants in the system, the first concurrency level can be set to 10, simultaneously starting 10 threads to process the assets within each tenant. If resources are limited, the concurrency level can also be set to 5, processing in two batches.

[0154] Based on some embodiments of this specification, by using the parallel partitioning of tenant granularity and the adjustable concurrency mechanism, system resources can be fully utilized during the inspection process, shortening the time for obtaining batch asset information, while avoiding resource overload due to excessive concurrency.

[0155] In one or more embodiments of this specification, the parallel execution of asset acquisition operations corresponding to each tenant group specifically includes: for each tenant group, dividing the published data assets within each tenant group according to the dataset granularity to obtain multiple dataset subgroups; and executing the asset acquisition operations corresponding to each dataset subgroup in parallel.

[0156] A dataset serves as an aggregated container for assets. Assets from different datasets within the same tenant do not interfere with each other, making them suitable for further parallel processing. Within the processing threads of a tenant group, asset retrieval operations for each dataset subgroup can be executed in parallel according to a second concurrency level, which can be less than or equal to the number of dataset subgroups. By configuring a second concurrency level, further concurrency control can be implemented within the tenant, further improving retrieval efficiency.

[0157] Based on some embodiments of this specification, parallel partitioning at the dataset granularity is introduced within tenant groups, realizing multi-level concurrent scheduling. This enables asset acquisition operations to proceed step by step from the tenant layer to the dataset layer, effectively utilizing the inherent business isolation attributes of assets and improving the overall throughput of inspections.

[0158] In one or more embodiments of this specification, the parallel execution of asset acquisition operations corresponding to each of the dataset subgroups specifically includes: dividing the published data assets within each dataset subgroup to obtain multiple asset groups; and performing quality detection operations corresponding to each asset group in parallel.

[0159] In one specific embodiment, the system adopts a three-tiered scheduling strategy. The first tier (tenant tier): Published data assets to be inspected are grouped by tenant, and each tenant group is processed in parallel. Assuming there are N tenants in the system, N tenant groups are formed. The system executes asset acquisition operations for each tenant group in parallel, i.e., multiple threads are started simultaneously, each thread responsible for acquiring all assets within a tenant. The second tier (dataset tier): Within each tenant group, assets are further grouped by their respective datasets, and each dataset subgroup is processed in parallel. Assuming a tenant has M datasets, M dataset subgroups are formed. The system executes asset acquisition operations for each dataset subgroup in parallel within its thread, i.e., multiple sub-threads are started simultaneously, each sub-thread responsible for acquiring all assets within a dataset. The third tier (asset tier): Within each dataset subgroup, assets are further grouped, and quality inspection operations for each asset group are executed in parallel. This tier can directly execute the inspection logic in parallel to further improve efficiency; alternatively, it can choose to execute sequentially based on resource availability to control resource consumption. Through this three-tier partitioning and hierarchical concurrent scheduling, tens of thousands of asset inspection tasks that originally required sequential execution can be completed in a shorter time. In addition, the system supports configuring concurrency parameters for each tier (such as first concurrency, second concurrency), so as to dynamically adjust the degree of parallelism according to the system resource status and balance detection efficiency and resource consumption.

[0160] During the inspection process, scheduling rules can be configured through the scheduling platform to achieve timed triggering and scheduling management of inspection tasks. In terms of execution efficiency, concurrent scheduling is achieved through layered scheduling to ensure the efficiency of batch asset quality inspection; at the same time, the concurrency parameter of each layer controls the concurrency of each layer, dynamically adjusting the concurrency to balance inspection efficiency and system resource consumption.

[0161] Based on some embodiments of this specification, the inspection task is decomposed into sub-tasks that can be executed independently and in parallel through a three-level hierarchical concurrent scheduling of tenants, datasets, and assets. This fully utilizes system resources while maintaining the flexibility of concurrency adjustment, thereby improving inspection efficiency and avoiding excessive resource consumption, and providing feasible technical support for large-scale asset inspection.

[0162] Figure 5 This is a schematic diagram of a hierarchical scheduling scheme for data asset quality inspection provided in the embodiments of this specification.

[0163] like Figure 5 As shown, in some embodiments, asset quality inspection adopts a hierarchical scheduling strategy, specifically including three levels.

[0164] The first layer is tenant-level segmentation. The system acquires all published data assets to be inspected and divides them according to the tenant to which the asset belongs, resulting in multiple tenant groups. A tenant is the basic unit of business isolation; assets from different tenants do not interfere with each other. Assuming there are M tenants in the system, M tenant groups can be formed. The system executes the asset acquisition operations corresponding to each tenant group in parallel, that is, it starts multiple threads simultaneously, with each thread responsible for acquiring all assets within a single tenant.

[0165] The second layer involves granular dataset segmentation. For each tenant group, the group is further divided according to the dataset to which the assets belong, resulting in multiple dataset subgroups. A dataset is an aggregate container for assets; assets from different datasets within the same tenant do not interfere with each other. Assuming a tenant has N datasets, this results in N dataset subgroups. Within the processing thread of each tenant group, the system executes the asset acquisition operations corresponding to each dataset subgroup in parallel; that is, multiple sub-threads are started simultaneously, each responsible for acquiring all assets within a single dataset.

[0166] The third layer involves granular asset processing. For each dataset subgroup, assets are divided into multiple asset groups, each containing several assets. The system can execute quality inspection operations for each asset group in parallel. That is, within the processing sub-thread of each dataset subgroup, multiple threads are started simultaneously, each responsible for the quality inspection operations of each asset within one asset group. The number of asset groups, X, can be determined based on the actual number of assets and preset concurrency parameters.

[0167] Through the above three-level hierarchical scheduling, the inspection task is broken down into sub-tasks that can be executed independently and in parallel, from the tenant layer to the dataset layer and then to the asset layer. The parallelism between each layer is controlled by the concurrency parameter, which can be dynamically adjusted according to the system resource status, thereby improving inspection efficiency while balancing resource consumption.

[0168] In one or more embodiments of this specification, after obtaining the quality inspection result of the data asset, the method further includes: recording the current quality inspection result of the data asset in the current inspection period.

[0169] In some embodiments, after obtaining the quality inspection results of the data assets, the method further includes: recording the current quality inspection results of the data assets in the current inspection cycle. The purpose of recording the inspection results includes recording daily execution results for periodically generating asset quality analysis reports, assessing overall asset quality trends and improvement effects, and providing data support for anomaly handling tracking. The inspection result record should include information such as the asset's unique identifier, the inspection date (e.g., daily granularity), whether the inspection passed, and details of the inspection result. In practical applications, the asset's unique identifier combined with the inspection date can be used as a uniqueness constraint to ensure that only one latest inspection result is retained for each asset per day. When multiple inspections (such as manual inspections and patrol inspections) occur on the same day, only the latest result is retained.

[0170] The test result record can meet the following conditions: it can be specific to a certain asset and a certain test, that is, record the asset's unique identifier and the test date (day granularity); it can identify the asset test result, that is, record whether the test passed and the test result details (if the test fails, record the result of each item in detail to guide the user in handling it); for abnormal test results, it can contact the person in charge and record the abnormal handling result, that is, record the asset owner's information and whether a reminder notification has been sent.

[0171] In practical applications, a unique asset identifier plus the inspection date can be used as a unique constraint to ensure that only the latest inspection result is retained for each asset per day. When multiple inspections (such as manual inspections and routine inspections) occur on the same day, only the latest result is retained.

[0172] Based on some embodiments of this specification, by recording the quality inspection results of assets in each inspection cycle, a historical data sequence of asset quality can be established, providing a reliable basis for subsequent continuous anomaly judgment, trend analysis and graded disposal, enabling asset management to move from single inspection to continuous monitoring.

[0173] In one or more embodiments of this specification, after obtaining the quality inspection result of the data asset, the method further includes: obtaining the historical quality inspection result of the data asset in at least one historical inspection period; determining the number of consecutive inspection failures of the data asset based on the current quality inspection result of the data asset in the current inspection period and the historical quality inspection result; and sending an abnormality reminder notification to the notification recipient associated with the data asset if the number of consecutive inspection failures is less than or equal to a preset threshold.

[0174] The number of consecutive failed tests refers to the cumulative number of times the asset fails tests across multiple consecutive testing cycles. For example, based on the current testing cycle, several cycles can be traced back to count the number of consecutive failed tests since the most recent successful test. A preset threshold can be set according to actual business needs, such as 3 times. When the number of consecutive failed tests does not exceed the threshold, the system only sends a notification, guiding the user to view the testing details and anomaly handling suggestions. The notification recipient can be the designated communication account of the person in charge or administrator associated with the data asset.

[0175] Based on some embodiments of this specification, by statistically analyzing and classifying the number of consecutive failed tests, timely warnings can be issued to relevant parties when asset quality shows continuous abnormalities, preventing problems from accumulating to the point of seriously affecting business operations. At the same time, non-urgent reminders reduce interference to users.

[0176] In one or more embodiments of this specification, the data quality detection method further includes: if the number of consecutive failed detections exceeds the preset threshold, setting the data asset to an offline state.

[0177] In some embodiments, if the number of consecutive failed tests exceeds the preset threshold, the data asset is set to an offline state. When an asset fails quality testing for multiple consecutive testing cycles (e.g., 4 consecutive days), it indicates that the underlying data of the asset has a persistent problem, and continuing to keep it in an available state may have an adverse impact on downstream business. At this time, the system automatically sets the asset's status to offline, blocks downstream access to it, and sends a forced offline notification to relevant parties. After being taken offline, the user needs to repair the asset according to the testing report, and it can only be brought back online after being resubmitted for testing and passing the test.

[0178] For example, using the current date as the base date, the test results of the asset for the most recent 4 days are retrieved. If all tests fail, the asset is taken offline and the processing method is recorded on the latest test results; otherwise, the asset quality test anomaly alert process is executed and the processing method is recorded on the latest test results.

[0179] Based on some embodiments of this specification, by setting an upper limit threshold for the number of consecutive failures and automatically taking the asset offline when the threshold is exceeded, it is possible to effectively prevent substandard assets from affecting downstream business in the long term, forming a closed-loop management from detection and early warning to disposal, and improving the overall availability and stability of data assets.

[0180] Figure 6 This is a schematic diagram of a data asset quality inspection process provided in an embodiment of this specification.

[0181] like Figure 6As shown, in some embodiments, the asset quality inspection process is triggered by an asset quality detection task, which is executed according to a preset cycle. The process first obtains a tenant list and processes each tenant in the list sequentially. For the currently processed tenant, it obtains a list of datasets that are in a valid state under that tenant, and then, for each dataset, it obtains a list of data assets (such as event assets) created by that dataset, and then performs asset quality detection on each data asset.

[0182] Taking event-based data assets as an example, the data quality inspection process can include several optional inspection steps such as data source access inspection, data format verification, data quality inspection, and data scale inspection.

[0183] In the data source access detection stage, the data source legality can be verified first, checking whether the tenant of the source table associated with the asset to be detected is consistent with the current operating tenant; then the data source legality can be verified, checking whether the storage engine of the source table conforms to the preset rules; after completing the above verification, the verification results of this stage are assembled.

[0184] In the data format verification stage, data format detection can be performed first to verify whether the data types of each column in the source table conform to preset rules; then data standard detection can be performed to verify whether the units of special fields conform to preset rules; after completing the above detection, the verification results of this stage can be assembled.

[0185] In the data quality inspection stage, a structured query statement (SQL) can be assembled first to check whether there are null values ​​in the entity fields of the data to be processed, and the returned query results can be received to perform data integrity verification. Then, SQL can be assembled to check whether there are outliers in special fields of the data to be processed, and the returned query results can be received to perform data accuracy verification. After completing the above verification, the verification results of this stage can be assembled.

[0186] In the data scale detection phase, the first step is to verify the number of columns in the source table to see if it exceeds a preset limit. Then, branching is performed based on the source table type and whether filters are configured. If the source table is a physical table and no filters are configured, the data volume of the specified partition is directly obtained, and the returned query results are received. If the source table is a view or a physical table with configured filters, SQL is assembled to query the data volume of the target partition, and the returned query results are received. Based on the obtained data volume, the row data volume is verified to meet preset rules. After completing the above verifications, the verification results for this phase are assembled.

[0187] After each of the above-mentioned checks is completed, the system combines the verification results of each step to form an asset quality inspection result, and records the result. It can also send a notification message to the user if the inspection fails.

[0188] Subsequently, if the inspection fails, the system can retrieve the asset's historical inspection results over the most recent inspection periods (e.g., the last 4 days) and determine the consecutive inspection failures based on the current and historical results. If the number of consecutive inspection failures meets a preset threshold (e.g., more than 3 times), the asset's status is changed to draft or it is taken offline, and the processing method is recorded as offline in the asset quality inspection results, with a notification sent to relevant parties. If the number of consecutive inspection failures does not exceed the preset threshold, a notification is sent to relevant parties, and the processing method is recorded as notification in the asset quality inspection results. After completing the above processing, the quality inspection process for the data asset ends, and the asset quality inspection task continues to process the next data asset, the next data table, or the next tenant until all assets to be inspected have been processed.

[0189] In some embodiments, the aforementioned asset quality inspection process, executed automatically and periodically, continuously monitors the quality status of published data assets, promptly identifying quality issues caused by changes in underlying data (such as table structure alterations or data volume increases). The process checks multiple dimensions, including data source availability, data integrity, data accuracy, and data scale, and handles issues tiered based on the number of consecutive failed checks (e.g., issuing alerts or automatic removal), forming a closed-loop mechanism of detection, recording, judgment, and handling. By recording historical inspection results and making decisions based on the number of consecutive failures, the impact of occasional anomalies on asset status can be reduced, while ensuring that persistent problems are addressed promptly. This process helps improve the quality stability and overall availability of existing assets, reduces manual inspection costs, and provides a data foundation for asset quality trend analysis.

[0190] In some embodiments, the data quality detection method further includes: based on the quality detection results, calculating the total number of detections and the number of successful detections for at least one asset type within a preset time period; and calculating the pass rate for the at least one asset type based on the total number of detections and the number of successful detections. The preset time period can be set by day, week, or month. The total number of detections includes the number of all pre-detections for that asset type within that time period (e.g., detections triggered in response to asset creation or editing requests), and the number of successful detections is the number of detections that resulted in a successful result. The pass rate (number of successful detections / total number of detections) reflects the effectiveness of pre-detection in intercepting abnormal assets. For different asset types (such as events, tags, and audiences), their respective pass rates can be calculated separately to more accurately assess the quality access status of different asset types.

[0191] In some embodiments, the data quality detection method further includes: based on the quality detection results, counting the number of assets that failed detection and the total number of assets that passed detection for at least one asset type within multiple statistical periods; and determining the trend of the abnormal asset ratio of the at least one asset type based on the number of assets that failed detection and the total number of assets that passed detection within the multiple statistical periods. The statistical period can be in days. Within each period, the number of assets that failed detection is the number of assets whose quality detection results were not passed within that period, and the total number of assets that passed detection is the total number of assets in a valid state within that period (e.g., the total number of published assets). By calculating the daily abnormal asset ratio (number of failed assets / total number of valid assets), the trend of this ratio over time can be observed, thereby assessing the overall asset quality changes and the effectiveness of anomaly handling measures in improving asset quality.

[0192] In some embodiments, the data quality detection method further includes: acquiring the number of assets that failed detection and the number of assets that were repaired within a preset time period for at least one asset type; and calculating the abnormal asset repair ratio for the at least one asset type based on the number of assets that failed detection and the number of assets that were repaired. The preset time period can be set by day, week, or month. The number of assets that failed detection is the number of assets marked as failing for the first time within that time period, and the number of assets that were repaired is the number of assets that were repaired and passed re-detection within that time period. The repair ratio (number of repaired assets / number of assets that failed detection) reflects the efficiency of repairing abnormal assets and can serve as an important indicator for measuring the effectiveness of optimization suggestions and the degree of asset quality improvement. For different asset types, their respective repair ratios can be calculated separately to improve the governance strategies for different asset types in a targeted manner.

[0193] Based on some embodiments in this specification, by constructing two stages—pre-entry inspection and in-process inspection—the quality of newly added assets and the quality of existing assets can be effectively controlled and improved. Pre-entry inspection provides clear quality assessment standards during the asset creation phase, enabling users to quickly complete data preparation; in-process inspection continuously monitors existing assets through layered concurrent scheduling, and promptly handles abnormal assets in conjunction with a tiered disposal mechanism, thereby improving asset availability and stability and enhancing the user experience during the data preparation phase.

[0194] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0195] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0196] Figure 7 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a data quality testing device.

[0197] like Figure 7 As shown, the device may include: The information acquisition module 702 is used to acquire the asset type identifier and source data information of the data asset to be detected; the source data information is used to describe the data source of the data asset. Parameter conversion module 704 is used to determine asset detection parameters based on the source data information; The detection rule determination module 706 is used to determine at least one quality detection rule corresponding to the data asset based on the asset type identifier; The rule detector creation module 708 is used to create rule detectors corresponding to each of the aforementioned quality detection rules. The quality inspection module 710 is used to take the asset inspection parameters as input, execute each of the rule detectors, and obtain the quality inspection results of the data assets.

[0198] based on Figure 7 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0199] Optionally, the asset detection parameters include at least one of the following: source table identifier, data filtering conditions, partition field identifier, number of partitions, source table field list, asset dataset identifier, and asset tenant identifier.

[0200] Optionally, determining at least one quality inspection rule corresponding to the data asset based on the asset type identifier specifically includes: Based on the asset type identifier, a quality inspection rule set corresponding to the asset type identifier is read from a preset rule base, and the quality inspection rule set contains at least one quality inspection rule.

[0201] Optionally, the quality inspection rule includes at least one of the following: rule type identifier, inspection dimension identifier, threshold parameter, rule description template, and repair suggestion template.

[0202] Optionally, the creation of rule detectors corresponding to each of the quality detection rules includes: Obtain the rule type identifier for each of the aforementioned quality inspection rules; Based on the rule type identifier, a rule detector instance corresponding to the rule type identifier is created by the rule engine.

[0203] Optionally, determining at least one quality inspection rule corresponding to the data asset based on the asset type identifier specifically includes: The quality detection rules corresponding to the data assets are obtained from multiple preset configuration levels; the multiple configuration levels include at least two of the following: global configuration level, tenant configuration level, dataset configuration level, physical table configuration level, and asset configuration level, which have a hierarchical relationship.

[0204] Optionally, obtaining the quality inspection rules corresponding to the data asset from a set of preset configuration levels includes: In the multiple configuration levels, the lower configuration level inherits the quality detection rules of the upper configuration level.

[0205] Optionally, within the same configuration level, one detection item corresponds to one detection rule; the step of obtaining the quality detection rule corresponding to the data asset from multiple preset configuration levels includes: When multiple quality detection rules exist for the same detection item across multiple configuration levels, the quality detection rule at the lower level is determined as the quality detection rule corresponding to the detection item.

[0206] Optionally, obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: In response to an asset creation request or an asset editing request, the asset type identifier and source data information of the data asset are obtained.

[0207] Optionally, after obtaining the quality inspection results of the data assets, the method further includes: Send the quality inspection results to the requesting party; the quality inspection results include whether the quality inspection passed or failed.

[0208] Optionally, the step of executing each of the rule detectors to obtain the quality inspection results of the data assets specifically includes: Perform the detection operations of each of the rule detectors; Obtain the execution results of each rule detector; Based on the execution results, a quality inspection result for the data asset is generated.

[0209] Optionally, obtaining the execution results of each of the rule detectors specifically includes: If the execution time of the detection operation does not exceed a preset time threshold, the execution results of each rule detector are obtained; If the execution time of the detection operation exceeds a preset time threshold, a detection record of the data asset is generated. The detection record includes an asynchronous execution identifier and first status information, whereby the first status information indicates that the detection is in progress. Based on the asynchronous execution identifier in the detection record, the execution results of each rule detector are obtained.

[0210] Optionally, obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: According to the preset cycle, obtain the asset type identifier and source data information of at least one published data asset.

[0211] Optionally, obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: The at least one published data asset is divided according to the tenant granularity to obtain multiple tenant groups; The asset acquisition operations corresponding to each tenant group are executed in parallel.

[0212] Optionally, the parallel execution of asset acquisition operations corresponding to each tenant group specifically includes: For each tenant group, the published data assets within each tenant group are divided according to the dataset granularity to obtain multiple dataset subgroups; The asset acquisition operations corresponding to each of the aforementioned dataset subgroups are executed in parallel.

[0213] Optionally, the parallel execution of asset acquisition operations corresponding to each of the dataset subgroups specifically includes: For each of the aforementioned dataset subgroups, the published data assets within the dataset subgroups are divided to obtain multiple asset groups; The quality inspection operations corresponding to each of the asset groups are executed in parallel.

[0214] Optionally, after obtaining the quality inspection results of the data assets, the method further includes: Record the current quality inspection results of the data asset in the current inspection cycle.

[0215] Optionally, after obtaining the quality inspection results of the data assets, the method further includes: Obtain the historical quality inspection results of the data assets for at least one historical inspection period; Based on the current quality inspection results of the data asset in the current inspection cycle and the historical quality inspection results, determine the number of consecutive inspection failures of the data asset; If the number of consecutive failed detections is less than or equal to a preset threshold, an anomaly alert notification is sent to the notification recipient associated with the data asset.

[0216] Optionally, after determining the number of consecutive failed detections of the data asset, the method further includes: If the number of consecutive failed detections exceeds the preset threshold, the data asset will be set to an offline state.

[0217] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0218] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0219] The above is a schematic scheme of a data quality detection device according to this embodiment. It should be noted that the technical solution of this data quality detection device and the technical solution of the data quality detection method described above belong to the same concept. For details not described in detail in the technical solution of the data quality detection device, please refer to the description of the technical solution of the data quality detection method described above.

[0220] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0221] Figure 8 A structural block diagram of a computing device provided according to an embodiment of this specification is shown.

[0222] The computing device 800 includes: Memory 810 and processor 820; The memory 810 is used to store computer programs / instructions, and the processor 820 is used to execute the computer programs / instructions, which, when executed by the processor 820, implement the steps of the data quality detection method.

[0223] Specifically, the components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and the database 850 is used to store data.

[0224] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0225] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0226] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0227] The processor 820 implements the steps of the data quality detection method when executing the computer instructions.

[0228] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data quality detection method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data quality detection method described above.

[0229] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data quality detection method as described above.

[0230] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data quality detection method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data quality detection method described above.

[0231] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the data quality detection method described above.

[0232] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-described data quality detection method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described data quality detection method.

[0233] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The apparatus, device and method provided in the embodiments of this specification are corresponding to each other, and therefore the apparatus and device also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus and device will not be repeated here.

[0234] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0235] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0236] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0237] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0238] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0239] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0240] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0241] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0242] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0243] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0244] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0245] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0246] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0247] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data quality inspection method, comprising: Obtain the asset type identifier and source data information of the data asset to be tested; The source data information is used to describe the data source of the data asset; Determine asset detection parameters based on the source data information; Determine at least one quality inspection rule corresponding to the data asset based on the asset type identifier; Create rule detectors corresponding to each of the aforementioned quality inspection rules; Using the asset detection parameters as input, each of the rule detectors is executed to obtain the quality detection results of the data assets.

2. The method as described in claim 1, wherein determining at least one quality inspection rule corresponding to the data asset based on the asset type identifier specifically includes: The quality detection rules corresponding to the data assets are obtained from multiple preset configuration levels; the multiple configuration levels include at least two of the following: global configuration level, tenant configuration level, dataset configuration level, physical table configuration level, and asset configuration level, which have a hierarchical relationship.

3. The method as described in claim 2, wherein obtaining the quality inspection rules corresponding to the data asset from a preset plurality of configuration levels includes: In the multiple configuration levels, the lower configuration level inherits the quality detection rules of the upper configuration level; Alternatively, the step of obtaining the quality inspection rules corresponding to the data asset from multiple preset configuration levels includes: When multiple quality detection rules exist for the same detection item across multiple configuration levels, the quality detection rule at the lower level is determined as the quality detection rule corresponding to the detection item.

4. The method as described in claim 1, wherein determining at least one quality inspection rule corresponding to the data asset based on the asset type identifier specifically includes: Based on the asset type identifier, a quality inspection rule set corresponding to the asset type identifier is read from a preset rule base, and the quality inspection rule set contains at least one quality inspection rule.

5. The method as described in claim 1, wherein the quality inspection rule includes at least one of the following: rule type identifier, inspection dimension identifier, threshold parameter, rule description template, and repair suggestion template; Alternatively, the creation of rule detectors corresponding to each of the aforementioned quality detection rules includes: Obtain the rule type identifier for each of the aforementioned quality inspection rules; Based on the rule type identifier, a rule detector instance corresponding to the rule type identifier is created by the rule engine.

6. The method as described in claim 1, wherein the asset detection parameters include at least one of the following: source table identifier, data filtering conditions, partition field identifier, number of partitions, source table field list, asset dataset identifier, and asset tenant identifier.

7. The method as described in claim 1, wherein obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: In response to an asset creation request or an asset editing request, the asset type identifier and source data information of the data asset are obtained.

8. The method as described in claim 1, wherein executing each of the rule detectors to obtain the quality inspection results of the data assets specifically includes: Perform the detection operations of each of the rule detectors; If the execution time of the detection operation does not exceed a preset time threshold, the execution results of each rule detector are obtained, and the quality detection results of the data asset are generated based on the execution results. If the execution time of the detection operation exceeds a preset time threshold, a detection record of the data asset is generated. The detection record includes an asynchronous execution identifier and first status information, whereby the first status information indicates that the detection is in progress. Based on the asynchronous execution identifier in the detection record, the execution results of each rule detector are obtained, and the quality detection results of the data asset are generated according to the execution results.

9. The method as described in claim 1, wherein obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: According to the preset cycle, obtain the asset type identifier and source data information of at least one published data asset.

10. The method as described in claim 9, wherein obtaining the asset type identifier and source data information of the data asset to be detected specifically includes: The at least one published data asset is divided according to the tenant granularity to obtain multiple tenant groups; For each tenant group, the published data assets within each tenant group are divided according to the dataset granularity to obtain multiple dataset subgroups; For each of the aforementioned dataset subgroups, the published data assets within the dataset subgroups are divided to obtain multiple asset groups; The quality inspection operations corresponding to each of the asset groups are executed in parallel.

11. The method of claim 9, further comprising, after obtaining the quality inspection result of the data asset: Record the current quality inspection results of the data asset in the current inspection cycle.

12. The method of claim 1, further comprising, after obtaining the quality inspection result of the data asset: Obtain the historical quality inspection results of the data assets for at least one historical inspection period; Based on the current quality inspection results of the data asset in the current inspection cycle and the historical quality inspection results, determine the number of consecutive inspection failures of the data asset; If the number of consecutive failed detections is less than or equal to a preset threshold, an abnormality alert notification is sent to the notification recipient associated with the data asset. If the number of consecutive failed detections exceeds the preset threshold, the data asset will be set to an offline state.

13. A data quality inspection device, comprising: The information acquisition module is used to acquire the asset type identifier and source data information of the data asset to be tested; The source data information is used to describe the data source of the data asset; The parameter conversion module is used to determine asset detection parameters based on the source data information; The detection rule determination module is used to determine at least one quality detection rule corresponding to the data asset based on the asset type identifier; The rule detector creation module is used to create rule detectors corresponding to each of the aforementioned quality detection rules. The quality inspection module is used to take the asset inspection parameters as input, execute each of the rule detectors, and obtain the quality inspection results of the data assets.

14. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

Images