Data processing method and apparatus, storage medium, and program product

Abstract

This application discloses a data processing method, apparatus, storage medium, and program product, belonging to the field of data quality inspection technology. The data processing method includes: acquiring a first quality inspection algorithm and a second quality inspection algorithm; the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed; the second quality inspection algorithm is constructed based on data processing rules input by the user and is used to detect target anomalies in the data to be processed; the data processing rules are determined based on the business information of the data to be processed; based on the business information of the data to be processed, the execution strategies of the first and / or second quality inspection algorithms are matched, and a target quality inspection algorithm is determined according to the execution strategies; based on the target quality inspection algorithm, quality analysis is performed on the data to be processed to generate a quality score for the data to be processed. The data processing method of this application achieves multi-faceted quality inspection, improving the coverage and accuracy of data quality inspection.

Description

Technical Field

[0001] This application belongs to the field of data quality inspection technology, and in particular relates to a data processing method, apparatus, storage medium and program product. Background Technology

[0002] As enterprises advance their digital transformation, data quality directly impacts data analysis and business operations. In big data processing scenarios, data sources are diverse, structures are complex, and scales are large, increasing the difficulty of identifying abnormal data. Related technologies often rely on manually preset, fixed rules, resulting in high maintenance costs and difficulty in covering undefined and complexly interconnected new anomaly patterns. Furthermore, the processes of data standard setting, anomaly identification, analysis, and reporting in related technologies are often independent, leading to difficulties in problem localization and delayed improvement measures. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a data processing method, apparatus, storage medium, and program product that improves the coverage and accuracy of data quality detection.

[0004] Firstly, this application provides a data processing method, which includes: A first quality inspection algorithm and a second quality inspection algorithm are obtained; the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed; the second quality inspection algorithm is constructed based on the data processing rules of the data to be processed input by the user and is used to detect target anomalies in the data to be processed; the data processing rules are determined based on the business information of the data to be processed. Based on the business information of the data to be processed, the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm are matched, and the target quality inspection algorithm is determined according to the execution strategies. The target quality inspection algorithm is used to perform quality analysis on the data to be processed, and a quality score is generated for the data to be processed.

[0005] According to the data processing method of this application, a first quality inspection algorithm and a second quality inspection algorithm are constructed based on a general anomaly detection algorithm and manually formulated and transformed data processing rules, respectively. This combines experience-based rule judgment with machine learning-based intelligent detection to form a multi-detection mechanism. The first quality inspection algorithm can perform general anomaly detection on the data, identifying hidden distribution anomalies, density anomalies, and pattern deviations—multimodal quality problems—without additional manual intervention. This compensates for the coverage blind spots of the second quality inspection algorithm, improving the overall data quality discovery capability and quality inspection coverage. The second quality inspection algorithm, combined with specific business information, performs precise screening of the data to be processed, thereby improving the accuracy and relevance of data verification. Based on business information, a target quality inspection algorithm is determined between the two, and quality analysis is performed on the data to be processed to generate a quality score. This achieves multi-faceted quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0006] According to one embodiment of this application, the step of matching the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm based on the business information of the data to be processed, and determining the target quality inspection algorithm according to the execution strategies, includes: If the data to be processed is determined to be a first type of business based on the business information, the first quality inspection algorithm and the second quality inspection algorithm are determined as the target quality inspection algorithms of the execution strategy; If the data to be processed is determined to be a second type of business based on the business information, the first quality inspection algorithm is determined as the target quality inspection algorithm of the execution strategy.

[0007] According to one embodiment of this application, the business information includes: business attributes, data classification, standard basis, technical attributes, quality attributes, security attributes, and management attributes.

[0008] According to one embodiment of this application, the second quality inspection algorithm is constructed based on data processing rules of the user-input data to be processed, including: Based on the data processing rules of the user-input data to be processed, determine the rule attribute information, rule parameter information, and rule constraint expression information in the data processing rules; Based on the rule attribute information, the rule parameter information, and the rule constraint expression information, a quality inspection rule template is constructed. Based on the quality inspection rule template, a reference relationship is established between the quality inspection rule template and the data to be processed, and the second quality inspection algorithm is generated.

[0009] According to one embodiment of this application, the method further includes: Get new data to be processed; If at least some of the business information in the new data to be processed matches the quality inspection rule template, the data fields of the new data to be processed are mapped to the variable parameters of the quality inspection rule template to generate a reusable quality inspection rule template. Based on the reused quality inspection rule template, the new data to be processed is subjected to quality analysis, and a quality score for the new data to be processed is generated.

[0010] According to one embodiment of this application, the step of performing quality analysis on the data to be processed based on the target quality inspection algorithm to generate a quality score for the data to be processed includes: Based on the security level of the target quality inspection algorithm, a first security weight is determined; A second security weight is determined based on the proportion of abnormal data in the data to be processed. The target quality inspection algorithm is used to perform quality analysis on the data to be processed, and a quality score is generated by combining the first security weight and the second security weight.

[0011] According to one embodiment of this application, after performing quality analysis on the data to be processed based on the target quality inspection algorithm and generating a quality score for the data to be processed, the method further includes: Based on the quality score and the preset quality dimensions, the corresponding anomaly detection mechanism is triggered to generate a visual quality analysis result corresponding to the data to be processed. Output the visualized quality analysis results.

[0012] Secondly, this application provides a data processing apparatus, which includes: The first processing module is used to acquire a first quality inspection algorithm and a second quality inspection algorithm; the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed; the second quality inspection algorithm is constructed based on data processing rules for the data to be processed input by the user and is used to detect target anomalies in the data to be processed; the data processing rules are determined based on the business information of the data to be processed. The second processing module is used to match the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm based on the business information of the data to be processed, and to determine the target quality inspection algorithm according to the execution strategies. The third processing module is used to perform quality analysis on the data to be processed based on the target quality inspection algorithm and generate a quality score for the data to be processed.

[0013] According to the data processing apparatus of this application, a first quality inspection algorithm and a second quality inspection algorithm are constructed based on a general anomaly detection algorithm and manually formulated and transformed data processing rules, respectively. This combines experience-based rule judgment with machine learning-based intelligent detection to form a multi-detection mechanism. The first quality inspection algorithm can perform general anomaly detection on the data, identifying hidden distribution anomalies, density anomalies, and pattern deviations—multimodal quality problems—without additional manual intervention. This compensates for the coverage blind spots of the second quality inspection algorithm, improving the overall data quality discovery capability and quality inspection coverage. The second quality inspection algorithm, combined with specific business information, performs precise screening of the data to be processed, thereby improving the accuracy and relevance of data verification. Based on business information, a target quality inspection algorithm is determined between the two, and quality analysis is performed on the data to be processed to generate a quality score. This achieves multi-faceted quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0014] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method described in the first aspect above.

[0015] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in the first aspect above.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect above.

[0017] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: A first quality inspection algorithm and a second quality inspection algorithm are constructed based on a general anomaly detection algorithm and manually formulated and transformed data processing rules, respectively. This combines experience-based rule judgment with machine learning-based intelligent detection to form a multi-layered detection mechanism. The first quality inspection algorithm can perform general anomaly detection, identifying hidden distribution anomalies, density anomalies, and pattern deviations—multimodal quality issues—without additional human intervention. This compensates for the coverage blind spots of the second quality inspection algorithm, improving the overall data quality discovery capability and inspection coverage. The second quality inspection algorithm, combined with specific business information, performs precise screening of the data to be processed, thereby improving the accuracy and relevance of data verification. Based on business information, a target quality inspection algorithm is determined between the two, performing quality analysis on the data to be processed to generate a quality score. This achieves multi-faceted quality inspection of different levels and types of data, improving the coverage and accuracy of data quality inspection.

[0018] Furthermore, based on clear business rules, manually defined data standards are transformed into executable rule constraints, quantifying business agreements from textual descriptions into logical expressions that can be calculated and executed (such as conditional constraint expressions). This allows the construction of a second quality inspection algorithm to perform quality analysis on the data to be processed and generate quality scores. Furthermore, the quality scores can serve as a quantitative representation of the degree to which business rules are met, supporting comparative evaluation and anomaly localization between different data objects. This enables multi-dimensional quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0019] Furthermore, by transforming business data standards into parameterizable and reusable quality inspection rule templates, when at least some business information in new data to be processed matches the quality inspection rule template, the corresponding quality inspection rule template is referenced. This allows similar business rules to be applied to different data objects without repeated construction, thereby improving the efficiency of rule migration and reuse. Furthermore, with the help of parameterized configuration and rule referencing mechanisms, the same rule can be flexibly adapted to different tables, different fields, and different data scenarios, reducing the costs of repeatedly creating rules, repeatedly configuring, and manually maintaining them, and improving the scalability and execution efficiency of the data quality inspection solution.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 2 This is a second schematic flowchart of the data processing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the business information of the data processing method provided in the embodiments of this application; Figure 4 This is one of the interface diagrams of the data processing method provided in the embodiments of this application; Figure 5 This is the third flowchart illustrating the data processing method provided in the embodiments of this application; Figure 6 This is the fourth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the data processing rules of the data processing method provided in the embodiments of this application; Figure 8 This is a second schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 9 This is the fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 10 This is the third schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 11 This is the fourth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 12 This is the fifth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 13 This is the sixth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 14 This is the seventh schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 15 This is the eighth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 16 This is the ninth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 17 This is the tenth schematic diagram of the interface of the data processing method provided in the embodiments of this application; Figure 18 This is a schematic diagram of the visualization quality analysis results of the data processing method provided in the embodiments of this application; Figure 19 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 20 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0023] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0024] The data processing method, data processing device, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0025] The data processing method can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0026] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets. It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer.

[0027] The following embodiments describe a display and a terminal. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0028] The data processing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the data processing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The data processing method provided in this application embodiment is described below using an electronic device as the execution subject.

[0029] During the research and development process, the inventors discovered that data quality is crucial for ensuring the reliability of analysis results in big data processing workflows. Current data quality management methods mainly fall into two categories. The first is the rule-based explicit method, which screens data using predefined data standards, business rules, and constraints (such as value range, non-empty constraints, and format specifications). While simple and intuitive, this method suffers from high rule maintenance costs and struggles to detect complex anomalies hidden deep within data relationships where rules are not defined. The second is the implicit method based on statistics and machine learning, which automatically identifies anomalies from data distribution using algorithms such as outlier detection (e.g., the PyOD library) and clustering. This type of method can effectively handle high-dimensional, non-linear data, but its results have weak interpretability and heavily rely on data quality and model selection. In the absence of a clear business context, it is prone to misjudging normal business fluctuations as anomalies.

[0030] In summary, rule-based and model-based methods in related technologies are often applied independently, creating breakpoints in the "data standard establishment - anomaly identification - root cause analysis - report feedback" process. This results in fragmented data quality inspection work, low efficiency, and difficulty in forming a closed management loop.

[0031] To address the issues of low efficiency and accuracy in data quality inspection, the inventors, after in-depth research, designed a data processing method comprising: acquiring a first quality inspection algorithm and a second quality inspection algorithm; the first quality inspection algorithm being constructed based on a preset set of anomaly detection rules and used to detect general anomalies in the data to be processed; the second quality inspection algorithm being constructed based on data processing rules input by the user and used to detect target anomalies in the data to be processed; the data processing rules being determined based on the business information of the data to be processed; and, based on the business information of the data to be processed, matching the execution strategies of the first and / or second quality inspection algorithms, and determining the target quality inspection algorithm according to the execution strategies. The target quality inspection algorithm is used to analyze the quality of the data to be processed and generate a quality score for the data.

[0032] According to the data processing method provided in this application, a first quality inspection algorithm and a second quality inspection algorithm are constructed based on a general anomaly detection algorithm and manually formulated and transformed data processing rules, respectively. This combines experience-based rule judgment with machine learning-based intelligent detection to form a multi-detection mechanism. The first quality inspection algorithm can perform general anomaly detection on the data, identifying hidden distribution anomalies, density anomalies, and pattern deviations—multimodal quality problems—without additional manual intervention. This compensates for the coverage blind spots of the second quality inspection algorithm, improving the overall data quality discovery capability and quality inspection coverage. The second quality inspection algorithm, combined with specific business information, performs precise screening of the data to be processed, thereby improving the accuracy and relevance of data verification. Based on the business information, a target quality inspection algorithm is determined between the two, and quality analysis is performed on the data to be processed to generate a quality score. This achieves multi-faceted quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0033] like Figure 1 As shown, the data processing method includes steps 110, 120 and 130.

[0034] Step 110: Obtain the first quality inspection algorithm and the second quality inspection algorithm; In this step, the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules, which is used to detect general anomalies in the data to be processed; wherein, the preset set of anomaly detection rules can be a model, a rule set, or an algorithm flow, etc.

[0035] The second quality inspection algorithm is constructed based on the data processing rules of the data to be processed input by the user, and is used to detect target anomalies in the data to be processed; wherein, the data processing rules are determined based on the business information of the data to be processed.

[0036] Target anomalies refer to data anomalies that correspond to the business scenario to which the data to be processed belongs and are defined by business rules, business standards, or business constraints. They are used to characterize data anomalies in the data to be processed under specific business semantics, business logic, or business value requirements.

[0037] Business information of data to be processed refers to descriptive information related to the business scenario to which the data belongs. It is used to characterize the business attributes of the data and its application background, including but not limited to the business type to which the data belongs, the meaning of the fields, the data source, the data usage scenario, and the corresponding data standards or constraints.

[0038] It should be noted that the first and second quality inspection algorithms are parallel quality inspection methods that complement each other and can perform quality inspection on the data to be processed from different dimensions.

[0039] In actual implementation, the first quality inspection algorithm is a general anomaly detection scheme for common business scenarios, used to identify anomalies in the data to be processed in a general dimension when business data standards are lacking or not refined. The second quality inspection algorithm is an anomaly detection scheme for specific business scenarios, built based on data processing rules corresponding to business information, used to identify anomalies in the data to be processed that are consistent with business data standards. Data standards are the foundation and basis of data quality inspection. Before carrying out data quality inspection tasks, data standards must be formulated for internal data based on business needs and relevant standards (i.e., business information). For example: When the business information pertains to human resources and the data to be processed is basic personnel information under that human resources business, the basic personnel information typically includes fields such as name, gender, and education level. The name, as the legal name registered with the public security department, should be consistent with the information on the identification document; the gender field should meet the preset enumeration value range and business coding requirements; and the education level field should conform to the educational level and corresponding coding standards recognized by the national education authorities.

[0040] Since the above fields all have clear value constraints and validation criteria, it is necessary to formulate corresponding data standards for basic personnel information.

[0041] It should be noted that in some embodiments, a first quality inspection algorithm can be used to perform quality analysis on the data to be processed and screen out the first abnormal results; combined with the first abnormal results, a second quality inspection algorithm can be used to perform quality analysis on the data to be processed to achieve anomaly identification for the target business scenario.

[0042] In this embodiment, the target business scenario can be based on business information or user-defined criteria. For example, for basic personnel information in human resources business, the gender field can be set to meet a preset enumeration value range and the education field can be set to meet a specific coding standard as the target business scenario; or the user can define the value range and condition constraint expression for specific numerical business data as the target business scenario.

[0043] In other embodiments, the target quality inspection algorithm may be set by default to either the first or second quality inspection algorithm to process all the data to be processed.

[0044] In actual implementation, the quality inspection methods of the first and second quality inspection algorithms are coupled and complementary to each other, and can be combined to perform parallel quality inspection.

[0045] For example, based on the general quality inspection algorithm (i.e., the first quality inspection algorithm) for performing general quality analysis on the data to be processed (i.e., performing quality inspection in general dimensions in the absence or lack of refined business data standards), and according to business needs, the target business scenario is clarified or specific constraint conditions are set. Then, based on the manually defined or transformed quality inspection algorithm (i.e., the second quality inspection algorithm), more business-targeted anomaly identification is performed on the data to be processed according to specific data standards.

[0046] In some embodiments, the business information includes: business attributes, data classification, standard basis, technical attributes, quality attributes, security attributes, and management attributes.

[0047] In this embodiment, the business attributes, data classification, standard basis, technical attributes, quality attributes, security attributes, and management attributes are the standard dimensions of the business information.

[0048] It can be understood that the data to be processed contains various different types of data. Before carrying out the data quality inspection task, the data standards for the data to be processed can be formulated according to business requirements and relevant standards.

[0049] The following takes Figure 3 the data standard example to illustrate the standard dimensions and corresponding standard constraints in the business information.

[0050] Such as Figure 3 shown, the standard dimensions include business attributes, data classification, standard basis, technical attributes, quality attributes, security attributes, and management attributes. Different standard dimensions correspond to different standard indicators.

[0051] Continuing to refer to Figure 3 and taking the business attributes as an example, the standard indicators of the business attributes include Chinese, English, Chinese definition, value range or business rules, which are used to uniformly describe the semantic meaning, naming specification, and value constraints of business fields.

[0052] Among them, in Standard Embodiment 1, the business field is "Name", its Chinese is "Name", its English is "FullName", the Chinese definition is "the name and title officially registered by a person in the public security department", and the corresponding business rule is "should be consistent with the name on the identity document"; In Standard Embodiment 2, the business field is "Gender Code", its Chinese is "Gender Code", its English is "GenderCode", the Chinese definition is "the classification code representing the physiological gender of a person", and the corresponding value range is the preset enumerated value range, for example, 1 represents male, 2 represents female, and 0 represents unknown.

[0053] In the actual execution process, the data standards established in the above embodiments can be managed based on the functional modules as shown in Figure 4 the figure.

[0054] refer to Figure 4 This functional module includes creating, editing, publishing, and describing standards. When creating a new standard, users can select the category of the standard code, the referenced standard version, and the associated standard basis, and maintain the corresponding encoding value items. For example, for the standard code of "academic qualifications", the encoding values ​​can be configured to include postgraduate, undergraduate, junior college, high school, junior high school, etc.

[0055] Continue to refer to Figure 4 When creating a new data standard, you can also associate it with relevant national standards, industry standards and other data standard files.

[0056] According to the data processing method provided in the embodiments of this application, different standard constraints are established for different dimensions of business information to achieve unified specifications for the data to be processed in terms of semantic definition, value range and business rules, etc., and to provide a basis for subsequent data processing rule generation and data quality detection.

[0057] In some embodiments, the second quality inspection algorithm is constructed based on data processing rules of the data to be processed input by the user, including: Based on the data processing rules of the user-input data to be processed, determine the rule attribute information, rule parameter information, and rule constraint expression information in the data processing rules; Based on rule attribute information, rule parameter information, and rule constraint expression information, a quality inspection rule template is constructed. Based on the quality inspection rule template, a reference relationship is established between the quality inspection rule template and the data to be processed, and a second quality inspection algorithm is generated.

[0058] In this embodiment, the rule attribute information is the basic attribute of the data processing rule, which is used to identify, classify and manage the rule; The rule parameter information is a parameterized description of the constraints configured based on the rule attribute information, which can be determined based on user-defined parameters or historical experience. Rule constraint expression information is the rule logic generated based on rule attribute information and rule parameter information. For example, it is used to construct the condition constraint part of the query statement, including WHERE conditions, HAVING conditions, and subquery conditions used to implement the condition constraints.

[0059] The quality inspection rule template is a reusable rule carrier built based on the aforementioned rule attribute information, rule parameter information, and rule constraint expression information. It is used to uniformly encapsulate, store, and invoke data processing rules. In practical applications, the quality inspection rule template can be instantiated through parameters and referenced to different data to be processed, thereby achieving the reuse of the same rule in different data tables, fields, or data scenarios. During actual execution, the rule attribute information includes, but is not limited to, rule number, rule type, applicable data quality dimension, and rule description information.

[0060] The rule parameter information includes, but is not limited to: field parameters for receiving the names of the fields being inspected, and constraint parameters for describing the range of values ​​or threshold conditions.

[0061] In actual implementation, constructing a second quality inspection algorithm corresponds to, for example: Figure 5 The process shown involves standard-rule transformation, rule-table referencing, and scheduled task execution. This process maps business data standards to quality inspection rule templates, then imports the quality inspection rule templates into matching table fields, establishing a reference relationship between the quality inspection rule templates and the data to be processed, thereby enabling flexible rule configuration with multiple variables.

[0062] The result of transforming standards into rules can be as follows: Figure 7 As shown, the rule attribute information includes the rule number, rule type, and rule description; the rule parameter information includes the field parameters corresponding to the field being inspected; and the rule constraint expression information is the rule logic generated based on the rule parameter information.

[0063] Taking "gender code" as an example, the rule number corresponding to this rule is PS002-R2, the rule type is validity, and the rule description is "the gender code must be within the value range ('0', '1', '2', '9')". Among them, the rule parameter information includes the field parameter ${currentField} used to indicate the field being inspected; the rule constraint expression information corresponds to the rule logic WHERE ${currentField} NOT IN ('0', '1', '2', '9'), which is used to filter data records that do not meet the value range constraint.

[0064] In some embodiments, Figure 5 Standard-rule transformation in (i.e.) Figure 6 The rule conversion process includes Figure 6 The functions include: standard rule conversion, creating new built-in rules, rule parameter configuration, detection script configuration, and generating rule library tables.

[0065] The generated rule base table is a collection of quality inspection rule templates based on the constructed second quality inspection algorithm. It is used to uniformly store and manage the rule attribute information, parameter information, and rule constraint expressions. In practical applications, the quality inspection rule templates can be instantiated by parameters and referenced to different data tables or fields, thereby realizing the reuse of one rule in multiple tables.

[0066] The detection script configuration includes: configuring the rule constraint expression information.

[0067] The following is based on Figure 8 Taking this as an example, we will explain the configuration of rule parameters in the standard-rule conversion.

[0068] After creating a new built-in rule, configuring the rule parameters for the new rule includes configuring the rule attribute information and the rule parameters.

[0069] Step 1: Configure rule attribute information. This step is used to determine the rule's name, data source adaptation, and quality inspection type, among other attribute information; Figure 8 Taking the "Value Range Validation" rule shown in number 1 as an example, this rule is used to constrain the value range of numeric fields.

[0070] In some embodiments, the quality inspection types include uniqueness, completeness, timeliness, accuracy, validity, and consistency, which are used for subsequent quality inspection reports to be classified and statistically analyzed according to the problem type.

[0071] Step 2: Configure rule parameter information. This step is used to determine the parameter variables in the rule. Figure 8 Taking number 2 as an example, the parameter variables ${minValue} and ${maxValue} are configured for the value range validation rule, where ${currentField} is a preset field parameter used to receive the field name to be inspected.

[0072] Step 3: Configure rule constraint expression information. This step generates rule logical expressions based on the rule parameter information. The rule logical expressions include expressions used to form the conditional constraint parts of the SQL statement (e.g., WHERE conditions). Figure 8 Taking the example shown in number 3, the rule logic can be expressed as: ${currentField}<${minValue} OR ${currentField}>${maxValue}, where the connector can be selected according to the actual scenario.

[0073] After configuring the rule parameters, refer to Figure 8Generating a rule base table means writing the configured rule attribute information, rule parameter information, and rule constraint expression information into the rule base table to form a reusable quality inspection rule template, which can be called later when referencing rules for different data to be processed.

[0074] As shown in the above embodiments, the rule conversion module is used to transform established data standards into reusable quality inspection rule templates. In this module, by configuring rule attribute information, rule parameter information, and rule constraint expression information, a quality inspection rule template is formed and stored in the rule base table. During the rule generation process, users can configure parameter variables according to the data standards. The system generates corresponding rule logical expressions based on these parameter variables. These rule logical expressions, as part of the quality inspection rule template, are used for subsequent rule referencing and execution.

[0075] According to the data processing method provided in the embodiments of this application, based on clear business rules, manually formulated data standards are transformed into executable rule constraints, quantifying business agreements from textual descriptions into logical expressions that can be calculated and executed (such as conditional constraint expressions), thereby constructing a second quality inspection algorithm to perform quality analysis on the data to be processed and generate quality scores; furthermore, the quality scores can serve as a quantitative representation of the degree of satisfaction of business rules, used to support comparative evaluation and anomaly location between different data objects, thereby achieving multi-dimensional quality inspection of data at different levels and of different types, and improving the coverage and accuracy of data quality inspection.

[0076] refer to Figure 5 After the standard-to-rule conversion is completed, the rule referencing stage begins. In this stage, the rule is invoked. Figure 8 The quality inspection rule template is generated and stored in the rule base table. The quality inspection rule template is then linked to the database table fields corresponding to the data to be processed, so that the quality inspection rule template can be reused for different data table fields of different types of databases.

[0077] In some embodiments, rule-table references (i.e.) Figure 9 (The rule references in the text) can include Figure 9 The functions include: creating a quality inspection plan, configuring quality inspection issues, configuring quality inspection rules, executing the quality inspection plan, and viewing the quality inspection results.

[0078] The following explains the configuration quality inspection issues and rules.

[0079] Step 1: Configure quality inspection issues; see reference Figure 10 Select the data source and data table to be inspected, and configure the inspection plan name, inspection plan type, data source, table name, and group SQL to group the data to be inspected.

[0080] Step 2: Configure quality inspection rules; see reference Figure 11 When configuring quality inspection rules, you can reference the quality inspection rule template in the pre-configured rule base for any field in the selected data table, establish a reference relationship between the quality inspection rule template and the target field, and configure information such as quality inspection issues, quality inspection objects, rule levels, issue levels, and rule levels. refer to Figure 12 When configuring quality inspection rules, you can do so using custom SQL. By enabling custom SQL and writing the quality inspection logic in the SQL script editing area, you can also execute SQL tests to verify the validity of the rules, thus meeting the needs of complex quality inspection scenarios, such as the correlation verification of multiple data tables or the combined constraint check across fields.

[0081] After configuring the quality inspection rules, the generated quality inspection plan can be displayed in the form of a list, card, or tree structure. Figure 13 Taking the quality inspection task configuration list shown as an example, the list displays the configured quality inspection schemes one by one. Each quality inspection scheme corresponds to a scheduling task, and displays the scheduling status, execution cycle, and next execution time of the quality inspection scheme.

[0082] According to the data processing method provided in the embodiments of this application, constraint checks on the same data item are realized from multiple dimensions by decoupling rule configuration and rule reference, and the same rule is also used for multi-table reference.

[0083] For example, in the "Personnel Basic Information" table - "Name" field, a quality control rule can be generated for the string length of "Name", such as "WHERE length(${currentField})>2". Alternatively, a quality control rule can be set to prevent "Name" from being nullable, generating the rule "WHERE ${currentField} IS NULL OR ${currentField} = ' '". These generated quality control rules can then be applied to other data tables containing a "Name" field.

[0084] In some embodiments, refer to the following before implementing the quality inspection plan: Figure 5 It also allows for the configuration of scheduled tasks for quality inspection plans.

[0085] In this embodiment, the scheduled task can be implemented by setting the automatic execution time of the quality inspection task based on the scheduling configuration.

[0086] In actual execution, the scheduling configuration can set a quality inspection plan for a person's basic information to be executed automatically according to a preset scheduling cycle, and display the next execution time as a specified date and time in the list.

[0087] For example, refer to Figure 14Configure the scheduling cycle status to enable or disable scheduled task scheduling, and set the execution cycle based on the preset scheduling expression; after completing the scheduling parameter settings, the next and subsequent execution times of the quality inspection task can be generated and displayed to verify the effective time of the scheduling configuration.

[0088] According to the data processing method provided in the embodiments of this application, by configuring the rule reference of the database table fields to be inspected, the table name and field name are dynamically passed to the quality inspection rule template, the variables are instantiated, the quality inspection rule template is transformed into an actual executable SQL statement, and the data that does not meet the rules is filtered out by executing the SQL statement, thereby realizing the quality inspection of the data to be processed.

[0089] In some embodiments, the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed.

[0090] In this embodiment, the preset set of anomaly detection rules may include one or more general anomaly detection algorithms and their corresponding detection parameter configurations, such as Isolation Forest, LOF, or AutoEncoder.

[0091] In actual execution, the first quality inspection algorithm can be built based on the platform's built-in executable environment (such as PyOD or scikit-learn), and the anomaly detection task can be executed through a unified interface call mechanism; for example, the data of the field to be inspected can be transformed into input data that the algorithm can process, and the anomaly detection result can be output based on the first quality inspection algorithm.

[0092] The following explanation uses the first quality inspection algorithm as an example to illustrate the anomaly recognition model generated by integrating multiple general anomaly detection algorithms.

[0093] In actual implementation, refer to Figure 15 , Figure 15 This is a configuration interface diagram of the anomaly detection model in the data development module. The data development module provides an executable environment for Python nodes to run the anomaly detection model. Based on this function, any algorithm model can be configured. The data development module integrates the DolphinScheduler scheduling tool and builds an anomaly data detection model based on the PyOD framework. The anomaly data detection model is embedded into the data development module for periodic scheduling and execution to detect data to be processed and store the detected anomaly data in the background database.

[0094] The aforementioned anomaly detection model built on the PyOD framework can integrate various anomaly detection algorithms, including Isolation Forest, LOF, or AutoEncoder. Through an unsupervised learning anomaly scoring mechanism and ensemble learning strategy, it can automatically identify deep quality problems such as distribution anomalies, density anomalies, and pattern deviations hidden in the data. It can effectively identify data distribution shifts and multimodal anomalies that are difficult to cover by traditional rules without the need for preset thresholds.

[0095] According to the data processing method provided in the embodiments of this application, the first quality inspection algorithm is constructed based on a set of general anomaly detection rules. It can be used to identify anomalies in data that lacks clear business rule constraints. It is suitable for outlier detection scenarios of numerical fields, such as amount and age. The first quality inspection algorithm can output suspicious data based on anomaly scores, and the detection sensitivity can be adjusted through parameter configuration to adapt to different data distributions and business categories. This enables multi-faceted quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0096] Step 120: Based on the business information of the data to be processed, match the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm, and determine the target quality inspection algorithm according to the execution strategy.

[0097] In this step, the execution strategy determines the invocation method of the first and second quality inspection algorithms based on different business scenarios, in order to generate the corresponding target quality inspection algorithm. The target quality inspection algorithm is used to perform quality analysis on the data to be processed.

[0098] In some embodiments, the target quality inspection algorithm may be a first quality inspection algorithm, a second quality inspection algorithm, or a combination of the first quality inspection algorithm and the second quality inspection algorithm.

[0099] It should be noted that, in some other embodiments, the target quality inspection algorithm may also be other quality inspection algorithms derived from the first quality inspection algorithm and / or the second quality inspection algorithm, and this application does not limit this.

[0100] In some embodiments, step 120 includes: If the data to be processed is determined to be of the first type of business based on business information, the first quality inspection algorithm and the second quality inspection algorithm are determined as the target quality inspection algorithms for the execution strategy. If the data to be processed is determined to be of the second type of business based on business information, the first quality inspection algorithm is selected as the target quality inspection algorithm for the execution strategy.

[0101] In this embodiment, the first type of business is a business scenario in which clear business data standards have been established. The data standards are formulated manually according to specific business needs. For example, for data with clear value specifications and business constraints, such as basic personnel information and business master data, the first quality inspection algorithm and the second quality inspection algorithm are used to inspect the data to be processed.

[0102] The second type of business is a business scenario where no clear business rules have been established. For example, a scenario for outlier identification of numerical fields, including fields such as amount and age. Due to the lack of clear business value constraints, in order to improve the coverage of quality inspection, the first quality inspection algorithm is used to inspect the data to be processed in this scenario.

[0103] In some embodiments, the data to be processed can be analyzed based on preset business judgment rules to determine whether the data to be processed is a first type of business or a second type of business.

[0104] In this embodiment, the preset business judgment rules can be determined based on user-defined rules or historical experience.

[0105] In some embodiments, the data processing method further includes: Get new data to be processed; If at least some of the business information in the new data to be processed matches the quality inspection rule template, the data fields of the new data to be processed are mapped to the variable parameters of the quality inspection rule template to generate a reusable quality inspection rule template. Based on the reused quality inspection rule template, quality analysis is performed on the new data to be processed, and a quality score is generated for the new data to be processed.

[0106] In actual implementation, refer to Figure 5 After completing the standard-rule conversion process, a first quality inspection algorithm can be built based on general rules, and a second quality inspection algorithm can be built based on the data processing rules corresponding to business information. The quality inspection rule templates corresponding to the first and second quality inspection algorithms are stored.

[0107] For newly acquired data to be processed, if at least some of the business information in the new data matches the quality inspection rule template, such as if both the new data to be processed and historical data contain similar business fields such as name, age, or date of birth, a reused quality inspection rule template can be generated through the rule-table reference process. Based on this, the reused first quality inspection algorithm and the reused second quality inspection algorithm can be obtained to perform quality analysis on the new data to be processed and generate a quality score for the new data to be processed.

[0108] According to the data processing method provided in the embodiments of this application, by converting business data standards into parameterizable and reusable quality inspection rule templates, when at least some business information in new data to be processed matches the quality inspection rule template, the corresponding quality inspection rule template is referenced, so that similar business rules can be applied to different data objects without repeated construction, thereby improving the efficiency of rule migration and reuse. Furthermore, with the help of parameterized configuration and rule reference mechanism, the same rule can be flexibly adapted to different tables, different fields and different data scenarios, reducing the cost of repeated rule creation, repeated configuration and manual maintenance, and improving the scalability and execution efficiency of the data quality inspection solution.

[0109] Step 130: Perform quality analysis on the data to be processed based on the target quality inspection algorithm to generate a quality score for the data to be processed.

[0110] In this step, the quality score of the data to be processed is determined based on the amount of outlier data and the security level of the data.

[0111] In some embodiments, step 130 includes: Based on the security level of the target quality inspection algorithm, determine the first security weight; The second security weight is determined based on the proportion of abnormal data in the data to be processed. The target quality inspection algorithm is used to analyze the quality of the data to be processed, and a quality score is generated by combining the first security weight and the second security weight.

[0112] In this embodiment, the security level is a preset grading index used to characterize the importance of the target quality inspection algorithm.

[0113] The amount of anomalous data is used to reflect the degree of anomalousness of the current overall data.

[0114] It should be noted that the first security weight and the second security weight have different sources and different dimensions of function, and they are not interchangeable.

[0115] In some embodiments, the security level of the target quality inspection algorithm can be divided into: a first security level, a second security level, and a third security level, and the security weights can be divided into: a first security weight, a second security weight, and a third security weight.

[0116] In actual implementation, the first security level corresponds to critical business data. When quality problems occur in this type of data, it may directly affect important decisions or cause significant business losses. For example, data used for core business decisions, regulatory reporting, or external audits; this level corresponds to the first security weight.

[0117] The second security level corresponds to important business data. When quality problems occur in this type of data, certain business risks may arise, but it usually does not cause immediate direct losses. For example, data that is of concern to regulators and core management. This level corresponds to the second security weight.

[0118] The third security level corresponds to general business data. When quality problems occur in this type of data, they mainly affect local or auxiliary business operations, with a relatively small impact on overall operation. This level corresponds to the third security weight.

[0119] Understandably, the higher the security level, the greater its corresponding security weight, and the greater its impact on the quality score calculation.

[0120] The calculation of security level, security weight, and quality score will be explained below using Table 1 as an example.

[0121]

[0122] In actual implementation, referring to Table 1, different rule levels correspond to different rule weights, representing the full score of the rule in the overall quality assessment. For example, high-level rules can be set with larger weight values ​​(e.g., 6), medium-level rules with smaller weight values ​​(e.g., 3), and low-level rules with even smaller weight values ​​(e.g., 1).

[0123] Referring to Table 1, under each rule level, multiple intervals are divided based on the proportion of abnormal data, and a pre-set deduction ratio is assigned to each interval. When the proportion of abnormal data is low, the corresponding deduction ratio is low or even 0; as the proportion of abnormal data increases, the deduction ratio gradually increases until it reaches the full deduction of the rule's weight.

[0124] Continuing with reference to Table 1, the actual deduction and actual score are determined based on the different levels of rules and the amount of abnormal data in the rules. The actual deduction is the actual deduction value calculated according to the rule weight and the deduction ratio; the actual score is the final score after subtracting the actual deduction from the rule weight.

[0125] Referring back to Table 1, each rule level has its specific weight, and points are deducted according to a certain percentage based on the proportion of outlier data, ultimately resulting in the actual score. The larger the amount of outlier data, the higher the deduction value, and the lower the actual score. Based on these rules, the actual scores of multiple rules can be weighted and summarized, for example: The table-level quality score is the ratio of the sum of the actual scores of all rules in the table to the sum of the weights of all rules. The entity (such as a company) level quality score is the ratio of the sum of the actual scores of all rules within its coverage to the sum of the weights of all rules.

[0126] Based on the above rules, the data in each table of Company A, Company B and Company C as shown in Table 2 were subjected to quality inspection.

[0127]

[0128] Referring to Table 2, for each company and its corresponding data table, the amount of abnormal data and monitored data were statistically analyzed, and the percentage of abnormal data was calculated. Then, considering the applicable rule level and rule type for that data table, the corresponding deduction value was determined, and the actual score was obtained by deducting from the rule weight. Because the percentage of abnormal data varies for different companies across different tables, there is a difference between the deduction and the actual score.

[0129] Based on the rules set in Table 2, the anomaly detection results of each company on different data tables were weighted and calculated to obtain the standardized score of each table and the comprehensive score of each company. The results are shown in Tables 3 and 4.

[0130]

[0131] Table 3 is a table-level score calculated by normalizing the actual score in Table 2 and the corresponding rule weight on a single table dimension. For example, the actual score in Table 2 is divided by the rule weight corresponding to the rule level and converted into a percentage score.

[0132]

[0133] Table 4 shows the comprehensive score obtained by summing the actual scores of the same company across multiple data tables according to the rule weights and then dividing by the sum of all rule weights.

[0134] The actual data and the corresponding scoring results show that the tiered and quantitative scoring mechanism can not only distinguish the data quality levels of different companies, but also better meet the needs of current actual business scenarios, which helps to promote companies to focus on the management and continuous improvement of important data.

[0135] According to the data processing method provided in the embodiments of this application, a comprehensive data quality score is given to the dataset through a scoring model to generate a data quality inspection report, which can promote various departments to carry out data quality rectification work and achieve a complete closed loop of data quality work.

[0136] In some embodiments, after performing quality analysis on the data to be processed based on the target quality inspection algorithm and generating a quality score for the data to be processed, the method further includes: Based on the quality score and preset quality dimensions, the corresponding anomaly detection mechanism is triggered to generate visual quality analysis results corresponding to the data to be processed. Output visualized quality analysis results.

[0137] In this embodiment, the visualized quality analysis results are used to show users the data quality quantification results of the data to be processed. The output format may include, but is not limited to: interactive result tables, downloadable data files, and charts or pictures for visualization. A report file can be generated through the export function for archiving.

[0138] The preset quality dimensions are used to classify and describe the quality issues of the data to be processed and to collect the results, so as to perform itemized analysis and visualization of abnormal data based on different quality dimensions.

[0139] The anomaly detection mechanism is used to further identify, aggregate, and analyze the number of abnormal rules, the proportion of abnormal rules, or the amount of abnormal data in the data to be processed, based on the quality score and in combination with preset quality dimensions, thereby generating corresponding visualized quality analysis results.

[0140] In some embodiments, the preset quality dimensions include, but are not limited to: data uniqueness, completeness, timeliness, accuracy, validity, and consistency.

[0141] In actual implementation, refer to Figure 2 The data processing method provided in the above embodiments can be integrated into a hierarchical and quantitative comprehensive scoring model, and a closed-loop process is formed by data standards, rules / algorithms, scheduling execution and result output; the closed-loop process includes at least data standard establishment, abnormal data detection, abnormal data analysis and automatic generation and output of quality inspection reports.

[0142] In actual implementation, refer to Figure 16 and Figure 17 It can visualize abnormal data based on preset data components (such as BI report components or dashboard components). The visualization quality analysis results can be displayed through the data quality dashboard, and it supports viewing from overview indicators to detailed anomalies step by step to realize the statistical analysis and display of abnormal data.

[0143] refer to Figure 16 , Figure 16 The interface provides a data quality overview, displaying the overall quality score, statistical indicators related to abnormal data, and the score distribution by quality dimension. It also supports filtering by department / scope and time. Additionally, users can jump from the overview entry to the abnormal table list to view information such as the score, number of abnormal rules, and percentage of abnormalities for each data table.

[0144] refer to Figure 17 , Figure 17 The interface for viewing anomaly details displays the anomaly table and its key indicators in a list format. It also allows users to access individual table detail pages to view the anomaly data items and corresponding data values ​​for specific fields, thus enabling result tracing and display from table-level summary to field-level details.

[0145] In actual implementation, refer to Figure 18 Based on the data quality dashboard component provided in the above embodiments, multiple visual quality analysis results can be automatically summarized and a structured data quality inspection report can be generated within a preset scheduling period, realizing the automated output of data quality results for different business units.

[0146] Figure 18 The report first provides an overall quality overview, including comprehensive scores, the scale of abnormal data, and the percentage of abnormal data. Then, it presents the test results by business domain or data table dimension, and scores and ranks the data quality status of different dimensions to identify objects with relatively prominent quality risks.

[0147] In the details section, the report lists the number and percentage of anomalies for each data table under different rule levels, and gives corresponding scores accordingly, supporting further identification of the problem distribution of specific data items from the summary results.

[0148] Meanwhile, the report also standardizes and weights the results based on a tiered quantitative scoring mechanism, forming multi-level scoring results that allow for horizontal comparison of different business units and vertical tracking of data quality changes of the same business unit at different times.

[0149] This enables the automatic integration of quality inspection, result summarization, grading and scoring, and report generation, and regularly outputs visualized quality analysis results of the data to be processed without the need for manual item-by-item statistics.

[0150] According to the data processing method provided in this application, based on a closed-loop process, rule transformation can be automatically or semi-automatically triggered after data standards are established or updated. With the coordination of rule referencing and scheduled tasks, quality analysis is performed on the data to be processed, generating quality scores and outputting visualized quality analysis results. By linking data standards, rule transformation, rule referencing, detection execution, and result output into a consistent processing chain, manual configuration and repetitive maintenance are reduced, transforming large-scale data quality inspection from manual drive to automatic operation based on configuration, thereby improving the coverage and accuracy of data quality inspection.

[0151] The data processing method provided in this application can be executed by a data processing device. This application uses an example of a data processing device executing the data processing method to illustrate the data processing device provided in this application.

[0152] This application also provides a data processing apparatus.

[0153] like Figure 19As shown, the data processing device includes: a first processing module 1910, a second processing module 1920, and a third processing module 1930.

[0154] The first processing module 1910 is used to acquire a first quality inspection algorithm and a second quality inspection algorithm. The first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed. The second quality inspection algorithm is constructed based on the data processing rules of the data to be processed input by the user and is used to detect target anomalies in the data to be processed. The data processing rules are determined based on the business information of the data to be processed. The second processing module 1920 is used to match the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm based on the business information of the data to be processed, and to determine the target quality inspection algorithm according to the execution strategy. The third processing module 1930 is used to perform quality analysis on the data to be processed based on the target quality inspection algorithm and generate a quality score for the data to be processed.

[0155] According to the data processing apparatus provided in this application embodiment, a first quality inspection algorithm and a second quality inspection algorithm are constructed based on a general anomaly detection algorithm and manually formulated and transformed data processing rules, respectively. This combines experience-based rule judgment with machine learning-based intelligent detection to form a multi-detection mechanism. The first quality inspection algorithm can perform general anomaly detection on the data, identifying hidden distribution anomalies, density anomalies, and pattern deviations—multimodal quality problems—without additional manual intervention. This compensates for the coverage blind spots of the second quality inspection algorithm, improving the overall data quality discovery capability and quality inspection coverage. The second quality inspection algorithm, combined with specific business information, performs precise screening of the data to be processed, thereby improving the accuracy and targeting of data verification. Based on business information, a target quality inspection algorithm is determined between the two, and quality analysis is performed on the data to be processed to generate a quality score. This achieves multi-faceted quality inspection of data at different levels and of different types, improving the coverage and accuracy of data quality inspection.

[0156] In some embodiments, the second processing module 1920 may also be used for: If the data to be processed is determined to be of the first type of business based on business information, the first quality inspection algorithm and the second quality inspection algorithm are determined as the target quality inspection algorithms for the execution strategy. If the data to be processed is determined to be of the second type of business based on business information, the first quality inspection algorithm is selected as the target quality inspection algorithm for the execution strategy.

[0157] In some embodiments, the first processing module 1910 may also be used for: Based on rule attribute information, rule parameter information, and rule constraint expression information, a quality inspection rule template is constructed. Based on the quality inspection rule template, a reference relationship is established between the quality inspection rule template and the data to be processed, and a second quality inspection algorithm is generated.

[0158] In some embodiments, the data processing apparatus further includes a fourth processing module for: Get new data to be processed; If at least some of the business information in the new data to be processed matches the quality inspection rule template, the data fields of the new data to be processed are mapped to the variable parameters of the quality inspection rule template to generate a reusable quality inspection rule template. Based on the reused quality inspection rule template, quality analysis is performed on the new data to be processed, and a quality score is generated for the new data to be processed.

[0159] In some embodiments, the third processing module 1930 may also be used for: Based on the security level of the target quality inspection algorithm, determine the security weight of the target quality inspection algorithm; The target quality inspection algorithm is used to analyze the quality of the data to be processed. By combining the amount of abnormal data and the security weight, a quality score is generated for the data to be processed.

[0160] In some embodiments, the device further includes a fifth processing module for: Based on the quality score and preset quality dimensions, the corresponding anomaly detection mechanism is triggered to generate visual quality analysis results corresponding to the data to be processed. Output visualized quality analysis results.

[0161] The data processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0162] The data processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0163] The data processing device provided in this application embodiment can achieve... Figures 1 to 18 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0164] In some embodiments, such as Figure 20 As shown, this application embodiment also provides an electronic device 2000, including a processor 2001, a memory 2002, and a computer program stored in the memory 2002 and executable on the processor 2001. When the program is executed by the processor 2001, it implements the various processes of the above-described data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0165] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0166] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0167] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0168] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method.

[0169] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0170] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0171] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0172] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0174] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0175] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0176] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A data processing method, characterized by, include: Obtain a first quality inspection algorithm and a second quality inspection algorithm; the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed; The second quality inspection algorithm is constructed based on the data processing rules of the data to be processed input by the user, and is used to detect target anomalies in the data to be processed; The data processing rules are determined based on the business information of the data to be processed. Based on the business information of the data to be processed, the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm are matched, and the target quality inspection algorithm is determined according to the execution strategies. The target quality inspection algorithm is used to perform quality analysis on the data to be processed, and a quality score is generated for the data to be processed.

2. The data processing method according to claim 1, characterized in that, The process of matching the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm with the business information based on the data to be processed, and determining the target quality inspection algorithm according to the execution strategies, includes: If the data to be processed is determined to be a first type of business based on the business information, the first quality inspection algorithm and the second quality inspection algorithm are determined as the target quality inspection algorithms of the execution strategy; If the data to be processed is determined to be a second type of business based on the business information, the first quality inspection algorithm is determined as the target quality inspection algorithm of the execution strategy.

3. The data processing method of claim 1, wherein, The business information includes: business attributes, data classification, standard basis, technical attributes, quality attributes, security attributes, and management attributes.

4. The data processing method according to any one of claims 1-3, characterized in that, The second quality inspection algorithm is constructed based on the data processing rules of the user-input data to be processed, including: Based on the data processing rules of the user-input data to be processed, determine the rule attribute information, rule parameter information, and rule constraint expression information in the data processing rules; Based on the rule attribute information, the rule parameter information, and the rule constraint expression information, a quality inspection rule template is constructed. Based on the quality inspection rule template, a reference relationship is established between the quality inspection rule template and the data to be processed, and the second quality inspection algorithm is generated.

5. The data processing method according to claim 4, characterized in that, The method further includes: Get new data to be processed; If at least some of the business information in the new data to be processed matches the quality inspection rule template, the data fields of the new data to be processed are mapped to the variable parameters of the quality inspection rule template to generate a reusable quality inspection rule template. Based on the reused quality inspection rule template, the new data to be processed is subjected to quality analysis, and a quality score for the new data to be processed is generated.

6. The data processing method according to any one of claims 1-3, characterized in that, The step of performing quality analysis on the data to be processed based on the target quality inspection algorithm to generate a quality score for the data to be processed includes: Based on the security level of the target quality inspection algorithm, a first security weight is determined; A second security weight is determined based on the proportion of abnormal data in the data to be processed. The target quality inspection algorithm is used to perform quality analysis on the data to be processed, and a quality score is generated by combining the first security weight and the second security weight.

7. The data processing method according to any one of claims 1 to 3, characterized in that, After performing quality analysis on the data to be processed based on the target quality inspection algorithm and generating a quality score for the data to be processed, the method further includes: Based on the quality score and the preset quality dimensions, the corresponding anomaly detection mechanism is triggered to generate a visual quality analysis result corresponding to the data to be processed. Output the visualized quality analysis results.

8. A data processing apparatus, characterized by, include: The first processing module is used to acquire a first quality inspection algorithm and a second quality inspection algorithm; the first quality inspection algorithm is constructed based on a preset set of anomaly detection rules and is used to detect general anomalies in the data to be processed; the second quality inspection algorithm is constructed based on the data processing rules of the data to be processed input by the user and is used to detect target anomalies in the data to be processed. The data processing rules are determined based on the business information of the data to be processed. The second processing module is used to match the execution strategies of the first quality inspection algorithm and / or the second quality inspection algorithm based on the business information of the data to be processed, and to determine the target quality inspection algorithm according to the execution strategy. The third processing module is used to perform quality analysis on the data to be processed based on the target quality inspection algorithm and generate a quality score for the data to be processed. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data processing method as described in any one of claims 1-7.

Images