Data processing method, cloud platform system, readable medium and program product
By introducing repair rules into the data quality management system, data that does not meet the quality rules are automatically processed, which solves the problem of increased work costs caused by developers' manual processing of data, and achieves more efficient data processing and resource utilization.
Patent Information
- Application Number
- CN202410181785.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-18
- Publication Date
- 2025-08-19
AI Technical Summary
In the existing data quality management method, developers need to manually process data that does not meet quality rules, resulting in increased work costs.
By introducing repair rules into the data quality management system, data that does not meet the quality rules are automatically processed, quality reports are generated, and processed data is sent.
It reduces the work costs of users and improves data processing efficiency and resource utilization.
Smart Images
Figure CN120508551A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management technology, and in particular to a data processing method, a cloud platform system, a readable medium, and a program product. Background Art
[0002] Data quality management (DQM) is widely used in industries such as electronics manufacturing, the Internet, chemicals, new materials, semiconductors, and automobiles. It is a process of identifying, monitoring, providing early warnings, and improving data quality at every stage of its life cycle: generation, storage, use, archiving, and deletion.
[0003] For example, before a client device uses a service provided by a cloud platform, such as generating a user profile based on customer usage data and generating a corresponding personalized recommendation strategy, it is necessary to perform quality management on the customer usage data through the cloud platform system's data quality management system (such as determining the integrity of the data), and then process unqualified data (such as data that does not meet integrity) based on the quality report generated by the data quality management system or the quality warning issued, and then use the corresponding cloud service based on the processed qualified data, such as generating a user profile and generating a corresponding personalized recommendation strategy. In the process of data quality management, if the data quality management system detects that the data is missing, it determines the missing data as dirty data, and identifies the dirty data in the generated quality report, or issues a warning message to notify the developer of the client device to process the dirty data.
[0004] Current data quality management methods involve identifying dirty data in quality reports or sending warnings to client developers when data doesn't meet quality rules pre-set by business platform developers, such as integrity rules. This leads to increased developer workload. Summary of the Invention
[0005] The purpose of this application is to provide a data processing method, a cloud platform system, a readable medium and a program product.
[0006] The first aspect of the present application provides a data processing method, which is applied to an electronic device, including: receiving a data management request from a client device, obtaining first data corresponding to the data management request; selecting data to be processed that does not meet quality rules from the first data; adopting a repair rule corresponding to the data to be processed, processing the data to be processed, and obtaining second data; and sending the second data to the client device.
[0007] In the embodiment of the present application, the first data may be data to be managed, the data to be processed may be problem data, and the second data may be processed data.
[0008] It can be understood that the data processing method of the present application selects problem data that does not meet the quality rules from the data to be managed, adopts the repair rules corresponding to the problem data to process the problem data, and sends the processed data to the client device, without the user having to manually process the data that does not meet the quality rules, which can effectively reduce the user's work costs.
[0009] In one possible implementation of the first aspect above, obtaining the first data corresponding to the data management request includes: obtaining the first data from a local memory of the client device based on the data management request; or obtaining the first data from a cloud database corresponding to the client device based on the data management request.
[0010] In a possible implementation of the first aspect above, selecting the data to be processed that does not meet the quality rules from the first data includes: determining a data management task corresponding to the data management request, wherein the data management task includes management rules corresponding to at least one target field of the first data, and the management rules include quality rules and repair rules; based on the quality rules corresponding to at least one target field of the first data, selecting the data to be processed that does not meet the quality rules from the first data.
[0011] In one possible implementation of the first aspect above, the quality rules include first quality rules and / or second quality rules; wherein, the first quality rules include at least one of the following: uniqueness rule, non-empty rule, value range rule, threshold rule, format rule, conditional rule, consistency rule, repeatability rule and integrity rule; the second quality rules include at least one of the following parameters: rule name, rule type, rule content; wherein, the expression form of the rule content includes at least one of the following: regular expression, custom statement.
[0012] In the embodiment of the present application, the first quality rule may be a basic quality rule, and the second quality rule may be a user-defined quality rule.
[0013] In a possible implementation of the first aspect above, the repair rule includes at least one of the following: modifying the rule, deleting the rule, and revising the rule.
[0014] In one possible implementation of the first aspect above, determining the data management task corresponding to the data management request includes: generating a rule script corresponding to the management rules corresponding to each target field based on the quality rules and repair rules corresponding to each target field of the first data; and generating a task script corresponding to the data management task based on at least one rule script.
[0015] It can be understood that based on the rule script of at least one target field of the data to be processed or managed, a task script corresponding to the data to be processed or managed is generated. In other words, a task script can correspond to multiple rules. This allows for concurrent computation of multiple rules within a single task, thereby improving resource utilization, reducing operating costs, and enhancing the computational efficiency of quality rule monitoring.
[0016] In one possible implementation of the first aspect above, a repair rule corresponding to the data to be processed is adopted to process the data to be processed to obtain second data, including: performing at least one of the following processing on the first sub-data to be processed in the data to be processed that meets the triggering conditions of the repair rule based on the repair rule: modification, deletion, and repair; and using the processed sub-data to be processed as the second data.
[0017] In an embodiment of the present application, the first sub-data to be processed may be first problematic sub-data. For a target field in the data to be managed that does not satisfy a corresponding quality rule but meets the matching criteria of a repair rule, the first problematic sub-data is processed based on the repair rule to ensure that it satisfies the corresponding quality rule. The processed data is then sent to the client device. The processing of the first problematic sub-data based on the repair rule includes, but is not limited to, modification, deletion, and correction.
[0018] In one possible implementation of the first aspect above, the management rules also include monitoring alarm rules, and corresponding to the presence of second to-be-processed sub-data in the to-be-processed data that does not meet the triggering conditions of the repair rules, an alarm message is sent to the client device based on the monitoring alarm rules.
[0019] In an embodiment of the present application, the second sub-data to be processed may be second problem sub-data. When the target field in the data to be managed contains the second problem sub-data that does not satisfy the corresponding quality rule and does not satisfy the matching condition of the repair rule, the second problem sub-data is saved and an alarm message is sent to the client device.
[0020] The second aspect of the present application provides a cloud platform system, including: a client device, used to send a data management request; a data quality management system, used to receive the data management request, obtain the first data corresponding to the data management request, and select the data to be processed that does not meet the quality rules from the first data; the data quality management system is also used to adopt the repair rules corresponding to the data to be processed, process the data to be processed, obtain second data, and send the second data; the client device is also used to receive the second data.
[0021] A third aspect of the present application provides a readable medium having instructions stored thereon. When the instructions are executed on an electronic device, the electronic device executes any one of the methods of the first aspect.
[0022] A fourth aspect of the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor for executing any one of the methods of the first aspect above.
[0023] A fifth aspect of the present application provides a program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements any one of the methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0025] Figure 1 According to an embodiment of the present application, a schematic diagram of an application scenario of a cloud platform is shown;
[0026] Figure 2 According to an embodiment of the present application, a schematic diagram of a data quality management method of a data quality management system is shown;
[0027] Figure 3 According to an embodiment of the present application, a schematic diagram of a visual interface for setting data quality rules is shown;
[0028] Figure 4 According to an embodiment of the present application, a schematic diagram of another visual interface for setting data quality rules is shown;
[0029] Figure 5 According to an embodiment of the present application, a schematic diagram of a data quality management method of another data quality management system is shown;
[0030] Figure 6 According to an embodiment of the present application, a flow chart of a data processing method is shown;
[0031] Figure 7 According to an embodiment of the present application, a schematic diagram of a visual interface for a data quality management system to determine quality rules and corresponding repair rules and generate corresponding rule scripts is shown;
[0032] Figure 8 According to an embodiment of the present application, a structural diagram of a data quality management system is shown. DETAILED DESCRIPTION
[0033] The illustrative embodiments of the present application include, but are not limited to, a data processing method, a cloud platform system, a readable medium, and a program product.
[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and comprehensively described below with reference to the accompanying drawings.
[0035] First, the cloud platform system involved in some embodiments of the present application is introduced.
[0036] Cloud service providers connect a large number of resources via a network to form a cloud platform, which they then centrally manage, allocate, and schedule to provide cloud services to users. A cloud platform includes various cloud service instances, such as computing resources, storage resources, and applications. Essentially, a cloud service instance is a virtualization of various resources on the cloud platform into containers or virtual machines.
[0037] Specifically, cloud services can provide basic hardware resource services. They can also offer software resource services such as operating systems, dependency libraries, and databases. This means the cloud host also provides a software environment for building applications. Cloud services can also directly provide executable applications, known as cloud applications. Users can download the corresponding micro-client for cloud applications to run them, eliminating the need to download the application to a local device. A micro-client refers to a miniature or thin client.
[0038] For example, Figure 1 According to an embodiment of the present application, a schematic diagram of a cloud platform system is shown. Figure 1 As shown, users can obtain these containers or virtual machines on the cloud platform 20 through the network, and can use the corresponding computing resources, storage resources or application functions on the client device 10.
[0039] The client device 10 may be a mobile phone, a tablet computer, a laptop computer or the like.
[0040] It can be understood that the cloud platform 20 can provide some business systems, such as a data quality management system, for implementing corresponding business functions using corresponding computing resources, storage resources, etc. The above-mentioned business system can be a general-purpose physical server, such as an ARM server or an X86 server. The above-mentioned business system can also be a virtual machine (VM) implemented by network function virtualization (NFV) technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. The above-mentioned business system can also be a server cluster, wherein each server in the server cluster can be implemented by the above-mentioned physical server or the above-mentioned virtual machine.
[0041] The following describes the process by which the data quality management system manages and monitors the data of client devices.
[0042] It is understandable that during the process of data collection and processing, data quality problems such as missing data may inevitably arise due to factors such as transmission channels, computing services, storage services, and other quality factors.
[0043] Client device 10 uses a service provided by cloud platform 20, such as generating a user profile and a corresponding personalized recommendation strategy based on data from the client's use of the client device. Missing or inconsistent user usage data can lead to biased data processing results, which in turn can affect the generated user profile and the corresponding personalized recommendation strategy.
[0044] Therefore, the cloud platform 20 usually performs quality management and monitoring on the data through a data quality management system so that the data used by the cloud platform during data processing is qualified, thereby making better use of the data.
[0045] like Figure 2 As shown in the figure, the data quality management method of the data quality management system usually includes determining quality rules, conducting quality audits, issuing quality warnings, generating quality reports, rectifying quality problems and optimizing processes.
[0046] Specifically, quality rules are used to determine whether data meets requirements for accuracy, completeness, consistency, uniqueness, timeliness, and validity. These rules primarily include uniqueness rules, non-null rules, regularity rules, and conditional rules. Determining quality rules involves determining preset quality rules selected by the user or custom quality rules entered by the user.
[0047] It can be understood that the "user" in the embodiment of the present application is used to refer to the developer of the client device. Unless otherwise specified, the "user" below refers to the developer of the client device.
[0048] For example, Figure 3 According to an embodiment of the present application, a schematic diagram of a visual interface for setting quality rules is shown.
[0049] like Figure 3 As shown, the interface 20a includes a "Define Relationship" display area and an "Output Result Description" display area. The "Define Relationship" display area is used to display the data table input by the user, as well as the fields in the data table and the corresponding relationship expressions. The "Output Result Description" display area is used to display the description of the output result corresponding to the relationship expression in the "Define Relationship" display area.
[0050] For example, if the relationship expression displayed in the "Define Relationship" display area is: select max(${Column1},min{${Column2})from${Schema_Table1}, then the "Output Result Description" display area can display: maximum value and minimum value.
[0051] Or for example, the relationship expression displayed in the "Define Relationship" display area is: select min(${Column1},max{${Column1},ROUND(avg(${Column1},)2),sum{${Column1})from${Schema_Table1}, and the "Output Result Description" display area can display "minimum value, maximum value, average value, sum".
[0052] For example, Figure 4 According to an embodiment of the present application, a schematic diagram of another visual interface for setting quality rules is shown.
[0053] like Figure 4As shown, the interface 30a includes setting areas such as "Template Name", "Rule Field", "Sampling Method", "Verification Type", "Test Method", and "Custom Statement". Among them, the "Template Name" area is used to display the name of the rule template entered by the user; the "Rule Field" area is used to display the data field corresponding to the rule selected by the user, such as "User-defined Structured Query Language (user-defined SQL)"; the "Sampling Method" is used to display the data collection method corresponding to the field selected by the user, such as "Custom SQL"; the "Verification Type" is used to display the data type of the field, such as numeric; the "Verification Method" is used to display the method of verifying the data selected by the user, such as comparison with a fixed value; the "Custom SQL" is used to display the user-defined statement entered by the user; the "Target Folder" is used to display the storage location of the data selected by the user, such as the "Test" folder, etc.
[0054] Taking the determination of data accuracy as an example, first, the data management system determines that the quality rules set by the user are threshold rules, for example, the numerical range of the data needs to be within the preset range of 20 to 100. Then, the data management system performs a quality audit on the data to be managed based on the quality rules, that is, determines whether the data to be managed meets the quality rules. If there is problem data that does not meet the quality rules in the data to be managed, for example, there is data with a value of 500 in the data to be managed, which does not meet the threshold rules, a quality alarm is issued, such as sending a notification to the user to inform the user that the problem data that does not meet the conditions needs to be processed, and a quality report is generated. Among them, in the quality report, the problem data that does not meet the quality rules in the data to be managed can be identified, or processing opinions can be provided based on the quality rules, etc. The user can process the data to be managed based on the quality report, or optimize the quality rules based on the quality report.
[0055] As mentioned above, in the current data quality management method, when the data does not meet the quality rules, the data quality management system will issue an early warning message, and the user will manually process the data that does not meet the quality rules, which increases the user's work cost.
[0056] In view of this, an embodiment of the present application provides a data processing method. In the above-mentioned data quality management method, the logic for determining data repair rules is added to the link of determining quality rules. That is, the quality rules set by the user and the corresponding repair rules are determined. When the data does not meet the quality rules set by the user, the data management system can process the data that does not meet the quality rules according to the corresponding repair rules to make it meet the quality rules without the need for manual processing by the user. Among them, quality rules may include: non-empty rules, threshold rules, regular rules, conditional rules, etc. Repair rules may include: modification rules, deletion rules, correction rules, etc.
[0057] For example, a non-empty rule is used to determine whether each field in the data contains an empty value. The corresponding repair rule for a non-empty rule can be a delete rule, which is used to delete data with empty value fields. The corresponding repair rule for a non-empty rule can also be a correction rule, which is used to fill in empty value fields in the data so that there are no empty value fields in the data.
[0058] For example, a threshold rule is used to determine whether data is within a threshold range. The corresponding repair rule for the threshold rule can be a modification rule, which is used to modify data that is not within the threshold range so that it meets the threshold range condition. Alternatively, the corresponding repair rule for the threshold rule can also be a deletion rule, which is used to delete data that is not within the threshold range.
[0059] For example, a regularization rule is used to determine whether data meets a set regularization specification. The corresponding repair rule for the regularization rule can be a modification rule, which is used to modify data that does not meet the regularization specification so that it meets the regularization specification. Alternatively, the corresponding repair rule for the regularization rule can also be a deletion rule, which is used to delete data that does not meet the regularization specification.
[0060] For another example, a conditional rule is used to determine whether data meets set constraints. The corresponding repair rule for the conditional rule can be a modification rule, which is used to modify data that does not meet the constraints so that it meets the constraints. Alternatively, the corresponding repair rule for the conditional rule can also be a deletion rule, which is used to delete data that does not meet the constraints.
[0061] like Figure 5 As shown, the data processing method provided in the embodiment of the present application specifically includes determining quality rules and corresponding repair rules, performing data processing, performing quality audits, issuing quality alarms, generating quality reports, rectifying quality problems, and optimizing processes.
[0062] Taking the example of determining data accuracy, the data management system first determines a quality rule as a threshold rule. For example, the data value range must be within a preset range of 20 to 100. The corresponding remediation rule for the threshold rule can be a modification rule. For example, if the data is within the range of 100 to 150, the data value is subtracted by 50 to make it meet the threshold rule. Next, based on the modification rule, the problematic data in the managed data that does not meet the threshold rule is modified to meet the threshold rule. Then, a quality audit is performed on the managed data to determine whether any problematic data in the managed data still fails to meet the quality rule. If any problematic data in the managed data still fails to meet the quality rule, such as data with a value of 300 that fails to meet the threshold rule, a quality alert is issued, such as a notification sent to the developer informing them that the problematic data needs to be addressed. Furthermore, a quality report is generated. The quality report can identify the problematic data in the managed data that does not meet the quality rule, provide treatment suggestions based on the quality rule, or propose optimization suggestions for the remediation rule corresponding to the quality rule.
[0063] It can be understood that the data processing method provided in the embodiment of the present application, by determining the quality rules and corresponding repair rules of the data to be managed, when there is data that does not meet the quality rules in the data to be managed, the data that does not meet the quality rules is processed based on the corresponding repair rules to make it meet the quality rules, without the need for the user to manually process the data that does not meet the quality rules, which can effectively reduce the user's work cost.
[0064] In order to better understand the technical solutions of the embodiments of the present application, some technical solutions of the present application are introduced in detail below.
[0065] Figure 6 According to the embodiment of the present application, a flow chart of a data processing method is shown. It can be understood that Figure 6 The execution body of each step of the process shown is the data quality management system 100. Figure 6 The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 6 As shown, the process includes but is not limited to the following steps:
[0066] S601: Acquire data to be managed.
[0067] In some embodiments, upon receiving a data management request from a client device, the data quality management system 100 obtains the data to be managed (as an example of first data) from a data source.
[0068] The data to be managed can be business data stored in a cloud database or transmitted from a client device connected to the data quality management system 100 via a network. In other words, the data source can be a cloud database or the local storage of the client device. It is understood that this application does not specifically limit the method for obtaining the data to be managed.
[0069] S602: Determine at least one quality rule and a corresponding repair rule corresponding to the data to be managed.
[0070] In some embodiments, the data quality management system 100 determines a data management task corresponding to a data management request. The data management task includes a quality rule and a repair rule. In other words, the data quality management system 100 determines a quality rule selected and / or input by a user corresponding to at least one target field of the data to be managed and a repair rule corresponding to the quality rule.
[0071] Specifically, the quality rule includes but is not limited to the following parameters: the name of the quality rule corresponding to each target field, the corresponding rule template (ie, rule type), and the rule content. The repair rule includes but is not limited to determining the following parameters: the name of the repair rule and the corresponding processing content.
[0072] In some embodiments, the quality rules may include basic quality rules (as an instance of a first quality rule) pre-set by the data quality management system 100 and user-defined quality rules (as an instance of a second quality rule). The basic quality rules include, but are not limited to, uniqueness rules, non-empty rules, foreign key rules, value range rules, threshold rules, format rules, conditional rules, consistency rules, repeatability rules, and integrity rules. User-defined quality rules are set in the form of regular expressions or custom SQL. In other words, when the user determines the quality rules, he or she can create general basic quality rules or user-defined quality rules according to different business needs.
[0073] It can be understood that the uniqueness rule can mean that the data of the target field must be unique and non-repeatable. The non-empty rule can mean that the target field cannot be empty. The threshold rule can be used to check whether the data of the target field is within the threshold range. The regular rule can be used to check whether the data of the target field meets the set regular specifications. The conditional rule can be used to check whether the data of the target field that meets specific conditions meets the constraints, and the condition content can be manually entered or selected by the user. This application does not limit the specific content of each quality rule.
[0074] In some embodiments, the repair rules corresponding to the basic quality rules can be pre-set by the data quality management system 100 or set by the user; the repair rules corresponding to the user-defined quality rules can be set by the user for each quality rule.
[0075] For example, a non-empty rule is used to determine whether the data in the target field is null. The corresponding repair rule for a non-empty rule can be a delete rule, which is used to delete data in a field with null values. The corresponding repair rule for a non-empty rule can also be a correction rule, which is used to fill in null fields in the data so that there are no null fields in the data.
[0076] For example, a threshold rule is used to determine whether the data in a target field is within a threshold range. The corresponding repair rule for the threshold rule can be a modification rule, which is used to modify the data that is not within the threshold range so that it meets the threshold range condition. Alternatively, the corresponding repair rule for the threshold rule can be a deletion rule, which is used to delete the data that is not within the threshold range.
[0077] For example, a regularization rule is used to determine whether the data in a target field meets the specified regularization specifications. The corresponding repair rule for the regularization rule can be a modification rule, which is used to modify data that does not meet the regularization specifications so that it meets the regularization specifications. Alternatively, the corresponding repair rule for the regularization rule can be a deletion rule, which is used to delete data that does not meet the regularization specifications.
[0078] For example, a conditional rule is used to determine whether the data in a target field meets the set constraints. The corresponding repair rule for the conditional rule can be a modification rule, which is used to modify the data that does not meet the constraints so that it meets the constraints. Alternatively, the corresponding repair rule for the conditional rule can be a deletion rule, which is used to delete the data that does not meet the constraints.
[0079] It is understandable that this application does not limit the specific content of the repair rules corresponding to each quality rule.
[0080] In other embodiments, the data quality management system 100 can also determine the monitoring alarm rules determined by the user, so that when the repair rules cannot process all data that does not meet the quality rules, an alarm can still be issued for the data that cannot be processed by the repair rules and does not meet the quality rules.
[0081] S603: Generate a corresponding rule script based on the determined quality rules and the corresponding repair rules.
[0082] In some embodiments, the data quality management system 100 generates a rule script corresponding to each target field based on the quality rule and corresponding repair rule of at least one target field of the data to be managed. The rule script may be, for example, an SQL script.
[0083] For example, Figure 7 According to an embodiment of the present application, a schematic diagram of a visual interface for a data quality management system 100 to determine quality rules and corresponding repair rules and generate corresponding rule scripts is shown.
[0084] like Figure 7 As shown, the visualization interface 70a includes a "verification logic" display area, a "monitoring alarm" display area, a "configuration repair mapping" display area, and a "rule script" display area.
[0085] The "verification logic" display area includes a field selection area, a rule name input area, a rule template selection area, a regular expression input area, and a temporary rule selection area. It can be understood that the "verification logic" display area in the visual interface 70a is used to display user-defined regular rules.
[0086] The "Monitoring Alarm" area includes the alarm threshold input area and the failure threshold input area.
[0087] The "Configure Repair Map" display area includes a default repair value selection area and a custom modification rule setting area. It can be understood that the "Configure Repair Map" display area in the visualization interface 70a is used to display the modification rules determined by the user.
[0088] The “Rule Script” display area generates a rule script corresponding to the target field in real time based on the various parameters of the quality rule determined by the user, as well as the monitoring alarm rule and the repair rule, and displays it in the visual interface 70a.
[0089] For example, the data quality management system 100 determines that the target field determined by the user is "survey_id", the quality rule is a regular rule, and the regular expression is "^(CS)(\d){8}$", and the modification rule is the default repair value "CS111". The corresponding rule script is:
[0090] select
[0091] case
[0092] when survey_id not rlike'^(CS)(\d){8}$'
[0093] then CS111
[0094] end
[0095] After the user clicks the confirmation control in the visual interface 70 a , the data quality management system 100 completes the creation of the rule script.
[0096] For example, the generation of rule scripts corresponding to some types of rule templates can refer to the following methods:
[0097] "Value comparison" type (e.g. compare with 2, the default fix value is 2):
[0098] Satisfies the conditions: fd is not null and (fd = 2)
[0099] The condition is not met: fd is null or not (fd = 2)
[0100] Rule script: case when fd is null or not(fd=2)then'2'else fd end
[0101] "Enumeration value" type (for example, if there are 14 enumeration values, the default fix value is 14):
[0102] Conditions are met: fd is not null and fd in ('14')
[0103] Condition not met: fd is null or fd not in ('14')
[0104] Rule script: case when fd is null or fd not in('14')then'14'else fd end
[0105] "Outside enumeration value" type (for example, if the enumeration has more than 15 values, the default fix value is 15):
[0106] Satisfies the condition: fd is null or fd not in('15')
[0107] The conditions are not met: fd is not null and fd in ('15')
[0108] Rule script: case when fd is not null and fd in('15')then'15'else fd end
[0109] "Regular" type (for example, the regular expression is '[1-9][0-9]{5}$', and the default repair value is 20):
[0110] Satisfies the conditions: fd is not null and fd rlike'[1-9][0-9]{5}$'
[0111] Conditions not met: fd is null or fd not rlike'[1-9][0-9]{5}$'
[0112] Rule script: case when fd is null or fd not rlike'[1-9][0-9]{5}$'then'20'else fd end
[0113] "Not meeting regularity" type (for example, the regular expression is '[1-9][0-9]{5}$', and the default repair value is 21):
[0114] Satisfies the following conditions: fd is null or fd not rlike'[1-9][0-9]{5}$'
[0115] The conditions are not met: fd is not null and fd rlike'[1-9][0-9]{5}$'
[0116] Rule script: case when fd is not null and fd rlike'[1-9][0-9]{5}$'then'21'else fd end
[0117] "In-range" type (for example, greater than or equal to 6 and less than or equal to 16, and greater than or equal to 7 and less than or equal to 17, the default repair value is 17):
[0118] Meet the conditions: (fd is not null and fd>=6and<=16)AND (fd is not null and fd>=7and<=17)
[0119] The conditions are not met: (fd is null or fd not(fd>=6and<=16))OR(fd is null or fdnot(fd>=7and<=17))
[0120] Rule script: case when(fd is null or fd not(fd>=6and<=16))OR(fd is nullor fd not(fd>=7and)
[0121] <=17))then'17'else fd end
[0122] "Out of range" type (for example, outside the range of greater than or equal to 6 and less than or equal to 16, the default repair value is 16):
[0123] Satisfy the conditions: fd is null or fd not (fd>=6and<=16
[0124] The conditions are not met: fd is not null and fd>=6and<=16
[0125] Rule script: case when fd is not null and fd>=6and<=16then'16'else fdend
[0126] "Custom expression" type (for example, if a>22, the default repair value is 22):
[0127] Satisfies the conditions: fd is not null and (a>22)
[0128] The condition is not met: fd is null or not (a>22)
[0129] Rule script: case when fd is null or not(a>22)then'22'else fd end
[0130] "Null Value" type (default fix value is 100):
[0131] Conditions met: fd is not null
[0132] Condition not met: fd is null
[0133] Rule script: case when fd is null then '100' else fd end
[0134] S604: Generate a task script corresponding to the data to be managed based on a rule script of at least one target field of the data to be managed, and run the task script to complete quality management of the data to be managed.
[0135] In some embodiments, the data quality management system 100 executes task scripts to complete quality audits, data repairs, and supervisory alerts for the data to be managed. Task scripts are task scripts for data management tasks (or data monitoring tasks) corresponding to the data to be managed.
[0136] Specifically, corresponding to the problem data (serving as the data to be processed) that does not meet the corresponding quality rules in the target field of the data to be managed, the problem data is managed accordingly.
[0137] In some embodiments, corresponding to a first problematic sub-data (serving as an instance of the first sub-data to be processed) that does not satisfy a corresponding quality rule and meets a matching condition (or trigger condition) of a repair rule in a target field of the data to be managed, the first problematic sub-data is processed based on the repair rule to satisfy the corresponding quality rule. The processed data (serving as an instance of the second data) is then sent to the client device. The processing of the first problematic sub-data based on the repair rule includes, but is not limited to, modification, deletion, and correction.
[0138] In other embodiments, corresponding to the target field in the data to be managed, there is second problem sub-data that does not meet the corresponding quality rules and does not meet the matching conditions of the repair rules. The second problem sub-data is saved and an alarm message is sent to the client device based on the monitoring alarm rules.
[0139] In other embodiments, the data quality management system 100 may also perform statistics and analysis on problem data found during the execution of the task script, and analyze the causes of the data quality problems based on the statistical results to generate a quality report.
[0140] It is understood that the data quality management system 100 generates a task script corresponding to the data to be managed based on the rule script of at least one target field of the data to be managed. In other words, a task script can correspond to multiple rules. As a result, the data quality management system 100 can implement concurrent calculation of multiple rules for a single task, thereby improving resource utilization, reducing operating costs, and increasing the efficiency of quality rule monitoring calculations.
[0141] It is understood that in other embodiments, according to actual needs, the above Figure 6 The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S602 and S603 can be combined into one step. This application does not impose any restrictions on this.
[0142] To sum up, the data processing method provided in the embodiment of the present application selects problem data that does not meet the quality rules from the data to be managed, processes the problem data using repair rules corresponding to the problem data, and sends the processed data to the client device without the user having to manually process the data that does not meet the quality rules, thereby effectively reducing the user's work costs.
[0143] Moreover, by determining the rule scripts corresponding to multiple target fields of the data to be managed, and then merging the multiple rule scripts to generate a complete task script corresponding to the data management task of the data to be managed, concurrent calculation of multiple rules for a single task can be achieved, thereby improving resource utilization, reducing operating costs, and improving the efficiency of quality rule monitoring calculations.
[0144] In an exemplary embodiment, the present application also proposes a data quality management system 100 .
[0145] For example, Figure 8 According to an embodiment of the present application, a structural diagram of a data quality management system 100 is shown.
[0146] like Figure 8 As shown, the data quality management system 100 includes: a rule management module 110, a task management module 120, a monitoring management module 130, a quality reporting module 140, and an interface module 150. Detailed descriptions of each module are as follows.
[0147] The rule management module 110 is used to determine quality rules and corresponding repair rules. Among them, the quality rules may include basic quality rules pre-set by the rule management module 110 and user-defined quality rules. Basic quality rules include but are not limited to uniqueness rules, non-empty rules, foreign key rules, value range rules, threshold rules, format rules, conditional rules, consistency rules, repeatability rules and integrity rules. User-defined quality rules are set in the form of regular expressions or custom SQL statements. In some embodiments, the rule management module 110 is also used to manage quality rules and import and export quality rules.
[0148] The task management module 120 provides full lifecycle management for data monitoring tasks, including querying, creating, modifying, deleting, scheduling, terminating, pausing, and manually executing data monitoring tasks. It also includes a data quality management engine. When creating a data monitoring task (task script), the data source and corresponding quality and repair rules must be configured, and then executed in the data quality management engine.
[0149] The monitoring management module 130 includes the functions of data repair management and alarm management. In some embodiments, during the execution of a data monitoring task, when problem data that does not meet the quality rules is found, such as incomplete data, incorrect format, etc., and the problem data meets the matching conditions of the corresponding repair rules, the monitoring management module 130 processes the problem data based on the repair rules to make it meet the quality rules. In other embodiments, during the execution of a data monitoring task, when problem data that does not meet the quality rules is found, and the problem data does not meet the matching conditions of the corresponding repair rules, the monitoring management module 130 saves the problem data and issues an alarm to the user, and the user chooses different ways to process the problem data.
[0150] The quality reporting module 140 is used to collect statistics and analyze problematic data discovered during the execution of data monitoring tasks. Based on the statistical results, it analyzes the causes of data quality issues and generates quality reports to further improve data quality, standardize data quality management, and enhance data application capabilities. In some embodiments, the data statistical analysis results are categorized by different data monitoring tasks, such as filtering and categorizing existing tasks and rules, allowing users to view specific data quality information for each data monitoring task individually.
[0151] The interface module 150 is used to provide a series of internal and external interfaces to realize data transmission and reception, as well as collaborative operations with other systems, such as collaborative operations with the resource management system of the cloud platform system.
[0152] It should be noted that the implementation of each module can also refer to Figure 6 The corresponding description of the method embodiment shown executes the method and functions performed by the data quality management system 100 in the above embodiment.
[0153] An embodiment of the present application further provides a readable medium having instructions stored thereon, which, when executed in an electronic device, enables the electronic device to implement the data processing method of the embodiment of the present application.
[0154] An embodiment of the present application further provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device; and a processor for implementing the data processing method of an embodiment of the present application. It is understood that the electronic device can be the client device 10 described above, a server, or other electronic device, and this application does not limit this.
[0155] An embodiment of the present application further provides a program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the data processing method of the embodiment of the present application.
[0156] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.
[0157] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
[0158] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.
[0159] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.
[0160] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.
[0161] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.
Claims
1. A data processing method, characterized in that: Used in electronic equipment, including: receiving a data management request from a client device, and obtaining first data corresponding to the data management request; Selecting data to be processed that does not meet the quality rules from the first data; Adopting a repair rule corresponding to the data to be processed, processing the data to be processed to obtain second data; The second data is sent to the client device.
2. The method according to claim 1, characterized in that The obtaining the first data corresponding to the data management request includes: Acquire the first data from the local storage of the client device based on the data management request; or The first data is obtained from a cloud database corresponding to the client device based on the data management request.
3. The method according to claim 1, characterized in that The selecting the data to be processed that does not meet the quality rule from the first data includes: Determining a data management task corresponding to the data management request, wherein the data management task includes a management rule corresponding to at least one target field of the first data, and the management rule includes a quality rule and a repair rule; Based on a quality rule corresponding to at least one target field of the first data, data to be processed that does not meet the quality rule is selected from the first data.
4. The method according to claim 3, characterized in that The quality rules include first quality rules and / or second quality rules; wherein, The first quality rule includes at least one of the following: a uniqueness rule, a non-empty rule, a value range rule, a threshold rule, a format rule, a condition rule, a consistency rule, a repeatability rule, and an integrity rule; The second quality rule includes at least one of the following parameters: rule name, rule type, and rule content; wherein the expression form of the rule content includes at least one of the following: regular expression and user-defined statement.
5. The method according to claim 3, characterized in that The repair rule includes at least one of the following: a modification rule, a deletion rule, and a correction rule.
6. The method according to claim 3, characterized in that The determining the data management task corresponding to the data management request includes: generating a rule script corresponding to the management rule corresponding to each target field based on the quality rule and the repair rule corresponding to each target field of the first data; Based on at least one of the rule scripts, a task script corresponding to the data management task is generated.
7. The method according to claim 3, characterized in that The adopting a repair rule corresponding to the data to be processed to process the data to be processed to obtain second data includes: Based on the repair rule, performing at least one of the following processing on the first sub-data to be processed in the data to be processed that meets the triggering condition of the repair rule: modifying, deleting, and repairing; The processed sub-data to be processed is used as the second data.
8. The method according to claim 7, characterized in that The management rules also include monitoring alarm rules, and, Corresponding to the presence of second to-be-processed sub-data in the to-be-processed data that does not satisfy the triggering condition of the repair rule, an alarm message is sent to the client device based on the monitoring alarm rule.
9. A cloud platform system, characterized in that: include: A client device for sending data management requests; a data quality management system, configured to receive the data management request, obtain first data corresponding to the data management request, and select data to be processed that does not meet quality rules from the first data; The data quality management system is further configured to adopt a repair rule corresponding to the data to be processed, process the data to be processed, obtain second data, and send the second data; The client device is further configured to receive the second data.
10. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 8.
11. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and, A processor, configured to execute the method according to any one of claims 1 to 8.
12. A program product, characterized in that The program product includes instructions, and when the instructions are executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 8.