Data quality management method and device

Abstract

The embodiment of the invention provides a data quality management method and device, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining to-be-verified target data; querying whether a verification result corresponding to the target data is stored in a verification cache library or not; under the condition that the verification result corresponding to the target data is not stored in the verification cache library, querying a rule corresponding to the target data in a rule library of the electronic equipment to obtain a rule group corresponding to the target data; the rules in the rule base comprise rules obtained based on historical data training and rules obtained from other data platforms; adjusting rule parameters of each rule in the rule group according to the target data; verifying the target data based on each rule in the adjusted rule group to obtain a verification result of the target data; and executing a quality management operation corresponding to the verification result on the target data. The method can be applied to scenes such as financial science and technology and artificial intelligence, and the efficiency of the data quality management method can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a data quality management method and device. BACKGROUND

[0002] With the deepening of financial informatization, the timeliness and accuracy requirements of regulatory authorities on data quality compliance are increasingly stringent, and higher requirements are put forward for data quality management of financial institutions.

[0003] At present, a large number of data quality management rules involved by financial institutions exist in data platforms. Taking the "EAST Data Quality 'Promotion Project'" as an example, regulatory documents require financial institutions to complete full-quantity data audit and repair defects within several weeks. The existing method often has long rule deployment chain and long verification period, resulting in low efficiency of data quality management of financial institutions. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a data quality management method and device, aiming to improve the efficiency of the data quality management method.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a data quality management method, which comprises: acquiring target data to be verified;

[0006] querying whether the verification result corresponding to the target data is stored in the verification cache library;

[0007] In the case that the verification result corresponding to the target data is not stored in the verification cache library, querying the rule corresponding to the target data in the rule library of the electronic device to obtain the rule group corresponding to the target data; wherein the rules in the rule library include rules obtained by the electronic device based on historical data training, and rules obtained by the electronic device from other data platforms;

[0008] According to the target data, adjusting the rule parameters of each rule in the rule group;

[0009] verifying the target data based on each rule in the adjusted rule group to obtain the verification result of the target data;

[0010] Performing quality management operations corresponding to the verification result on the target data.

[0011] In some embodiments, the acquiring target data to be verified comprises at least one of the following:

[0012] determining the first data generated by the electronic device as the target data to be verified;

[0013] In a case where a preset quality management time is reached, the second data that is not verified by the electronic device is determined as the target data to be verified.

[0014] The data of the electronic device is monitored by an anomaly detection algorithm, and third data that is monitored as being abnormal is determined as the target data to be verified.

[0015] In some embodiments, the rule base includes a plurality of rules, and each rule corresponds to a first identity tag and a first function tag;

[0016] The rule corresponding to the target data is queried in the rule base of the electronic device, and a rule group corresponding to the target data is obtained.

[0017] A second identity tag and a second function tag of the target data are obtained, wherein the second identity tag is used to identify the application scenario and the influence range of the target data, and the second function tag is used to identify the function to be verified of the target data.

[0018] Based on the second identity tag, the first identity tag is matched in the rule base to obtain a first rule.

[0019] Based on the second function tag, the first function tag is matched in the first rule to obtain the rule group corresponding to the target data.

[0020] In some embodiments, the rule parameters of each rule in the rule group are adjusted according to the target data, comprising:

[0021] The business characteristics of the target data are obtained.

[0022] The business characteristics and each rule in the rule group are input into a first model, and the rule parameters of each rule in the rule group are adjusted based on the business characteristics by the first model, to obtain each rule in the rule group after the rule parameters are adjusted.

[0023] In some embodiments, the target data is verified based on each rule in the rule group after the adjustment, comprising:

[0024] The dependency relationship and the priority of each rule in the rule group are obtained.

[0025] According to the load and resource state of the electronic device, the execution frequency and the priority of each rule in the rule group are adjusted.

[0026] After each rule in the rule group is filtered by the execution frequency, the target data is verified by calling each of the filtered rules in the rule group in parallel through a distributed computing framework according to the priority and the dependency relationship.

[0027] In some embodiments, performing the quality management operation on the target data corresponding to the verification result includes:

[0028] If the verification result is satisfactory, the target data is released.

[0029] If the verification result is unsuccessful, a warning notification will be sent to the user.

[0030] In some embodiments, the method further includes:

[0031] The second model reads compliance requirements from regulatory documents and generates the first new rule for the electronic device.

[0032] The third model generates a second set of new rules for the electronic device based on the regulatory requirements of different regions.

[0033] Combine the first new rule and the second new rule to obtain the combined new rule;

[0034] The newly added rules after combination are merged into the rule base, and the updates of the rule base are recorded.

[0035] In some embodiments, the method further includes:

[0036] Based on the visual interface of the electronic device, a third new rule of the electronic device is generated by the user triggering the movement operation of the visual interface;

[0037] The visualization interface simulates and displays the verification results of the target data of the electronic device by the third newly added rule.

[0038] Based on the visual interface of the electronic device, the user adjusts the rule parameters of each rule in the rule group to obtain the adjusted rule parameters;

[0039] The adjusted rule parameters are updated in the rule, and the updated rule is stored in the rule base.

[0040] In some embodiments, the method further includes:

[0041] Obtain the tenant tag of the electronic device;

[0042] Based on the tenant tag, determine the private rule base corresponding to the tenant in the rule base;

[0043] obtaining a visibility label of the rule in the private rule library; wherein the visibility label is used to define access authority of the rule between tenants;

[0044] determining a shared rule library of the electronic device based on the visibility label; wherein the shared rule library is used to check the target data.

[0045] To achieve the above object, a second aspect of the embodiment of the present application provides a data quality management device, the device comprising:

[0046] an obtaining module, configured to obtain target data to be checked;

[0047] a querying module, configured to query whether a check result corresponding to the target data is stored in a check cache library;

[0048] a first obtaining module, configured to query a rule corresponding to the target data in a rule library of an electronic device in a case where the check cache library does not store the check result corresponding to the target data, and obtain a rule group corresponding to the target data; wherein the rules in the rule library include rules trained by the electronic device based on historical data and rules obtained by the electronic device from other data platforms;

[0049] an adjusting module, configured to adjust rule parameters of each rule in the rule group according to the target data;

[0050] a second obtaining module, configured to check the target data based on each rule in the adjusted rule group, and obtain a check result of the target data;

[0051] an executing module, configured to perform a quality management operation corresponding to the check result on the target data.

[0052] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect when executing the computer program.

[0053] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program being executed by a processor to implement the method of the first aspect.

[0054] To achieve the above object, a fifth aspect of the embodiment of the present application provides a computer program product, the program product being stored in a storage medium, and the program product being executed by at least one processor to implement the method of the first aspect.

[0055] The data quality management method, device, equipment, medium and product provided by the application obtain target data to be verified; query whether the verification cache library stores a verification result corresponding to the target data; in the case that the verification cache library does not store the verification result corresponding to the target data, query the rule corresponding to the target data in the rule library of the electronic device to obtain a rule group corresponding to the target data; wherein the rules in the rule library include rules trained by the electronic device based on historical data, and rules obtained by the electronic device from other data platforms; adjust the rule parameters of each rule in the rule group according to the target data; verify the target data based on each rule in the adjusted rule group to obtain a verification result of the target data; and perform a quality management operation corresponding to the verification result on the target data. The application can realize the reuse of historical verification results by querying the verification cache library, alleviate the problem of long verification period, and then trigger data verification of the target data only when the cache is not hit, automatically perform the corresponding quality management operation without manual intervention, and shorten the deployment chain. At the same time, the rule library of the electronic device includes the rules of other data platforms, which can centralize the rules scattered in multiple data platforms and further improve the data quality management efficiency of the electronic device. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is an optional flowchart of the data quality management method provided by the application embodiment;

[0057] Figure 2 is a flowchart of step S101 in Figure 1

[0058] Figure 3 is a flowchart of step S103 in Figure 1

[0059] Figure 4 is a flowchart of step S104 in Figure 1

[0060] Figure 5 is a flowchart of step S105 in Figure 1

[0061] Figure 6 is a flowchart of step S106 in Figure 1

[0062] Figure 7 is another optional flowchart of the data quality management method provided by the application embodiment;

[0063] Figure 8 is another optional flowchart of the data quality management method provided by the application embodiment; ​​​​​

[0064] Figure 9 is another optional flowchart of the data quality management method provided by the embodiments of the present application;

[0065] Figure 10 is a schematic diagram of synchronizing the rule table to multiple data platforms provided by the embodiments of the present application;

[0066] Figure 11 is a visual interface diagram of adding a rule provided by the embodiments of the present application;

[0067] Figure 12 is an interface diagram of adjusting a rule parameter provided by the embodiments of the present application;

[0068] Figure 13 is a rule group detail interface diagram provided by the embodiments of the present application;

[0069] Figure 14 is an interface diagram of executing rule group information and executing data details provided by the embodiments of the present application;

[0070] Figure 15 is a structural schematic diagram of the data quality management device provided by the embodiments of the present application;

[0071] Figure 16 is a hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0072] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0073] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0075] First, the several terms involved in the present application are analyzed:

[0076] Artificial intelligence (AI): is a new technical science of studying, developing the theory, method, technology and application system for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, the research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0077] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.), NLP is a branch of artificial intelligence, and is a cross discipline of computer science and linguistics, and is often referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0078] Information extraction (Information Extraction): a text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, chapters, and text information is composed of small specific units, such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, places, etc. from text data is text information extraction, of course, the information extracted by text information extraction technology can be various types of information.

[0079] The insurance system, the bank system, the transaction system, the order system and the like are core business systems in the financial institutions, each of which bears different functions: the insurance system is responsible for policy management, claim settlement, renewal and actuarial, and generates data such as policy information, claim settlement records, premium water; the bank system covers account management, deposit and loan, payment clearing and risk control, and generates data such as account balance, transaction water, loan contract; the transaction system handles the matching, market and clearing business of securities, funds and foreign exchange, and outputs data such as transaction return, order form, position information; the order system faces e-commerce or financial products, completes the ordering, canceling and reconciliation process, and generates data such as order status, payment voucher, commodity or product details.

[0080] With the deepening of financial informatization, the timeliness and accuracy requirements of the regulatory authorities on data quality compliance are increasingly stringent, and higher requirements are put forward for the data quality management of the financial institutions.

[0081] At present, a large number of data quality management rules involved by the financial institutions exist in the data platform. Taking the EAST data quality "improvement project" as an example, the regulatory documents require the financial institutions to complete the full data audit and repair defects within several weeks. The existing method often has long rule deployment chain and long verification period, resulting in low efficiency of data quality management of the financial institutions.

[0082] Based on this, the embodiment of the present application provides a data quality management method and device, aiming to improve the efficiency of the data quality management method.

[0083] The data quality management method and device provided by the embodiment of the present application are specifically explained by the following embodiments, and first, the data quality management method in the embodiment of the present application is described.

[0084] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0085] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0086] The data quality management method provided in the embodiments of the present application relates to the technical field of artificial intelligence. The data quality management method provided in the embodiments of the present application can be applied to an electronic device, and further can be applied to software of the electronic device. The electronic device can be a terminal or a server. In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, etc. The server end can be configured as a stand-alone physical server, or can be configured as a server cluster or a distributed system formed by multiple physical servers, or can be configured as a cloud server for providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The software can be an application for implementing the data quality management method, but is not limited to the above forms.

[0087] The present application can be applied in numerous general purpose or special purpose computer system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0088] It should be noted that in each of the specific embodiments of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0089] The present application can be applied in the fields of financial technology and artificial intelligence, and the present application is not limited to specific application scenarios.

[0090] Figure 1is an optional flowchart of a data quality management method provided by the embodiment of the application, and the method is applied to an electronic device. Figure 1 The method in the embodiment can include, but is not limited to, steps S101 to S106.

[0091] In step S101, target data to be verified is obtained.

[0092] In step S101 of some embodiments, the service record or data block that needs to be quality verified is obtained by means of real-time stream access, periodic extraction or exception triggering.

[0093] In some embodiments, the real-time stream and the offline batch file can be combined into the same batch.

[0094] In other embodiments, the target data can be prioritized, and the high-priority service data can be inserted.

[0095] For example, the bank core system 2024-7-28 10:00:00 generates 500,000 transfer transaction records; the scheduler takes the incremental transaction records from 10:00 to 10:05 as a "target data".

[0096] In step S102, it is determined whether the verification result corresponding to the target data is stored in the verification cache library.

[0097] In step S102 of some embodiments, it is determined whether the target data has a corresponding verification result in the verification cache library. The verification cache library stores the verification results of historical target data.

[0098] In step S103, when the verification cache library does not store the verification result corresponding to the target data, the rule corresponding to the target data is queried in the rule library of the electronic device to obtain the rule group corresponding to the target data; wherein the rules in the rule library include the rules obtained by the electronic device based on historical data and the rules obtained by the electronic device from other data platforms.

[0099] In step S103 of some embodiments, when the target data does not have a corresponding verification result in the verification cache library, the rule group is queried in the rule library of the electronic device.

[0100] It should be noted that the rule library can be a unified metadata warehouse, which can include rule scripts, identity tags, function tags, version numbers, tenant tags and visibility tags. The rule group can be a collection of multiple rules required by the verification task, or it can be a rule.

[0101] In some embodiments, the identity tag and the function tag can be used for retrieval to obtain the rule group.

[0102] In some embodiments, the rule set is obtained by full-text search plus tag joint search.

[0103] Step S104, adjusting the rule parameters of each rule in the rule set according to the target data;

[0104] In some embodiments of step S104, the rule parameters are adjusted by the target data.

[0105] It should be noted that the rule parameters can be thresholds, window sizes, filtering conditions, and other variables.

[0106] In some embodiments, business features (such as null rate, distribution mean) are extracted, and the business features are input into a target model (such as XGBoost regression), and the target model outputs new rule parameters. The new rule parameters are written into the rules.

[0107] In some embodiments, the rule parameters can be adjusted by manual review.

[0108] For example, the original threshold of the date reasonableness rule is ±30 days, and the model is based on 99% of the transactions in the last 7 days falling within ±7 days, so the threshold of the date reasonableness rule is automatically reduced to ±10 days.

[0109] Step S105, verifying the target data based on each rule in the adjusted rule set, and obtaining a verification result of the target data;

[0110] In some embodiments of step S105, the target data is verified using the adjusted rules, and the result is obtained.

[0111] In some embodiments, the rules and their dependencies are modeled as a directed acyclic graph, and a scheduler collects CPU and memory indicators in real time, calculates the concurrency degree for parallel nodes, reduces concurrency in high load, and improves concurrency in low load. Nodes without dependencies are run in parallel; nodes with dependencies are run in series.

[0112] In some embodiments, GPU / FPGA can be used for acceleration.

[0113] Step S106, performing a quality management operation corresponding to the verification result on the target data.

[0114] In some embodiments of step S106, a quality management operation is performed, wherein the quality management operation can include release, alarm, block, and automatic repair.

[0115] In some embodiments, the test result is passed to the downstream table. The test result is not passed to trigger an SMS / email / WeChat robot, and the abnormal details are attached.

[0116] In other embodiments, the quality management operation can be manually involved.

[0117] The steps S101 to S106 shown in the embodiments of the present application are as follows: obtaining target data to be verified; querying whether the verification cache library stores a verification result corresponding to the target data; in the case that the verification cache library does not store the verification result corresponding to the target data, querying a rule corresponding to the target data in a rule library of the electronic device to obtain a rule group corresponding to the target data; wherein the rules in the rule library include rules trained by the electronic device based on historical data and rules obtained by the electronic device from other data platforms; adjusting rule parameters of each rule in the rule group according to the target data; verifying the target data based on each rule in the adjusted rule group to obtain a verification result of the target data; and performing a quality management operation corresponding to the verification result on the target data. The present application can realize the reuse of historical verification results by querying the verification cache library, alleviate the problem of long verification period, and then trigger data verification of the target data only when the cache is not hit, automatically perform the corresponding quality management operation without manual intervention, and shorten the deployment chain. At the same time, the rule library of the electronic device includes rules of other data platforms, which can centralize the rules scattered in multiple data platforms and further improve the data quality management efficiency of the electronic device.

[0118] Please refer to Figure 2 In some embodiments, step S101 can include but is not limited to steps S201 to S203:

[0119] Step S201: determining first data generated by the electronic device as the target data to be verified;

[0120] In step S201 of some embodiments, real-time data generated by the electronic device is used as the data to be verified.

[0121] It should be noted that the first data can be newly added or changed records generated in real time during the operation of the business system, such as transfer flow, policy issuance, and order placement.

[0122] Step S202: determining second data not verified by the electronic device as the target data to be verified when a preset quality management time is reached;

[0123] In step S202 of some embodiments, historical or incremental data blocks that have been stored in the database but have not been verified are used as the data to be verified under the premise of a period defined by the scheduler.

[0124] Step S203, monitoring the data of the electronic device by an anomaly detection algorithm, and determining third data monitored as abnormal as the target data to be verified.

[0125] In step S203 of some embodiments, by the anomaly detection algorithm, data with an algorithm output anomaly score greater than a threshold value is determined as the data to be verified.

[0126] In the embodiments of the present application, through real-time data, a zero-delay closed loop of "data generation and verification" is realized to prevent the spread of dirty data. Through periodic batch processing, bottom-up verification can be ensured to guarantee that historical or offline data also meet the quality standards. Through anomaly triggering, potential data quality problems are actively discovered to reduce post-repair costs.

[0127] Please refer to Figure 3 In some embodiments, the rule library includes a plurality of rules, and a first identity tag and a first function tag corresponding to each rule; and step S103 can include but is not limited to steps S301 to S304:

[0128] Step S301, obtaining a second identity tag and a second function tag of the target data; wherein the second identity tag is used to identify the application scenario and the impact range of the target data; and the second function tag is used to identify the function to be verified of the target data.

[0129] In step S301 of some embodiments, the identity tag and the function tag are automatically generated according to the data content or the metadata before the data enters the verification link.

[0130] It should be noted that the identity tag can be an enumeration key composed of "application scenario + impact range". The application scenario can include compliance, accuracy, consistency, and timeliness. The impact range can include the institutional level, the business line level, the channel level, and the customer level.

[0131] The function tag can be a business dimension described by the rule, such as "date reasonableness", "data integrity", "correlation verification", "value range verification", etc.

[0132] Step S302, matching the first identity tag in the rule library based on the second identity tag to obtain a first rule;

[0133] In step S302 of some embodiments, all rule sets (first rules) satisfying the second identity tag are queried, wherein the condition is that the first identity tag and the second identity tag correspond to the same.

[0134] Step S303, matching the first function tag in the first rule based on the second function tag to obtain a rule group corresponding to the target data.

[0135] In step S303 of some embodiments, the first rule is matched again for the function label to obtain a rule related to the target data service dimension.

[0136] In the embodiments of the present application, the two-level index of the identity label coarse screening and the function label fine screening can complete rule matching, ensure accurate and efficient verification, and improve rule management efficiency.

[0137] Please refer to Figure 4 In some embodiments, step S104 can include but is not limited to steps S401 to S403:

[0138] In step S401, the service characteristics of the target data are obtained.

[0139] In step S401 of some embodiments, the service characteristics of the data are extracted in real time or offline in batches.

[0140] It should be noted that the service characteristics can be a set of quantitative indicators capable of describing the distribution, quality risk and compliance sensitivity of the target data. The service characteristics can include null rate, distribution quantile, time span, etc.

[0141] In step S402, the service characteristics and each rule in the rule group are input into a first model, the rule parameters of each rule in the rule group are adjusted based on the service characteristics by the first model, and each rule in the rule group after adjustment of the rule parameters is obtained.

[0142] In step S402 of some embodiments, the "adjustable parameter range" declared in the rule metadata is called to prevent the first model from outputting out-of-range values. The new parameters of each rule are calculated by the first model based on the service characteristics within the adjustable parameter range. The parameter change record (old value, new value, change reason, timestamp) is generated and written into the version library. The new parameters are hot-updated to the rule instance and written back to the cache. In addition, if performance or false alarm abnormalities are detected in subsequent batches, the system can be rolled back to the previous version in seconds.

[0143] It should be noted that the first model can be an online or quasi-real-time parameter tuner.

[0144] In some embodiments, the verification result can be fed back to the model in real time after the verification result is obtained, to realize closed-loop learning.

[0145] In some other embodiments, the new threshold value can be used only for the flow of the preset threshold value, and the monitoring index is monitored before being put into full production.

[0146] The embodiments of the present application provide high-fidelity input for parameter adjustment model through acquisition of service features, to avoid false positives / false negatives caused by fixed threshold. The threshold range of the verification is automatically adjusted under the premise of ensuring the compliance bottom line, to improve the regulatory pass rate.

[0147] Please refer to Figure 5 In some embodiments, step S105 can further include, but is not limited to, steps S501-S502:

[0148] Step S501: acquiring the dependency relationship and priority of each rule in the rule group;

[0149] It should be noted that the dependency relationship can be an edge in a directed acyclic graph, indicating a precondition that the rule must be executed first, such as table-in verification first and then inter-table verification. The priority can be the weight of the rule in the queue.

[0150] Step S502: adjusting the execution frequency of each rule in the rule group and the priority according to the load and resource state of the electronic device;

[0151] In step S502 of some embodiments, the load and resource state of the electronic device are collected, and the execution frequency and priority are adjusted according to the load and resource state.

[0152] It should be noted that the load can be CPU usage, memory occupancy, network I / O throughput, disk I / O latency, etc. The resource state can be the number of remaining available CPU cores, the amount of remaining available memory, the number of current concurrent task slots of the cluster, and the node health state (normal / degraded / faulty), etc.

[0153] Step S503: after each rule in the rule group is filtered by the execution frequency, each filtered rule in the rule group is called in parallel according to the priority and the dependency relationship through a distributed computing framework, to verify the target data.

[0154] In step S503 of some embodiments, when the load and resource state are good, the frequency is maintained and improved; otherwise, the frequency is reduced, delayed, or even skipped. The execution order of the rules filtered by the frequency is determined according to the priority, and the priority is adjusted in real time according to the resource state, and then the rules are executed according to the dependency relationship in the rules. The rules are executed in parallel through a distributed computing framework. For rules of the same priority, the execution order of the rules is determined according to the amount of resource state used by the rules.

[0155] The embodiments of the present application can ensure correct execution order, avoid conflicts and redundancies in rule execution through dependency relationship and priority. According to load and resource state adjustment, high-priority tasks can be prioritized to avoid cluster collapse when resources are tight. Through parallel execution, the throughput can be maximized under the premise of ensuring correct order, and the overall verification time can be shortened.

[0156] Referring to Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S602:

[0157] Step S601, if the verification result is passed, the target data is released;

[0158] Step S602, if the verification result is not passed, a warning notification is sent to the user.

[0159] It should be noted that the warning notification can be reached through multiple channels such as short message, email, enterprise WeChat, etc.

[0160] In some embodiments, in the case where the verification result is not passed, an email can be sent to the data responsible person in the case where the warning level is general.

[0161] In some other embodiments, in the case where the verification result is not passed, a short message can be sent to the data responsible person in the case where the warning level is serious.

[0162] The embodiments of the present application can ensure that compliant and clean data enters the production link, reducing downstream business errors. At the same time, through the verification result, data quality problems can also be exposed, triggering an early warning and improving the response speed of data quality management.

[0163] Referring to Figure 7 The embodiments of the present application also provide another optional flowchart of the data quality management method, Figure 7 The method in the above embodiment can include but is not limited to steps S701 to S704.

[0164] Step S701, generating a first newly added rule of the electronic device based on the compliance requirements read by the second model according to the regulatory documents;

[0165] In step S701 of some embodiments, the compliance requirements (extraction results) are obtained by parsing the regulatory documents, and the extraction results are input into the model to obtain a machine-readable logical expression (first newly added rule).

[0166] It should be noted that the regulatory documents can be official documents published in PDF, DOCX, web pages or structured interfaces, such as "EAST Data Quality Improvement Project Notice".

[0167] The second model can be an NLP extraction model, and the task type is "information extraction + text generation".

[0168] In step S702, a third model is used to generate a second newly added rule of the electronic device based on regulatory requirements of different regions.

[0169] In step S702 of some embodiments, the regulatory requirements of different regions are parsed, and the obtained text results are input into a model to obtain rules corresponding to the regulatory requirements of the regions.

[0170] It should be noted that the regulatory requirements of different regions can be additional indicators issued by provincial banking and insurance regulatory bureaus and central bank branches, such as "Supplementary Requirements of Chongqing Banking and Insurance Regulatory Bureau on EAST". The third model can also be an NLP extraction model.

[0171] In step S703, the first newly added rule and the second newly added rule are combined to obtain a combined newly added rule.

[0172] In step S703 of some embodiments, the first newly added rule and the second newly added rule are combined to obtain a combined newly added rule.

[0173] In step S704, the combined newly added rule is merged into the rule library, and the update of the rule library is recorded.

[0174] In step S704 of some embodiments, the combined newly added rule is merged into the rule library, and the version number, change time, and change reason of the rule library are recorded.

[0175] According to the embodiments of the present application, the regulatory files and regional regulatory requirements are converted into newly added rules, which can reduce the workload of manual interpretation and rule writing, reduce the compliance landing cost, and ensure the compliance of the rule library.

[0176] Please refer to Figure 8 Another optional flowchart of the data quality management method provided by the embodiments of the present application is provided. Figure 8 The method in the flowchart can include but is not limited to steps S801 to S804.

[0177] In step S801, based on the visual interface of the electronic device, a third newly added rule of the electronic device is generated by triggering an operation of moving the visual interface by a user.

[0178] In step S801 of some embodiments, in a web-based drag-and-drop rule builder, a new verification logic that has not been stored in the database is generated by mouse / touch dragging on the canvas, nodes, connections, and attribute panels.

[0179] Step S802, simulating and displaying, by the visualization interface, a checking result of the third newly-added rule on the target data of the electronic device.

[0180] In step S802 of some embodiments, the visualization interface simulates and displays the checking result.

[0181] Step S803, adjusting, by the user, a rule parameter of each rule in the rule group based on the visualization interface of the electronic device, to obtain an adjusted rule parameter.

[0182] In step S803 of some embodiments, the user adjusts the rule group parameter through the visualization interface. For example, double-clicking the rule node pops up the parameter panel; the user drags the slider to change the threshold from 1 million to 800,000; the preview button runs the sample again to show the new result.

[0183] Step S804, updating the adjusted rule parameter in the rule, and storing the updated rule in the rule library.

[0184] In step S804 of some embodiments, the adjusted rule parameter is updated in the rule and stored in the rule library.

[0185] In the embodiments of the present application, through the intuitive rule visualization interface, the user can quickly build rule logic through drag-and-drop operation and preview the rule execution effect in real time. Through interactive debugging of the rule, real-time feedback is provided to help the user quickly optimize the rule.

[0186] Please refer to Figure 9 Another optional flowchart of the data quality management method provided by the embodiments of the present application can include but not limited to the steps S901 to S904. Figure 9

[0187] Step S901, obtaining a tenant label of an electronic device;

[0188] It should be noted that the tenant label can be a global identifier, and can be a combination of a financial institution code and a business line.

[0189] Step S902, determining, based on the tenant label, a private rule library corresponding to the tenant in the rule library;

[0190] In step S902 of some embodiments, through the tenant label, independent rule libraries are provided for different financial institutions or business departments.

[0191] It should be noted that the private rule library can be a rule subset visible and writable only by the current tenant, and can be an independent table, an independent index or a logical partition in physics.

[0192] ​Step S903, obtaining a visibility label of the rule in the private rule library; wherein the visibility label is used to limit the access authority of the rule between tenants;

[0193] Step S904, determining a shared rule library of the electronic device based on the visibility label; wherein the shared rule library is used to check the target data.

[0194] In step S904 of some embodiments, the visibility label of the sharing level of the rule between tenants is defined to obtain a rule subset that can be used by all tenants of the electronic device.

[0195] It should be noted that the rule library of the electronic device is logically divided into a private rule library and a shared rule library, and a single electronic device maintains the two types of rules at the same time; when checking, the private rule library of the current tenant is pulled first, and then the shared rule library is pulled; the two parts of the rule are combined to form a final rule group, and then enter the subsequent parameter adjustment, parallel checking and other processes.

[0196] Embodiments of the present application provide independent rule libraries for different financial institutions or business departments to ensure the privacy and isolation of the rules. By supporting the sharing of common rules between different tenants, the rule reuse efficiency is improved.

[0197] Embodiments of the present application also provide a specific embodiment of a data quality management method, including intelligent generation and optimization of rules, multi-dimensional rule classification and label management, cross-platform dynamic synchronization of rules, rule caching and execution acceleration, intelligent arrangement and execution of rule groups, real-time monitoring and early warning of data quality, version management and traceability of rules, compliance automation support of rules, visualization and interactive debugging of rules, and multi-tenant support of rules.

[0198] I. Intelligent generation and optimization of rules. That is, the rules in the rule library include rules trained by the electronic device based on historical data; according to the target data, the rule parameters of each rule in the rule group are adjusted.

[0199] 1. Rule discovery based on machine learning:

[0200] By analyzing historical data and business patterns, machine learning algorithms (such as clustering and classification) are used to automatically discover potential rule patterns and generate new check rules. For example, the system (electronic device) can identify the correlation rules between tables and automatically generate the corresponding check logic.

[0201] 2. Dynamic optimization of rules:

[0202] The system supports dynamic adjustment of rule parameters (such as threshold, filtering condition) according to real-time data characteristics, ensuring the applicability of rules in different business scenarios. For example, when the data distribution changes, the system can automatically adjust the threshold range of date reasonableness verification.

[0203] II. Multi-dimensional rule classification and labeling management. That is, querying the rule corresponding to the target data in the rule library of the electronic device to obtain the rule group corresponding to the target data.

[0204] 1. Rule classification system:

[0205] In addition to traditional intra-table and inter-table rule classification, the system also supports multi-dimensional classification based on rule application scenarios (such as compliance, accuracy, consistency) and impact range (such as institution level, business line).

[0206] 2. Intelligent labeling management:

[0207] The system automatically assigns labels to each rule (such as "date reasonableness", "data integrity", "correlation verification"), supports users to quickly retrieve and combine rules through labels, and improves rule management efficiency.

[0208] III. Cross-platform dynamic synchronization of rules. That is, the rules in the rule library include rules obtained by the electronic device from other data platforms.

[0209] Multi-platform rule synchronization mechanism:

[0210] The system supports synchronization of the rule table to multiple data platforms (such as MySQL, Oracle, Hive, Hadoop), and ensures efficient execution of rules on different platforms through dynamic adapters. For example, Figure 10 as shown, Figure 10 Schematic diagram for synchronizing the rule table to multiple data platforms.

[0211] IV. Rule caching and execution acceleration. That is, querying whether the verification result corresponding to the target data is stored in the verification cache library.

[0212] 1. Rule parallel execution:

[0213] The system supports parallel execution of rules, and improves rule execution efficiency through distributed computing frameworks (such as Spark, Flink).

[0214] 2. Rule caching mechanism:

[0215] The system supports caching of rule execution results, avoiding repeated calculation and reducing system load.

[0216] V. Intelligent arrangement and execution of rule groups. That is, verifying the target data based on each rule in the adjusted rule group.

[0217] 1. Intelligent orchestration of rule groups:

[0218] The system supports automatic orchestration of rule groups based on their dependencies and execution priorities, avoiding conflicts and redundancies in rule execution. For example, the system can automatically identify that a table-level validation rule needs to be executed first, followed by an inter-table dependency rule.

[0219] 2. Dynamic task scheduling:

[0220] The system supports dynamic adjustment of the execution frequency and priority of rule groups based on task load and resource status, ensuring efficient task completion.

[0221] Six, real-time data quality monitoring and early warning. That is, performing quality management operations corresponding to the verification results on the target data.

[0222] 1. Real-time data quality monitoring:

[0223] The system supports real-time monitoring of data quality indicators (such as data completeness, accuracy, and consistency), and displays data quality status through a visual dashboard.

[0224] 2. Intelligent anomaly detection:

[0225] Anomaly detection algorithms (such as Isolation Forest and Time Series Analysis) are used to identify data quality anomalies in real time, and automatically trigger rule verification tasks to quickly locate the root cause of the problem.

[0226] 3. Hierarchical early warning mechanism:

[0227] According to the severity of the anomaly, the system can trigger hierarchical warning notifications (such as SMS and email) and provide repair suggestions to improve the response speed of data quality management.

[0228] Seven, version management and traceability of rules. That is, recording the updates of the rule library.

[0229] 1. Rule version control:

[0230] The system supports versioned management of rules, recording the change history of each rule modification to ensure traceability and recoverability of rules.

[0231] 2. Change impact analysis:

[0232] When a rule is modified, the system can automatically analyze the impact of the change on other rules or business processes, reducing the risk caused by rule modification.

[0233] Eight, compliance automation support for rules. That is, through the second model, read the compliance requirements based on regulatory documents, and generate the first new rules for electronic devices; through the third model, generate the second new rules for electronic devices based on regulatory requirements in different regions.

[0234] 1. Automatic identification of regulatory rules:

[0235] The system supports extracting key compliance requirements from regulatory documents (such as the "EAST Data Quality 'Upgrade Project' Notice") and automatically generating corresponding verification rules, reducing the workload of manual interpretation and rule writing.

[0236] 2. Cross-regional rule adaptation:

[0237] The system supports dynamically adjusting the rule library according to the regulatory requirements of different regions to ensure the compliance of the rule library and reduce the compliance risk of financial institutions.

[0238] Nine, visualization and interactive debugging of rules. That is, based on the visual interface of the electronic device, the third new rule of the electronic device is generated by the user triggering the operation of moving the visual interface; the verification result of the target data of the electronic device by the third new rule is simulated and displayed through the visual interface; based on the visual interface of the electronic device, the adjusted rule parameters are obtained by the user adjusting the rule parameters of each rule in the rule group; the adjusted rule parameters are updated in the rule.

[0239] 1. Rule visualization interface:

[0240] The system provides an intuitive rule visualization interface, and users can quickly build rule logic (such as filtering conditions, association relationships) through drag-and-drop operations and preview rule execution effects in real time. As shown in Figure 11 , the visualization interface diagram of the new rule is shown in Figure 11 , in which, in the case of a new rule, the following table, Table 1, can be entered in the rule page:

[0241]

[0242] 2. Interactive debugging tool:

[0243] The system supports users to interactively debug rules and provides real-time feedback (such as rule matching results, abnormal data samples) to help users quickly optimize rules. As shown in Figures 12-14 , the interface diagram of rule parameter adjustment is shown in Figure 12 ; the rule group detail interface diagram is shown in Figure 13 ; and the execution rule group information and execution data detail interface diagram is shown in Figure 14 .

[0244] Ten, multi-tenant support of rules. That is, obtaining the tenant tag of the electronic device; based on the tenant tag, determining the private rule library corresponding to the tenant in the rule library; obtaining the visibility tag of the rule in the private rule library; based on the visibility tag, determining the shared rule library of the electronic device.

[0245] 1. Institution-level rule isolation:

[0246] The system supports a multi-tenant mode, providing independent rule bases for different financial institutions or business departments, to ensure the privacy and isolation of rules.

[0247] 2. Shared rule base:

[0248] The system supports sharing of common rules (such as date reasonableness verification rules) between different tenants, to improve rule reuse efficiency.

[0249] Please refer to Figure 15 The embodiments of the present application also provide a data quality management device, which can implement the above-mentioned data quality management method, and the device comprises:

[0250] The acquisition module 1501 is configured to acquire target data to be verified.

[0251] The query module 1502 is configured to query whether the verification cache library stores a verification result corresponding to the target data.

[0252] The first obtaining module 1503 is configured to query a rule corresponding to the target data in a rule base of an electronic device in a case where the verification cache library does not store a verification result corresponding to the target data, to obtain a rule group corresponding to the target data; wherein the rules in the rule base include rules obtained by the electronic device based on historical data, and rules obtained by the electronic device from other data platforms.

[0253] The adjustment module 1504 is configured to adjust rule parameters of each rule in the rule group according to the target data.

[0254] The second obtaining module 1505 is configured to verify the target data based on each rule in the adjusted rule group, to obtain a verification result of the target data.

[0255] The execution module 1506 is configured to perform a quality management operation corresponding to the verification result on the target data.

[0256] The specific implementation of the data quality management device is basically the same as the specific embodiments of the above-mentioned data quality management method, and will not be repeated here.

[0257] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data quality management method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0258] Please refer to Figure 16 ,Figure 16 The hardware structure of an electronic device is illustrated, and the electronic device includes:

[0259] The processor 1601 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0260] The memory 1602 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1602 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1602 and are called and executed by the processor 1601 to implement the data quality management method of the embodiments of the present application.

[0261] The input / output interface 1603 is configured to implement information input and output.

[0262] The communication interface 1604 is configured to implement the communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0263] The bus 1605 is configured to transmit information between various components (for example, the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604) of the device.

[0264] The processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604 are connected to each other by the bus 1605 to realize the communication connection between them in the device.

[0265] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned data quality management method.

[0266] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0267] The data quality management method and device provided by the embodiments of the present application can achieve reuse of historical verification results by querying the verification cache library, alleviate the problem of long verification period, and then trigger data verification of the target data only when cache miss occurs, automatically perform corresponding quality management operations, without manual intervention, and shorten the deployment chain. Meanwhile, the rule library of the electronic device includes rules of other data platforms, which can enable rules scattered in multiple data platforms to be centralized, and further improve the data quality management efficiency of the electronic device.

[0268] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0269] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0270] The apparatus embodiments described above are merely exemplary, and units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0271] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0272] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0273] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are a "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0274] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0275] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0276] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0277] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0278] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A data quality management method, applied to electronic devices, characterized in that, The method includes: Obtain the target data to be verified; Check whether the verification cache contains a verification result corresponding to the target data; If no verification result corresponding to the target data is stored in the verification cache, the rule corresponding to the target data is queried in the rule base of the electronic device to obtain the rule group corresponding to the target data; wherein, the rules in the rule base include rules trained by the electronic device based on historical data, as well as rules obtained by the electronic device from other data platforms; Based on the target data, the rule parameters of each rule in the rule group are adjusted; The target data is validated based on each rule in the adjusted rule group to obtain the validation result of the target data. Perform quality management operations on the target data corresponding to the verification results.

2. The method according to claim 1, characterized in that, The acquisition of the target data to be verified includes at least one of the following: The first data generated by the electronic device is determined as the target data to be verified. If the preset quality management time is reached, the second data of the electronic device that has not been verified will be determined as the target data to be verified. The data of the electronic device is monitored by an anomaly detection algorithm, and the third data detected as abnormal is identified as the target data to be verified.

3. The method according to claim 1, characterized in that, The rule base contains multiple rules, as well as a first identity tag and a first function tag corresponding to each rule; The step of querying the rule base of the electronic device for the rule corresponding to the target data to obtain the rule group corresponding to the target data includes: Obtain a second identity tag and a second function tag for the target data; wherein, the second identity tag is used to identify the application scenario and scope of influence of the target data; and the second function tag is used to identify the function of the target data to be verified. Based on the second identity tag, the first identity tag is matched in the rule base to obtain the first rule; Based on the second functional tag, the first functional tag is matched in the first rule to obtain the rule group corresponding to the target data.

4. The method according to claim 1, characterized in that, The step of adjusting the rule parameters of each rule in the rule group based on the target data includes: Obtain the business characteristics of the target data; The business features and each rule in the rule group are input into the first model. The first model adjusts the rule parameters of each rule in the rule group based on the business features to obtain each rule in the rule group after the rule parameters are adjusted.

5. The method according to claim 1, characterized in that, The validation of the target data based on each rule in the adjusted rule group includes: Obtain the dependencies and priorities of each rule in the rule group; Adjust the execution frequency and priority of each rule in the rule group according to the load and resource status of the electronic device; After each rule in the rule group is filtered by the execution frequency, the target data is verified by calling each of the filtered rules in the rule group in parallel through a distributed computing framework according to the priority and the dependency relationship.

6. The method according to claim 1, characterized in that, The step of performing quality management operations on the target data corresponding to the verification result includes: If the verification result is satisfactory, the target data is released. If the verification result is unsuccessful, a warning notification will be sent to the user.

7. The method according to claim 1, characterized in that, The method further includes: The second model reads compliance requirements from regulatory documents and generates the first new rule for the electronic device. The third model generates a second set of new rules for the electronic device based on the regulatory requirements of different regions. Combine the first new rule and the second new rule to obtain the combined new rule; The newly added rules after combination are merged into the rule base, and the updates of the rule base are recorded.

8. The method according to claim 1, characterized in that, The method further includes: Based on the visual interface of the electronic device, a third new rule of the electronic device is generated by the user triggering the movement operation of the visual interface; The visualization interface simulates and displays the verification results of the target data of the electronic device by the third newly added rule. Based on the visual interface of the electronic device, the user adjusts the rule parameters of each rule in the rule group to obtain the adjusted rule parameters; The adjusted rule parameters are updated in the rule, and the updated rule is stored in the rule base.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the tenant tag of the electronic device; Based on the tenant tag, determine the private rule base corresponding to the tenant in the rule base; Obtain the visibility tags of the rules in the private rule base; wherein, the visibility tags are used to limit the access permissions of the rules between tenants; Based on the visibility tag, a shared rule base for the electronic device is determined; wherein, the shared rule base is used to verify the target data.

10. A data quality management device, characterized in that, The device includes: The acquisition module is used to acquire the target data to be verified. The query module is used to query whether the verification cache library stores a verification result corresponding to the target data; The first obtaining module is used to query the rule base of the electronic device for the rule corresponding to the target data when the verification cache does not store the verification result corresponding to the target data, and obtain the rule group corresponding to the target data; wherein, the rules in the rule base include the rules trained by the electronic device based on historical data, and the rules obtained by the electronic device from other data platforms; The adjustment module is used to adjust the rule parameters of each rule in the rule group according to the target data; The second obtaining module is used to verify the target data based on each rule in the adjusted rule group, and obtain the verification result of the target data; The execution module is used to perform quality management operations on the target data corresponding to the verification result.

Images