Data rule matching method, device and storage medium

Abstract

The present application provides a data rule matching method, device and storage medium, the method comprising: obtaining multiple first metadata in a rule knowledge base; wherein each first metadata includes an index identifier of a database table and a rule identifier corresponding to a rule type associated with the index identifier; determining and counting the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtaining a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers; screening all first frequencies, and obtaining a first data range of a target sub-identifier number for sub-identifiers included in a decision item set based on the number of index sub-identifiers corresponding to the screened first frequencies; combining and screening all sub-identifiers in the first metadata based on the first data range to obtain a target sub-identifier number item set; and determining a matching result obtained by obtaining a matching rule for the index information to be matched based on the target sub-identifier number item set.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology in financial technology (Fintech), and involve but are not limited to a data rule matching method, device and storage medium. Background Art

[0002] With the advancement of computing, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually shifting towards financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, Fintech also places higher demands on technology. Within the Fintech field, with the development of the big data ecosystem, effective quality testing of massive amounts of data through matching rules has become a prerequisite for realizing the value of data.

[0003] In related technologies, the method for matching data with rules involves first collecting, cleaning, and mining data to identify key data, then presetting rules corresponding to the data for verification, thereby effectively testing the data quality through matching rules. However, this method has at least one problem: low efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a data rule matching method, device, and storage medium to solve the problem of low efficiency when performing data rule matching in related technologies.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method, including:

[0007] Obtaining a plurality of first metadata in a rule knowledge base; wherein each first metadata includes an index identifier of a database table and a rule identifier corresponding to a rule type associated with the index identifier;

[0008] Determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain the first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers;

[0009] Filtering all first frequencies, and obtaining a first data range of a target number of sub-identifiers for the decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies, wherein the sub-identifiers include the index sub-identifiers and the rule sub-identifiers in the rule identifiers;

[0010] Based on the first data range, all sub-identifiers in the first metadata are combined and filtered to obtain a target sub-identifier quantity item set;

[0011] Based on the target sub-identification quantity item set, a matching result obtained by the obtained matching rule of the index information to be matched is determined.

[0012] The present invention provides a data rule matching device, including:

[0013] An acquisition module, configured to acquire a plurality of first metadata from a rule knowledge base; wherein each first metadata includes an index identifier of a database table and a rule identifier corresponding to a rule type associated with the index identifier;

[0014] a determination module, configured to determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers;

[0015] a processing module, configured to filter all first frequencies and obtain a first data range of a target number of sub-identifiers for a decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies, wherein the sub-identifiers include the index sub-identifiers and the rule sub-identifiers in the rule identifier;

[0016] The processing module is further configured to combine and filter all sub-identifiers in the first metadata based on the first data range to obtain a target sub-identifier quantity item set;

[0017] The determination module is further configured to determine, based on the target sub-identification quantity item set, a matching result obtained by the obtained matching rule of the index information to be matched.

[0018] An embodiment of the present application provides a device, including:

[0019] a memory for storing executable instructions;

[0020] The processor is configured to implement the above method when executing the executable instructions stored in the memory.

[0021] An embodiment of the present application provides a storage medium storing executable instructions for causing a processor to execute the above method.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] The present application obtains a first data range of the target sub-identifier number for the sub-identifiers included in the decision item set based on the number of frequently used index sub-identifiers stored in the rule knowledge base. That is to say, the value range of the target sub-identifier number is dynamically determined according to the frequency of use of the index sub-identifier number in the actual rule knowledge base, and the number of the obtained target sub-identifier number item set contains a certain range, which is more adaptable to the identification and matching of index identifiers extracted from actual index information. In this way, the problem of at least low efficiency in data rule matching in related technologies is solved, and the target sub-identifier number item set that is consistent with the actual data situation is determined according to the target sub-identifier number, and it is ensured that the system has objective fault tolerance and strong adaptability; in addition, the consumption of system resources is reduced through fast matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of an optional architecture of a terminal provided in an embodiment of the present application;

[0025] Figure 2 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of the rules corresponding to the rule identifiers provided in the embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of performing null value checking on annotators in a test data table in a test database provided by an embodiment of the present application;

[0028] Figure 5 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0029] Figure 6 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0030] Figure 7 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0031] Figure 8 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0032] Figure 9 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application;

[0033] Figure 10 This is an optional flowchart of the data rule matching method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0036] In order to better understand the data rule matching method provided in the embodiments of the present application, the data rule matching method in the related art is first described:

[0037] With the development of the big data ecosystem, effective data quality testing of massive amounts of data has become essential for unlocking its value. Related technologies typically perform data quality testing by first collecting, cleaning, and mining data to identify key data, then pre-setting corresponding validation strategies and performing various data dimension tests, either on a sampling or full-scale basis. It's important to emphasize that data preprocessing, such as cleaning and mining, and / or pre-setting validation strategies, inadvertently increase labor costs and risks. Semi-automated data quality testing can easily expose issues with timely human intervention, efficiency, and accuracy in validation strategy management and implementation.

[0038] The following describes an exemplary application of the data rule matching device provided in an embodiment of the present application. The data rule matching device provided in an embodiment of the present application can be implemented as any terminal with a screen display function, such as a laptop computer, a tablet computer, a desktop computer, an intelligent robot, or as a server. The following describes an exemplary application of the data rule matching device when implemented as a terminal.

[0039] See also Figure 1 , Figure 1 is a schematic diagram of the structure of the terminal 100 provided in an embodiment of the present application, Figure 1The terminal 100 shown includes: at least one processor 110, at least one network interface 120, a user interface 130 and a memory 150. The various components in the terminal 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140 is not described in detail. Figure 1 Various buses are labeled as bus system 140 .

[0040] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0041] The user interface 130 includes one or more output devices 131 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0042] The memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. The memory 150 may optionally include one or more storage devices physically located away from the processor 110. The memory 150 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 150 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0043] Operating system 151, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0044] A network communication module 152 for reaching other computing devices via one or more (wired or wireless) network interfaces 120 , illustratively including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0045] The input processing module 153 is configured to detect one or more user inputs or interactions from one of the one or more input devices 132 and to translate the detected inputs or interactions.

[0046] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 1 A data rule matching device 154 stored in the memory 150 is shown. The data rule matching device 154 may be a data rule matching device in the terminal 100 and may be software in the form of a program or plug-in. The data rule matching device 154 includes the following software modules: an acquisition module 1541, a determination module 1542, and a processing module 1543. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0047] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the data rule matching method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0048] The data rule matching method provided by the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the terminal 100 provided by the embodiment of the present application. Figure 2 , Figure 2 This is an optional flow chart of the data rule matching method provided in the embodiment of the present application, which will be combined with Figure 2 The steps shown are explained,

[0049] Step 201: obtain a plurality of first metadata in a rule knowledge base.

[0050] Each first metadata includes an index identifier of the database table and a rule identifier corresponding to a rule type associated with the index identifier.

[0051] In the embodiment of the present application, the rule knowledge base includes a certain scale of data rule information collection, and the data rule information includes but is not limited to the hierarchical name of the database table, the name of the database table field, and the corresponding verification rule type. For example, the data rule information stored in the rule knowledge base is shown in Table 1, and the rule reference corresponding to the rule identifier is Figure 3 As shown, it is important to emphasize that only by associating rules with data can the rules be used to actually verify the data.

[0052] rule_1 dbName=tctp_ods_save,tableName=cus_kpi_single rule_1 dbName=tc_ods_sa,tableName=c_kpi_single rule_2 dbName=ttp_dm_save,tableName=one_workflow_tmp rule_3 dbName=ccp_ads_tmp,tableName=ph_salary_one rule_1 dbName=ai_ods_bak,tableName=app_salary_le rule_3 dbName=cp_ods_bak,tableName=c_kpi_m rule_3 dbName=c_ods_bak,tableName=c_kpi_mn rule_3 dbName=tip_ads_save,tableName=n_kpi_sa rule_1 dbName=fp_ads_bak,tableName=q_salary_sa

[0053] Table 1

[0054] In the embodiments of the present application, metadata refers to the data obtained by decomposing the data rule information stored in the rule knowledge base and extracting key information; metadata includes the index identifier of the database table and the rule identifier corresponding to the rule type to be verified. Here, first metadata can be understood as the data rule information already stored in the rule knowledge base, obtained by decomposing the data rule information and extracting key information from the data rule information.

[0055] In an embodiment of the present application, an index identifier refers to a key index term in a database table that has business and hierarchical meanings, and the index identifier includes at least one index sub-identifier. A rule identifier refers to identification information corresponding to a rule type, and the rule identifier includes at least one rule sub-identifier.

[0056] In the embodiment of the present application, the process of obtaining multiple first metadata in the rule knowledge base can be achieved through the following steps:

[0057] The first step is to traverse the data rule information of the offline database table generated by the rule knowledge base.

[0058] The second step is to decompose and extract index identifiers with business meanings and / or hierarchical meanings from the data information in the data rule information, and to decompose and extract rule identifiers from the rule information in the data rule information.

[0059] The third step is to construct the first metadata based on the index identifier and the rule identifier.

[0060] In the embodiment of the present application, the offline library table is also called a database table. The data rule information or historical data rule records of the offline library table generated by the rule knowledge base can be multiple and are used to construct the first data set.

[0061] In an embodiment of the present application, the business-meaning index identifier contains certain business information, which can directly reflect the importance of the table, extract representative strings from the library table name, and discard customized characters that do not represent meaning. The business-meaning index identifier includes, but is not limited to, at least one business index sub-identifier such as the library name, table name, and field name of the database table. For example, the database table includes a key performance indicator (KPI) table, a salary table, and a workflow table, and the corresponding business index sub-identifiers of the business meaning include KPI, salary, and workflow.

[0062] The above-mentioned hierarchical index identifier refers to the hierarchical index sub-identifier corresponding to the data in the database table when it is processed at different levels in the data warehouse. Each layer of data has different data usage requirements. The purpose of obtaining the hierarchical identifier is to facilitate quick positioning and understanding when using the data table. Here, hierarchical index sub-identifiers include but are not limited to the operation data store (ODS), data warehouse detail (DWD), data warehouse base (DWB), data warehouse service (DWS), and application data store (ADS). The ODS layer is used to store data tables for raw data and directly loads raw logs and data, meaning that the data remains unchanged. The DWD layer cleans the data in the ODS layer, such as removing null values, dirty data, and data that exceeds the limit range. The data structure and granularity of the DWD layer remain consistent with the raw data tables. The DWB layer stores objective data and is generally used as an intermediate layer, which can be considered as a data layer for a large number of indicators. The DWS layer performs light aggregation of the basic data in the DWB layer, such as aggregation to granularity such as user-day or device-day. At this layer, a cross-theme wide table is formed based on a certain dimension. The ADS layer provides data for various statistical tables. It should be emphasized that the use of data warehouses can break down complex tasks into multiple steps to complete, and each layer in the data warehouse only handles a single step, which is relatively simple and convenient for locating problems; at the same time, standardizing data stratification and passing through the middle layer data can greatly reduce repeated calculations, increase the reusability of a single calculation structure, and isolate the calculated statistical data from the original data, realizing the decoupling of statistical data from the original data.

[0063] Here, we take a feasible application scenario as an example, referring to Figure 4 As shown, Figure 4The figure shows that a null value check is performed on the annotator in the test data table in the test database. The data rule information includes "'bdc_op_dm_safe.bdc_bl_court_announcement_ods.annoperson' uses null value check". The index identifiers with business meaning and / or hierarchical meaning that are decomposed and extracted from the data information "bdc_op_dm_safe.bdc_bl_court_announcement_ods.annoperson" in the data rule information are "bdc", "dm", "court", "ods", and "annoperson". Furthermore, since the rule type is that the field cannot be empty, it is recorded as "column_not_null". Therefore, the rule identifier "column_not_null" is decomposed and extracted from the rule information "uses null value check" in the data rule information. Based on the index identifiers "bdc", "dm", "court", "ods", and "annoperson" and the rule identifier "column_not_null", the first metadata constructed is {"bdc", "dm", "court", "ods", "annoperson", and "column_not_null"}. Here, the process of obtaining the first metadata based on the data rule information is shown in Table 2.

[0064]

[0065] Table 2

[0066] Here, in order to strengthen the understanding of obtaining the first metadata based on the data rule information, the embodiment of the present application further explains the first metadata obtained based on the data rule information in the above Table 1, traverses the data rule information of the offline library table generated by the rule knowledge base; extracts the representative index identifiers or key strings in the data information in the data rule information, discards the custom characters that do not have representative meanings, and the extracted key strings are as follows: ods, dm, ads, kpi, salary, workflow; it can be intuitively understood that only when the index identifiers or key strings of the above words appear, the rules corresponding to the associated rule identifiers will be automatically used for verification. Here, the above Table 1 is updated based on the extracted key strings to obtain Table 3.

[0067] rule_1 db_ods,table_kpi rule_1 db_ods,table_kpi rule_2 db_dm,table_workflow rule_3 db_ads,table_salary rule_1 db_ods,table_salary rule_3 db_ods,table_kpi rule_3 db_ods,table_kpi rule_3 db_ads,table_kpi rule_1 db_ads,table_salary

[0068] Table 3

[0069] Furthermore, based on the index identifier and the rule identifier, first metadata is constructed. Here, the first metadata is composed of the index identifier and the rule identifier, and the index identifier and the rule identifier are separated by a setting symbol such as "|". The same index identifiers in Table 3 are merged to finally obtain multiple first metadata, as shown in Table 4 below.

[0070] First metadata db_ods,table_kpi|rule_1,rule_3 db_ods,table_kpi|rule_1,rule_3 db_dm,table_workflow|rule_2 db_ads,table_salary|rule_1,rule_3 db_ads,table_kpi|rule_3 db_ods,table_salary|rule_1

[0071] Table 4

[0072] Step 202: Determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers.

[0073] In the embodiment of the present application, the index sub-identifier includes a business index sub-identifier and a hierarchical index sub-identifier, and the number of index sub-identifiers refers to the number of all index sub-identifiers included in the index identifier in the first metadata.

[0074] In the embodiment of the present application, the first frequency refers to the number of times the same index sub-identifier quantity appears in all index sub-identifier quantities.

[0075] In an embodiment of the present application, after obtaining a plurality of first metadata from a rule knowledge base, the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier is determined and counted, and a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers is determined. Of course, an embodiment of the present application may also obtain a set of index sub-identifier numbers after obtaining a plurality of first metadata from a rule knowledge base and determining and counting the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier; and based on the values ​​of the index sub-identifier numbers included in the set of index sub-identifier numbers, determine the first frequency of occurrence of each value in the set of index sub-identifier numbers.

[0076] In a feasible application scenario, assuming that the number of first metadata in the rule knowledge base is N=10,000, the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier is determined and counted, and the statistical results of the first frequency of the same index sub-identifier number appearing in all index sub-identifier numbers are obtained, as shown in Table 5 below.

[0077] Number of index sub-identifiers First frequency 4 2000 5 4000 6 3000 3 1000

[0078] Table 5

[0079] It should be emphasized that in order to enhance the credibility of the association relationship, what kind of index identifier can obtain what kind of rule, and the information of the previous index identifier cannot be too little. If there is only one index sub-identifier in the index identifier to associate the rule, the credibility is not high. The more index sub-identifiers there are, the more common matching rules are extracted from the historical records, and the more credible the trained association rules are.

[0080] Step 203 : Filter all first frequencies, and obtain a first data range of a target number of sub-identifiers for the decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies.

[0081] The sub-identifier includes the index sub-identifier and the rule sub-identifier in the rule identifier.

[0082] In the embodiment of the present application, the purpose of screening all first frequencies is to select a number of frequently used index sub-identifiers to ensure high credibility of the data association rule.

[0083] In the embodiment of the present application, the number of target sub-identifiers refers to the sum of the number of index sub-identifiers and rule sub-identifiers contained in the combined item set. Here, the number of target sub-identifiers can be represented by the symbol K.

[0084] In the embodiment of the present application, the first data range refers to the range of possible values ​​of the number of target sub-identifiers.

[0085] In an embodiment of the present application, after determining and counting the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtaining the first frequency of the same index sub-identifier number appearing in all index sub-identifier numbers, all first frequencies are filtered, and based on the number of index sub-identifiers corresponding to the filtered first frequencies, a first data range of the target number of sub-identifiers for the decision item set is obtained.

[0086] Step 204 : Based on the first data range, all sub-identifiers in the first metadata are combined and filtered to obtain a target sub-identifier quantity item set.

[0087] In the embodiment of the present application, the target sub-identification quantity item set can be understood as data with high usage frequency and strong correlation in the association relationship between data and rules in the historical data rule records.

[0088] In an embodiment of the present application, when all first frequencies are screened and a first data range of a target sub-identifier number of sub-identifiers for a decision item set is obtained based on the number of index sub-identifiers corresponding to the screened first frequencies, all sub-identifiers in the first metadata are combined and screened based on the first data range to obtain a target sub-identifier number item set, and the target sub-identifier number item set is stored in a system or terminal, so that when a new index identifier needs to match a rule, the rule identifier corresponding to the new index identifier can be quickly found in the target sub-identifier number item set, thereby using the rule corresponding to the rule identifier to perform quality detection on the data corresponding to the new index identifier.

[0089] Step 205 : Based on the target sub-identification quantity item set, a matching result obtained by the obtained matching rule of the index information to be matched is determined.

[0090] In the embodiment of the present application, the index information to be matched refers to the information corresponding to the data that needs to be quality tested.

[0091] An embodiment of the present application provides a data rule matching method, which obtains multiple first metadata in a rule knowledge base; wherein each first metadata includes an index identifier of a database table, and a rule identifier corresponding to a rule type associated with the index identifier; determines and counts the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtains a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers; filters all first frequencies, and based on the number of index sub-identifiers corresponding to the filtered first frequencies, obtains a first data range of a target sub-identifier number of sub-identifiers for a decision item set, wherein the sub-identifier includes an index sub-identifier and a rule sub-identifier in a rule identifier; based on the first data range, combines and filters all sub-identifiers in the first metadata to obtain a target sub-identifier number item set; based on the target sub-identifier number item set, determines The matching result obtained by the matching rule of the index information to be matched is obtained; that is, based on the number of frequently used index sub-identifiers stored in the rule knowledge base, a first data range of the target sub-identifier number of sub-identifiers included in the decision item set is obtained, that is, the value range of the target sub-identifier number is dynamically determined according to the frequency of use of the index sub-identifier number in the actual rule knowledge base, and the number of the obtained target sub-identifier number item set contains a certain range, which is more adaptable to the identification and matching of the index identifier extracted from the actual index information. In this way, the problem of at least low efficiency in data rule matching in the related technology is solved, and the target sub-identifier number item set that is consistent with the actual data situation is determined according to the target sub-identifier number, and it is ensured that the system has objective fault tolerance and strong adaptability; in addition, the consumption of system resources is reduced through fast matching.

[0092] See also Figure 5 , Figure 5 This is an optional flow chart of the data rule matching method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained,

[0093] Step 301: Obtain a plurality of first metadata in a rule knowledge base.

[0094] Each first metadata includes an index identifier of the database table and a rule identifier corresponding to a rule type associated with the index identifier.

[0095] Step 302: Determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers.

[0096] Step 303: Filter out first frequencies greater than a frequency threshold from all first frequencies.

[0097] In the embodiment of the present application, the frequency threshold is a preset threshold. For example, when the number N of first metadata is 10,000, the frequency threshold can be set to 1,500.

[0098] Step 304: Obtain the minimum number of index sub-identifiers and the maximum number of index sub-identifiers among the filtered numbers of index sub-identifiers corresponding to the first frequency.

[0099] Step 305: Determine a first data range of the target sub-identifier quantity based on the minimum index sub-identifier quantity and the maximum index sub-identifier quantity.

[0100] In an embodiment of the present application, first frequencies greater than a frequency threshold are filtered out from all first frequencies to obtain filtered first frequencies, and a minimum number of index sub-identifiers among the numbers of index sub-identifiers corresponding to the filtered first frequencies and a maximum number of index sub-identifiers among the numbers of index sub-identifiers corresponding to the filtered first frequencies are obtained; further, based on the minimum number of index sub-identifiers and the maximum number of index sub-identifiers, a first data range of the target sub-identifier number is determined.

[0101] Here, using Table 5 as an example, first frequencies greater than a frequency threshold, such as 1500, are screened from the four first frequencies. Specifically, the screened first frequencies are 2000, 3000, and 4000. Furthermore, the minimum number of index sub-identifiers, such as 4, and the maximum number of index sub-identifiers, such as 6, are obtained for the numbers of index sub-identifiers corresponding to the screened first frequencies, such as 2000, 3000, and 4000, respectively. Finally, a first data range for the target number of sub-identifiers K is determined based on the minimum number of index sub-identifiers, such as 4, and the maximum number of index sub-identifiers, such as 6.

[0102] Here, by setting the frequency threshold, the number of frequently used index sub-identifiers stored in the rule knowledge base can be determined. The value range of the target sub-identifier number K determined by this is that the number of item sets contains a certain range, which is more adaptable to the recognition and matching of actual data extraction fragments and has objective fault tolerance in the system.

[0103] Step 306: Based on the first data range, all sub-identifiers in the first metadata are combined and filtered to obtain a target sub-identifier quantity item set.

[0104] In other embodiments of the present application, in the process of obtaining the target sub-identification quantity item set, each combination item in the target sub-identification quantity item set can be converted into a preset data format, and each combination item in the target sub-identification quantity item set is stored in the system database. Here, the preset data format is shown in Table 6.

[0105] Frequency Index ID Rule Identifier Frequency of use int set set float

[0106] Table 6

[0107] Among them, frequency represents the number of times the combination item appears in multiple first metadata, and the number is an integer, that is, it can be represented by the character type int; the index identifier is the set of all index sub-identifiers in the combination item, and the index identifier can be identified by p_fragment_set; the rule identifier is the set of all rule sub-identifiers in the combination item, and the rule identifier can be identified by r_fragment_set. It should be noted that the index identifier p_fragment_set and the rule identifier r_fragment_set together constitute a combination item, which are stored separately here; the usage frequency is the ratio of the frequency num of the combination item appearing in the first metadata to the number of data N in the first metadata, that is, the percentage value of num / N. The usage frequency is greater than 0 and less than 1, and the usage frequency can be represented by support.

[0108] As can be seen above, a custom data structure is used to store the item set's hit count (num), index identifier (set), rule identifier (set), and usage frequency (support). Each time data is identified and matched against a rule, if an item set is hit, the hit data populates the rule knowledge base. Through simple calculations, the occurrence frequency and usage frequency attributes of the combined items in the item set can be simultaneously updated, preventing accuracy loss with the addition of new data. This allows already calculated item sets to be reused and updated promptly, provided the target sub-identifier number K remains unchanged.

[0109] Step 307: extract the to-be-matched index identifier with a specific meaning from the to-be-matched index information.

[0110] In the embodiments of the present application, specific meanings include but are not limited to business meanings and hierarchical meanings.

[0111] In the embodiment of the present application, the to-be-matched index information is decomposed according to character strings, and character strings with business meanings and / or hierarchical meanings are extracted from the decomposed character strings as to-be-matched index identifiers.

[0112] Step 308: Obtain a search result of searching for the to-be-matched index identifier from the target sub-identifier quantity item set.

[0113] Step 309: If the search result indicates that there is an index identifier to be matched, obtain a target rule identifier corresponding to the target rule type associated with the index identifier to be matched, call the target rule corresponding to the target rule identifier, and verify the business corresponding to the index information to be matched.

[0114] In an embodiment of the present application, after extracting the to-be-matched index identifier with a specific meaning from the to-be-matched index information, a search result of searching for the to-be-matched index identifier from the target sub-identifier quantity item set is obtained. If the search result indicates that there is an to-be-matched index identifier, that is, there is a rule matching the to-be-matched index identifier in the target sub-identifier quantity item set, then a target rule identifier corresponding to the target rule type associated with the to-be-matched index identifier is obtained, and the target rule corresponding to the target rule identifier is called to perform quality verification on the business corresponding to the to-be-matched index information.

[0115] From the above, it can be seen that the embodiment of the present application determines the number of index sub-identifiers corresponding to the index sub-identifiers in each index identifier and the first frequency of the same index sub-identifier number in all index sub-identifier numbers based on multiple first metadata recorded in the rule knowledge base; further, based on the minimum number of index sub-identifiers and the maximum number of index sub-identifiers in the number of index sub-identifiers corresponding to the first frequency greater than the frequency threshold selected, the first data range of the target sub-identifier number is determined; in this way, by setting the frequency threshold, the number of index sub-identifiers with high usage frequency stored in the rule knowledge base can be determined, and the value range of the target sub-identifier number K determined thereby, the number of item sets contains a certain range, which is more adaptable to the recognition and matching of actual data extraction fragments, and has objective fault tolerance in the system.

[0116] Here, combined Figure 6 The process of determining the first data range of the target sub-identifier quantity based on the minimum index sub-identifier quantity and the maximum index sub-identifier quantity in step 305 is further described.

[0117] Step 401: Obtain the number of rule types.

[0118] In the embodiment of the present application, the number of rule types is the number of rule types corresponding to the rules used for verifying services in the system. For example, an index identifier may involve six types of rule types for verification.

[0119] Step 402: Determine the sum of the minimum index sub-identifier quantity and the initial type quantity as the starting value of the first data range.

[0120] In the embodiments of this application, the initial number of types refers to the minimum number of rule types for quality verification of the data corresponding to the index identifier. The initial number of types can be one or at least two, and this application does not impose specific restrictions. Here, the initial number of types is 1 for exemplary description.

[0121] Step 403: Determine the sum of the maximum number of index sub-identifiers and the number of types as the end value of the first data range.

[0122] In an embodiment of the present application, the number of types of the rule type is obtained, the sum of the minimum number of index sub-identifiers and the initial number of types is determined as the starting value p of the first data range, and the sum of the maximum number of index sub-identifiers and the number of types is determined as the ending value q of the first data range, thereby obtaining the first data range [p, q] corresponding to the target sub-identifier number K.

[0123] Here, the above-determined minimum index sub-identifier number of 4 and the maximum index sub-identifier number of 6 are used as an example to illustrate that the sum of the minimum index sub-identifier number 4 and the initial type number 1 is determined to be the starting value 5 of the first data range, and the sum of the maximum index sub-identifier number 6 and the type number 6 is determined to be the ending value 12 of the first data range. In this way, the first data range corresponding to the target sub-identifier number K is [5,12].

[0124] It should be noted that when determining the first data range of the target sub-identifier number K, the target sub-identifier number K is greater than (the minimum index sub-identifier number + 1), the minimum index sub-identifier number is 4, and at least one rule identifier is obtained, so the minimum value of the target sub-identifier number K, that is, the starting value, is the minimum index sub-identifier number + 1; if there are too many index sub-identifiers and many associated rules, the data rule information will explode, and the number of associated rules is limited to a maximum of 6, so the maximum value of the target sub-identifier number K, that is, the ending value, is the target sub-identifier number K+6.

[0125] From the above, it can be seen that when determining the number of frequently used index sub-identifiers stored in the rule knowledge base, considering that the rule type corresponding to the rule used for verification is type 1 to 6, that is, the number of sub-identifiers composed of the index sub-identifier and the rule sub-identifier is at least the minimum index sub-identifier number + 1; the number of sub-identifiers composed of the index sub-identifier and the rule sub-identifier is at most the maximum index sub-identifier number + 6. The value range of the target sub-identifier number K is determined by this. It should be noted that the value of K is flexibly adjusted to better adapt to the actual situation of new data.

[0126] Here, combined Figure 7 The process of combining and filtering all sub-identifiers in the first metadata based on the first data range to obtain a target sub-identifier quantity item set in step 306 is further described.

[0127] Step 501: When the number of target sub-identifiers is a first value in a first data range, obtain all initial sub-identifiers in first metadata and the number of data in the first metadata.

[0128] The first value is a starting value in the first data range.

[0129] In the embodiment of the present application, all initial sub-identifiers refer to all index sub-identifiers and all rule sub-identifiers in the first metadata.

[0130] In the embodiment of the present application, the data quantity of the first metadata refers to the number of data of the first metadata, and the data quantity of the first metadata can be represented by the symbol N.

[0131] Step 502: Select the number of initial sub-identifiers corresponding to the first value from all initial sub-identifiers and combine them to obtain a first value item set.

[0132] The first numerical item set includes at least one first numerical combination item.

[0133] In an embodiment of the present application, when the number of target sub-identifiers is the first numerical value in the first data range, that is, the starting value, all initial sub-identifiers in the first metadata and the data quantity of the first metadata are obtained, and from all the initial sub-identifiers, the number of initial sub-identifiers corresponding to the starting value are selected for combination to obtain at least one first numerical combination item, and further obtain a first numerical item set composed of the at least one first numerical combination item.

[0134] Step 503 : determining and counting the second frequency of occurrence of the combination identifier in each first numerical combination item in the first metadata, and determining the ratio of the second frequency to the data quantity as the first usage frequency corresponding to each first numerical combination item.

[0135] In an embodiment of the present application, when a number of initial sub-identifiers corresponding to the first numerical value are selected from all initial sub-identifiers for combination to obtain a first numerical item set, the second frequency of the combination identifier in each first numerical combination item in the first numerical item set appearing in the first metadata is determined and counted, and the ratio of the second frequency to the data quantity N is determined as the first usage frequency corresponding to each first numerical combination item.

[0136] Step 504 : Filter out first numerical combination items whose first usage frequency is greater than or equal to a frequency threshold from at least one first numerical combination item to obtain a target first numerical item set.

[0137] The target first numerical item set includes at least one first numerical combination item.

[0138] In the embodiment of the present application, the frequency threshold refers to a pre-set frequency threshold. It should be noted that the setting of the frequency threshold is related to timeliness. If there is a timeliness issue when performing quality verification on the data, that is, the faster the quality verification, the better, then the frequency threshold can be set relatively high, such as the frequency threshold can be greater than or equal to 50%; if there is no timeliness issue when performing quality verification on the data, then the combined rules will be more and the quality verification time will be longer, such as the frequency threshold can be greater than or equal to 30%. It should be noted that the lower the frequency threshold, the more associated rules there are, and more quality verification can be done without being destructive, but the quality verification time is also relatively longer.

[0139] In an embodiment of the present application, after determining and counting the second frequency of the combination identifier in each first numerical combination item appearing in the first metadata, and determining the ratio of the second frequency to the amount of data as the first usage frequency corresponding to each first numerical combination item, first numerical combination items whose first usage frequency is greater than or equal to the frequency threshold are screened out from at least one first numerical combination item to obtain a target first numerical item set.

[0140] Step 505: When the number of target sub-identifiers is the second value in the first data range, all sub-identifiers in the target numerical item set corresponding to the previous value of the second value are combined and filtered to obtain the target second numerical item set, until the target second numerical item set corresponding to the second value being the end value in the first data range is obtained.

[0141] The second value is greater than the starting value and less than or equal to the ending value.

[0142] In the embodiment of the present application, the second value is any value in the range determined by being greater than the starting value p and less than or equal to the ending value q.

[0143] In an embodiment of the present application, when the number of target sub-identifiers is the second numerical value corresponding to the first digit in the first data range, first, the previous numerical value of the second numerical value corresponding to the first digit, that is, the first numerical value, is obtained, and all the first sub-identifiers in the corresponding target first numerical item set are obtained; secondly, from all the first sub-identifiers, the number of first sub-identifiers corresponding to the second numerical value corresponding to the first digit is selected and combined to obtain the second numerical item set corresponding to the first digit, wherein the second numerical item set corresponding to the first digit includes at least one second numerical combination item corresponding to the first digit; then, the third frequency of the combination identifier in each second numerical combination item corresponding to the first digit in the first metadata is determined and counted, and the ratio of the third frequency to the number of data is determined, which is the second usage frequency corresponding to each second numerical combination item corresponding to the first digit; finally, from at least one second numerical combination item corresponding to the first digit, the second numerical combination item corresponding to the first digit whose second usage frequency is greater than or equal to the frequency threshold is screened out to obtain the target second numerical item set corresponding to the first digit.

[0144] Furthermore, in the case where the target number of sub-identifiers is the second numerical value corresponding to the second digit in the first data range, first, the previous numerical value of the second numerical value corresponding to the second digit, that is, the second numerical value corresponding to the first digit, is obtained, and all second sub-identifiers in the target second numerical value item set corresponding to the first digit are obtained; secondly, from all the second sub-identifiers, the number of second sub-identifiers corresponding to the second numerical value corresponding to the second digit is selected and combined to obtain the second numerical value item set corresponding to the second digit, wherein the second numerical value item set corresponding to the second digit includes at least one second numerical value combination item corresponding to the second digit; then, the fourth frequency of the combination identifier in each second numerical value combination item corresponding to the second digit in the first metadata is determined and counted, and the ratio of the fourth frequency to the number of data is determined, which is the third usage frequency corresponding to each second numerical value combination item corresponding to the second digit; finally, from at least one second numerical value combination item corresponding to the second digit, the second numerical value combination item corresponding to the second digit whose third usage frequency is greater than or equal to the frequency threshold is screened out to obtain the target second numerical value item set corresponding to the second digit; until the target second numerical value item set corresponding to the end value in the first data range is obtained.

[0145] Step 506 : Remove the combination items that do not include the rule identifier from all combination items in the target first numerical item set and the target second numerical item set, and obtain the target sub-identifier quantity item set composed of the remaining combination items.

[0146] In the embodiment of the present application, all combination items include combination items that do not include rule identifiers, and remaining combination items except the combination items that do not include rule identifiers, and each combination item in the remaining combination items includes a rule identifier.

[0147] In an embodiment of the present application, from all combination items in the target first numerical item set and the target second numerical item set, the combination items that do not include the rule identifier are removed to obtain a target sub-identification quantity item set composed of the remaining combination items in all combination items; in this way, the combination items that do not include the rule identifier are removed to obtain a target sub-identification quantity item set composed of the remaining combination items that include the rule identifier.

[0148] In an achievable application scenario, referring to the following Table 7, taking the first data range corresponding to the target sub-identifier quantity K as [5, 12] and the usage frequency threshold as 50% as an example, the process of combining and filtering all sub-identifiers in the first metadata based on the first data range to obtain the target sub-identifier quantity item set is described as follows.

[0149]

[0150]

[0151] Table 7

[0152] When the target number of sub-identifiers K is 5 in the first data range [5, 12], all initial sub-identifiers in the first metadata, namely ads, kpi, kaf, id, name, age, rule_1, rule_2, rule_3, rule_4, rule_5, and rule_6, are obtained, and the number of data N in the first metadata is 9. Furthermore, 5 initial sub-identifiers are selected from all the initial sub-identifiers and combined to obtain a 5-item set with K = 5. Then, the second frequency of occurrence of the combination identifier in each 5-item combination in the first metadata is determined and counted, and the ratio of the second frequency to the number of data is determined as the usage frequency corresponding to each 5-item combination. From at least one 5-item combination, 5 combination items whose usage frequency is greater than or equal to the frequency threshold are screened to obtain the target 5-item set. Due to the large amount of data, only combination items that meet the usage frequency requirements and each combination item includes a rule sub-identifier are shown here. The resulting target 5-item set is shown in Table 8. Each combination item in the target 5-item set is converted into a pre-set format as shown in Table 9.

[0153] ads,kpi,kaf,id,rule_1 6 ads,kpi,kaf,id,rule_2 6 ads,kpi,kaf,id,rule_3 5 ads,kpi,kaf,id,rule_4 5

[0154] Table 8

[0155] num p_fragment_set r_fragment_set support 6 {ads, kpi, kaf, id} {rule_1} 6 / 9 6 {ads, kpi, kaf, id} {rule_2} 6 / 9 5 {ads, kpi, kaf, id} {rule_3} 5 / 9 5 {ads, kpi, kaf, id} {rule_4} 5 / 9

[0156] Table 9

[0157] Furthermore, according to the same method as above, the target 6-item set is obtained based on the target 5-item set as shown in Table 10, and each combination item in the target 6-item set is converted into a preset format as shown in Table 11.

[0158] ads, kpi, kaf, id, name, rule_1 5 ads, kpi, kaf, id, name, rule_2 5

[0159] Table 10

[0160] num p_fragment_set r_fragment_set support 5 {ads, kpi, kaf, id, name} {rule_1} 5 / 9 5 {ads, kpi, kaf, id, name} {rule_2} 5 / 9

[0161] Table 11

[0162] Furthermore, according to the same method as above, the target 7-item set is obtained based on the target 6-item set as shown in Table 12, and each combination item in the target 7-item set is converted into a preset format as shown in Table 13.

[0163] ads, kpi, kaf, id, name, rule_1, rule_2 5 ads, kpi, kaf, id, name, rule_1, rule_2 5

[0164] Table 12

[0165] num p_fragment_set r_fragment_set support 5 {ads, kpi, kaf, id, name} {rule_1,rule_2} 5 / 9 5 {ads, kpi, kaf, id, name} {rule_1,rule_2} 5 / 9

[0166] Table 13

[0167] It should be emphasized that the corresponding item sets obtained when the number of target sub-identifiers K is K=8, K=9, K=10, K=11, and K=12 do not have a usage frequency greater than or equal to the frequency threshold of 50% support, so they are not listed here.

[0168] Finally, the target 5-item set, target 6-item set and target 7-item set are taken as the target sub-identifier quantity item sets. Therefore, based on the first data range [5,12] corresponding to the target sub-identifier quantity K and the usage frequency threshold, the target sub-identifier quantity item sets are obtained, that is, the target 5-item set and the target 6-item set are used to obtain the target 7-item set.

[0169] In another feasible application scenario, referring to Table 4 above, and taking the first data range corresponding to the target sub-identifier quantity K as [3, 4] and the usage frequency threshold as 30% (here, since the number of data N in the first metadata is 6 and the usage frequency threshold is 30%, it can be understood that the corresponding usage frequency threshold is 6×30%=1.8), statistics are performed on the item sets with the target sub-identifier quantities K=1, K=2, K=3, and K=4, so as to understand the process of combining and filtering all sub-identifiers in the first metadata to obtain the item sets with the target sub-identifier quantity.

[0170] When the number of target sub-identifiers K is 1, all initial sub-identifiers in the first metadata, namely, db_ods, db_ads, db_dm, table_kpi, table_salary, table_workflow, rule_1, rule_2, and rule_3, are obtained, and the number of data in the first metadata, N, is 6. Furthermore, one initial sub-identifier is selected from all initial sub-identifiers to obtain a 1-item set with K = 1. Then, the frequency of occurrence of the initial sub-identifier in each 1-group item in the 1-item set in the first metadata is determined and counted. From at least one 1-group item, 1-group items whose frequency is greater than or equal to a frequency threshold are selected to obtain the target 1-item set. As shown in Table 14, Table 14 includes the frequencies corresponding to all sub-identifiers when K = 1, i.e., the 1-item set corresponding to K = 1. It should be emphasized that in all the following tables, bolded items indicate items that do not meet the frequency threshold and need to be eliminated. The remaining 1-item sets in the 1-item set correspond to the target 1-item set.

[0171] db_ods 3 db_ads 2 table_kpi 3 table_salary 2 rule_1 4 rule_3 4 rule_2 1 db_dm 1 table_workflow 1

[0172] Table 14

[0173] Furthermore, when the number of target sub-identifiers K is 2, all first sub-identifiers in the target 1-item set, namely, db_ods, db_ads, table_kpi, rule_1, and rule_3, are obtained. Furthermore, two first sub-identifiers are selected from all first sub-identifiers and permuted and combined to obtain a 2-item set with K = 2. Then, the frequency of occurrence of the combination identifiers in each 2-item combination in the 2-item set in the first metadata is determined and counted. From at least one 2-item combination, 2-item combinations with a frequency greater than or equal to a frequency threshold are selected to obtain the target 2-item set. As shown in Table 15, Table 15 includes the frequencies corresponding to all combination sub-identifiers when K = 2. It should be emphasized that the bolded items in the table below indicate combination items among the listed items that do not meet the frequency threshold and need to be eliminated. Here, the remaining combination items in the 2-item set correspond to the target 2-item set.

[0174] db_ods, rule_1 3 db_ads, rule_1 1 table_kpi, rule_1 2 rule_3, rule_1 3 db_ods, rule_3 2 db_ads, rule_3 2 table_kpi, rule_3 3 db_ods, db_ads 0 db_ods, table_kpi 2 db_ads, table_kpi 1

[0175] Table 15

[0176] Furthermore, according to the same method as above, the target 3-item set with K=3 is obtained based on the target 2-item set as shown in Table 16.

[0177] db_ods, rule_1, rule_3 2 table_kpi, rule_1, rule_3 2 db_ads, rule_1, rule_3 1 db_ods, db_ads, table_kpi 0 db_ods, table_kpi, rule_1 2 db_ads, table_kpi, rule_3 2

[0178] Table 16

[0179] Furthermore, according to the same method as above, the target 4-item set with K=4 is obtained based on the target 3-item set as shown in Table 17.

[0180] db_ods, table_kpi, rule_1, rule_3, 2 db_ads, table_kpi, rule_1, rule_3 0 db_ods, db_ads, table_kpi, rule_3 0 db_ods, db_ads, table_kpi, rule_1 0

[0181] Table 17

[0182] From the results of the target 3-item set corresponding to K=3 and the target 4-item set corresponding to K=4, it can be seen that when the to-be-matched index identifier hits db_ods and table_kpi, it is reasonable to automatically match rule_1 and rule_3 to perform data quality verification on the business corresponding to the to-be-matched index identifier.

[0183] Here, combined Figure 8 Further explain the process after calling the target rule corresponding to the target rule identifier and verifying the business corresponding to the matching index information.

[0184] Step 601: Based on the information quantity of the to-be-matched index information, the data quantity of the first metadata is incremented to obtain the incremented data quantity.

[0185] Here, illustratively, if the number of information in the to-be-matched index information is 1, the number of data N in the first metadata is incremented by 1 to obtain the incremented number of data N+1.

[0186] Step 602: Based on the information quantity, increment the frequencies of the matched combination items corresponding to the to-be-matched index identifiers in the target sub-identifier quantity item set, and the frequencies of the subset of the matched combination items, respectively, to obtain a first incremented frequency of the matched combination items and a second incremented frequency of the subset of the matched combination items.

[0187] Here, exemplarily, if the information quantity of the index information to be matched is 1, the frequency of the matched combination item corresponding to the index identifier to be matched is num1, and the frequency of the subset of the matched combination item is num2; based on the information quantity 1, the frequency num1 of the matched combination item corresponding to the index identifier to be matched and the frequency num2 of the subset of the matched combination item in the target sub-identifier quantity item set are increased by 1, and the first incremented frequency num1+1 of the matched combination item and the second incremented frequency num2+1 of the subset of the matched combination item are obtained.

[0188] Step 603: Update the usage frequency of the matched combination item to the ratio of the frequency after the first increment to the number of data after the increment, and update the usage frequency of the subset of the matched combination item to the ratio of the frequency after the second increment to the number of data after the increment.

[0189] Here, the usage frequency support1 of the matched combination items is updated to be the ratio of the frequency num1+1 after the first increment and the number of data after the increment N+1, that is, support1=(num1+1) / (N+1), and the usage frequency support2 of the subset of the matched combination items is updated to be the ratio of the frequency num2+2 after the second increment and the number of data after the increment N+1, that is, support2=(num2+1) / (N+1).

[0190] Step 604: Update the usage frequency of the remaining combination items in the target sub-identification quantity item set to the ratio of the frequency of the remaining combination items to the number of incremented data.

[0191] The remaining combination items include the combination items in the target sub-identification quantity item set except the matched combination items and the subset of the matched combination items.

[0192] Here, the usage frequency support3 of the remaining combination items in the update target sub-identification quantity item set is the ratio of the frequency num3 of the remaining combination items to the value-added data quantity N+1, that is, support3=num3 / (N+1).

[0193] Step 605 : Based on the usage frequency of the combination item in the target sub-identification quantity item set, the target sub-identification quantity item set is updated to obtain an updated target sub-identification quantity item set.

[0194] In an embodiment of the present application, after the usage frequency of the combination items in the target sub-identification quantity item set is updated, the combination items whose usage frequency is greater than or equal to the frequency threshold are filtered out from the updated combination items; that is, after other new data arrives, it is determined that the usage frequency of the remaining combination items in the target sub-identification quantity item set has not increased, that is, the probability of hitting is getting smaller and smaller, which has no guiding significance, so the combination items in the target sub-identification quantity item set that are less than the frequency threshold support are removed from the system database.

[0195] Exemplarily, the target sub-identifier quantity K item set includes the target 5-item set, the target 6-item set, and the target 7-item set. If the to-be-matched index identifier p_fragment_set corresponding to the to-be-matched index information is found from the target sub-identifier quantity K item set, such as ads, kpi, kaf, id, name, then the rule identifier r_fragment_set corresponding to the to-be-matched index identifier such as ads, kpi, kaf, id, name is taken out, such as rule_1, and the rule corresponding to the rule identifier rule_1 is used to perform rule automatic discovery verification on the business corresponding to the to-be-matched index information; further, K =6, the usage frequency of the combination "ads,kpi,kaf,id,name,rule_1" in the target 6-item set corresponding to K = 6 is adjusted from 5 / 9 to 5 + 1 / 9 + 1 = 6 / 10. Furthermore, the usage frequency of the combination "ads,kpi,kaf,id,name,rule_1" in the subset of "ads,kpi,kaf,id,name,rule_1" corresponding to K = 6, that is, in the target 5-item set corresponding to K = 5, is adjusted to 6 + 1 / 9 + 1 = 7 / 10. The usage frequency of the remaining combinations in the target set of K sub-identifiers is updated to support = num / (N + 1). Thus, after the arrival of new data, the usage frequency of the remaining combinations in the target set of K sub-identifiers does not increase, that is, the probability of hitting is decreasing, and they are no longer useful for guidance and are therefore removed.

[0196] As can be seen from the above, after the target sub-identifier K item set is calculated, each time new data, namely the to-be-matched index identifier corresponding to the to-be-matched index information, is identified and matched. The ratio (also known as the usage frequency) of the combination items in the target sub-identifier K item set that share the same to-be-matched index identifier is correspondingly increased, as well as the ratio of the subset of such combination items. Similarly, the ratios of the remaining combination items are reduced. Based on the adjusted ratios, the combination items in the target sub-identifier K item set are dynamically removed, thus avoiding repeated scans of the rule knowledge base. The final ratios determine whether to retain the combination items in the target sub-identifier K item set, ensuring that the data-rule relationship expressed by the several target sub-identifier K item sets trained across the entire rule knowledge base is real-time and close to the actual data.

[0197] See also Figure 9 , Figure 9 This is an optional flow chart of the data rule matching method provided in the embodiment of the present application, which will be combined with Figure 9 The steps shown are explained,

[0198] Step 701: Obtain a plurality of first metadata in a rule knowledge base.

[0199] Each first metadata includes an index identifier of the database table and a rule identifier corresponding to a rule type associated with the index identifier.

[0200] Step 702: Determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers.

[0201] Step 703 : Filter all first frequencies, and obtain a first data range of a target number of sub-identifiers for the decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies.

[0202] Step 704: Based on the first data range, all sub-identifiers in the first metadata are combined and filtered to obtain a target sub-identifier quantity item set.

[0203] Step 705: extract the to-be-matched index identifier with a specific meaning from the to-be-matched index information.

[0204] Step 706: Obtain a search result of searching for the to-be-matched index identifier from the target sub-identifier quantity item set.

[0205] Step 707: When the frequency of the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifiers in the to-be-matched index identifier is greater than the frequency threshold, if the search result indicates that there is no to-be-matched index identifier, adjust the first data range of the number of target sub-identifiers based on the number of to-be-matched index sub-identifiers.

[0206] In an embodiment of the present application, when the frequency of the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifier in the to-be-matched index identifier is greater than the frequency threshold, if the search result indicates that there is no to-be-matched index identifier, that is, there is no rule matching the to-be-matched index identifier in the target sub-identifier quantity set stored in the system database, then the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifier in the to-be-matched index identifier is obtained, and based on the number of to-be-matched index sub-identifiers, the first data range of the target sub-identifier quantity is adjusted.

[0207] Here, if the quantity value corresponding to the number of index sub-identifiers to be matched is greater than the start value of the first data range, the start value is adjusted to the sum of the number of index sub-identifiers to be matched and the initial type number, wherein the end value of the first data range remains unchanged.

[0208] If the quantity value is greater than the end value, the end value is adjusted to the sum of the number of index sub-identifiers to be matched and the number of rule types, wherein the start value of the first data range remains unchanged.

[0209] For example, assuming that a large number of to-be-matched index sub-identifiers of to-be-matched index sub-identifiers included in the to-be-matched index identifiers appear, and the frequency of occurrence of the to-be-matched index identifiers is greater than a frequency threshold, the first data range [p, q] of the target sub-identifier quantity K is adjusted. That is, if the quantity value z corresponding to the number of to-be-matched index sub-identifiers is greater than the starting value p of the first data range, the starting value p is adjusted to the sum of the number of to-be-matched index sub-identifiers z and the initial number of types, such as 1; wherein the ending value q of the first data range remains unchanged; that is, the adjusted first data range of the target sub-identifier quantity K is [z+1, q]. If the quantity value z is greater than the ending value q, the ending value q is adjusted to the sum of the number of to-be-matched index sub-identifiers z and the number of types of the rule type, 6. wherein the starting value p of the first data range remains unchanged; that is, the adjusted first data range of the target sub-identifier quantity K is [p, z+6], so as to obtain a candidate sub-identifier quantity item set based on the adjusted data range.

[0210] Step 708: When the to-be-matched index identifier has matched the association rule, obtain second metadata constructed by the to-be-matched index identifier and the rule identifier corresponding to the rule type of the matched association rule.

[0211] In the embodiment of the present application, the matched association rule of the to-be-matched index identifier can be understood as manually associating the to-be-matched index identifier with the rule identifier corresponding to the rule type, thereby completing the matching rule for the to-be-matched index identifier.

[0212] Step 709 : Based on the adjusted first data range, all sub-identifiers in the first metadata and the second metadata are combined and screened to obtain a candidate sub-identifier quantity item set.

[0213] Step 710: Merge the candidate sub-identification quantity item set with the target sub-identification quantity item set to update the target sub-identification quantity item set.

[0214] In this embodiment, when an association rule has been matched for an index identifier to be matched, second metadata is obtained, constructed from a rule identifier corresponding to the index identifier to be matched and the rule type of the matched association rule. Based on the adjusted first data range, all sub-identifiers in the first and second metadata are combined and filtered to obtain a candidate sub-identifier quantity item set. The candidate sub-identifier quantity item set is then merged with the target sub-identifier quantity item set to update the target sub-identifier quantity item set. This target sub-identifier quantity item set is thus always capable of identifying and matching all data, allowing for automated rule verification.

[0215] From the above, it can be seen that after the calculation of the target sub-identifier quantity K item set is completed, if there is no index identifier in the target sub-identifier quantity K item set that is the same as the to-be-matched index identifier corresponding to the new data, that is, the to-be-matched index information, and the frequency of the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifier of the to-be-matched index identifier is greater than the frequency threshold, the value range of the target sub-identifier quantity K is readjusted according to the number of to-be-matched index sub-identifiers, and then the calculation of the new item set is triggered, and then the processing of the new situation is included. This process is automated and has a certain correction capability.

[0216] Here, see Figure 10 , further explains the overall solution for data rule matching provided in this application.

[0217] Step 801: Obtain new library table fields.

[0218] Here, the new library table field corresponds to the aforementioned index information to be matched.

[0219] Step 802: Decompose and extract segments with specific meanings in the new database table fields to obtain data segments to be matched.

[0220] Here, the data segment to be matched corresponds to the aforementioned index identifier to be matched.

[0221] Step 803: Determine whether the data segment to be matched matches the data segment in the target sub-identification quantity item set.

[0222] Here, the data segment corresponds to the above index identifier. If the data segment to be matched matches the data segment in the target sub-identifier number item set, step 804 is executed; if the data segment to be matched does not match the data segment in the target sub-identifier number item set, step 806 is executed.

[0223] Step 804: perform rule verification on the business corresponding to the new database table field through the rule fragment associated with the data fragment in the target sub-identification quantity item set, and update the usage frequency of each combination item in the target sub-identification quantity item set.

[0224] Here, the rule fragment corresponds to the above rule identifier.

[0225] Step 805: Data identification and rule matching of the new database table fields are completed.

[0226] Step 806: Count the frequency of occurrence of the number of the to-be-matched data sub-segments in the to-be-matched data segment.

[0227] Step 807: Determine whether the frequency exceeds the frequency threshold.

[0228] Here, if the frequency does not exceed the frequency threshold, step 808 is executed; if the frequency exceeds the frequency threshold, step 809 is executed.

[0229] Step 808: No processing is performed.

[0230] Step 809 : Based on the number of data sub-segments to be matched, the value range of K is updated, and based on the value range of K, a corresponding candidate sub-identifier quantity item set is obtained, and the candidate sub-identifier quantity item set is added to the target sub-identifier quantity item set.

[0231] In this way, the problem of at least low efficiency when matching data rules in related technologies is solved, and the target sub-identifier quantity item set that is consistent with the actual data situation is determined based on the target sub-identifier quantity, and it is ensured that the system has objective fault tolerance and strong adaptability; in addition, through fast matching, the consumption of system resources is reduced.

[0232] The following continues to describe the exemplary structure of the data rule matching device 154 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 1 As shown, the software module stored in the data rule matching device 154 of the memory 150 may be the data rule matching device in the terminal 100, including:

[0233] The obtaining module 1541 is configured to obtain a plurality of first metadata in the rule knowledge base, wherein each first metadata includes an index identifier of a database table and a rule identifier corresponding to a rule type associated with the index identifier;

[0234] Determining module 1542, configured to determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain a first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers;

[0235] Processing module 1543 is configured to filter all first frequencies and obtain a first data range of a target number of sub-identifiers for the decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies, where the sub-identifiers include index sub-identifiers and rule sub-identifiers in the rule identifier.

[0236] The processing module 1543 is further configured to combine and filter all sub-identifiers in the first metadata based on the first data range to obtain a target sub-identifier quantity item set;

[0237] The determination module 1542 is further configured to determine a matching result obtained by the obtained matching rule of the to-be-matched index information based on the target sub-identification quantity item set.

[0238] In some embodiments, the processing module 1543 is further used to filter out first frequencies greater than a frequency threshold from all first frequencies; the acquisition module 1541 is further used to obtain the minimum number of index sub-identifiers and the maximum number of index sub-identifiers among the numbers of index sub-identifiers corresponding to the filtered first frequencies; the determination module 1542 is further used to determine a first data range of the target sub-identifier number based on the minimum number of index sub-identifiers and the maximum number of index sub-identifiers.

[0239] In some embodiments, the acquisition module 1541 is also used to obtain the number of types of the rule type; the determination module 1542 is also used to determine the sum of the minimum number of index sub-identifiers and the initial number of types, which is the starting value of the first data range; and determine the sum of the maximum number of index sub-identifiers and the number of types, which is the ending value of the first data range.

[0240] In some embodiments, the acquisition module 1541 is further used to obtain all initial sub-identifiers in the first metadata and the data quantity of the first metadata when the number of target sub-identifiers is the first numerical value in the first data range; wherein the first numerical value is the starting value in the first data range; the processing module 1543 is further used to select the number of initial sub-identifiers corresponding to the first numerical value from all initial sub-identifiers for combination to obtain a first numerical item set; the first numerical item set includes at least one first numerical combination item; the determination module 1542 is further used to determine and count the second frequency of the combination identifier in each first numerical combination item appearing in the first metadata, and determine the ratio of the second frequency to the data quantity as the first usage frequency corresponding to each first numerical combination item ... Module 1543 is further used to filter out first numerical combination items whose first usage frequency is greater than or equal to the frequency threshold from at least one first numerical combination item to obtain a target first numerical item set; when the target number of sub-identifiers is the second numerical value in the first data range, all sub-identifiers in the target numerical item set corresponding to the previous numerical value of the second numerical value are combined and filtered to obtain a target second numerical item set, until a target second numerical item set corresponding to the end value whose second numerical value is within the first data range is obtained; wherein the second numerical value is greater than the starting value and less than or equal to the end value; and from all combination items in the target first numerical item set and the target second numerical item set, remove the combination items that do not include the rule identifier to obtain a target sub-identifier quantity item set composed of the remaining combination items.

[0241] In some embodiments, the acquisition module 1541 is also used to extract the to-be-matched index identifier with a specific meaning from the to-be-matched index information; obtain the search result of searching for the to-be-matched index identifier from the target sub-identifier quantity item set; if the search result indicates that there is an to-be-matched index identifier, obtain the target rule identifier corresponding to the target rule type associated with the to-be-matched index identifier, and call the target rule corresponding to the target rule identifier to verify the business corresponding to the to-be-matched index information.

[0242] In some embodiments, the processing module 1543 is further configured to perform increment processing on the data quantity of the first metadata based on the information quantity of the index information to be matched, to obtain the incremented data quantity; perform increment processing on the frequency of the matched combination item corresponding to the index identifier to be matched and the frequency of the subset of the matched combination items in the target sub-identifier quantity item set based on the information quantity, to obtain a first incremented frequency of the matched combination item and a second incremented frequency of the subset of the matched combination item; update the usage frequency of the matched combination item to be the ratio of the first incremented frequency to the incremented data quantity, and update the usage frequency of the subset of the matched combination item to be the ratio of the second incremented frequency to the incremented data quantity; update the usage frequency of the remaining combination items in the target sub-identifier quantity item set to be the ratio of the frequency of the remaining combination items to the incremented data quantity; wherein the remaining combination items include the combination items in the target sub-identifier quantity item set excluding the matched combination item and the subset of the matched combination items; and update the target sub-identifier quantity item set based on the usage frequency of the combination items in the target sub-identifier quantity item set to obtain an updated target sub-identifier quantity item set.

[0243] In some embodiments, the frequency of the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifier in the to-be-matched index identifier is greater than a frequency threshold, and the processing module 1543 is further used to adjust the first data range of the number of target sub-identifiers based on the number of to-be-matched index sub-identifiers if the search result indicates that there is no to-be-matched index identifier; in the case where the to-be-matched index identifier has matched the association rule, obtain the second metadata constructed by the rule identifier corresponding to the to-be-matched index identifier and the rule type of the matched association rule; based on the adjusted first data range, combine and filter all sub-identifiers in the first metadata and the second metadata to obtain a candidate sub-identifier quantity item set; merge the candidate sub-identifier quantity item set with the target sub-identifier quantity item set to update the target sub-identifier quantity item set.

[0244] In some embodiments, the processing module 1543 is also used to adjust the starting value to the sum of the number of index sub-identifiers to be matched and the initial number of types if the quantity value corresponding to the number of index sub-identifiers to be matched is greater than the starting value of the first data range; wherein the ending value of the first data range remains unchanged; if the quantity value is greater than the ending value, adjust the ending value to the sum of the number of index sub-identifiers to be matched and the number of types of the rule type; wherein the starting value of the first data range remains unchanged.

[0245] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0246] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 2 、 5 -10 shows the method.

[0247] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0248] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0249] By way of example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). By way of example, executable instructions may be deployed for execution on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0250] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data rule matching method, characterized in that: include: Obtaining a plurality of first metadata in a rule knowledge base; wherein each first metadata includes an index identifier of a database table and a rule identifier corresponding to a rule type associated with the index identifier; Determine and count the number of index sub-identifiers corresponding to the index sub-identifier in each index identifier, and obtain the first frequency of occurrence of the same index sub-identifier number in all index sub-identifier numbers; Filtering all first frequencies, and obtaining a first data range of a target number of sub-identifiers for the decision item set based on the number of index sub-identifiers corresponding to the filtered first frequencies, wherein the sub-identifiers include the index sub-identifiers and the rule sub-identifiers in the rule identifiers; Based on the first data range, all sub-identifiers in the first metadata are combined and filtered to obtain a target sub-identifier quantity item set; Based on the target sub-identification quantity item set, a matching result obtained by the obtained matching rule of the index information to be matched is determined.

2. The method according to claim 1, characterized in that The filtering process is performed on all first frequencies, and based on the number of index sub-identifiers corresponding to the filtered first frequencies, a first data range of a target number of sub-identifiers for the decision item set is obtained, including: Filtering out first frequencies greater than a frequency threshold from all the first frequencies; Obtaining the minimum number of index sub-identifiers and the maximum number of index sub-identifiers among the numbers of index sub-identifiers corresponding to the first frequency after screening; The first data range of the target sub-identifier quantity is determined based on the minimum index sub-identifier quantity and the maximum index sub-identifier quantity.

3. The method according to claim 2, characterized in that The determining the first data range of the target sub-identifier quantity based on the minimum index sub-identifier quantity and the maximum index sub-identifier quantity includes: Get the type number of the rule type; Determine the sum of the minimum number of index sub-identifiers and the initial number of types as the starting value of the first data range; The sum of the maximum number of index sub-identifiers and the number of types is determined as the end value of the first data range.

4. The method according to claim 1, wherein The combining and screening of all sub-identifiers in the first metadata based on the first data range to obtain a target sub-identifier quantity item set includes: When the number of target sub-identifiers is a first value in the first data range, obtaining all initial sub-identifiers in the first metadata and the number of data in the first metadata; wherein the first value is a starting value in the first data range; Selecting the number of initial sub-identifiers corresponding to the first value from all the initial sub-identifiers and combining them to obtain a first numerical item set; the first numerical item set includes at least one first numerical combination item; Determining and counting a second frequency of occurrence of the combination identifier in each first numerical combination item in the first metadata, and determining a ratio of the second frequency to the amount of data as a first usage frequency corresponding to each first numerical combination item; Filtering out, from the at least one first numerical combination item, first numerical combination items whose first usage frequency is greater than or equal to a frequency threshold, to obtain a target first numerical item set; When the target number of sub-identifiers is a second value within the first data range, all sub-identifiers in the target numerical item set corresponding to the previous value of the second value are combined and filtered to obtain a target second numerical item set, until a target second numerical item set corresponding to an end value within the first data range is obtained in which the second value is greater than the start value and less than or equal to the end value; From all combination items in the target first numerical item set and the target second numerical item set, the combination items not including the rule identifier are removed to obtain the target sub-identifier quantity item set composed of the remaining combination items.

5. The method according to any one of claims 1 to 4, characterized in that The determining, based on the target sub-identification quantity item set, a matching result obtained by the obtained matching rule of the index information to be matched includes: Extracting a to-be-matched index identifier with a specific meaning from the to-be-matched index information; the specific meaning includes: business meaning and hierarchical meaning; Obtaining a search result of searching for the to-be-matched index identifier from the target sub-identifier quantity item set; If the search result indicates that the to-be-matched index identifier exists, a target rule identifier corresponding to the target rule type associated with the to-be-matched index identifier is obtained, and a target rule corresponding to the target rule identifier is called to verify the business corresponding to the to-be-matched index information.

6. The method according to claim 5, characterized in that After calling the target rule corresponding to the target rule identifier and verifying the business corresponding to the to-be-matched index information, the method further includes: Based on the information quantity of the to-be-matched index information, performing value-added processing on the data quantity of the first metadata to obtain a value-added data quantity; Based on the information quantity, performing increment processing on the frequencies of the matched combination items corresponding to the to-be-matched index identifiers and the frequencies of the subsets of the matched combination items in the target sub-identifier quantity item set, respectively, to obtain a first incremented frequency of the matched combination items and a second incremented frequency of the subsets of the matched combination items; Updating the usage frequency of the matched combination item to the ratio of the frequency after the first increment to the number of data after the increment, and updating the usage frequency of the subset of the matched combination item to the ratio of the frequency after the second increment to the number of data after the increment; The usage frequency of the remaining combination items in the target sub-identification quantity item set is updated to be the ratio of the frequency of the remaining combination items to the amount of the value-added data; wherein the remaining combination items include the combination items in the target sub-identification quantity item set excluding the matched combination items and the subset of the matched combination items; Based on the usage frequency of the combination items in the target sub-identification quantity item set, the target sub-identification quantity item set is updated to obtain an updated target sub-identification quantity item set.

7. The method according to claim 5, characterized in that The frequency of the number of to-be-matched index sub-identifiers corresponding to the to-be-matched index sub-identifiers in the to-be-matched index identifier is greater than a frequency threshold, and the method includes: If the search result indicates that the to-be-matched index identifier does not exist, adjusting the first data range of the number of target sub-identifiers based on the number of the to-be-matched index sub-identifiers; In a case where the to-be-matched index identifier has matched an association rule, obtaining second metadata constructed from a rule identifier corresponding to the to-be-matched index identifier and a rule type of the matched association rule; Based on the adjusted first data range, all sub-identifiers in the first metadata and the second metadata are combined and screened to obtain a candidate sub-identifier quantity item set; The candidate sub-identification quantity item set is merged with the target sub-identification quantity item set to update the target sub-identification quantity item set.

8. The method according to claim 7, characterized in that The adjusting the first data range of the number of target sub-identifiers based on the number of index sub-identifiers to be matched includes: If the quantity value corresponding to the number of index sub-identifiers to be matched is greater than the starting value of the first data range, the starting value is adjusted to the sum of the number of index sub-identifiers to be matched and the initial type number; wherein the ending value of the first data range remains unchanged; If the quantity value is greater than the end value, the end value is adjusted to the sum of the number of index sub-identifiers to be matched and the number of types of the rule type; wherein the starting value of the first data range remains unchanged.

9. A data rule matching device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

10. A storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement the method according to any one of claims 1 to 8.

Images