Field rule matching recommendation method and device based on multi-dimensional feature fusion

Abstract

The application relates to the technical field of data processing, and provides a field rule matching recommendation method and device based on multi-dimensional feature fusion. User input information is acquired, the user input information comprising a target data table and a current business target; a field feature extraction module is used for performing feature extraction on fields in the target data table to obtain field multi-dimensional features; a rule attribute modeling is used for performing attribute modeling on data quality rules in a preset rule library to obtain rule multi-dimensional attributes; a multi-dimensional scoring model is used for calculating matching scores of the field multi-dimensional features and the rule multi-dimensional attributes; and a rule recommendation engine is used for dynamically adjusting the weights of each dimension in the multi-dimensional scoring model according to a business target priority, and outputting Top-N recommended rules based on the adjusted weights and the matching scores. The scheme can solve the problems of low matching precision, poor efficiency and insufficient scene coverage of traditional methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a field rule matching recommendation method and device based on multi-dimensional feature fusion. BACKGROUND

[0002] Data quality governance is a core link of enterprise digital transformation, and one of its core problems is to automatically recommend applicable quality rules for data fields. Traditional methods have the following defects:

[0003] (1) Rule matching relies on manual experience: experts need to manually configure the mapping relationship between rules and fields, which is low in efficiency and easy to miss.

[0004] (2) Matching logic is single: most systems only perform simple matching based on field names or types, ignoring semantic similarity, business goal adaptability and other key factors.

[0005] (3) Insufficient scene coverage: existing rule libraries are mostly general rules, lacking adaptation ability for specific business processes (such as financial audit, supply chain management).

[0006] (4) Weak dynamic adjustment capability: unable to dynamically adjust the recommendation strategy according to the user's input real-time business goal (such as "priority to ensure data accuracy").

[0007] Therefore, a method is needed to solve the problems of low accuracy, poor efficiency and insufficient scene coverage of traditional quality rule matching methods. SUMMARY

[0008] The embodiments of the present application provide a field rule matching recommendation method and device based on multi-dimensional feature fusion, which can solve the problems of low accuracy, poor efficiency and insufficient scene coverage of traditional quality rule matching methods.

[0009] According to an aspect of the present application, a field rule matching recommendation method based on multi-dimensional feature fusion is provided, comprising:

[0010] Obtaining user input information, the user input information including a target data table and a current business goal;

[0011] Extracting features of fields in the target data table through a field feature extraction module to obtain field multi-dimensional features;

[0012] Attribute modeling of data quality rules in a preset rule library through a rule attribute modeling module to obtain rule multi-dimensional attributes;

[0013] Calculating the matching score of the field multi-dimensional features and the rule multi-dimensional attributes through a multi-dimensional scoring model;

[0014] The rule recommendation engine dynamically adjusts the weight of each dimension in the multi-dimensional scoring model according to the business target priority, and outputs Top-N recommended rules based on the adjusted weight and the matching score.

[0015] Optionally, the field multi-dimensional characteristics include text semantic characteristics, data type characteristics, statistical characteristics, and business process labels, wherein the text semantic characteristics are obtained by converting the field description into a high-dimensional semantic vector through a BERT pre-training model; the data type characteristics are data type identifiers of the field, and the data type includes numerical type, date type, and string type; the statistical characteristics include the missing rate, the proportion of outliers, the average value, the standard deviation, and the number of unique values of the field sample value; and the business process label is an identifier of a business process to which the field belongs, and the business process includes a sales process, a supply chain management process, and a financial audit process.

[0016] Optionally, the application logic of the statistical characteristics includes: when the missing rate of the field is greater than a preset threshold, automatically marking the field to be matched with a non-empty check type rule after field characteristic extraction; and when the proportion of outliers of the field is greater than a preset threshold, automatically marking the field to be matched with an abnormal value detection type rule.

[0017] Optionally, the rule multi-dimensional attributes include semantic characteristics, action types, priorities, specialization degrees, and complexities, wherein the semantic characteristics are obtained from the core business semantics of the rule name; the action type is a data type identifier applicable to the rule, corresponding to the data type characteristics of the field; the priority is an importance level of the rule in data quality governance; the specialization degree is a specificity level of the applicable scenario of the rule; and the complexity is a technical difficulty and a calculation resource consumption level of the implementation of the rule.

[0018] Optionally, the evaluation dimensions of the multi-dimensional scoring model include semantic similarity, universality, priority, specialization degree, complexity, and business target alignment degree.

[0019] The semantic similarity is obtained by calculating the cosine similarity between the field text semantic vector and the rule semantic feature vector, and the value range is [0, 1]; the universality is obtained by predefining a rule action type and field data type matching matrix and normalizing, and the value range is [0, 1]; the priority is obtained by combining the pre-defined priority level of the rule and the dynamic adjustment of the field statistical characteristics and normalizing, and the value range is [0, 1]; the specialization degree is obtained by reverse normalization, converting the specialization degree to a value range of [0.2, 1.0]; the complexity is obtained by converting it into a cost coefficient and normalizing it to a value range of [0.2, 1.0]; and the business target alignment degree is obtained by constructing a business process and rule mapping matrix and combining TF-IDF text similarity calculation, and the value range is [0, 1];

[0020] The matching score is calculated by a weighted linear combination model, and the calculation formula is: Sere=∑(Wi×Di), wherein Wi is the weight of each evaluation dimension, the sum of all dimension weights is 1, and Di is the normalized score of each evaluation dimension.

[0021] Optionally, the rule recommendation engine adopts an analytic hierarchy process to realize dynamic weight distribution of each evaluation dimension.

[0022] In the core business scenario, the weight of the semantic similarity dimension is adjusted to 30%, and the weight of the rule priority dimension is adjusted to 25%; in the general governance scenario, the weight of the type compatibility dimension is adjusted to 20%, and the weight of the implementation efficiency is adjusted to 10%.

[0023] The output content of the rule recommendation engine includes a Top-N recommended rule list arranged in descending order of matching scores and a dimension score radar chart of each recommended rule.

[0024] The rule recommendation engine supports dynamic optimization of the model, including training the weight coefficients of each evaluation dimension through historical matching data, and updating the parameters of the multi-dimensional scoring model every month; and supports manual calibration of multi-dimensional attributes of rules, including adjusting the priority, specialization degree and complexity level of the rules to adapt to changes in business requirements.

[0025] Optionally, the method further comprises:

[0026] editing the attributes of the rules in the rule library, including modifying the semantic features, action types, priorities, specialization degrees and complexity levels of the rules;

[0027] adding new data quality rules to the rule library and automatically completing multi-dimensional attribute modeling of the new rules;

[0028] adapting the semantic descriptions of the multi-language fields to the semantic descriptions of the multi-language rules, and realizing semantic matching of the fields and the rules in different languages through the multi-language capability of the BERT pre-training model.

[0029] According to another aspect of the present application, a field rule matching recommendation device based on multi-dimensional feature fusion is provided, comprising:

[0030] An acquisition unit is configured to acquire user input information, wherein the user input information includes a target data table and a current business target.

[0031] An extraction unit is configured to perform feature extraction on the fields in the target data table through a field feature extraction module to obtain field multi-dimensional features.

[0032] A modeling unit is configured to perform attribute modeling on data quality rules in a preset rule library through a rule attribute modeling to obtain rule multi-dimensional attributes.

[0033] a computing unit configured to calculate a matching score of the field multi-dimensional feature and the rule multi-dimensional attribute by using a multi-dimensional scoring model;

[0034] an output unit configured to dynamically adjust weights of each dimension in the multi-dimensional scoring model according to a business target priority by using a rule recommendation engine, and output Top-N recommended rules based on the adjusted weights and the matching score.

[0035] According to another aspect of the present application, an electronic device is provided, which comprises:

[0036] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the field rule matching recommendation method based on multi-dimensional feature fusion according to any one of the embodiments of the present application.

[0037] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the field rule matching recommendation method based on multi-dimensional feature fusion according to any one of the embodiments of the present application.

[0038] The embodiments of the present application provide a field rule matching recommendation method and device based on multi-dimensional feature fusion, and Top-N applicable rules can be automatically recommended by inputting a data table and a business target, and six dimensions including semantic similarity, generality, priority, rule specialization degree, complexity and business target adaptability are comprehensively considered to improve matching accuracy.

[0039] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is a flowchart of a field rule matching recommendation method based on multi-dimensional feature fusion provided by an embodiment of the present application;

[0042] Figure 2 is a structural schematic diagram of a field rule matching recommendation device based on multi-dimensional feature fusion provided by an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of an electronic device implementing a field rule matching recommendation method based on multi-dimensional feature fusion of an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] As shown in Figure 1 , the embodiments of the present application provide a field rule matching recommendation method based on multi-dimensional feature fusion, which can include the following steps:

[0047] S110, obtaining user input information, the user input information including a target data table and a current business target.

[0048] The user needs to provide two types of key information: one is the target data table, i.e. the specific data table that needs to be governed by data quality, which contains multiple fields to be matched rules and is the only data source for subsequent field feature extraction; the other is the current business target, i.e. the specific governance direction that the user expects to achieve through rule matching, which directly determines the weight adjustment logic of each dimension in the subsequent multi-dimensional scoring model, ensuring that the recommended result fits the user's core needs.

[0049] S120, performing feature extraction on the fields in the target data table through a field feature extraction module to obtain field multi-dimensional features.

[0050] The core target of the field feature extraction module is to mine key information reflecting business attributes, data characteristics and scene attribution from the fields of the target data table, form a complete feature system, and provide field-side basis for accurate matching with rules.

[0051] In the embodiment of the application, the field multi-dimensional features include text semantic features, data type features, statistical features and business process labels, wherein the text semantic features are obtained by converting the field description into a high-dimensional semantic vector through a BERT pre-training model; the data type features are data type identifiers of the fields, and the data types include numerical type, date type and string type; the statistical features include the missing rate, the proportion of outliers, the average value, the standard deviation and the number of unique values of the field sample values; and the business process label is an identifier of a business process to which the field belongs, and the business process includes a sales process, a supply chain management process and a financial audit process.

[0052] The text semantic features are core features reflecting the business attributes of the fields, and the core target thereof is to realize accurate matching with rule semantics by quantifying the semantic information of the field description, and to avoid deviation caused by only relying on literal matching of the field name. Taking the field description as a data source, such description directly contains the business meaning of the field and is a core carrier of semantic matching. The BERT pre-training model is used for processing, and the natural language text of the field description is input into the BERT model, and the model converts the text into a high-dimensional semantic vector by understanding the context semantics. Such a vector can not only capture the literal meaning, but also identify deep business associations. As a core basis for matching the field with the rule semantics, the subsequent rule attribute modeling will extract the semantic features of the rule, and by calculating the similarity between the field semantic vector and the rule semantic vector, it is determined whether the business meanings of the two are consistent.

[0053] The data type feature is a basic identity identifier of the field, directly determines the applicable range of the rule, and is a hard threshold for field-rule matching, avoiding invalid matching caused by incompatible rule and field type. The data type identifier of the field is obtained by automatic recognition or manual annotation of the system, and the data type category to which the field belongs is directly used as a direct basis for matching the action type of the rule. The data type feature includes three types of core data types, including:

[0054] Numerical type: suitable for indicating quantifiable fields such as quantity, amount, index, etc., such as “order amount”, “inventory quantity” and “product unit price”;

[0055] Date type: suitable for indicating time and date fields, such as “order creation date”, “contract expiration time” and “financial settlement date”; String type: suitable for indicating non-quantitative fields such as text, code and identification, such as “customer number”, “product model” and “invoice number”.

[0056] The rule attribute modeling defines the action type of the rule. Only when the field data type feature is consistent with the rule action type, the rule has the basic condition to adapt to the field.

[0057] The statistical feature is a feature extracted from the field sample value, reflecting the data distribution law. The core goal is to dynamically adjust the recommended priority of the rule according to the data quality status of the field, so that the matching result is more in line with the actual governance needs of the field.

[0058] The sample value of the field generates features through statistical analysis of the sample value, including five specific statistical indicators, which are:

[0059] Missing rate: The proportion of null values (or invalid values) in the total sample number in the field sample value, reflecting the data integrity of the field (for example, the missing rate of a certain "contact information" field is 30%, indicating that there are more data missing in this field);

[0060] Outlier proportion: The proportion of values that exceed the normal range (such as determined by box plot and standard deviation method) in the total sample number in the field sample value, reflecting the data accuracy of the field (for example, the outlier proportion of a certain "product unit price" field is 15%, indicating that there are more abnormal price data in this field);

[0061] Average value: For numerical fields, calculate the arithmetic mean of sample values, reflecting the overall level of the field data;

[0062] Standard deviation: For numerical fields, calculate the dispersion of sample values, reflecting the data volatility;

[0063] Number of unique values: The number of non-repeated values in the field sample value, reflecting the diversity of the field data.

[0064] Dynamic adjustment of rule priority, the subsequent multi-dimensional scoring model will adjust the priority score of the rule combined with the statistical characteristics, to ensure that the recommended rule can accurately solve the data quality problem of the field.

[0065] Business process label is a feature that identifies the business scenario to which the field belongs. Its core goal is to make field rule matching more in line with specific business needs, and avoid the problem of insufficient adaptability of general rules in vertical business scenarios. The identification of the business process to which the field belongs, that is, by associating the field with its application scenario in the enterprise business system, the field is labeled with a clear business process label, which is the core basis for rule scenario matching. It includes three core business processes, which are:

[0066] Sales process: applicable to fields related to the sales process of the enterprise, such as "order number", "sales amount", "customer order time", "product sales";

[0067] Supply chain management process: suitable for fields related to the supply chain of the enterprise, such as "inventory quantity", "purchase order number", "logistics delivery time", "supplier code";

[0068] Financial audit process: suitable for fields related to the financial and audit aspects of the enterprise, such as "invoice amount", "tax amount", "financial closing date", "cost accounting value".

[0069] In the embodiments of the present application, the application logic of the statistical features includes: when the missing rate of the field is greater than the preset threshold, the field is automatically marked to need to match the non-empty check type rule in priority after field feature extraction; when the proportion of outliers of the field is greater than the preset threshold, the field is automatically marked to need to adapt to the outlier detection type rule.

[0070] After completing the field statistical feature extraction, the missing rate and the proportion of outliers of the field are judged based on the preset threshold, and the corresponding rule matching priority is marked accordingly. If the field missing rate exceeds the preset threshold (such as the threshold is set to 20%), it means that there are many null values in the field, and the data integrity is insufficient, so the system will automatically mark the field to need to match the non-empty check type rule in priority, to ensure that the data missing problem is solved in priority; if the proportion of outliers of the field exceeds the preset threshold (such as the threshold is set to 15%), it means that there are many abnormal data in the field, and the data accuracy is questionable, so the system will automatically mark the field to need to adapt to the outlier detection rule (such as "box plot outlier identification" "standard deviation method outlier filtering"), to solve the data abnormal problem. The core of this logic is to make the rule matching strongly related to the actual data quality problem of the field, to avoid indiscriminate recommendation, and to improve the matching accuracy.

[0071] S130, attribute modeling of data quality rules in the preset rule library is performed through rule attribute modeling, to obtain rule multi-dimensional attributes.

[0072] For the data quality rules in the preset rule library, the applicable scene, importance, and implementation difficulty of each rule are determined through attribute decomposition, to form an attribute system on the rule side, to provide a basis on the rule side for matching with the field.

[0073] In the embodiments of the present application, the rule multi-dimensional attributes include semantic features, action types, priorities, specialization degrees, and complexities; wherein the semantic features are obtained from the core business semantics of the rule name; the action type is a data type identifier applicable to the rule, corresponding to the data type feature of the field; the priority is the importance level of the rule in data quality governance; the specialization degree is the specificity level of the applicable scene of the rule; and the complexity is the technical difficulty and the level of calculation resource consumption of the implementation of the rule.

[0074] The semantic feature is a core business semantic extracted from the rule name, and is a key basis for judging whether the rule and the field business logic are consistent. The extraction logic is: directly stripping the non-core information from the rule name, retaining the core business direction, and forming the semantic identification of the rule. The core role of this attribute is to match the text semantic features of the field. By calculating the semantic similarity between the two, rules consistent with the business meaning of the field are selected, avoiding the problem of disconnection between rule function and field business attribute.

[0075] The action type is a data type identification applicable to the rule, directly corresponding to the data type feature of the field, and is a hard screening condition for rule matching. The definition logic is: according to the rule function to determine the applicable field data type, forming the type adaptation identification of the rule. The core role of this attribute is to quickly filter rules that are not compatible in type. Only when the rule action type is consistent with the field data type feature, the rule has a matching basis, avoiding invalid matching.

[0076] The priority is the importance level of the rule in data quality governance, used to determine the recommended order when multiple rules are adapted. The division logic is: based on the influence degree of the rule on the core target of data quality to set the level. The core role of this attribute is to dynamically adjust the recommended weight combined with the field statistical features. High-priority rules that have a greater impact on data quality will get a higher score in matching, ensuring that key data quality problems are solved through rules first.

[0077] The specialization degree is the specificity level of the rule applicable scenario, used to adapt to the field matching needs in different business scenarios. The division logic is: according to the width of the rule applicable range to set the level. The narrower the applicable scenario, the higher the specialization degree; the wider the applicable scenario, the lower the specialization degree. The core role of this attribute is to match the business process label of the field. High-specialization-degree rules will preferentially match fields corresponding to the business process, improving the matching accuracy of vertical business scenarios.

[0078] The complexity is the technical difficulty and computational resource consumption level of rule implementation, used to balance the rule effect and the feasibility of landing. The division logic is: based on the technical means and computational resources required for rule implementation to set the level. Rules with low implementation difficulty and low resource consumption have low complexity; rules requiring complex algorithms have high complexity. The core role of this attribute is to adjust the recommended tendency combined with business goals. In resource-limited scenarios, low-complexity rules will get a higher score, ensuring that rules can be efficiently landed, and avoiding the difficulty of implementation due to high complexity.

[0079] S140, calculating the matching score of the multi-dimensional features of the field and the multi-dimensional attributes of the rule through a multi-dimensional scoring model.

[0080] By comprehensively evaluating the dimensions, the degree of fit between the two is quantified and the matching score is output.

[0081] In the embodiments of the present application, the evaluation dimensions of the multi-dimensional scoring model include semantic similarity, generality, priority, specialization, complexity, and business goal alignment.

[0082] The semantic similarity is obtained by calculating the cosine similarity of the field text semantic vector and the rule semantic feature vector, and the value range is [0, 1]. The generality is obtained by pre-defined rule action type and field data type matching matrix normalization processing, and the value range is [0, 1]. The priority is obtained by combining rule pre-set priority level and field statistical characteristics after dynamic adjustment and normalization, and the value range is [0, 1]. The specialization is obtained by reverse normalization processing, which converts the specialization to a value range of [0.2, 1.0]. The complexity is obtained by converting it into a cost coefficient and normalizing it to a value range of [0.2, 1.0]. The business goal alignment is obtained by constructing a business process and rule mapping matrix combined with TF-IDF text similarity calculation, and the value range is [0, 1].

[0083] The matching score is calculated by a weighted linear combination model, and the calculation formula is: Sere=Σ(Wi×Di), where Wi is the weight of each evaluation dimension, the sum of all dimension weights is 1, and Di is the normalized score of each evaluation dimension.

[0084] The semantic similarity matches the business meaning of the field and the rule, which is based on the text semantic features of the field (high-dimensional semantic vector converted by BERT model) and the semantic features of the rule (vector converted after extracting the core semantics from the rule name). The similarity of the two vectors is calculated by the cosine similarity algorithm, and the closer the value is to 1, the more consistent the business meaning of the field and the rule is. The value range is [0, 1], 0 means complete mismatch of business meaning, and 1 means complete match.

[0085] The generality is used to judge the type compatibility of the field and the rule, which is based on the pre-defined rule action type and field data type matching matrix. First, it is judged whether the types of the two are compatible according to the matching matrix, and the basis score is 1 if they are compatible, and 0 if they are not compatible. Then it is normalized. The value range is [0, 1], 0 means complete incompatibility of types, and 1 means complete compatibility.

[0086] The priority represents the importance weight of the dynamically adjusted rule, which combines two factors, one is the pre-set priority level of the rule, and the other is the statistical characteristics of the field. First, the pre-set priority level of the rule is dynamically adjusted according to the statistical characteristics of the field, such as adjusting the pre-set 5-level "non-empty check" rule to 5.5 due to high field missing rate, and then normalizing the adjusted level to the interval [0, 1], such as mapping the 1-5 level and the adjusted level to the score of 0-1. The value range is [0, 1], 0 means very low importance of the rule, and 1 means very high importance.

[0087] The specialization degree represents the adaptability of the rule in the highlighted vertical scene. Based on the preset specialization degree level of the rule, it is defined as level 1-5, and level 5 represents the most specific applicable scene, such as rules that only adapt to the financial audit scene; level 1 represents the most general applicable scene, such as general non-empty check rules. By using reverse normalization processing, the specialization degree level of 1-5 is converted according to the mapping relationship of “level 1→0.2, level 2→0.4, level 3→0.6, level 4→0.8, and level 5→1.0”, to ensure that the high specialization degree rule has a higher score in the field matching of the corresponding business scene. The value range is [0.2, 1.0], and 0.2 represents that the rule has a very wide applicable scene, and 1.0 represents that the rule has a very specific applicable scene.

[0088] The complexity represents the balance between the cost and feasibility of implementing the rule. Based on the preset complexity level of the rule, it is defined as level 1-5, and level 1 represents the lowest difficulty and the least resource consumption, such as non-empty check; level 5 represents the highest difficulty and the most consumption, such as time series trend analysis. First, the complexity level is converted into a cost coefficient, and the formula is “6-complexity level”, such as level 1→5 and level 5→1. Then, the cost coefficient is normalized to the interval [0.2, 1.0] to ensure that the low complexity rule has a higher score. The value range is [0.2, 1.0], and 0.2 represents that the rule has a very high implementation cost, and 1.0 represents that the rule has a very low implementation cost.

[0089] The business target alignment degree represents the matching of the field and the scene target of the rule. Two factors are combined, one is the constructed business process-rule mapping matrix, and the other is the TF-IDF text similarity, which calculates the text similarity between the field business process label and the rule applicable scene description. First, the rules that adapt to the field business process are selected through the mapping matrix, and then the TF-IDF text similarity is combined to optimize the score, and finally normalized to the interval [0, 1]. The value range is [0, 1], 0 represents that the rule is completely inconsistent with the business target of the field, and 1 represents that it is completely consistent.

[0090] The final matching score of the field and the rule is integrated by a weighted linear combination model of the scores of the six dimensions, and the formula strictly follows the file definition:

[0091] The calculation formula is: Sere=Σ(Wi×Di);

[0092] Wherein, Wi: the weight of each evaluation dimension, the total weight of all dimensions is 1, and the weight will be dynamically adjusted according to the user input business target, such as increasing the semantic similarity weight to 30% in the core business scene and increasing the generality weight to 20% in the general scene;

[0093] Di: the normalized score of each evaluation dimension, which is the score calculated by the above six dimensions and is in the corresponding value range.

[0094] S150, dynamically adjusting the weight of each dimension in the multi-dimensional scoring model according to the business target priority through the rule recommendation engine, and outputting Top-N recommended rules based on the adjusted weight and the matching score.

[0095] By dynamically adjusting the weight, the recommended result is adapted to the user's business target, and the optimal rule is finally output.

[0096] In the embodiment of the application, the rule recommendation engine uses the analytic hierarchy process to realize dynamic weight distribution of each evaluation dimension.

[0097] In the core business scenario, the weight of the semantic similarity dimension is adjusted to 30%, and the weight of the rule priority dimension is adjusted to 25%; in the general governance scenario, the weight of the type compatibility dimension is adjusted to 20%, and the weight of the implementation efficiency is adjusted to 10%.

[0098] The output content of the rule recommendation engine includes a Top-N recommended rule list ranked in descending order of matching score, and a dimension score radar chart of each recommended rule.

[0099] The rule recommendation engine supports dynamic optimization of the model, including training the weight coefficient of each evaluation dimension through historical matching data, and updating the parameters of the multi-dimensional scoring model every month; and supporting manual calibration of multi-dimensional attributes of rules, including adjusting the priority, specialization degree and complexity level of the rules, to adapt to changes in business requirements.

[0100] Using the analytic hierarchy process, the weights of the six evaluation dimensions are adjusted according to different scenarios. The core business scenario focuses on business adaptation, with a semantic similarity weight of 30% and a rule priority weight of 25%; the general governance scenario focuses on compatibility and efficiency, with a type compatibility weight of 20% and an implementation efficiency weight of 10%.

[0101] The output content is a Top-N recommended rule list sorted in descending order of matching score, and a dimension score radar chart of each rule, which intuitively displays the advantages and disadvantages of the rules.

[0102] The dimension weight coefficient is trained through historical matching data, and the scoring model parameters are updated every month; on the other hand, the rule attributes are manually calibrated, and the priority, specialization degree, complexity level of the rules can be adjusted to adapt to changes in business requirements.

[0103] In the embodiment of the application, the method can further include the following steps:

[0104] The attributes of the rules in the rule library are edited, including modifying the semantic features, action type, priority, specialization degree and complexity of the rules.

[0105] New data quality rules are added to the rule library, and multi-dimensional attribute modeling of the new rules is automatically completed.

[0106] The semantic adaptation of the multilingual field description and the multilingual rule description is achieved by the multilingual capability of the BERT pre-training model to match the semantics of the fields and rules in different languages.

[0107] For the existing data quality rules in the rule library, manual modification of core attributes is supported to ensure that rule attributes can be dynamically adjusted according to business demand changes. The specific editing range strictly corresponds to the multi-dimensional attribute system of the rule.

[0108] Semantic feature editing: The core business semantics extracted from the rule name can be modified, such as adjusting the semantic feature of date format verification (YYYY-MM-DD) from date format verification to financial scenario date format verification, to ensure that the semantic description is more consistent with the business scenario; The data type identifier applicable to the rule can be adjusted, such as extending the original range verification rule applicable to numerical values to simultaneously adapt to numerical values and dates.

[0109] Priority editing can reset the importance level of rules in data quality governance, such as upgrading the cross-border order amount verification rule from level 3 to level 5, ensuring that core business data is prioritized for governance.

[0110] Specialization editing can adjust the specificity level of the rule applicable scenario, such as upgrading the specialization of the original general invoice amount verification rule from level 2 to level 4, limiting it to cross-border e-commerce invoice amount verification, and adapting to vertical scenarios.

[0111] Complexity editing can adjust the technical difficulty and resource consumption level of the rule according to the implementation technology upgrade, such as reducing the complexity of the time series anomaly detection rule from level 4 to level 2 due to algorithm optimization, reducing implementation costs.

[0112] When adding a new data quality rule to the rule library, the rule multi-dimensional attribute modeling is automatically completed without manual configuration one by one, and the modeling logic is consistent with the existing rule attribute system. When adding a new rule, the user only needs to input the rule name, applicable data type, automatically extract the rule semantic feature, automatically match and generate the action type, and assign the initial priority, specialization, and complexity based on the rule function. After automatic modeling, the new rule is directly included in the rule library and can participate in subsequent field-rule matching. It also supports manual adjustment of the initial attributes based on business needs.

[0113] The semantic matching of field description and rule description in different languages is realized through the multilingual capability of the BERT pre-training model, and the language barrier is broken. When the field description is not in Chinese and the rule description is in Chinese, or vice versa, the system calls the multilingual BERT model. The model converts the field description and the rule description in different languages into high-dimensional semantic vectors in a unified dimension. Subsequently, the cosine similarity of the semantic vectors in different languages is calculated to judge the business meaning fit degree of the field and the rule, so that accurate semantic matching can still be realized in a multilingual scene, and the global data governance needs of enterprises are adapted.

[0114] As shown in Figure 2 The embodiment of the present application provides a field rule matching recommendation device based on multi-dimensional feature fusion, which comprises:

[0115] The acquisition unit 210 is configured to acquire user input information, and the user input information comprises a target data table and a current business target.

[0116] The extraction unit 220 is configured to perform feature extraction on the fields in the target data table through a field feature extraction module to obtain field multi-dimensional features.

[0117] The modeling unit 230 is configured to perform attribute modeling on the data quality rules in the preset rule library through a rule attribute modeling to obtain rule multi-dimensional attributes.

[0118] The calculation unit 240 is configured to calculate the matching score of the field multi-dimensional features and the rule multi-dimensional attributes through a multi-dimensional scoring model.

[0119] The output unit 250 is configured to dynamically adjust the weight of each dimension in the multi-dimensional scoring model according to the business target priority through a rule recommendation engine, and output Top-N recommended rules based on the adjusted weight and the matching score.

[0120] It can be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the field rule matching recommendation device based on multi-dimensional feature fusion. In other embodiments of the present application, the field rule matching recommendation device based on multi-dimensional feature fusion can include more or fewer components than the diagram, or combine certain components, or split certain components, or different component arrangement. The components shown in the diagram can be realized by hardware, software or a combination of software and hardware.

[0121] The information interaction, execution process and the like between the units in the above device are based on the same concept as the method embodiment of the present application, and the specific content can be referred to the description in the method embodiment of the present application, which will not be described here.

[0122] Figure 3A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0123] As shown in Figure 3 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected in communication with the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0125] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the field rule matching recommendation method based on multi-dimensional feature fusion.

[0126] In some embodiments, the field rule matching recommendation method based on multi-dimensional feature fusion can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the field rule matching recommendation method based on multi-dimensional feature fusion described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the field rule matching recommendation method based on multi-dimensional feature fusion by any other suitable means, e.g., by way of firmware.

[0127] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0128] Computer programs implementing methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or the block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, or entirely on a remote machine or server.

[0129] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0131] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0133] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0134] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A field rule matching recommendation method based on multi-dimensional feature fusion, characterized in that, include: Obtain user input information, which includes the target data table and the current business objective; The field feature extraction module extracts features from the fields in the target data table to obtain multi-dimensional features of the fields; By performing attribute modeling on the data quality rules in the preset rule base, multi-dimensional attributes of the rules are obtained. The matching score between the multi-dimensional features of the field and the multi-dimensional attributes of the rule is calculated using a multi-dimensional scoring model. The rule-based recommendation engine dynamically adjusts the weights of each dimension in the multi-dimensional scoring model according to the priority of business objectives, and outputs the Top-N recommendation rules based on the adjusted weights and matching scores. The multi-dimensional features of the field include textual semantic features, data type features, statistical features, and business process tags. The textual semantic features are obtained by converting the field description into a high-dimensional semantic vector using a BERT pre-trained model. The data type features are the data type identifiers of the field, including numeric, date, and string types. The statistical features include the missing rate, outlier ratio, mean, standard deviation, and number of unique values ​​for the field's sample values. The business process tags are the identifiers of the business process to which the field belongs, including sales processes, supply chain management processes, and financial audit processes. The rule's multi-dimensional attributes include semantic features, action type, priority, specialization, and complexity. The semantic features are derived from the core business semantics extracted from the rule name. The action type is the data type identifier to which the rule applies, corresponding to the data type features of the field. The priority is the importance level of the rule in data quality governance. The specialization is the specificity level of the rule's applicable scenario. The complexity is the level of technical difficulty and computational resource consumption in implementing the rule. The evaluation dimensions of the multi-dimensional scoring model include semantic similarity, generality, priority, specialization, complexity, and alignment with business objectives. The semantic similarity is obtained by calculating the cosine similarity between the field text semantic vector and the rule semantic feature vector, with a value range of [0,1]. The generality is obtained by normalizing the predefined rule action type and field data type matching matrix, with a value range of [0,1]. The priority is obtained by normalizing the rule's preset priority level and field statistical features after dynamic adjustment, with a value range of [0,1]. The specialization degree is obtained by reverse normalization, converting the specialization degree to a value range of [0.2,1.0]. The complexity is obtained by converting it into a cost coefficient and normalizing it to a value range of [0.2,1.0]. The business goal alignment degree is obtained by constructing a business process and rule mapping matrix and calculating it using TF-IDF text similarity, with a value range of [0,1]. The matching score is calculated using a weighted linear combination model. The calculation formula is: Sere=Σ(Wi×Di), where Wi is the weight of each evaluation dimension, the sum of the weights of all dimensions is 1, and Di is the normalized score of each evaluation dimension.

2. The method according to claim 1, characterized in that, The application logic of the statistical features includes: when the missing rate of a field is greater than a preset threshold, the field is automatically marked as needing to be preferentially matched with non-empty check rules after the field features are extracted; when the proportion of outliers in a field is greater than a preset threshold, the field is automatically marked as needing to be matched with outlier detection rules.

3. The method according to claim 1, characterized in that, The rule recommendation engine uses the hierarchical analysis method to achieve dynamic weight allocation for each evaluation dimension; In core business scenarios, the weight of semantic similarity is adjusted to 30%, and the weight of rule priority is adjusted to 25%; in general governance scenarios, the weight of type compatibility is adjusted to 20%, and the weight of implementation efficiency is adjusted to 10%. The output of the rule recommendation engine includes a list of Top-N recommendation rules sorted in descending order of matching score, and a radar chart of the dimensional scores of each recommendation rule; The rule recommendation engine supports dynamic model optimization, including training the weight coefficients of each evaluation dimension using historical matching data and updating the parameters of the multi-dimensional scoring model monthly. It also supports manual calibration of rule multi-dimensional attributes, including adjusting the priority, specialization, and complexity level of rules to adapt to changes in business needs.

4. The method according to claim 1, characterized in that, The method further includes: Edit the attributes of rules in the rule base, including modifying the semantic features, action type, priority, specialization degree, and complexity of the rules; Add data quality rules to the rule base and automatically complete multi-dimensional attribute modeling for the new rules; Semantic adaptation of multilingual field descriptions and multilingual rule descriptions is achieved by leveraging the multilingual capabilities of the BERT pre-trained model to realize semantic matching of fields and rules in different languages.

5. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the field rule matching recommendation method based on multi-dimensional feature fusion as described in any one of claims 1-4.

6. A computer-readable medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the field rule matching recommendation method based on multi-dimensional feature fusion as described in any one of claims 1-4.

Images