A data asset management method, apparatus, and storage medium

Abstract

This invention discloses a data asset management method, comprising the following steps: determining a rule expression for data asset management; constructing a rule engine adapted to the data assets and a data model corresponding to the rule engine; inputting the data model and the rule expression into the rule engine; using the rule expression in the rule engine to iteratively calculate the dataset corresponding to the data model, and using the result of the iterative calculation of the dataset as the result of managing the data assets contained in the data model. This invention also discloses a data asset management device and a storage medium.

Description

Technical Field

[0001] This invention relates to the field of big data rule calculation technology, and in particular to a data asset management method, apparatus, and storage medium. Background Technology

[0002] With the development of computer technology, computer software can now independently complete some data processing based on various rules input by humans, and non-technical personnel can operate it. Especially in fields where the rules and logic are frequently changing and complex, computer computing technology has an advantage that cannot be matched by manual calculation.

[0003] However, in existing data rule calculations, the typical approach is to retrieve the dataset to be calculated using a single SQL statement, and then perform rule calculations on the dataset using a single rule. When there are multiple rules in the database, multiple SQL statements need to be compiled to ensure a one-to-one correspondence between the data model and the rules in the rule base.

[0004] The drawbacks of existing technologies are that using multiple SQL statements to process data makes it difficult to handle the interrelationships between data; because each SQL statement is bound to a specific rule, the reusability of SQL statements is poor; and multiple SQL statements in the rule engine query the same data source, resulting in a large amount of repeated data scanning and wasting computing resources and disk I / O. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a data management method, a data management device, and a readable storage medium storing code for implementing the data management method.

[0006] In a first aspect, the present invention provides a data asset management method, comprising: determining a rule expression for data asset management; constructing a rule engine adapted to the data asset and a data model corresponding to the rule engine; inputting the data model and the rule expression into the rule engine; using the rule expression in the rule engine to iteratively calculate the dataset corresponding to the data model, and using the result of iteratively calculating the dataset as the result of managing the data assets contained in the data model.

[0007] Furthermore, inputting the data model and the rule expression into the rule engine includes: compiling the rule expression into a rule set; matching the data model with the rule expression in the rule set through the rule engine to obtain a matching result; and using the matching result as the basis for iterative calculation of the dataset corresponding to the data model.

[0008] Furthermore, constructing the data model corresponding to the rule engine includes: constructing a basic data model corresponding to the rule engine through a scripting language or database management system; storing the structure and relationships of the data in the data assets in the basic data model; and converting the basic data model into multiple sub-data models contained under the basic data model.

[0009] Furthermore, the iterative calculation of the dataset corresponding to the data model using the rule expressions in the rule engine includes: determining whether the basic data model contains the sub-data model; if a sub-data model exists, then using the dataset corresponding to the sub-data model as the input source of the rule engine, and using the rule set corresponding to the sub-data model to perform iterative calculation on the dataset corresponding to the sub-data model; if no sub-data model exists, then using the dataset corresponding to the basic data model as the input source of the rule engine, and using the rule set corresponding to the basic data model to perform iterative calculation on the dataset corresponding to the basic data model.

[0010] Further, after obtaining the sub-data model or the rule set corresponding to the data model, the iterative calculation of the dataset corresponding to the data model or the sub-data model using the rule set includes: determining whether there are preset global class rules in the rule set; if there are global class rules, compiling the global class rules and prioritizing the calculation of global class rules on the dataset; if there are no global class rules, publishing the preset detailed class rules to the distributed data nodes and compiling the detailed class rules on the data nodes; obtaining detailed data as input data and performing detailed class rule calculations.

[0011] Furthermore, the iterative calculation of the dataset corresponding to the data model using the rule engine further includes: compiling the data model to be used, compiling the rule set corresponding to the data model on the distributed data node, and monitoring the rule change log storing rule expression change information in real time; acquiring real-time data, inputting the real-time data into the data model for conversion to form a dataset corresponding to the data model; and performing real-time iterative calculation of the dataset using the rule set.

[0012] Furthermore, the real-time iterative calculation of the dataset using the rule set includes: storing a state table generated by the real-time iterative calculation of the dataset and storing the real-time calculation results, and performing global rule iterative calculation or detailed rule iterative calculation using the state table as the input source.

[0013] In one embodiment, the data asset management method of the present invention further includes: performing indicator system calculation, data asset rule calculation, and process management on the dataset corresponding to the data model through the rule engine.

[0014] In a second aspect, the present invention provides a data asset management device, the device including a memory and a processor, the memory storing program instructions executable by the processor, the processor calling the program instructions to execute the data asset management method described in the first aspect.

[0015] Thirdly, the present invention provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data asset management method described in the first aspect.

[0016] Compared to existing technologies, the data management method provided by this invention constructs a corresponding rule engine based on the data processing purpose, builds a corresponding data model based on the rule engine, compiles the rule expressions required for data processing into the rule engine for storage, and when data is input into the rule engine, it uses the data model to transform the data into a corresponding dataset, and processes the dataset by matching the corresponding rules. By storing rule sets and data models in the rule engine, and matching the input data with the rule expressions, rule calculations on the same basic data across different dimensions become convenient and efficient. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the data asset management method provided in an embodiment of the present invention;

[0018] Figure 2 for Figure 1 A flowchart illustrating the batch rule calculation method included in step S3 of the data asset management method flowchart.

[0019] Figure 3 for Figure 1 A flowchart illustrating the real-time rule calculation method included in step S3 of the data asset management method flowchart.

[0020] Figure 4 This is a schematic diagram of the device structure of the data asset management device provided in an embodiment of the present invention. Detailed Implementation

[0021] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of preferred embodiments with reference to the accompanying drawings. Through the description of the specific embodiments, a more in-depth and specific understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the present invention.

[0022] Example 1

[0023] This invention provides a data asset management method; please refer to the appendix. Figure 1 Data asset management methods include the following steps:

[0024] Step S1: Determine the rule expression used for data asset management;

[0025] Step S2: Construct a rule engine adapted to the data assets and a data model corresponding to the rule engine, and input the data model and the rule expression into the rule engine;

[0026] Step S3: In the rule engine, the rule expression is used to iteratively calculate the dataset corresponding to the data model, and the result of the iterative calculation of the dataset is used as the result of managing the data assets contained in the data model.

[0027] It is understood that determining the rule expression used for data asset management in step S1 includes determining the matching pattern, conditions, and logical relationships of the rule expression; and selecting the rule expression language, which may include, but is not limited to, SQL, Scala, MEVALE, etc.

[0028] After determining the rule expression, the process also includes verifying the accuracy and effectiveness of the rule expression using test data and real-world cases to ensure that the rule expression can correctly match the target pattern or condition; debugging and optimizing the rule expression, embedding the rule expression into the rule engine, and making necessary corrections; and maintaining and updating the rule expression to ensure that the rule expression remains consistent with changes in the system.

[0029] Optionally, step S2, inputting the data model and the rule expression into the rule engine, includes: compiling the rule expression into a rule set; matching the data model with the rule expression in the rule set through the rule engine to obtain a matching result; and using the matching result as the basis for iterative calculation of the dataset corresponding to the data model.

[0030] Furthermore, compiling the rule expression into a rule set includes: predefining rule expressions with the same constraint rules in the rule expressions used for data asset management, generating a predefined rule containing multiple identical rule expressions, referencing the rule expression by calling fields in the predefined rule; using logical operators to establish logical relationships between multiple rule expressions to form a composite rule for satisfying the computational requirements of composite conditions; and compiling the predefined rule and the composite rule to form the rule set.

[0031] Specifically, the predefined rules reduce and classify rule expressions with the same constraints in the rule expressions used to input the rule engine, grouping rule expressions with the same constraints into one category. Each category is compiled only once, thus saving computing resources and disk I / O. In addition, predefining rule expressions enables unified management of rules, ensuring consistency of rules throughout the system. Predefined rule expressions are shared and reused by multiple components and systems, improving code maintainability and scalability.

[0032] For the same rule expression, a repetition rule is predefined. When referencing a repetition rule for a specific data field in the dataset corresponding to the data model, the reference is made by passing the parameter of the specific data field represented by the corresponding parameter name. When the referenced rule is a constraint rule for an empty field, the field name is passed as a parameter to reference that field.

[0033] Optionally, after compiling the rule expressions into a rule set, the following operations can be performed on the rule set: Define operations for each rule expression in the rule set; the operation definition specifies the specific action corresponding to the condition in the rule expression that should be executed when the condition in the rule expression is met; operations can include triggering events, calling functions, modifying data, sending notifications, etc.; rule set organization: after the rule expressions are determined, they need to be organized and managed, which includes determining the priority and execution order of the rules, and organizing related rules together to form a rule set; rule set maintenance and updates: rule sets are dynamic and need to be maintained and updated; as business needs change or new rules are introduced, the rule set may need to be modified, added, or deleted.

[0034] Further, in step S3, constructing the data model corresponding to the rule engine includes: constructing a basic data model corresponding to the rule engine through a scripting language or database management system; storing the structure and relationships of the data in the data assets into the basic data model; and converting the basic data model into multiple sub-data models contained under the basic data model.

[0035] Optionally, the scripting language used to build the basic data model can be SQL, Scala, MEVALE, etc., and the database management system can be MySQL, PostgreSQL, Oracle Database, MongoDB, Redis, etc. In practical applications, the appropriate database can be selected based on factors such as application requirements, performance requirements, data model, and budget.

[0036] Further, step S3, which uses the rule engine to iteratively calculate the dataset corresponding to the data model, includes: determining whether the basic data model contains the sub-data model; if it contains the sub-data model, then using the dataset corresponding to the sub-data model as the input source of the rule engine, and using the rule set corresponding to the sub-data model to iteratively calculate the dataset corresponding to the sub-data model; if there is no sub-data model, then using the dataset corresponding to the basic data model as the input source of the rule engine, and using the rule set corresponding to the basic data model to iteratively calculate the dataset corresponding to the basic data model.

[0037] Understandably, the rule expressions provided in this embodiment include global rules and detailed rules.

[0038] Global rules are rules that apply to the entire dataset or data stream. They typically focus on logic and conditions across multiple data objects or attributes. Global rules can be used to perform overall validation, computation, and decision-making for a dataset.

[0039] Global rules are suitable for scenarios where rule evaluation and processing need to be based on the overall dataset or the relationships and interactions between multiple data objects. Examples include risk assessment of all orders in a dataset and calculating the total sales revenue.

[0040] Detailed rules are rules applied to individual data objects. They focus on the attributes and conditions of a specific data object and execute corresponding logic and operations based on these attributes and conditions. Detailed rules are typically used for validation, calculation, and decision-making on individual data objects. They are suitable for scenarios that require rule evaluation and processing of individual data objects. Examples include validating whether a customer's name conforms to a specified format and calculating discounts on order amounts.

[0041] Therefore, the data asset management method provided in this embodiment combines global rules and detailed rules, enabling the data asset management method provided by the present invention to not only have the high computational efficiency of global rules and unified calculation of large amounts of data, but also the accuracy and flexibility of detailed rules in data processing. The present invention makes the quality assessment of data assets and the calculation of data indicators in the field of big data efficient, accurate and flexible.

[0042] Further, after obtaining the sub-data model or the rule set corresponding to the data model, the iterative calculation of the dataset corresponding to the data model or the sub-data model using the rule set includes the following operations: determining whether there are preset global class rules in the rule set; if there are global class rules, compiling the global class rules and prioritizing the calculation of global class rules on the dataset; if there are no global class rules, publishing the preset detailed class rules to the distributed data nodes and compiling the detailed class rules on the data nodes; obtaining detailed data as input data and performing detailed class rule calculations.

[0043] Preferably, in this embodiment, when both global rules and detailed rules exist in the rule set corresponding to the data model, the calculation of global rules is performed first.

[0044] Furthermore, step S3, which uses the rule engine to iteratively calculate the dataset corresponding to the data model, also includes: compiling the data model to be used, compiling the rule set corresponding to the data model on the distributed data node, and monitoring the rule change log storing rule expression change information in real time; acquiring real-time data, inputting the real-time data into the data model for conversion to form a dataset corresponding to the data model; and using the rule set to perform real-time iterative calculation on the dataset.

[0045] Furthermore, performing real-time iterative calculations on the dataset using the rule set includes the following operations: storing a state table containing real-time calculation results generated from the real-time iterative calculations on the dataset, and performing global rule iterative calculations or detailed rule iterative calculations using the state table as the input source.

[0046] In essence, a state table is a data structure used to record and manage the state information generated by the rule engine during rule execution. It stores and tracks the rules applied by the rule engine when processing input data, the evaluation results of those rules, and related contextual information.

[0047] By maintaining a status table generated from real-time rule computation, the rule engine can dynamically update and query status information during execution. This helps monitor the progress of rule execution, track rule results, and troubleshoot and handle errors. The status table can also be used for purposes such as generating reports, logging, and performance analysis.

[0048] Furthermore, in some embodiments, the data asset management method of the present invention may also include: performing indicator system calculation, data asset rule calculation, and process management on the dataset corresponding to the data model through the rule engine.

[0049] Specifically, the rule engine can perform indicator calculations on the dataset, including but not limited to performance indicators, financial indicators, market indicators, risk indicators, IT indicators, etc.

[0050] Specifically, using a rule engine to perform data asset rule calculations on a dataset can include, but is not limited to: verifying data quality, including assessing and improving data quality, such as verifying data integrity, accuracy, consistency and validity, detecting duplicate data, missing values ​​and outliers, etc.; classifying and labeling data, data anonymization, data cleaning and transformation, data incompleteness, etc.

[0051] Specifically, using a rule engine to manage the workflow of a dataset can include, but is not limited to, data validation and verification, data transformation and mapping, data splitting and merging, data filtering and screening, and data anomaly handling.

[0052] Example 2

[0053] Please see Figure 2 , Figure 2 A schematic diagram of a batch rule calculation process in step S3 of the data asset management method provided in the foregoing embodiments is shown.

[0054] Includes the following steps:

[0055] Step S301: Obtain the specified rule set;

[0056] Step S302: Obtain the data model corresponding to the rule set;

[0057] Step S303: Determine whether the data model has a sub-data model;

[0058] Step S304: If yes, then obtain the rule set corresponding to the sub-model;

[0059] Step S305: If not, obtain the rule set corresponding to the data model;

[0060] Step S306: Determine if a global class rule exists;

[0061] Step S307: If so, use global class rules to iteratively calculate the dataset corresponding to the data model;

[0062] Step S308: If not, use the detail class rules to iteratively calculate the dataset corresponding to the data model.

[0063] It can be understood that step S303 determines whether a sub-data model exists in the data model. If the determination result is "no", then the rule set obtained in step S305 is the same as the rule set obtained in step S301.

[0064] If step S303 determines whether a sub-data model exists in the data model and the result is "yes", then the model conversion rules are compiled, the sub-data model is obtained using the compiled model conversion rules, the dataset corresponding to the sub-data model is generated, and then the rule set corresponding to the sub-data model is obtained through step S304.

[0065] In step S306, it is determined whether a global rule exists in the rule set. If a global rule exists, it is compiled in step S307, and the global rule calculation is performed using the dataset corresponding to the data model. The results of the global rule calculation are then integrated. If no global rule exists, detailed rules are published to the distributed data nodes in step S308, and the rules are compiled on each data node. Input data is obtained from the detailed data, and detailed rule calculation is performed. The calculation results are then integrated as the output of data asset management.

[0066] Example 3

[0067] Please see Figure 3 , Figure 3 A schematic diagram of a real-time rule calculation process based on step S3 of the data asset management method provided in the foregoing embodiments is shown.

[0068] Includes the following steps:

[0069] S3001: Compile the data model and rule set to be used;

[0070] S3002: Determine if a global rule exists in the rule set;

[0071] S3003: If so, then use the real-time status table as the input source to perform global rule calculation;

[0072] S3004: If not, use the change data in the real-time status table as the input source to perform the detailed rule calculation.

[0073] In step S3001, the data model to be used and the rule set corresponding to the data model are compiled in advance, the compiled rule set and data model are loaded into the distributed nodes, and the rule change log is monitored in real time to ensure that the latest rule set is used in the real-time calculation process.

[0074] It should be noted that step S3001 also includes: real-time monitoring of rule change logs. The rule change log records the identity information of the user or administrator performing the rule change operation, such as username, ID, or other identifiers; identifies the type of rule change operation, such as adding a rule, deleting a rule, or modifying a rule; records the specific details of the rule change, which may include the rule's name, description, conditions, and actions; and the rule's state before and after the change: recording the rule's state before the change operation for comparison with the changed rule; and recording the rule's state after the change operation to show the effect of the rule change. Monitoring the rule change log helps ensure the security, reliability, and performance of the rule engine and provides comprehensive visualization of rule modification operations, thereby better managing and maintaining the rules of the rule engine.

[0075] After compiling the data model and rule set to be used, the compilation also includes compiling the rule set on each distributed node. When real-time data flows in, the data model is used to transform the real-time data to form a dataset corresponding to the data model. The dataset containing the processing results of the real-time data is stored in a real-time state table. Therefore, the rule set related to the data model can be used to perform real-time iterative calculation on the real-time state table.

[0076] Before performing the calculation, step S3002 involves determining whether a global rule exists in the rule set. If the result is "yes," step S3003 uses the real-time status table as data input to calculate the global rule. If the result is "no," step S3004 uses detailed rules to calculate the detailed rules for the change details data in the real-time status table. After the calculation is completed, the results are integrated, the calculation result is output, and the calculation result is used as the management result of the data asset.

[0077] Example 4

[0078] Based on the inventive concept of the foregoing embodiments, the present invention provides a data asset management device for implementing the data asset management method described in Embodiment 1. Please refer to the appendix. Figure 4 .

[0079] The data asset management device includes a processor and a memory storing computer programs, wherein, Figure 4 The processors illustrated here do not refer to a single processor, but rather to the positional relationship of the processor relative to other devices. In practical applications, there can be one or more processors; similarly, Figure 4The memory illustrated in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of the memory relative to other devices. In practical applications, there can be one or more memories. The data asset management device may also include at least one network interface for connection to the host. When the processor runs the computer program, the data asset management method as described in Embodiment 1 is implemented.

[0080] Example 5

[0081] Based on the inventive concept of the foregoing embodiments, the present invention also provides a readable storage medium. Specifically, the readable storage medium stores a computer program. The readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the readable storage medium is run by a processor, it implements the steps of the fault detection method as described in the above embodiments. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 The description of the illustrated embodiments will not be repeated here.

[0082] Compared to existing technologies, the data asset management method provided by this invention has the following advantages: It creates a rule engine based on the management needs of data assets. Using a data model as a foundation, and combining big data distributed computing and model transformation rules, it transforms data into a dataset corresponding to the data model. The dataset is then iteratively computed using rule expressions within the rule engine. Therefore, through the rule-based configuration of model transformation, rule calculations on different dimensions of the same basic data become accurate and efficient. Furthermore, by using big data caching technology, large-scale distributed iterative computation is performed on the corresponding dataset, saving disk I / O and computing resources, and reducing redundant data reading and transformation. Finally, this invention maps the rule set to the data model, increasing the maintainability and comprehensibility of the rule set.

[0083] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data asset management method, characterized in that, Includes the following steps: Determine the rule expressions used for data asset management; Construct a rule engine adapted to the data assets and a data model corresponding to the rule engine, and input the data model and the rule expression into the rule engine; The rule expressions in the rule engine are used to iteratively calculate the dataset corresponding to the data model, and the result of the iterative calculation of the dataset is used as the result of managing the data assets contained in the data model; The data model for constructing the rule engine includes: A basic data model corresponding to the rule engine is constructed using a scripting language or database management system, and the structure and relationships of the data in the data assets are stored in the basic data model. The basic data model is converted into multiple sub-data models contained under the basic data model; The iterative calculation of the dataset corresponding to the data model using the rule expressions in the rule engine includes: Determine whether the basic data model contains the sub-data model; If a sub-data model exists, the dataset corresponding to the sub-data model is used as the input source of the rule engine, and the dataset corresponding to the sub-data model is iteratively calculated using the rule set corresponding to the sub-data model. If no sub-data model exists, the dataset corresponding to the basic data model is used as the input source of the rule engine, and the dataset corresponding to the basic data model is iteratively calculated using the rule set corresponding to the basic data model. After obtaining the sub-data model or the rule set corresponding to the data model, iterative calculations are performed on the dataset corresponding to the data model or the sub-data model using the rule set corresponding to the data model or the sub-data model, including: Determine whether a preset global class rule exists in the rule set; If a global rule exists, the global rule is compiled, and the global rule calculation is performed on the dataset first. If no global class rule exists, the preset detailed class rule is published to the distributed data nodes and the detailed class rule is compiled on the data nodes; detailed data is obtained as input data, and the detailed class rule calculation is performed.

2. The data asset management method according to claim 1, characterized in that, The step of inputting the data model and the rule expression into the rule engine includes: Compile the rule expression into a rule set; The data model is matched with the rule expressions in the rule set using the rule engine to obtain the matching result; The matching results are used as the basis for iterative calculations on the dataset corresponding to the data model.

3. The data asset management method according to claim 1, characterized in that, The iterative calculation of the dataset corresponding to the data model using the rule engine includes: Compile the data model to be used, compile the rule set corresponding to the data model on the distributed data node, and monitor the rule change log that stores the rule expression change information in real time. Acquire real-time data, input the real-time data into the data model, and transform it to form a dataset corresponding to the data model; The dataset is subjected to real-time iterative computation using the rule set.

4. The data asset management method according to claim 3, characterized in that, The real-time iterative computation of the dataset using the rule set includes: The system stores a state table containing real-time calculation results generated by real-time iterative calculation of the dataset, and uses the state table as the input source for global rule iterative calculation or detailed rule iterative calculation.

5. The data asset management method according to claim 1, characterized in that, Also includes: The rule engine performs indicator system calculations, data asset rule calculations, and process management on the dataset corresponding to the data model.

6. A data asset management device, characterized in that, The apparatus includes a memory and a processor, the memory storing program instructions executable by the processor, the processor invoking the program instructions to perform the data asset management method as described in any one of claims 1 to 5.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data asset management method as described in any one of claims 1 to 5.

Images