Data asset management system and corresponding computer device and storage medium

Abstract

The application discloses a data asset management system and corresponding computer equipment and storage medium, wherein the system comprises a data asset meta model, a data asset instance model, a tenant and data asset and version relationship model and is used for maintenance of data asset and data version, adding, modifying and / or deleting corresponding data records to the tenant and data asset and version relationship model, publishing the obtained data asset model after maintenance, and notifying the corresponding tenant of the change of the data asset according to the obtained data asset model after maintenance and the tenant and data asset and version relationship model. The application simplifies the maintenance of the data asset model and version, and can realize the modifiability and deletability of the business attribute of the data asset model in the multi-tenant scenario.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and in particular to a data asset management system and corresponding computer equipment and computer-readable storage medium. Background Technology

[0002] Data assets refer to data resources, either physically or electronically, that are owned or controlled by an enterprise and can generate future economic benefits for the enterprise. After data governance and data asset management, data assets can be monetized by providing data sharing services to external parties.

[0003] Similar "data versioning" issues arise during data asset management. For example: 1. The business attributes of master data in the organizational information systems of large and medium-sized enterprises may be adjusted and updated annually, such as accounting subjects, products, suppliers, and personnel. The master data used in business processes needs to adapt to these version changes and remain consistent with the master data definitions and versions for the corresponding period, as well as the statistical dimensions used in data analysis. 2. Different enterprise business systems, especially legacy systems, may use different data versions for the same data asset at different times, but it is required that operations on the data asset by each system do not affect each other. 3. Due to enterprise data governance, the data asset model currently in use may need to be readjusted and republished, but it can only be rebuilt, not modified; data migration and re-maintenance are required, resulting in high change costs and significant implementation efficiency and costs.

[0004] The above issues require addressing two key points: multi-tenancy and changes to the data asset model and its versions.

[0005] Multi-tenancy requires that for the same data asset, different tenants may choose different data versions to use based on their own needs, and the operations between tenants should not affect each other. Traditional multi-tenancy has two approaches. The first is to manage the data asset model, its attributes, and data versions uniformly, adding a tenant ID attribute for differentiation; the second is for the data asset management system to assign each tenant an independent data asset model, achieving maximum data isolation, and tenants can only operate on data assets within their authorized scope. In the first approach, the tenant ID is a direct attribute of the data asset model, tightly bound to it, which can introduce complexity into model design and implementation. In the second approach, while the software data architecture is clear and straightforward, the implementation structure is complex, resulting in redundant and duplicated data asset storage, leading to resource waste and management difficulties.

[0006] The data version of a data asset (hereinafter referred to as data version) includes the data asset model version (hereinafter referred to as model version) and the data asset data record version (hereinafter referred to as record version). Changes to the model version will cause changes to the record version. The traditional approach to changes to the data asset model and its version is to avoid them as much as possible in business operations. This means that data governance and software implementation mandate that once a data asset model is released, it cannot be adjusted or modified; changes must be rebuilt, and data migration is required. In single-user scenarios, data asset models are only allowed to add or delete data, but not modify business attributes. Modifying business attributes will affect the data columns corresponding to the original business attributes currently in use. For example, the original English name of the "Department Code" attribute field in a company's department master data is dptcode. According to data asset naming conventions, it needs to be changed to the new English name DEPT_CODE, which will affect the normal access of the original dptcode field for current users. In a multi-tenant scenario, there are two situations: 1) Each tenant manages its own data asset model, which is the same as the single-tenant scenario, and will bring business complexity and technical implementation complexity to the management of data asset models; 2) Data asset models are managed uniformly. In this case, business attributes cannot be deleted or modified, because modifying business attributes will affect the data columns corresponding to the original business attributes that other tenants are using. Business attributes can only be deleted after all tenants confirm that they are no longer in use. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a data asset management system and corresponding computer equipment and storage media, which simplifies the maintenance of data asset models and versions, enables unified management of data asset models, and solves the problem that business attributes cannot be modified or deleted, and is applicable to multi-tenant scenarios.

[0008] In a first aspect of the present invention, a data asset management system is provided, comprising:

[0009] The data asset meta-model has at least the following attributes: data asset code, data asset model version number, and attribute name; wherein, data assets include data asset models and data asset instances, and the data asset model includes the data asset meta-model and data asset metadata, and the data asset metadata is defined based on the attributes of the data asset meta-model;

[0010] A data asset instance model, wherein data asset instances are dynamically generated based on the data asset metadata;

[0011] A model of the relationship between tenants and data assets and their versions, whose attributes include at least tenant ID, data asset code, and data asset model version number;

[0012] The maintenance module is used for the maintenance of the data assets and their data versions, including adding, modifying and / or deleting.

[0013] The allocation module is used to add, modify, and / or delete corresponding data records in the relationship model between the tenant and the data assets and their versions.

[0014] The publishing module is used to publish the data asset model obtained by the maintenance module;

[0015] The notification module is used to notify the corresponding tenants of changes in data assets based on the data asset model obtained by the maintenance module and the relationship model between the tenant and the data asset and its version.

[0016] In a second aspect of the invention, a computer device is provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the functions of a system according to a first aspect of the invention.

[0017] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the functions of a system according to a first aspect of the present invention.

[0018] According to this invention, by maintaining data assets and their versions based on a data asset meta-model, a data asset instance model, and a relationship model between tenants and data assets and their versions, and by adding, modifying, and / or deleting corresponding data records in the relationship model between tenants and data assets and their versions, the maintained data asset model is published, and changes to data assets are notified to the corresponding tenants based on the maintained data asset model and the relationship model between tenants and data assets and their versions. This enables a solution for data asset model and version management in multi-tenant scenarios, which is different from traditional implementation solutions. It solves the problem that the business attributes of data asset models cannot be modified or deleted. Combining the characteristics of row and column storage, it simplifies the logical and physical design of data asset models, data asset instances, and their relationships with tenants, simplifies the maintenance operation process of data assets and versions, and facilitates tenants to retrieve and query a specific version of the data asset model and its data, greatly improving production efficiency.

[0019] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] Figure 1 This is a block diagram of an embodiment of the system according to the present invention;

[0021] Figure 2A diagram illustrating the relationship between data asset models, data versions, and tenants;

[0022] Figure 3 This is a schematic diagram of a data asset model and data storage mode.

[0023] For clarity, these figures are schematic and simplified, showing only the details necessary for understanding the invention, while omitting other details. Detailed Implementation

[0024] The embodiments and examples of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] The scope of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific examples illustrate preferred embodiments of the invention, they are given for illustrative purposes only.

[0026] Figure 1 A block diagram of a preferred embodiment of a data asset management system according to the present invention is shown. This embodiment of the data asset management system includes a maintenance module 100, an allocation module 200, a publishing module 300, and a notification module 400.

[0027] The maintenance module 100 is used for the maintenance of data assets and their data versions. Maintenance includes adding, modifying, and deleting data asset models. Data assets are determined by data asset models and data asset instances. The data version (also called data version) of a data asset includes the data asset model version (also called model version) and the data asset data record version (also called record version). This application only discusses the model version. It should be noted that if the model version changes, the record version will also change by default; changes to the record version are not specifically described here. The data asset model includes the data asset meta-model and data asset metadata. The data asset model is defined and managed uniformly, and the data asset metadata is defined based on the attributes of the data asset meta-model.

[0028] Data asset instances are created and dynamically managed based on the data asset metadata in the data asset meta-model.

[0029] In one embodiment, the attributes of a data asset meta-model include at least: asset code, asset business code, asset Chinese name, model version number (i.e., data asset model version number), attribute name, attribute Chinese name, data type, data length, data precision, dependent attribute name, and whether it is stopped. The unique key of the data asset meta-model is asset code + model version number + attribute name. Each data asset's asset code is globally unique, and its naming convention is four characters: the first character is 26 uppercase letters, and the second and third characters are 26 uppercase letters and 10 digits from 0 to 9. The model version number is an internally defined version number, an integer starting from 1, and is not a business-related version number. The data type is abstracted into three types: character, number, and date. Numbers include integers and double precision. The date format is YYYY-MM-DD HH24:MI:SS, and the specific date type is determined based on the data length. The "dependent attribute name" indicates which existing attribute the current attribute is based on for modification, referencing the "attribute name" of its own model. The "whether it is stopped" indicates whether the data asset's attribute is available. The data asset meta-model is defined as follows:

[0030]

[0031] Based on the data asset metadata model above, the metadata definition for a data asset such as "department" is as follows:

[0032]

[0033] The data records in the data asset meta-model include all metadata of the data assets and the definitions of all versions. Each data record is an attribute definition of the data asset metadata for a specific model version. A change in the model version of a data asset is a complete change to all attributes. That is, adding, modifying, or deleting one or more attributes in a version will create a new model version, adding data records for all metadata attributes (including those that have changed) for that version. For example: when adding a new attribute, the new version copies the metadata definition of the previous version and adds the attribute; when deleting an existing attribute, the new version copies the metadata definition of the previous version but does not include the original attribute; when modifying an existing attribute, the new version copies the metadata definition of the previous version, modifies the corresponding information of the original attribute, and fills in the original attribute name in the "Dependent Attribute Name" field.

[0034] The attributes of the data asset instance model are dynamically generated. Their attributes are determined by the metadata attribute set of the data asset meta-model. The naming convention for attributes is: asset code + attribute name.

[0035]

[0036] Based on the asset code of the data asset meta-model and the data record with model version number = 1 (i.e., the metadata definition with model version number 1), an initial version of the data asset instance can be established. Each data asset instance initially includes all attributes of the initial version. Subsequent changes in the attribute rows caused by incremental changes in the model version of the data asset within the data asset meta-model will dynamically update the attribute columns of the data asset instance, and the data in the relevant attribute columns will change accordingly. For example: when version 2 adds an attribute A (attribute name A), the attribute named A is dynamically added to the data asset instance; when version 3 deletes an existing attribute B (attribute name B), the attributes of the data asset instance do not change; when version 4 modifies an existing attribute C (attribute name C) to a new attribute name C1, the attribute C1 (attribute name C1) is dynamically added to the data asset instance, and the data in the original attribute C column is synchronously copied to the new attribute C1 column. Adding and modifying actions are transactional.

[0037] The relationship between a tenant and the data asset model determines which data assets and their versions the tenant owns. A tenant's attribute must include at least a tenant ID (globally unique, integer data type). The relationship model between a tenant and its data assets and versions is as follows:

[0038] Attribute Name Data types Data width Remark Asset Code character type 50 Model version number Integer 8 Tenant ID Integer 8

[0039] Figure 2 The diagram illustrates the relationship between data asset models, data versions, and tenants. To operate on a specific version of a data asset, a tenant obtains its metadata definition by accessing the asset code and model version number of that data asset in the data asset metadata model. Then, they query the corresponding metadata attribute columns for that version within the corresponding data asset instance to obtain the dataset.

[0040] The physical model design for the data asset meta-model and the relationship model between tenants and data assets and their versions can employ row-based storage and traditional database table design. The data asset instance model can utilize column-based storage and partitioned table storage. Traditional database design typically aims to minimize data redundancy. Partitioned or column-based storage can mitigate redundancy, trading space for time to improve storage efficiency. Column-based storage is more conducive to the dynamic expansion of instance attribute columns, facilitates data compression, and offers higher storage efficiency than row-based storage. Furthermore, when tenants query data based on the data asset's version number, column-based storage can quickly retrieve the attribute set columns for that version, thereby improving data query efficiency.

[0041] The data asset model and overall data storage design can be tailored to the quantity of data assets and the scale of data records. Storage management can combine various storage methods, such as single-table, merged-table, single-table sharding, and merged-table sharding, allowing for selection based on specific needs during implementation.

[0042] - Data asset instances with few data records can be stored in a single table, with each data asset instance stored in a physical table;

[0043] - Data asset instances with large data records can be stored using a single shard, or (distributed) sharded storage using the version number as the key;

[0044] Considering the large number of data assets, a merged table can be used for storage, that is, multiple data asset instances can be combined into a single merged data asset. Based on the merged table, sharded storage can also be implemented, using the asset code as the key.

[0045] The diagram illustrates the data asset model and data storage modes. A001 represents single-table storage, B001 represents single-shard storage, C001 and D001 represent merged table storage, and E001, F001, and G001 represent merged table shard storage.

[0046] The allocation module 200 is used to add, modify, and / or delete corresponding data records in the relationship model between the tenant and the data asset and its version when a data asset already has a data version and a data asset instance, thereby realizing the allocation of the data asset and its version number to the tenant.

[0047] The publishing module 300 is used to publish the maintained data asset model. Publishing, for example, involves switching the editing state to indicate that the data asset model is available for use.

[0048] The notification module 400 is used to notify relevant tenants of changes to data assets based on the maintained data asset model and the relationship model between tenants and data assets and their versions. Notifications typically rely on a notification platform or engine and can be delivered via methods such as email or SMS. The notification informs tenants that the data attributes of the assets they are using have changed, and tenants can decide whether to acquire the new data assets.

[0049] Specifically, under normal circumstances, the main business operation process of data asset management personnel includes at least adding, modifying, and deleting data asset models, among which adding and modifying data asset models are transactional operations.

[0050] The process for adding a new data asset model is as follows: First, a new data asset metadata definition is added and saved. All attributes of this metadata are added, and the model version number is assigned a value of 1. Model version number 1 represents the initial version. Based on all the added attributes, a data asset instance is created, and attribute columns are added. Then, the initial version is published, notifying tenants of the new data asset.

[0051] Modifying a data asset model requires that a data asset already has a data version and a data asset instance.

[0052] - When a new attribute is added to the data asset meta-model, the data asset meta-model adds the attribute set of the previous version and the new attribute, increments the model version number by 1, adds an attribute column corresponding to the new attribute to the data asset instance, publishes the new model version number, and finally notifies the tenant of the version change of the data asset based on the relationship between the tenant and the data asset and its version.

[0053] - When an existing attribute is modified and saved in the data asset meta-model, the data asset meta-model adds the attribute set of the previous version and modifies the corresponding existing attribute. The model version number is incremented by 1. The modified attribute column is added to the data asset instance. Then, the contents of the original attribute column are copied to the newly added attribute column. Then, the data asset version is published. Finally, based on the relationship between the tenant and the data asset and its version, the tenant is notified of the version change of the data asset.

[0054] - When an existing attribute is deleted in the data asset meta-model, the data asset meta-model adds a new attribute set from the previous version but does not include the deleted original attribute. The model version number is incremented by 1, and then the new model version number is published. Finally, based on the relationship between the tenant and the data asset and its version, the tenant is notified of the version change of the data asset.

[0055] The data asset deletion process is a logical deletion: it requires that a data asset already has a data version and a data asset instance, and that the deletion operation has been approved.

[0056] - When a data asset is deleted from the data asset metadata model, all versions and instances of that data asset model become unavailable. The "Stop" attribute of all metadata of that data asset is modified to "Yes". Then, based on the relationship between the tenant and the data asset and its version, the tenant is notified of the version change of the data asset.

[0057] - When a version of a data asset is deleted, the "Stop" attribute of the metadata of that version of the data asset is modified to "Yes". The model and its instances of that version are no longer available. Then, based on the relationship between the tenant and the data asset and its version, the tenant is notified of the version change of the data asset.

[0058] The primary operational process for tenants involves querying data asset models and data. In this embodiment, the data asset management system also includes a query module, used by tenants to query corresponding data asset metadata and / or data asset records based on the relationship model between the tenant and the data asset and its version. Tenants' queries of historical version data records are unaffected by model or version changes. For example, tenant A has been using version 1 of data asset A (containing attributes A, B, and C). The data version has now been updated (containing attributes A, B1, C, and D), meaning attribute D has been added, and the attribute name of B has been changed to B1. The latest metadata attribute columns are A, B, C, D, and B1, but tenant A can still query data records for version 1 (containing attributes A, B, and C).

[0059] Data asset metadata query process: Based on the relationship model between tenants and data assets and their versions, tenants query the authorized data asset code and version number, and then query the data asset attributes of that version from the data asset metadata model based on the data asset code and version number.

[0060] The data record query process for data assets is as follows: Based on the relationship model between tenants and data assets and their versions, the tenant queries the authorized data asset code and version number, queries the data asset attribute set for that version from the data asset meta-model based on the data asset code and version number, and then queries the data of the corresponding attribute columns based on the data asset attribute set for that data asset instance.

[0061] The data asset management system of this invention allows data asset managers to maintain data asset models and their data versions in a unified manner, such as adding, modifying, and deleting data, according to data asset norms and standardized management systems. Data asset models and their data versions are assigned to corresponding tenants, and tenants are promptly notified of model or version changes. Data asset managers and tenants can maintain data asset records; historical versions of data records can only be queried, not modified, to maintain data integrity and overcome the limitation that the business attributes of data asset models cannot be modified or deleted.

[0062] The main operational process of the data asset model and its versions can be implemented via API services, with no restrictions on software implementation, generally using REST API or Web service.

[0063] In another embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to achieve [the desired result]. Figure 1 The functions of the system embodiments or other corresponding system embodiments shown are not described in detail here.

[0064] In another embodiment, the present invention provides a computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to achieve [a specific action / combination]. Figure 1 The functions of the system embodiments or other corresponding system embodiments shown are not described in detail here.

[0065] The various embodiments described herein, or their specific features, structures, or characteristics, may be suitably combined in one or more embodiments of the invention. Furthermore, in some cases, the order of steps described in the flowcharts and / or pipeline processes may be modified where appropriate, and they need not be performed in the exact order described. Additionally, various aspects of the invention may be implemented using software, hardware, firmware, or combinations thereof, and / or other computer-implemented modules or devices that perform the described functions. Software implementations of the invention may include executable code stored in a computer-readable medium and executed by one or more processors. Computer-readable media may include computer hard disk drives, ROM, RAM, flash memory, portable computer storage media such as CD-ROM, DVD-ROM, flash drives, and / or other devices having a Universal Serial Bus (USB) interface, and / or any other suitable tangible or non-transitory computer-readable medium or computer memory on which executable code can be stored and executed by a processor. The invention may be used in conjunction with any suitable operating system.

[0066] Unless explicitly stated otherwise, the singular forms “a” and “the” used herein include the plural meaning (i.e., meaning “at least one”). It should be further understood that the terms “having,” “comprising,” and / or “including” as used in the specification indicate the presence of the described features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The term “and / or” as used herein includes any and all combinations of one or more of the listed related items.

[0067] The foregoing has described some preferred embodiments of the present invention. However, it should be emphasized that the present invention is not limited to these embodiments, but can be implemented in other ways within the scope of the present invention. Those skilled in the art can make various modifications and variations to the present invention based on the inventive concept and without departing from the scope of the present invention, and such modifications or variations still fall within the protection scope of the present invention.

Claims

1. A data asset management system, characterized in that, The system includes: The data asset meta-model has at least the following attributes: data asset code, data asset model version number, and attribute name; wherein, data assets include data asset models and data asset instances, and the data asset model includes the data asset meta-model and data asset metadata, wherein the data asset metadata is defined based on the attributes of the data asset meta-model; A data asset instance model, wherein the data asset instance is dynamically generated based on the data asset metadata; A model of the relationship between tenants and data assets and their versions, whose attributes include at least tenant ID, data asset code, and data asset model version number; The maintenance module is used for maintaining the data assets and their data versions, including adding, modifying and / or deleting. The allocation module is used to add, modify, and / or delete corresponding data records in the relationship model between the tenant and the data assets and their versions. The publishing module is used to publish the data asset model obtained by the maintenance module; The notification module is used to notify the corresponding tenants of changes in data assets based on the data asset model obtained by the maintenance module and the relationship model between the tenant and the data asset and its version.

2. The system according to claim 1, characterized in that, The maintenance module is used for: In response to the addition of a new data asset metadata definition to the data asset metadata model, all attributes of the data asset metadata are added and the data asset model version number is assigned the value 1. Based on all the newly added attributes, create corresponding data asset instances and add attribute columns.

3. The system according to claim 1, characterized in that, The maintenance module is used for: In response to the addition of new attributes to the data asset meta-model, the data asset meta-model adds the attribute set of the previous version and the new attributes, and the data asset model version number is incremented by 1; The corresponding data asset instance will have a new attribute column added to match the new attribute.

4. The system according to claim 3, characterized in that, The maintenance module is also used for: In response to changes in the original attributes of the data asset meta-model, the data asset meta-model adds the attribute set of the previous version and modifies the corresponding original attributes, and the data asset model version number is incremented by 1; Added or modified attribute columns for data asset instances; Copy the contents of the original attribute column to the newly added attribute column.

5. The system according to claim 3 or 4, characterized in that, The maintenance module is also used for: In response to the deletion of existing attributes in the data asset meta-model, the data asset meta-model adds a new attribute set from the previous version, but does not include the deleted existing attributes, and the data asset model version number is incremented by 1.

6. The system according to claim 1, characterized in that, The attributes of the data asset meta-model also include a "stop" attribute to indicate whether the data asset's attributes are available. The maintenance module is used for: In response to the deletion of a data asset, the "Stop" attribute of the corresponding metadata of the data asset is set to "Yes".

7. The system according to claim 1, characterized in that, The system also includes: The query module is used by tenants to query the corresponding data asset metadata and / or data records of data assets based on the relationship model between the tenant and the data assets and their versions.

8. The system according to claim 1, characterized in that, The physical model design of the data asset meta-model and the relationship model between the tenant and the data asset and its version adopts row-based storage, while the data asset instance model adopts column-based storage.

9. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the functions of the system according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the functions of the system according to any one of claims 1-8.

Images