Discrete Data Model Methods and Systems
Through topology structure and virtual/graph model technology for discrete data sets, cross-domain correlation analysis and heterogeneity problems of discrete data sets are solved, dynamic expansion and scalability of data models are realized, and data analysis efficiency and accuracy are improved. sex.
Patent Information
- Application Number
- CN201910410407.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-17
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-05-17
AI Technical Summary
Existing data modeling technology is difficult to solve the problems of cross-domain correlation analysis of discrete data sets, dynamic expansion and scalability of heterogeneous data models, data distributiveness, data heterogeneity and multi-own characteristics of discrete data sets, which makes traditional technology not suitable for Discrete data analysis scenario.
By determining the topological nodes and their tree topology structures for discrete data sets, establishing the topology of the data model, and using virtual data model or graph model to shield the heterogeneity within each topological node, establishing vertical and horizontal relationships to realize the data model dynamic expansion and scalability.
It effectively solves the data modeling problem in discrete data analysis scenarios, improves the efficiency of the industry chain, provides users with accurate services, and supports cross-domain correlation analysis and data heterogeneity processing.
Smart Images

Figure CN111949741B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data modeling, and in particular to a global data model method and system for discrete data sets. Background Art
[0002] In the era of big data, data is distributed in a "discrete" manner, and each "discrete point" corresponds to a certain data island, such as Figure 1 As shown. In reality, each data island can be understood as a collection of private data resources and computing resources under the jurisdiction of an owner (such as an enterprise or a department, etc.). In the present invention, a data set that serves an analysis subject and is distributed in a "discrete state" is defined as a discrete data set; further, the joint data analysis across data islands for discrete data sets is defined as discrete data analysis.
[0003] In reality, discrete data analysis has far-reaching application value: through discrete data analysis, enterprises can establish synergy based on data sets of upstream and downstream of the industrial chain or cross-industry partners, effectively solving the problem of insufficient competitiveness caused by the lack of their own data resources and information asymmetry. For example, pharmaceutical factories can establish data synergy across hospitals, insurance companies and other links in the industrial chain (i.e., joint data analysis across data islands) in some form, thereby greatly improving the efficiency of the entire industrial chain, reducing ineffective waste, and providing users with precise services.
[0004] The data analysis process usually requires first establishing a data model according to the needs of the analysis topic, and then establishing a calculation model based on the data model to complete the execution of the subsequent analysis and calculation process. However, compared with the "non-discrete" data set, the data analysis and calculation process of the "discrete" data set has the following significant differences:
[0005] 1) Cross-domain correlation analysis
[0006] "Domain" here specifically refers to the boundaries of data islands, which are generally reflected in enterprise boundaries, department boundaries or network boundaries. The security constraints of data in a domain are the same. Compared with "non-discrete" data sets that usually have unified data security constraints, correlation analysis between "discrete" data sets usually needs to be completed across domains. To complete cross-domain correlation analysis of discrete data sets, it is necessary to overcome technical obstacles such as cross-domain heterogeneous data correlation expression, cross-domain computing task scheduling, and cross-domain data security transmission.
[0007] 2) Dynamic expansion and scalability of heterogeneous data models during computing
[0008] In general, "non-discrete" data sets do not support dynamic expansion of data models (including heterogeneous and homogeneous data models) when the computing model is running, and can only support the expansion of data sets based on existing data models. During the analysis process of "discrete" data sets, it is necessary to face dynamically connected or disconnected heterogeneous data sets (with heterogeneous data models), that is, the dynamic expansion and scalability of heterogeneous data models.
[0009] 3) Data distribution
[0010] A significant difference between "discrete" data sets and "non-discrete" data sets is that discrete data sets are usually distributed in multiple places, that is, they are distributed in different places. This brings about technical problems such as network heterogeneity, QoS of computing tasks under narrow bandwidth, transmission reliability and security that cannot be avoided in the analysis process of "discrete" data sets.
[0011] 4) Data Heterogeneity
[0012] Compared with "non-discrete" data sets, "discrete" data sets have more significant heterogeneous characteristics. They not only include the "data heterogeneity with different semantics" problem of non-discrete data sets, such as multiple heterogeneous data tables in the database are related by foreign keys; they also include the "data heterogeneity with the same semantics" problem, such as the problem of multi-source master data with the same semantics but different encodings (or understood as the same encoding but representing different semantics in different data sources).
[0013] 5) Multi-ownership of data
[0014] Compared with "non-discrete" data sets with unified ownership, "discrete" data sets usually have multi-owner characteristics, that is, the data sets come from multiple owners. This characteristic inherently limits the security constraints and ownership of the data, and its data modeling and analysis operation methods and space are significantly different from "non-discrete" data sets with unified data ownership.
[0015] In view of the differences in the above technical characteristics, traditional data modeling technologies used for data analysis are no longer suitable for discrete data analysis scenarios, and data modeling methods for discrete data analysis are urgently needed. Summary of the invention
[0016] In order to solve the data modeling problem in discrete data analysis scenarios that cannot be solved by existing data modeling technical means, the present invention provides a discrete data model method, comprising the following steps:
[0017] According to the needs of the analysis subject, the topological nodes and their tree-like topological structures are determined for discrete data sets, and the topology of the data model is established, which specifically includes: the topological nodes are based on the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain, and the data resources and computing resources in the problem domain are cut into independent topological nodes; according to the data security boundary, the data resources and related computing resources of the business department are integrated together and expressed as an independent topological node respectively; according to the logical union boundary, a higher-level topological node is established on the topological node of the business department, and corresponding computing resources are allocated; according to the management boundary, the sets of data sets managed by each are divided into independent topological nodes for expression; according to the trust boundary, the sets of data resources and computing resources managed by each are divided into independent topological nodes for expression; according to the interest boundary, the sets of data sets managed by each are divided into independent topological nodes for expression; according to the data analysis hierarchy requirements of the discrete data analysis problem domain, the topological nodes are distributed to the leaf nodes and parent nodes of the tree topological structure, and are expanded in the horizontal and vertical directions;
[0018] Establishing a data model set inside each topological node, specifically: establishing a data model set inside each topological node based on the principle of establishing a minimum number of logical data models, or establishing at least one virtual data model using a data virtualization method, wherein the virtual data model is used to shield the heterogeneity of multiple physical data models inside each topological node; wherein the logical data model refers to a data model without an actual data object instance, including a virtual data model and a graph model; and
[0019] Establishing the association relationship of the data model set specifically includes: establishing a vertical association relationship between the physical data model set and the virtual data model within a topological node; establishing a vertical association relationship between a low-level root model mapping and the virtual data model; establishing a horizontal association relationship of data models on the virtual data model plane based on the association relationship between data object attributes contained in the virtual data model; or, establishing a vertical association relationship between the physical data model set and the virtual data model within a topological node; establishing a vertical association relationship between the virtual data model and the graph model within a topological node; establishing a vertical association relationship between a low-level root model mapping and the graph model; and establishing a horizontal association relationship of data models on the graph model plane based on vertex and edge objects in the graph model.
[0020] The present invention also provides a discrete data model system, comprising:
[0021] The first establishment module is used to determine the topological nodes and their topological structures for discrete data sets according to the needs of the analysis subject, and establish the topology of the data model; the topological nodes are based on the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain, and the data resources and computing resources in the problem domain are cut into independent topological nodes; according to the data security boundary, the data resources and related computing resources of the business department are integrated together and expressed as an independent topological node respectively; according to the logical union boundary, a higher-level topological node is established on the topological node of the business department, and the corresponding computing resources are allocated; according to the management boundary, the data set collection managed by each is divided into independent topological nodes for expression; according to the trust boundary, the data resources and computing resources collection managed by each is divided into independent topological nodes for expression; according to the interest boundary, the data set collection managed by each is divided into independent topological nodes for expression; according to the data analysis hierarchy requirements of the discrete data analysis problem domain, the topological nodes are distributed to the leaf nodes and parent nodes of the tree topological structure, and are expanded in the horizontal and vertical directions;
[0022] a second establishment module, for establishing a data model set inside each topological node in the topology established by the first establishment module, specifically: establishing a data model set inside each topological node based on the principle of establishing a minimum number of logical data models, or using a data virtualization method to establish at least one virtual data model to achieve this, wherein the virtual data model is used to shield the heterogeneity of multiple physical data models inside each topological node; wherein the logical data model refers to a data model without an actual data object instance, including a virtual data model and a graph model; and
[0023] The third establishment module is used to establish the association relationship of the data model set established by the second establishment module, specifically including: establishing a vertical association relationship between the physical data model set and the virtual data model within the topological node; establishing a vertical association relationship between the low-level root model mapping and the virtual data model; establishing a horizontal association relationship between the data models on the virtual data model plane according to the association relationship between the data object attributes contained in the virtual data model; or, establishing a vertical association relationship between the physical data model set and the virtual data model within the topological node; establishing a vertical association relationship between the virtual data model and the graph model within the topological node; establishing a vertical association relationship between the low-level root model mapping and the graph model; and establishing a horizontal association relationship between the data models on the graph model plane according to the vertex and edge objects in the graph model.
[0024] The discrete data model method and system provided by the present invention determine the topological nodes and their topological structures for discrete data sets, establish the topology of the data model, the data model set within each topological node in the topological structure, and the association relationship between the data model sets, thereby realizing the establishment of a discrete data model, solving the data modeling problem in discrete data analysis scenarios that cannot be solved by existing data modeling technical means, and providing an effective solution for enterprises or business departments to solve the problem of insufficient competitiveness caused by their own scarce data resources and information asymmetry, thereby effectively improving the efficiency of the industrial chain and providing users with accurate services. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the discrete distribution of existing data;
[0026] Figure 2 This is a schematic diagram of integrating data islands within leaf nodes in this embodiment;
[0027] Figure 3 is a method flow chart of the discrete data model of this embodiment;
[0028] Figure 4 Schematic diagram of the tree topology structure of the discrete data model of this embodiment;
[0029] Figure 5 This is a schematic diagram of an example application of a tree topology structure of a discrete data model according to this embodiment;
[0030] Figure 6 This is a schematic diagram of the horizontal association relationship of the data model established by using the virtual model method in this embodiment;
[0031] Figure 7 This is a schematic diagram of the vertical association relationship of the data model established by using the virtual model method in this embodiment;
[0032] Figure 8 This is a schematic diagram of establishing a vertical association relationship of a data model using a graph model method in this embodiment;
[0033] Fig. 9 This is a schematic diagram of establishing a horizontal association relationship of a data model using a graph model method in this embodiment;
[0034] Fig.10 It is a structural diagram of the discrete data model system of this embodiment. DETAILED DESCRIPTION
[0035] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0036] The discrete data model of the embodiment of the present invention is a data model with a complex topological structure and nested relationships. In order to more clearly explain the technical solution of the embodiment of the present invention, the embodiment of the present invention is now defined as follows:
[0037] 1) Physical data model: In an embodiment of the present invention, the physical data model corresponds to a data island participating in discrete data analysis, and is used to describe the data structure of the data island; for example, if the data island is a database, then the corresponding physical data model can be the physical data schema of the database, including the semantics of the relevant table structure. In an embodiment of the present invention, the physical data model can be oriented to data with structured semantics, including but not limited to structured data (such as relational databases or graph databases, etc.), or oriented to unstructured data with structured semantics (such as JSON, CSV, XML, pictures, etc.), or stream data with data structure.
[0038] 2) Virtual data model: In the embodiment of the present invention, the virtual data model is an abstract view of data virtualization, which corresponds to the physical data model of one or more data islands involved in discrete data analysis, and is compatible with the data structures and types covered by the physical data model. In view of the fact that data virtualization technology is now very mature, there are many commercial products and solutions such as JBOSS's Data Virtualization and Tibco's Data Virtualization, and the details are not repeated here. It should be noted that the virtual data model is essentially consistent with the physical data model in the semantics of data object expression. Taking the structured relational database as an example, both can be expressed in the form of ER model. The essential difference is that the data objects in the physical data model have corresponding data instances (generally stored in the underlying medium, such as a database), while the virtual data model does not.
[0039] 3) Root model: In the embodiment of the present invention, the root model refers to a data model used to describe all data objects and data object associations contained in each topological node of the discrete data model. The above data objects are derived from the physical data model of the local data island contained in each topological node, the data model mapped into the external data island, etc.; in practical applications, the above data objects are usually only the minimum set of data objects involved in the analysis subject to which the discrete data model belongs. The root model can be expressed using existing expression methods of data models including virtual data models, graph models, etc.
[0040] In practice, the establishment of a data model for discrete data sets needs to follow some rules to ensure its operability. As far as the embodiment of the present invention is concerned, in order to illustrate the feasibility of the steps of the embodiment of the present invention, the following establishment rules are agreed upon: 1) Each topological node has and only has one root model; 2) A vertical association relationship between data models is established across levels, and a low-level topological node can only map to its parent node with the root model as output; 3) The minimum principle: Under the technical goal of completing the establishment of data model association relationships, the minimum logical data model is established; the so-called logical data model refers to a data model without an actual data object instance, such as the virtual data model and graph model used to associate data models in the embodiment of the present invention; 4) Establish a set of internal data models of the topological node. If the logical data model is used as an intermediate model, the underlying physical data model cannot bypass the intermediate model and directly establish a vertical association relationship with the top-level logical data model. For example, in the following step S102, the graph model method is used to establish a set of internal data models of the topological node. According to this rule, it is necessary to first establish a virtual data model, and then establish a root model (i.e., a graph model) based on the virtual data model. Direct mapping from the internal physical data model to the root model is not allowed.
[0041] The discrete data model of the embodiment of the present invention includes information such as a description of a data model topology structure, a data model set within a topology node, and association relationships between data models within a topology node and across topology nodes.
[0042] See also Figure 3 The discrete data model method provided by the embodiment of the present invention comprises the following steps:
[0043] Step S101: Establish the topology of the data model for discrete data sets according to the analysis subject requirements.
[0044] The topology of the data model for discrete data sets can be expressed in many ways in practice, but no matter which expression is used, it needs to meet the technical requirements of "discrete state" data analysis, especially the technical requirements of dynamic and scalable performance. The embodiment of the present invention uses a tree topology expression to demonstrate the feasibility of the steps of the embodiment of the present invention, such as Figure 4 As shown; in the embodiment of the present invention, the leaf nodes, parent nodes and root nodes of the tree topology structure are collectively referred to as topology nodes. The embodiment of the present invention establishes a tree topology of a data model of a discrete data set, including the following steps:
[0045] Step S1011: Determine the topological nodes of the tree topological structure.
[0046] In the embodiment of the present invention, the topological node is to cut the data resources and computing resources in the problem domain into independent topological nodes for abstract expression according to the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain. An example is as follows:
[0047] 1) Establish topological nodes based on data security boundaries: Assume that the data resources of various business departments within the jurisdiction of a prefecture-level city (i.e., data islands, usually in the form of databases, the same below) exist in a discrete data analysis problem domain. Since there are data security boundaries between these business departments, that is, data cannot be moved across departments for joint analysis due to data security restrictions, it is necessary to integrate the data resources of each department (with consistent security constraints and seamless movement) and the associated computing resources (such as the database analysis engine and hardware equipment) and express them as an independent topological node;
[0048] 2) Establish topological nodes based on logical joint boundaries: Taking the above scenario as an example, there are unified data security constraints between business departments and prefecture-level administrative units in the vertical direction (data can be migrated from departments to prefecture-level governments). In order to meet the needs of logical joint analysis of these (department-level) topological nodes, it is necessary to establish a higher-level (city-level) topological node on top of these (department-level) topological nodes and allocate corresponding computing resources;
[0049] 3) Establish topological nodes based on management boundaries: Assume that a provincial unit and the data sets of multiple prefecture-level units it manages exist in a discrete data analysis problem domain. Although the provincial unit and the prefecture-level units have the same data security boundary and trust boundary, due to the limitations of the management boundary, the data sets managed by each unit must still be divided into independent topological nodes and expressed independently in the discrete data model;
[0050] 4) Establish topological nodes based on trust boundaries: Assuming that the data sets of upstream and downstream enterprises in the industrial chain exist in a discrete data analysis problem domain, although there is a dependency relationship between them, due to the trust boundary between them (the trust is based on a contract within a certain time range), the data resources and computing resource sets they manage must still be divided into independent topological nodes and expressed independently in the discrete data model;
[0051] 5) Establish topological nodes based on interest boundaries: Assuming that competitor data sets in the same industry exist in a discrete data analysis problem domain, due to their respective interest boundaries (to protect their respective business secrets), they still need to divide their respective managed data sets into independent topological nodes and express them independently in the discrete data model.
[0052] Step S1012: Determine the topology structure.
[0053] The topological structure refers to the distribution of topological nodes. Usually, according to the hierarchical requirements of data analysis in the discrete data analysis problem domain, the topological nodes are distributed in the leaf nodes and parent nodes of the tree topological structure, and expanded along the X-axis and Y-axis directions. Figure 5 Taking the tree topology of the data model shown as an example, assuming that the problem domain of the analysis subject is to conduct joint data analysis across the four levels of prefecture-level cities, provinces, ministries, and the central government, a discrete data analysis system is considered due to network transmission bandwidth, data real-time, management authority, etc. Due to management requirements - cross-prefecture-level city-level collaborative data resources, cross-provincial collaborative data resources at the ministry level, and cross-ministerial collaborative data resources at the central level, the topological nodes representing the prefecture-level city data island set are placed at the 0th layer (i.e., the leaf node layer), the topological nodes representing the provincial data island set are placed at the 1st layer (i.e., the first-level parent node), the topological nodes representing the ministry data island set are placed at the 2nd layer (i.e., the second-level parent node), and the topological nodes representing the central government are placed at the 3rd layer. The topological nodes at the 3rd layer are the root nodes of the entire tree.
[0054] Regarding the topological structure (distribution mode) of the topological nodes, the following points are explained here:
[0055] (1) A data model for discrete data sets has a tree topology with only one root node.
[0056] (2) The data analysis level requirements are usually determined by the analysis topic; under different analysis topics, even for the same data set, there may be different data analysis level requirements. Figure 5 Taking the tree topology of the data model for discrete data sets shown in the figure as an example, if the analysis problem requires a flat management method, that is, the central government directly manages the prefecture-level city through the ministry, eliminating the provincial hierarchical structure in cross-regional data analysis, then the analysis level requirement may change to: the topological nodes representing the set of provincial data islands (such as Province 1, Province 2... Province M-1, Province M) will also be placed at the 0th layer, as leaf nodes and placed at the same level as the prefecture-level city topological nodes.
[0057] (3) In a discrete data analysis system, nodes of the same level that do not have a common parent node cannot perform joint data analysis. Therefore, this situation needs to be considered when determining the data analysis hierarchy requirements.
[0058] (4) In theory, a low-level topological node can be associated with multiple high-level topological nodes (for example, a leaf node can be associated with multiple parent nodes). However, in practice, in order to ensure the manageability of the entire system, it is agreed that a low-level node can only be associated with one high-level topological node. Joint analysis between topological nodes of the same level can only be completed through the common high-level topological nodes with which they are associated, for example Figure 4In the example, topological node A and topological node A' (assuming that there is an association between the two data sets) need to be jointly analyzed, which can only be completed through their common "ancestor" topological node H. The joint analysis between cross-level topological nodes includes two situations: ① There is a direct "parent-child" relationship between the two, for example Figure 4 The topological node E and the topological node A in the , so that the joint analysis task between the two is completed by the topological node E; ② There is no direct "parent-child" relationship between the two, for example Figure 4 The topological node A and topological node F, or topological node A and topological node E', need to be completed by their common "ancestor" topological node G or topological node H.
[0059] It should be noted that, through the tree topology established above, the data model of discrete data sets has the ability to dynamically expand and scale, that is, to add or reduce topological nodes on the "tree", thereby supporting the flexibility of the discrete data analysis and calculation model.
[0060] Step S102: Establishing a data model set inside each topological node in the topological structure.
[0061] In the tree topology structure of the embodiment of the present invention, the topology node is an abstract expression of the integration of data resources and computing resources involved in the problem domain. In other words, the topology node includes one or more data islands, and each data island (such as a database) may include one or more physical data models (such as the schema of the database).
[0062] There are two methods to establish a data model set within each topology node:
[0063] 1. Virtual model method
[0064] The virtual model method uses the virtual data model as the root model of the topological node to establish a set of internal data models of the topological node, which specifically includes the following steps:
[0065] 1) Establish a set of physical data models.
[0066] The method of establishing a physical data model set inside a topological node is to copy the existing physical data models inside the data island involved into the discrete data model data structure according to the needs of subject analysis. It should be noted that the number of elements (i.e., physical data models) contained in the physical data model set inside the topological node will be affected by the position of the topological node: ① If the topological node is a leaf node, then the number of elements contained in the physical data model set inside the leaf node is greater than or equal to 1, that is, it contains at least one physical data model. In actual applications, although the physical data models all come from data islands (assuming that each data island contains only one physical data model), the number of physical data models contained in the physical data model set is not necessarily equal to the number of data islands. The reason is: Figure 2 As shown in the figure, due to historical development, business needs and other reasons, some or all data islands are integrated into a central data storage medium (such as a data warehouse) through ETL, stream computing and other technical means, but the physical data models of the data islands involved will not be merged. Therefore, although the data island node participating in discrete data analysis has become one, that is, the central data storage medium, the number of physical data models involved has not changed. ② If the topological node is a parent node or a root node, the number of elements contained in the physical data model set inside the parent node or the root node can be zero, that is, it does not contain a physical data model (only the root model and the root model mapping of the lower-level topological node) because: in reality, there is no data island participating in the data analysis at the parent node or the root node.
[0067] 2) Establish a virtual data model.
[0068] Using the data virtualization method, a virtual data model associated with a set of physical data models is established; wherein the virtual data model is used to shield the heterogeneity of multiple physical data models within each topological node. Data virtualization related technical means are common knowledge in the industry, and the details are not repeated here. It should be noted that, in theory, multiple virtual data models can be established inside a topological node to associate multiple physical data models respectively, but according to the "minimum principle" in the above-mentioned establishment rules, each topological node contains only one newly created virtual data model, that is, the root model, and the initial state of the established root model is empty. In actual operation, a global virtual data model (the initial state is also empty) can be established for the entire discrete data model, and then placed in each topological node as a root model, because: the "local" root model of each topological node is essentially a part of the above-mentioned global virtual data model, and the virtual data model of the root node must be the global virtual data model, and the "local" root models of all topological nodes will be mapped here (see the relevant description of step S103 for details).
[0069] After using the virtual model method to establish the internal data model set of the topological node, the hierarchical relationship of the internal data model of the topological node is from low to high: (if any) physical data model set → virtual data model (root model); if there is an externally accessed root model, then it is at the same level as the internal physical data model set and is uniformly merged into the root model of the topological node.
[0070] 2. Graphical Model Method
[0071] The graph model method uses the graph model as the root model to establish a set of internal data models of topological nodes, which specifically includes the following steps:
[0072] 1) Establish a set of physical data models.
[0073] The specific content of establishing the physical data model set in this step is the same as that of establishing the physical data model set in the "virtual model method", which will not be repeated here.
[0074] 2) Establish a virtual data model.
[0075] The purpose of establishing a virtual data model inside a topological node is mainly to shield the heterogeneity of multiple physical data models inside the topological node, and to ensure the uniformity of the graph model method (the method steps will not be inconsistent due to data heterogeneity), as well as to support the technical requirements of dynamic and extensibility of discrete data analysis. The method of establishing a virtual data model based on a set of physical data models, as described in the "virtual model method", can apply the mature data virtualization technology in the industry, and the details will not be repeated here. It should be noted that in theory, multiple virtual data models can be established inside a topological node to associate multiple physical data models respectively, but according to the "minimum principle" in the aforementioned discrete data model establishment rules, only one virtual data model is allowed to be established inside a topological node, and all physical data models are mapped to the virtual data model. In addition, if there is no physical data model inside the topological node (such as a parent node or a root node, and the local physical data island does not participate in the discrete data analysis process), then there is no virtual data model inside the topological node.
[0076] 3) Establish a graph model.
[0077] Using the graph computing method, a graph model associated with a set of physical data models and a virtual data model is established. The technical means related to graph computing are common knowledge in the industry, and the details are not repeated here. It should be noted that, in theory, multiple graph models can be established inside a topological node to associate multiple physical data models or virtual data models respectively, but according to the "minimum principle" in the above-mentioned establishment rules, each topological node contains only one newly created graph model, namely the root model, and the initial state of the established root model is empty. In actual operation, a global graph model (the initial state is also empty) can be established for the entire discrete data model, and then placed in each topological node as a root model. The reason is that the "local" root model of each topological node is essentially a part of the above-mentioned global graph model, and the root model of the root node must be the global graph model, and the "local" root models of all topological nodes will be mapped here (see the relevant instructions of step S103 for details).
[0078] After using the graph model method to establish the internal data model set of the topological node, the hierarchical relationship of the internal data model of the topological node is from low to high: (if any) physical data model set → (if any) virtual data model → graph model (root model); if there is an externally accessed root model (graph model), then it is at the same level as the internal virtual data model and is uniformly merged into the root model of the topological node.
[0079] Step S103: Establishing a vertical association relationship of data models.
[0080] There are two methods for establishing the internal data model set of topological nodes: virtual model method and graph model method. Correspondingly, there are two methods for establishing the vertical association relationship of data models:
[0081] 1. Virtual model method
[0082] After completing the establishment of the internal data model set of the topological node using the virtual model method, you need to further follow Figure 4 The vertical relationship of the data model is established in the Y-axis direction as shown, as follows:
[0083] 1) The vertical relationship between the data model in the leaf node: Figure 7 As shown in (1), there is a vertical association relationship between multiple physical data models and the root model (virtual data model).
[0084] 2) The vertical association relationship of the data model involved in the parent node or root node mainly includes two scenarios:
[0085] ① Such as Figure 7 As shown in (2), multiple vertical associations between root model mappings from low-level topological nodes and local root models;
[0086] ② If Figure 7As shown in (3), multiple root model mappings originating from low-level topology nodes, and vertical associations between multiple local physical data models and local root models.
[0087] From the above two scenarios, it can be seen that the establishment of vertical association relationships of data models under the virtual model law system depends on the implementation of the following two technical scenarios:
[0088] (1) Establishing a vertical association relationship between the physical data model set and the virtual data model within the topological node;
[0089] (2) Establish a vertical association relationship between virtual data models, that is, a vertical association relationship between the low-level root model mapping (after entering the topology node, it still appears as a virtual data model) and the local root model.
[0090] Since the virtual data model is essentially consistent with the physical data model in terms of data object expression semantics, the above two technical scenarios can be implemented using a unified technical means, namely, data virtualization technology. In the embodiment of the present invention, data virtualization technology is used to mainly serve the following two technical goals:
[0091] (1) Establishing the mapping relationship between heterogeneous physical data models and virtual data models. In the data virtualization method system, the principle of establishing the mapping relationship of data models is to map based on the semantics of data object attributes contained in the same data model;
[0092] (2) Establishing heterogeneous object conversion calculation logic. In the above data object mapping process, if the data objects are heterogeneous, such as different data object types and different data object structures (for example, two attributes of a low-level data model data object correspond to one attribute of a high-level virtual data model data object), then the heterogeneity of the data objects is eliminated through conversion calculation logic.
[0093] Since the technical details related to data virtualization are already mature and are described in detail in products such as Informatica Data Virtualization, we will not go into details here.
[0094] 2. Graphical Model Method
[0095] After using the graph model method to complete the establishment of the internal data model set of the topological node, you need to further follow Figure 4 The vertical relationship of the data model is established in the Y-axis direction as shown, as follows:
[0096] (1) Establish the vertical association relationship of the internal data model of the leaf node: Figure 8As shown in (1), multiple local physical data models and a virtual data model are established, and then a vertical association relationship is established between the root model. In this scenario, the method for establishing a vertical association relationship between multiple local physical data models and a local virtual data model can be referred to the "virtual model method".
[0097] (2) Establishing a vertical association relationship between the data models involved in the parent node or root node, including the following two scenarios:
[0098] ① Such as Figure 8 As shown in (2), multiple vertical association relationships are established between the root model mappings from the low-level topological nodes and the local root model.
[0099] ② If Figure 8 As shown in (3), multiple vertical associations are established between the root model mapping of the low-level topology node, the local virtual data model and the local root model. In this scenario, it is necessary to first establish vertical associations between multiple local physical data models and the local virtual data model. For related methods, see the "Virtual Model Method".
[0100] From the above two scenarios, it can be seen that the establishment of vertical associations between data models under the graph model system depends on the implementation of the following two technical scenarios:
[0101] (1) Establish a vertical association between the virtual data model and the graph model within the topology node. The virtual data model is consistent with the physical data model in semantic expression, and a vertical relationship between the virtual data model and the graph model is achieved, that is, a mapping relationship between two heterogeneous data models is established. Given that the relevant technology is mature, the implementation details have been described in the Teleporter module of the OrientDB product, and the details will not be repeated here.
[0102] (2) Establish a vertical relationship between graph models, that is, a vertical relationship between the low-level root model mapping (after entering the topology node, it still appears as a graph model) and the local root model. The data object mapping technology used to establish a vertical relationship between graph models has the following two main implementation methods according to the needs of the analysis topic:
[0103] ① Model-level mapping: The low-level graph model is treated as a virtual vertex and mapped to the high-level graph model in the form of a graph model vertex data object. The graph model vertex data object contains the attribute sets related to all vertices and edge objects of the low-level graph model. Since the relevant technology is already common knowledge in the industry, the details will not be repeated here.
[0104] ② Object-level mapping: Map the data objects (vertex objects and edge objects) contained in the low-level graph model to the high-level graph model in the form of graph model data objects. The object properties and behaviors remain unchanged after mapping. Graph models are all isomorphic. When two or more low-level graph models contain the same vertex or edge objects, they need to be merged and expressed after being mapped to the high-level graph model (generally marked in the discrete data model context information). Since the relevant technology is already common knowledge in the industry, the details will not be repeated here.
[0105] Step S104: Establishing a horizontal association relationship of data models.
[0106] After establishing the vertical relationship of the data model, you need to further follow Figure 4 The horizontal association relationship between data models is established in the X-axis direction as shown. Since the horizontal association relationship of data models is reflected in the topological node root model, it is sufficient to perform related operations at this level.
[0107] There are two methods for establishing the internal data model set of topological nodes: virtual model method and graph model method. Correspondingly, there are two methods for establishing the horizontal association relationship of data models:
[0108] 1. Virtual model method
[0109] The horizontal association relationship between data models is on the virtual data model plane and can be established through the association relationship between the data object attributes contained in the virtual data model.
[0110] by Figure 6 Taking the virtual data model in as an example (the example scenario is expressed in ER diagram semantics, the same below), its three associated local physical data models DM1, DM2, and DM3 belong to three different data islands, namely Figure 7 (1) Scenario. After the three local physical data models are mapped to the virtual data model using data virtualization technology to establish a vertical association relationship (assuming a 1:1 mapping of data objects), the horizontal association relationship between the three local physical data models is established at the virtual data model level through the following intrinsic association relationship between the data object attributes they contain, such as Figure 6 The dashed line shows:
[0111] DM1.School.SchoolNo→DM2.Class.SchoolNo
[0112] (The SchoolNo attribute of the data object School of DM1 is associated with the SchoolNo attribute of the data object Class of DM2, the same below)
[0113] DM2.StudentLog-in.StudentNo→DM3.CourseChoosing.SN
[0114] DM2.Class.ClassNo→DM3.CourseChoosing.TeachingCN
[0115] In view of the semantic consistency between the virtual data model and the physical data model, similarly, the above method is also applicable to establishing horizontal association relationships between multiple virtual data models, and between multiple virtual data models and physical data models.
[0116] Similarly, if Figure 7 (2) Figure 7 (3) Use the virtual model method to establish vertical associations between data models. As described in the above example, all data objects are mapped to the virtual data model level. Then, a similar method is used to establish associations between data objects at the virtual data model level (across data models) to complete the establishment of horizontal associations between data models.
[0117] 2. Graphical Model Method
[0118] The horizontal association relationship between data models is on the graph model plane and can be established through the association relationship between the data object attributes contained in the graph model.
[0119] Taking the "virtual model method" scenario as an example, here we only take the vertical relationship from the virtual data model to the graph model as an example to describe how multiple local physical data models establish horizontal association relationships through the virtual data model and the graph model, that is, Figure 8 (1) Scenario: First, the local physical data model is mapped to the only virtual data model inside the topological node to establish a vertical association between the two (the virtual data model has no dotted line in the initial state, that is, the physical data models have no horizontal association at the virtual data model level); then the virtual data model to graph model mapping method is applied (see step S103) to map the virtual data model data object to the graph model; finally, based on the intrinsic association relationship of the attributes of the vertex object (corresponding to the table object on the virtual data model, which in turn corresponds to the table object of the associated physical data model) (similar to the virtual model method, which will not be repeated here), the edge objects between the vertices are established, thus completing the establishment of the horizontal association between the data models, as shown in FIG. Fig. 9 In the same way, Figure 8 (2) Figure 8(3) Establish vertical associations between data models. As described in the above example, all data objects are mapped to the graph model level. Then, a similar method is used to establish edge objects between vertices at the graph model level to complete the establishment of horizontal associations between data models.
[0120] It should be noted that in the process of establishing the above data model association relationships (vertical association relationships and horizontal association relationships), virtual data models are introduced. The introduction of virtual data models can effectively shield the impact of data heterogeneity on the discrete data analysis process, especially the impact of "data heterogeneity with the same semantics", and further support the dynamic extensibility and scalability of discrete data models.
[0121] After the above method steps are completed, the discrete data model of the embodiment of the present invention is established.
[0122] In practical applications, the discrete data model of the embodiment of the present invention has the following characteristics:
[0123] 1. Discrete data models can be used for one analysis topic or multiple analysis topics;
[0124] 2. Discrete data models may also include necessary contextual information: data silos / data sources corresponding to the objects involved in the analysis subject, as well as contextual information such as the storage location, access method, and security requirements of these data silos / data sources, as well as information such as the correspondence and mutual conversion logic between the objects and the data objects in the physical data model of the data silos / data sources.
[0125] The discrete data modeling method provided by the present invention is significantly different from the traditional data modeling method:
[0126] 1) Different characteristics of data sets: The data modeling method proposed in the embodiment of the present invention is oriented to "discrete" data sets, and is compatible with the five characteristics of "discrete" data analysis and calculation: cross-domain correlation analysis, dynamic expansion and scalability of heterogeneous data models during calculations, data distribution in different locations, data heterogeneity, and data multi-owner characteristics. Traditional data models cannot support the technical characteristics of these five dimensions.
[0127] 2) The structure and establishment method of the data model are different: the discrete data model includes a topological structure for discrete data sets (an embodiment of the present invention proposes a topological structure expression method, namely a tree topological structure), and the method of establishing the topological structure includes a topological node establishment method (based on the data security boundary, management boundary, logical union boundary, trust boundary, and interest boundary of the problem domain) and a topological node distribution method, which are technical features that traditional data models do not have.
[0128] 3) Different data model supporting elements: Discrete data model is a complex nested structure, which is built and supported by data models (including physical data models, virtual data models, and graph models); while traditional data models are built and supported by data objects.
[0129] 4) The methods of establishing element association relationships in data models are different: In the process of establishing association relationships between discrete data models (elements), graphical model methods can be used, especially model-level mapping and object-level mapping methods can be used to establish association relationships. These technical features are not available in traditional data models.
[0130] 5) Data model dynamics, extensibility and scalability are different: In order to support dynamic extensibility in the discrete data analysis process and shield the impact of data heterogeneity, the discrete data model itself has dynamic extensibility and scalability through the (tree) topology structure and the introduction of virtual data model. These are not available in traditional data models.
[0131] As described in the overview, the discrete data model method provided by the example of the present invention establishes the topology of the data model by determining the topological nodes and their topological structures for discrete data sets, and establishes a data model set inside each topological node in the topological structure, as well as an association relationship between the data model sets, thereby realizing the establishment of a data model for a "discrete state" data set, solving the problem that existing data modeling techniques cannot solve the data modeling problem in discrete data analysis scenarios, and supporting the technical requirements of discrete data analysis for data models in five dimensions, including cross-domain correlation analysis, dynamic expansion and scalability of heterogeneous data models during calculations, data distribution in different locations, data heterogeneity, and multi-owner characteristics of data. The discrete data model of the embodiment of the present invention can be used in correspondence with one or more analysis topics, providing an effective solution for enterprises or business departments to solve the problem of insufficient competitiveness caused by their own lack of data resources and information asymmetry, thereby effectively improving the efficiency of the industrial chain and providing users with accurate services.
[0132] See also Fig.10 , an embodiment of the present invention further provides a discrete data model system, including:
[0133] The first establishment module 1001 is used to determine the topological nodes and their topological structures for discrete data sets according to the analysis subject needs, and establish the topology of the data model;
[0134] The second establishing module 1002 is used to establish a data model set inside each topological node in the topology established by the first establishing module 1001; and
[0135] The third establishing module 1003 is used to establish the association relationship of the data model set established by the second establishing module 1002.
[0136] The first establishing module 1001 further includes:
[0137] A storage unit is used to store the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain;
[0138] The topological node unit is used to cut the data resources and computing resources in the problem domain into independent topological nodes according to the storage requirements of the storage unit;
[0139] The topology structure unit is used to place the topology nodes in the topology structure and establish the topology of the data model according to the data analysis level requirements of the discrete data analysis problem domain.
[0140] The second establishing module 1002 further includes: a physical data model set unit, which is used to copy the existing physical data models in the data island involved into the data model data structure according to the analysis subject needs, and establish a physical data model set; and
[0141] The virtualization unit is used to use a data virtualization method to establish a virtual data model associated with the physical data model set established by the physical data model set unit.
[0142] Preferably, when the topology of the data model established by the topology structure unit is a tree topology structure, the third establishing module 1003 further includes:
[0143] A vertical association relationship unit, used to establish a vertical association relationship between the physical data model set established by the physical data model set unit and the virtual data model established by the virtualization unit within the topology node, and to establish a vertical association relationship between the low-level root model mapping and the virtual data model; and
[0144] The horizontal association relationship unit is used to establish a horizontal association relationship of the data model on the virtual data model plane according to the association relationship between the data object attributes contained in the virtual data model established by the virtualization unit.
[0145] Alternatively, the second establishing module 1002 further includes:
[0146] The physical data model collection unit is used to copy the existing physical data models in the data island involved into the discrete data model data structure according to the analysis subject needs, and to establish a physical data model collection;
[0147] a virtualization unit, configured to use a data virtualization method to establish a virtual data model associated with the physical data model set established by the physical data model set unit; and
[0148] The graph model unit is used to use a graph computing method to establish a graph model associated with the physical data model set established by the physical data model set unit and the virtual data model established by the virtualization unit.
[0149] Preferably, when the topology of the data model established by the topology structure unit is a tree topology structure, the third establishing module 1003 further includes:
[0150] A vertical association relationship unit, used to establish a vertical association relationship between a virtual data model established by a virtualization unit and a graph model established by a graph model unit within a topology node, and to establish a vertical association relationship between a low-level root model mapping and a graph model; and
[0151] The horizontal association relationship unit is used to establish a horizontal association relationship of the data model on the graph model plane according to the edge objects between vertices in the graph model established by the graph model unit.
[0152] The discrete data model system provided by the embodiment of the present invention determines the topological nodes and their topological structures for the discrete data set through the first establishment module, establishes the topology of the data model, establishes a data model set inside each topological node in the topological structure through the second establishment module, and establishes the association relationship between the data model sets through the third establishment module, thereby realizing the establishment of a discrete data model and solving the data modeling problem in discrete data analysis scenarios that cannot be solved by existing data modeling technical means.
[0153] In practical applications, each functional module and unit involved in this embodiment can be implemented by a computer program running on computer hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiments of the above methods. Among them, the hardware refers to a server or desktop computer, laptop computer, etc. containing one or more processors and storage media; the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM); the computer program is implemented by computer languages such as C, C++, etc.
[0154] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A data modeling method in a discrete data analysis scenario, characterized in that: include: According to the needs of the analysis subject, the topological nodes and their tree-like topological structures are determined for discrete data sets, and the topology of the data model is established, which specifically includes: the topological nodes are based on the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain, and the data resources and computing resources in the problem domain are cut into independent topological nodes; according to the data security boundary, the data resources and related computing resources of the business department are integrated together and expressed as an independent topological node respectively; according to the logical union boundary, a higher-level topological node is established on the topological node of the business department, and the corresponding computing resources are allocated; according to the management boundary, the data set collection managed by each is divided into independent topological nodes for expression; according to the trust boundary, the data resources and computing resources collection managed by each is divided into independent topological nodes for expression; according to the interest boundary, the data set collection managed by each is divided into independent topological nodes for expression; according to the data analysis hierarchy requirements of the discrete data analysis problem domain, the topological nodes are distributed to the leaf nodes and parent nodes of the tree topological structure, and are expanded in the horizontal and vertical directions; the data resources are embodied in the form of a database, and the computing resources are the analysis engine and hardware equipment of the database; Establishing a data model set inside each topological node, specifically: establishing a data model set inside each topological node based on the principle of establishing a minimum number of logical data models, or establishing at least one virtual data model using a data virtualization method, wherein the virtual data model is used to shield the heterogeneity of multiple physical data models inside each topological node; wherein the logical data model refers to a data model without an actual data object instance, including a virtual data model and a graph model; and Establishing the association relationship of the data model set specifically includes: establishing a vertical association relationship between the physical data model set and the virtual data model within a topological node; establishing a vertical association relationship between a low-level root model mapping and the virtual data model; establishing a horizontal association relationship of data models on the virtual data model plane based on the association relationship between data object attributes contained in the virtual data model; or, establishing a vertical association relationship between the physical data model set and the virtual data model within a topological node; establishing a vertical association relationship between the virtual data model and the graph model within a topological node; establishing a vertical association relationship between a low-level root model mapping and the graph model; and establishing a horizontal association relationship of data models on the graph model plane based on vertex and edge objects in the graph model.
2. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The step of establishing the data model set association relationship specifically adopts the root model method to establish the data model set association relationship; wherein the root model refers to a data model used to describe all data objects and data object association relationships contained in each topological node.
3. The data modeling method in a discrete data analysis scenario according to claim 2, characterized in that: The method of using the root model method to establish the data model set association relationship further includes: establishing a global root model, and placing the global root model in each topological node of the data model as a root model.
4. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The step of establishing a data model set within each topological node based on the principle of establishing a minimum number of logical data models specifically includes: According to the needs of the analysis subject, copy the existing physical data model in the data island involved into the data model data structure to establish a physical data model set; and A data virtualization method is used to establish a virtual data model associated with the physical data model set; wherein the virtual data model is a topological node root model.
5. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The step of establishing a data model set within each topological node based on the principle of establishing a minimum number of logical data models specifically includes: According to the needs of the analysis subject, the existing physical data model in the data island involved is copied to the data model data structure to establish a physical data model set; Using a data virtualization method, establishing a virtual data model associated with the set of physical data models; and Using a graph computing method, a graph model associated with the physical data model set and the virtual data model is established; wherein the graph model is a topological node root model.
6. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The low-level root model mapping means that the low-level topological node can only be mapped to its parent node with the root model as output.
7. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The step of establishing a vertical association relationship between the low-level root model mapping and the graph model is specifically as follows: taking the low-level graph model as a virtual vertex and mapping it to the high-level graph model in the form of the graph model vertex data object; the graph model vertex data object includes all vertex and edge object-related attribute sets of the low-level graph model.
8. The data modeling method in a discrete data analysis scenario according to claim 1, characterized in that: The step of establishing a vertical association relationship between the low-level root model mapping and the graph model is specifically: mapping the data objects contained in the low-level graph model to the high-level graph model in the form of graph model data objects.
9. A data modeling system for discrete data analysis scenarios, characterized in that: include: The first establishment module is used to determine the topological nodes and their topological structures for discrete data sets according to the analysis subject needs, and to establish the topology of the data model; The topological nodes are divided into independent topological nodes based on the data security boundary, management boundary, logical union boundary, trust boundary or interest boundary requirements of the analysis subject in the problem domain. According to the data security boundary, the data resources and related computing resources of the business department are integrated together and expressed as an independent topological node respectively. According to the logical union boundary, a higher-level topological node is established on the topological node of the business department, and the corresponding computing resources are allocated. According to the management boundary, the data set collection managed by each is divided into independent topological nodes for expression. According to the trust boundary, the data resources and computing resource sets managed by each are divided into independent topological nodes for expression; according to the interest boundary, the data set sets managed by each are divided into independent topological nodes for expression; according to the data analysis hierarchy requirements of the discrete data analysis problem domain, the topological nodes are distributed to the leaf nodes and parent nodes of the tree topological structure, and are expanded in the horizontal and vertical directions; the data resources are embodied in the form of a database, and the computing resources are the analysis engine and hardware equipment of the database; a second establishment module, for establishing a data model set inside each topological node in the topology established by the first establishment module, specifically: establishing a data model set inside each topological node based on the principle of establishing a minimum number of logical data models, or using a data virtualization method to establish at least one virtual data model to achieve this, wherein the virtual data model is used to shield the heterogeneity of multiple physical data models inside each topological node; wherein the logical data model refers to a data model without an actual data object instance, including a virtual data model and a graph model; and The third establishment module is used to establish the association relationship of the data model set established by the second establishment module, specifically including: establishing a vertical association relationship between the physical data model set and the virtual data model within the topological node; establishing a vertical association relationship between the low-level root model mapping and the virtual data model; establishing a horizontal association relationship between the data models on the virtual data model plane according to the association relationship between the data object attributes contained in the virtual data model; or, establishing a vertical association relationship between the physical data model set and the virtual data model within the topological node; establishing a vertical association relationship between the virtual data model and the graph model within the topological node; establishing a vertical association relationship between the low-level root model mapping and the graph model; and establishing a horizontal association relationship between the data models on the graph model plane according to the vertex and edge objects in the graph model.
