Discrete indexing method and system

By projecting data objects onto the graph model and performing decomposition and calculation, the data analysis efficiency is improved, and the problem that existing indexing technology cannot effectively serve data analysis and calculation is solved. The data island association relationship is decoupled through the discrete indexing method. Analysis of heterogeneous data objects on multiple data islands.

CN111949830BActive Publication Date: 2024-11-22BIGKNOW (BEIJING) DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910410408.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-17
Publication Date
2024-11-22
Estimated Expiration
2039-05-17

AI Technical Summary

Technical Problem

Existing indexing technology cannot effectively improve data analysis and computing efficiency, especially due to the uncertainty and complexity of the analysis and computing logic, it is difficult to define the list of computing logic pointers in a unified and clear manner, and cannot serve the improvement of data analysis efficiency.

Method used

Using the discrete indexing method, the problem domain data object is projected to the graph model, the graph model data object is established through projection calculation logic, the graph model is instantiated, and the graph model is decomposed and calculated, the discrete index data is output, and the data island relationship is decoupled to realize The allocation and migration of computing logic improves data analysis efficiency.

Benefits of technology

It significantly improves data analysis efficiency, decouples data island association relationships through distributed computing, and is suitable for heterogeneous data object analysis of multiple data islands, improving overall analysis performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111949830B_ABST
    Figure CN111949830B_ABST
Patent Text Reader

Abstract

The present invention discloses a discrete indexing method and system, which belongs to the field of data indexing technology. The method comprises: according to the projection calculation logic, projecting the problem domain data object to the graph model, establishing the graph model data object; according to the projection calculation logic, instantiating the graph model; performing decomposition calculation according to the instantiated graph model, outputting and storing discrete index data. The system comprises a projection establishment module, an instantiation module and a calculation storage module. The discrete indexing method and system provided by the present invention can effectively decouple the association relationship of data islands, realize the allocation and migration of calculation logic to data, and then significantly improve the overall data analysis efficiency through distributed computing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data indexing technology, and in particular to a data indexing method and system for serving joint data analysis of heterogeneous data islands in different locations. Background Art

[0002] In a relational database, an index is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. It is a collection of attribute values ​​of one or more columns in a table and a corresponding list of logical pointers to the data pages that physically identify these values ​​in the table. The role of an index is equivalent to the catalog of a book, and the required content can be quickly searched based on the page numbers in the catalog. In other technical fields, such as search engines, the meaning of indexes is similar to that of the above-mentioned relational databases.

[0003] The definition of the query and analysis calculation process is as follows: given an input set M, the output set N is obtained from the data set X. If the calculation process of obtaining N only includes traversal and extraction calculation operations for the data set X, then the calculation process of obtaining N is called "query"; if the calculation process of obtaining N also includes additional data calculation and processing links, then the calculation process of obtaining N is called "analysis". For example, from a social database, a set of ID card numbers are input to obtain the name, age and other attribute data corresponding to the ID card number. Since the whole process only includes traversal (traversal of the ID card number set) and extraction (name, age attributes, etc. corresponding to each ID card) operations, the calculation process belongs to "query"; if the social circle of friends that meets the conditions and contains "the name of the person is Zhang San" is found from the database for the same input, since the whole process includes complex data calculation logic for obtaining the social group information of "Zhang San" in addition to traversal and extraction calculation operations, the calculation process belongs to "analysis". In the present invention, a storage structure that serves to improve the efficiency of query calculation (i.e., traversal calculation link) is defined as a query index; similarly, a storage structure that serves to improve the efficiency of analysis calculation (i.e., non-traversal calculation link) is defined as an analysis index.

[0004] The existing indexing technologies are all used to improve the efficiency of querying and retrieving (i.e. traversing) data / information. According to the above definition, they belong to "query indexing". However, in real applications, the "analysis" calculation process is more time-consuming and labor-intensive than the "query" calculation process, and it is more necessary to improve efficiency through "indexing" technology (indexing itself is a performance improvement technology that "trades space for time"). In addition, since the analysis and calculation logic usually has uncertainty due to different actual problem-solving scenarios, it is difficult to unify and clearly define it like the "query" calculation process (i.e. traversal calculation). Therefore, the existing indexing technology cannot provide a unified and general method system to establish a logical pointer list of data object instances and calculation logic fragments to serve the improvement of data analysis and calculation efficiency. Summary of the invention

[0005] In order to solve the problem that the existing indexing technology cannot serve to improve the efficiency of data analysis and computing, the present invention provides an analysis indexing method that can serve to improve the efficiency of data analysis and computing, that is, a discrete indexing method, comprising the following steps:

[0006] According to the projection calculation logic, the problem domain data object is projected onto the graph model to establish the graph model data object; the problem domain data object refers to the data object involved in the analysis subject described in the form of a data model, including entity objects and relationship objects; the projection calculation logic includes virtual object calculation logic and projection method logic, the virtual object calculation logic includes virtual entity object calculation logic and virtual relationship object calculation logic, the projection method logic includes instantiation projection method logic and abstraction projection method logic; the data model is a virtual data model aggregated from multiple data island local physical data models;

[0007] According to the attribute association relationship of the vertex objects in the graph model data object, the calculation logic of the data island association relationship is established;

[0008] Instantiate the graph model according to the projection calculation logic and the calculation logic of the data island association relationship;

[0009] Perform decomposition calculation according to the instantiated graph model, and output and store discrete index data;

[0010] The step of projecting the problem domain data object onto the graph model according to the projection calculation logic includes:

[0011] According to the virtual entity object calculation logic, the entity object attribute set actually defined and saved in the data model is processed and calculated to obtain a virtual entity data object; the virtual entity object is a data object that does not exist in the data model but needs to be established in the graph model according to the problem analysis and has the same abstract semantics as the entity object;

[0012] According to the virtual relationship object calculation logic, two entity object attribute sets actually defined and saved in the data model are processed and calculated to obtain a virtual relationship data object;

[0013] According to the logic of the instantiation projection method, the virtual entity data object and the virtual relationship data object are used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in a 1:1 ratio;

[0014] According to the logic of the abstract projection method, the virtual entity data object and the virtual relationship data object are used to establish a vertex and edge object set with the same attribute set or a partially identical attribute set on the graph model in an N:1 ratio;

[0015] According to the logic of the instantiation projection method, the problem domain data objects are used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set in a 1:1 ratio on the graph model;

[0016] According to the logic of the abstract projection method, the problem domain data objects are used to establish a set of vertices and edge objects with the same attribute set or partially the same attribute set in the graph model in an N:1 ratio;

[0017] Where N is a natural number greater than 1;

[0018] The data structure of the discrete index data includes at least four dimensions, namely, data object instance dimension, data island dimension, data object dimension and computing logic fragment dimension; the data island dimension includes independent data islands or logical data islands; the logical data island specifically refers to the physical data island set in the problem domain in the form of an abstract expression of a "logical" object.

[0019] The present invention also provides a discrete indexing system, comprising:

[0020] A projection establishment module is used to project the problem domain data object onto the graph model according to the projection calculation logic, and establish the graph model data object; the problem domain data object refers to the data object involved in the analysis subject described in the form of a data model, including entity objects and relationship objects; the data model is a virtual data model aggregated from multiple data island local physical data models;

[0021] A logic establishment module, used to establish a calculation logic of the data island association relationship according to the attribute association relationship of the vertex objects in the graph model data object established by the projection establishment module;

[0022] An instantiation module, used to instantiate the graph model according to the projection calculation logic and the calculation logic established by the logic establishment module;

[0023] A computing and storage module, used for performing decomposition calculations according to the graph model instantiated by the instantiation module, and outputting and storing discrete index data;

[0024] The projection establishment module comprises:

[0025] A virtual object computing logic unit, used to store virtual entity object computing logic and virtual relationship object computing logic;

[0026] A projection calculation logic unit, used to store instantiated projection method logic and abstracted projection method logic;

[0027] A virtual entity object projection unit, used to process and calculate the entity object attribute set actually defined and stored in the data model according to the virtual entity object calculation logic stored in the virtual object calculation logic unit to obtain a virtual entity data object; the virtual entity object is a data object that does not exist in the data model but needs to be established in the graph model according to the problem analysis and has the same abstract semantics as the entity object;

[0028] A virtual relationship object projection unit, configured to process and calculate the two entity object attribute sets actually defined and stored in the data model to obtain a virtual relationship data object according to the virtual relationship object calculation logic stored in the virtual object calculation logic unit;

[0029] A first instantiation projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on a graph model in a 1:1 ratio for the virtual entity data object and the virtual relationship data object according to the instantiation projection method logic stored in the projection calculation logic unit;

[0030] A first abstract projection unit is used to establish a vertex and edge object set with the same attribute set or part of the same attribute set on the graph model in N:1 for the virtual entity data object and the virtual relationship data object according to the abstract projection method logic stored in the projection calculation logic unit;

[0031] A second instantiation projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in a 1:1 ratio for the problem domain data object according to the instantiation projection method logic stored in the projection calculation logic unit;

[0032] A second abstract projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in N:1 for the problem domain data object according to the abstract projection method logic stored in the projection calculation logic unit;

[0033] Where N is a natural number greater than 1;

[0034] The data structure of the discrete index data includes at least four dimensions, namely, data object instance dimension, data island dimension, data object dimension and computing logic fragment dimension; the data island dimension includes independent data islands or logical data islands; the logical data island specifically refers to the physical data island set in the problem domain in the form of an abstract expression of a "logical" object.

[0035] The discrete indexing method and system provided by the present invention projects the problem domain data object to the graph model according to the projection calculation logic, and completes the instantiation of the graph model according to the projection calculation logic, thereby realizing the decomposition calculation of the graph model to obtain the discrete index. The discrete indexing provided by the present invention can effectively decouple the association relationship of data islands, realize the allocation and migration of calculation logic to data, and thus significantly improve the overall data analysis efficiency through distributed computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the discrete distribution of existing data;

[0037] Figure 2 Schematic diagram of the principle of using the discrete index of this embodiment to implement discrete data analysis;

[0038] Figure 3 It is a schematic diagram of the projection relationship between the problem domain data object and the graph model data object of this embodiment;

[0039] Figure 4 This is a schematic diagram of the principle of synchronizing data on a data island in this embodiment;

[0040] Figure 5 This is a schematic diagram of the principle of data island asynchronous data in this embodiment;

[0041] Figure 6 This is a schematic diagram of the existing electronic checkpoint facilities for urban road traffic;

[0042] Figure 7 This is a schematic diagram of the existing urban road traffic electronic checkpoint network topology;

[0043] Figure 8 is a schematic diagram of the social network analysis community discovery GN algorithm of this embodiment;

[0044] Fig. 9 is a schematic diagram of the gang relationship analysis algorithm of this embodiment;

[0045] Fig.10 is a schematic diagram of the data model of the "discrete" data warehouse of this embodiment;

[0046] Fig.11 is a schematic diagram of a graph model of a discrete index in this embodiment;

[0047] Fig.12 is a flow chart of the discrete indexing method of this embodiment;

[0048] Fig.13 is a schematic diagram of the structure of the discrete index system of this embodiment;

[0049] Fig.14 It is a schematic diagram of the nested expression of data islands in this embodiment. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] In reality, due to data security and management barriers, data islands are isolated from each other, unable to transmit data to each other, and distributed in a "discrete state", such as Figure 1 However, the data objects contained in these data islands are associated with each other, and the joint data analysis of the associated data islands is defined as discrete data analysis in the embodiment of the present invention. In order to improve the efficiency of discrete data analysis, the embodiment of the present invention proposes an analysis index technology, namely discrete index.

[0052] In order to meet the technical requirements of discrete data analysis in terms of data isolation, analysis efficiency and data security, the following can be adopted: Figure 2 The data analysis process shown is as follows: according to the needs of the problem being analyzed, a discrete index is established for the input data set (which may span multiple data islands with network and data security boundaries) through decomposition calculations, and then the data analysis logic fragments are assigned to each data object instance, so as to realize the distributed calculation of each data instance calling the assigned calculation logic, output the intermediate result set, and finally output the final analysis result set by summarizing and processing the intermediate result sets returned by each data object. Discrete data analysis has many application scenarios in reality. It has become a necessary technical means for various enterprises and institutions to solve their own data resource shortages, improve their own competitiveness, and discover new data value. For example, the analysis of discrete data sources across the industry chain links such as the proprietary databases of various hospitals, pharmaceutical factories, and insurance companies will greatly improve the efficiency of the entire industry chain, reduce ineffective waste, and provide users with more accurate services.

[0053] See also Fig.12 The discrete indexing method provided by the embodiment of the present invention comprises the following steps:

[0054] Step S101: Establish a graph model .

[0055] Model of this embodiment The definitions of vertex objects and edge objects are applicable and are completely consistent with the technical terms of existing standard graph models. Vertex objects and edge objects can be described using the Web Ontology Language (OWL); in OWL semantics, vertex objects correspond to entity objects and edge objects correspond to relationship objects. The OWL language has been widely used in the field of knowledge management and the details will not be repeated here. The relevant information can be stored in the backend graph database, or in a database that can express the semantics of the graph model (such as a key-value database, a document database, etc.). It can correspond to one discrete index or multiple discrete indexes (i.e., multiple discrete indexes reuse one graph model); for the sake of convenience, the subsequent steps of this embodiment are limited to a one-to-one correspondence, and the one-to-many case is different from the one-to-one case only in the initial state of the graph model, which does not affect the feasibility of the actual method steps. The initial state can be empty or non-empty. A graph model with a non-empty initial state is generally a case where the graph model is reused. Compared with the case where the graph model has an empty initial state, there is no essential difference in the method steps for establishing a discrete index. For the convenience of discussing subsequent method steps, the following only takes the case where the graph model has an empty initial state as a premise, that is, a new discrete index is created.

[0056] Step S102: Establishing a projection calculation logic of the problem domain data object to the graph model data object, and according to the projection calculation logic, projecting the problem domain data object to the graph model to establish the graph model data object.

[0057] In the embodiment of the present invention, the problem domain data object specifically refers to the data object involved in the analysis subject, which is usually described in the form of a data model. Here, the data model can be a physical data model of a data island, or a virtual data model aggregated from multiple local physical data models of data islands. It should be noted that the virtual data model is essentially consistent with the physical data model in terms of the semantics of data object expression. Taking a structured relational database as an example, both can be expressed in the form of an ER model. The essential difference is that the data object in the physical data model has a corresponding data instance (usually stored in the underlying medium, such as a database), while the virtual data model does not. In general, a virtual data model is established based on multiple physical data models, and data virtualization technology is adopted. Given that data virtualization technology is currently very mature, there are already many commercial products and solutions such as JBOSS's Data Virtualization and Tibco's Data Virtualization, and the details will not be repeated here.

[0058] In addition, the data islands mentioned in the embodiments of the present invention are usually data sources that need to be jointly analyzed, and are generally separated by network boundaries or user permission boundaries. In practical applications, data islands can exist in storage forms such as databases and file systems. For the convenience of describing this embodiment, the data islands referred to in the following are only represented by databases, and data islands in other storage forms can be converted into similar database forms, and then the relevant method steps can be reused.

[0059] The problem domain data objects can include entity objects, which specifically refer to the abstract description of objective things at the data model level, and have a set of attributes that characterize the object characteristics. They are generally entity data objects that have been saved and defined in the data model. If the data model adopts the database ER model semantic description, then the entity object can be a table object in the ER model; if the data model adopts the graph model semantic description, then the entity object can be a vertex object in the graph model. In practical applications, the data categories of entity objects include structured data, such as table objects in databases, and unstructured data described in a structured form, such as JSON data objects, XML data objects, and data objects such as pictures, voices, and videos described in a structured manner.

[0060] The problem domain data objects can also include relationship objects, which specifically refer to the abstract description of the relationship between entity objects, have a set of attributes that characterize the relationship characteristics, and are generally relational data objects that have been saved and defined in the data model; if the data model adopts the database ER model semantic description, then the relationship object can be an association table object of the ER model (a table object used to represent the relationship between two tables and related attributes, such as Fig.10 The "TeachingClass" table object in the graph describes the relationship and attributes between the two table objects "Course" and "Teacher"); if the data model adopts the semantic description of the graph model, then the relationship object can be an edge object in the graph model.

[0061] In actual applications, the above-mentioned entity objects and relationship objects can exist in a static form, a dynamic form, or a mixed form of static and dynamic forms; static form means that the entity or relationship object does not change during the data analysis process (such as file form), and dynamic form means that the entity or relationship object changes during the data analysis process (such as stream data form).

[0062] The projection calculation logic may include the virtual object calculation logic adopted according to the analysis problem needs. The virtual object calculation logic includes the virtual entity object calculation logic and the virtual relationship object calculation logic:

[0063] 1) Virtual entity object calculation logic: Virtual entity objects do not exist in the data model, but need to be established in the graph model based on problem analysis. Virtual entity object calculation logic is used to establish virtual entity objects, which is usually based on the processing calculation logic of the entity object attribute set actually defined and saved in the data model. Fig.10 "Student" entity object (related Fig.10The meaning of each data object is explained below). If the target problem is to find the names of students who live in the same dormitory as each teacher (there should be no less than 2 students in the same dormitory in the result set), then it is also necessary to establish a "dormitory" virtual entity object. The calculation logic of this virtual entity object is the method of calculating the virtual entity object "dormitory" based on the "DormitoryNo" attribute of the entity object "Student". Since the relevant technology is already a consensus in the industry, the details will not be repeated here.

[0064] 2) Virtual relationship object calculation logic: Virtual relationship objects do not exist in the data model, but need to be established in the graph model based on problem analysis. The virtual relationship object calculation logic is used to establish the virtual relationship object, which is usually based on the processing calculation logic of the two entity object attribute sets actually defined and saved in the data model. Fig.10 The relationship between the "StudentLog-in" entity object and the "CourseChoosing" entity object in the graph model does not have a corresponding relationship object definition in the data model. In order to accurately define and describe it in the graph model, it is necessary to establish a virtual relationship object "StudentCourseChoosing" based on the above entity objects (i.e. Fig.11 The virtual relationship object is obtained through the JOIN association calculation between the two (based on the common attribute of "StudentNo"). Since the relevant technology is a consensus in the industry, the details will not be repeated here.

[0065] It should be pointed out that after adopting the virtual object computing logic, the generated virtual entity objects and virtual relationship objects need to be projected into the graph model in order to establish the graph model data object.

[0066] The projection calculation logic may include the projection method logic of projecting the problem domain data object to the graph model data object according to the analysis problem needs. The projection method logic includes the instantiation projection method logic and the abstraction projection method logic, such as Figure 3 As shown:

[0067] 1) Instanced projection method logic: Based on the problem domain entity and relationship object instances, vertex and edge object instances with the same attribute set or partially the same attribute set are established in a 1:1 ratio on the graph model. For example, the above entity or relationship object instances are Fig.10 The "Student" entity object and the "TeachingClass" relationship object actually correspond to a table of data in the database, so each row of data record in the table is an instance of the entity object or relationship object.

[0068] 2) Abstract projection method logic: Based on the problem domain entities and relationship object instances, vertex and edge object instances with the same attribute set or partially the same attribute set are established in the graph model in N:1; where N is a natural number greater than 1. Take the abstract projection of relationship objects as an example: Assuming that the vertex objects in the graph model are passengers A and B (uniquely identified by ID card ID), the edge object instance definition that records the traveling relationship between passengers A and B on the graph model can adopt the abstract projection method logic, that is, no matter how many times passengers A and B have traveled together (which can be considered as data island relationship object instances), there is only one "traveling relationship exists" relationship object (i.e., graph model edge object) instance in the graph model. It should be noted that: in the case of adopting the abstract projection method logic, it is also necessary to define the calculation method related to the abstract projection. For example, the hash table calculation method can be adopted to merge the travel record instances of passengers A and B. Since the calculation method of merging such data instances is already common knowledge in the industry, the details will not be repeated here.

[0069] In practical applications, if the projection process has data security considerations, the projection calculation logic can include data security calculation logic, thereby protecting data privacy while ensuring that it does not affect the subsequent decomposition calculation results of the graph model. Data security calculation logic includes but is not limited to calculation methods such as hashing, encryption, and desensitization. For example, if the above-mentioned passengers A and B are identified on the graph model and recorded in the form of ID cards, it is easy to expose related privacy. At this time, their ID cards can be hashed (generally hash calculations are irreversible) to desensitize the data, and then projected. This not only achieves the same technical effect, but also effectively protects data privacy.

[0070] In practical applications, given that virtual entity objects, virtual relationship objects are consistent with entity objects and relationship objects in the semantic expression of the data model, the projection method logic and virtual object calculation logic can be superimposed and applied according to the needs of solving the problem. That is, the virtual object calculation logic is first performed based on the data model data object, and then the obtained virtual data object is cached, and then the projection method logic is called to project it to the graph model to establish the corresponding graph model data object.

[0071] In actual applications, if the (data island) update data needs to be synchronously projected to the graph model according to the needs of analyzing the problem and receiving the input parameter list, the projection calculation logic may include the synchronization calculation logic. For example, the data island (assuming that the data island exists in the form of relational database storage) updates the data using a two-dimensional table data structure, which needs to be converted into a graph model data structure through the synchronization calculation logic before being synchronously projected to the graph model.

[0072] In practical applications, the various logics included in the projection calculation logic usually need to be determined according to the needs of the analysis problem. For example, in the case of analyzing and searching for communities, the projection calculation logic may include projection method logic and security calculation logic; in the case of analyzing and searching for cloned license plates, the projection calculation logic may include projection method logic, security calculation logic and synchronization calculation logic; in the case of analyzing gang relationships, the projection calculation logic may include projection method logic, security calculation logic and synchronization calculation logic, etc.

[0073] According to the projection calculation logic, the problem domain data objects are projected onto the graph model to form graph model data objects, namely vertex objects and edge objects. It should be noted that the attributes of the graph model data objects (vertex objects, edge objects) may include all or part of the attributes of the data objects of the corresponding data model, that is, the attributes of the graph model data objects (vertex objects, edge objects) are a subset of the attributes of the data objects of the corresponding data model; in addition, the attributes of the graph model data objects may further include custom attributes that need to be established in the graph model analysis.

[0074] S103: Establishing the calculation logic of the data island association relationship according to the attribute association relationship of the vertex objects in the graph model data object.

[0075] The graph model data objects include all data objects of the data model (corresponding to the data islands involved) required for analysis; in the graph model, the data island association relationship is described and characterized by the association relationship between the data objects contained in the data model, and the calculation logic of the association relationship between data objects (i.e. the calculation logic of the data island association relationship) is established based on the attribute association relationship of the vertex objects (derived from different data models) in the graph model. For example, suppose Fig.10 The "StudentLog-in" entity object in the data model of the data island (database, the same below) in domain 2 uses the instantiation projection method logic to establish the corresponding vertex object on the graph model; similarly, the "CourseChoosing" entity object in the data model of the data island in domain 3 also uses the instantiation projection method logic to establish the corresponding vertex object on the graph model; then based on the same "StudentNo." attribute of the two types of vertex objects, the following can be established: Fig.11 The graph model "optional" edge object, the edge object instance set represents the above data island association relationship. Since the technology of establishing graph model edge object instances based on attribute data calculation is common knowledge in the industry, the details will not be repeated here.

[0076] In practical applications, the calculation logic for establishing data island association relationships is optional rather than mandatory, which is usually determined by the needs of analyzing the problem. For example, the following example of step S105 "dynamic topology and asynchronous data" combination conditions (such as Fig. 9As shown in the figure), the data island association relationship is reflected by the overlap of entity object instances (i.e., multiple data islands have citizens with the same ID card ID), without the need to establish association relationships between objects across data islands (i.e., edge objects on the graph model), that is, there is no need to establish the calculation logic of the data island association relationship. However, as a comparison, in the example under the combination condition of "static topology and asynchronous data" in step S105, if the need for analyzing the problem is to output an analysis report across data islands, then it is necessary to establish the calculation logic of the data island association relationship, that is, it is necessary to establish the calculation logic of the data island association relationship.

[0077] Step S104: extract data and transmit it to the target location, and call the projection calculation logic and the calculation logic of the data island association relationship to complete the graph model instantiation.

[0078] The graph model instantiation process is the process of assigning data instances to graph model object definitions. Since the object instantiation process is common knowledge in the industry, the details are not repeated here. The instantiation process needs to first extract the data object instance defined in the projection calculation logic in step S102, and then call the calculation logic of the projection calculation logic and the data island association relationship to complete the instantiation.

[0079] The transmission of extracted data can be carried out within a network domain or across network domains; it can be transmitted directly or forwarded through another computer (such as a front-end system); it can be transmitted in plain text or in cipher text.

[0080] The target location may be a memory or a persistent storage medium; the persistent storage medium may include but is not limited to a file system or a database.

[0081] The calculation logic of the relationship between the projection calculation logic and the data island can use Hadoop ecosystem data analysis tools including but not limited to Apache Flink, Apache Spark, etc., or can use other data analysis tools including but not limited to MPI, machine learning, etc.

[0082] In the process of graph model instantiation, mature ETL (Extract-Transform-Load) methods, data cleaning methods, data synchronization (Change Data Capture, CDC) and other technical means can be used. Since the relevant technologies are mature, the details are not repeated here. Graph model instantiation can be automatically started according to user-defined rules. For example, in real applications, there is a scenario where updated data on the data island side is continuously input into the graph model (such as the scenario under the synchronization data constraint in step S105). The user-defined rule can be that the graph model automatically starts running after the input data increment reaches the user-defined threshold, or the graph model instantiation can be manually started by the user.

[0083] Step S105: Perform decomposition calculation according to the instantiated graph model and output discrete index data.

[0084] The purpose of decomposing the instantiated graph model is to decompose the overall analysis and calculation logic for the global data set into calculation logic fragments for each data instance, and then output discrete index data, such as Figure 2 The functional relationship between discrete index data and graph model can be expressed as:

[0085]

[0086] in: It is discrete index data; To decompose the calculation function; It is an instantiated graph model, including the topological structure of the graph, the set of vertex and edge instances that constitute the topology, and the attributes of each element of the vertex and edge instance set; The input data during the decomposition calculation is derived from the updated data of the data island during the decomposition calculation (usually, the updated data needs to call the corresponding calculation logic and then project it into the graph model, such as Figure 4 and Figure 5 as shown).

[0087] In the decomposition calculation function During runtime, the instantiated graph model (Decomposition calculation function There are only two situations for the input of : no change or change. exist The situation that does not change during operation is defined as static topology, and the situation that does not change during operation is defined as dynamic topology. Changes can include changes to the properties of vertex and edge instances and / or the graph model The topological structure changes. In the decomposition calculation function During operation, whether the updated data of the data island is converted into a decomposition calculation function Input data There are also two situations: "conversion" or "no conversion". In this embodiment, after receiving the updated data, the data island projects all or part of the updated data onto the graph model within a preset period and converts it into a decomposition calculation function. Input data during operation , it is defined as synchronous data, otherwise it is asynchronous data. The existence form of only converges to 4 intervals: under the constraints of static topology and synchronous data combination conditions, under the constraints of dynamic topology and asynchronous data combination conditions, under the constraints of dynamic topology and synchronous data combination conditions, and under the constraints of static topology and asynchronous data combination conditions.

[0088] The following examples are given for the above four combined condition constraint intervals to illustrate the decomposition calculation function. Feasibility:

[0089] 1. Static topology and synchronous data combination conditions

[0090] A typical example of this combination of conditions is to find a fake license plate vehicle. The scenario is described as follows: A fake license plate vehicle refers to a vehicle that has been forged and illegally obtained from a real license plate vehicle, with the license plate number, model, and color being used to make smuggled, assembled, scrapped, or stolen vehicles appear "legal." A fake license plate is a label that refers to a fake license plate with the same number on another vehicle. The method for finding a fake license plate vehicle is to use an electronic card slot (such as Figure 6 as shown) to identify the license plate number of the passing vehicle; if the city electronic card network (such as Figure 7 If the electronic checkpoints at adjacent time nodes (as shown in the figure) recognize the same license plate, and the average vehicle speed obtained by dividing the shortest distance between the two electronic checkpoints by the above adjacent time difference exceeds the acceptable speed range (for example, 60 kilometers per hour), then the license plate is a suspected duplicate license plate vehicle; then the relevant detailed information (such as the electronic checkpoint’s photo of the vehicle license plate and the driver, etc.) will be retrieved for further identification.

[0091] For searching for fake license plate vehicles, the process of establishing discrete index data in this embodiment is as follows:

[0092] 1) Define input variables And instantiate: According to this example, the problem needs to establish a graphical model ; Then establish the projection calculation logic:

[0093] ① Virtual object calculation logic: This instance does not need to establish a virtual object;

[0094] ② Projection method logic: This example adopts the instantiation projection method logic. The vertex instance of the graph model corresponds to the electronic checkpoint, which is uniquely identified by the electronic checkpoint ID; the edge instance of the graph model corresponds to the connection line (i.e., road) between each adjacent electronic checkpoint in the urban electronic checkpoint network topology, and the attribute of the edge instance is the distance between adjacent checkpoints;

[0095] ③Secure calculation logic: If information security is required, the above-mentioned electronic card ID and the license plate number involved in the calculation can be hashed. This processing step does not affect the subsequent The operation of the computational logic;

[0096] ④Synchronous calculation logic: that is, the input variables mentioned later Definition of;

[0097] After establishing the projection calculation logic, you also need to establish the calculation logic of the data island association relationship: cross-data island edge objects (i.e. Figure 7 The data may come from different electronic checkpoints (local databases / applications). The establishment method is to perform association calculation based on the association attributes (i.e., associated roads) of vertex instances with associated relationships (there are road association relationships across electronic checkpoints) (the left and right vertex instances of the road correspond to two electronic checkpoint IDs in different data islands).

[0098] Based on the established projection calculation logic and the calculation logic of data island association relationship, the graph model The vertex objects and edge objects can be created. During runtime, the graph model It does not change, so it belongs to the static topology situation. The instantiation method is to import the relevant electronic checkpoint ID, the network topology of the city network electronic checkpoint (including the cross-electronic checkpoint database / cross-city (cross-data island) electronic checkpoint network topology), and the distance between adjacent checkpoints as a static file (such as CSV, JSON or XML), call the above projection calculation logic and the calculation logic of the data island association relationship, and assign it to the graph model. Vertex and edge instances in the graph complete the instantiation of the graph model.

[0099] 2) Define input variables : In actual applications, each electronic checkpoint transmits the license plate number and vehicle / driver photo data to the local system (database / application) in real time. The local system is the data island described in this embodiment. To support the above analysis logic, it is necessary to During operation, the updated data (captured and recognized license plate and time information) received by the data island is synchronously input to the , to support real-time detection of suspicious fake license plate vehicles. Refer to the above about input data The definition of This is a synchronous data situation.

[0100] 3) Determine The calculation logic: After receiving After that, based on the electronic checkpoint ID' where the license plate is located at the most recent time point, the shortest distance between the electronic checkpoint ID and ID' (ID and ID' are also the unique identifiers of the vertex instances of the graph model) is calculated on the graph model, and then based on the shortest distance-time difference-average speed correlation, it is determined whether the license plate is a suspected duplicate license plate; if so, the license plate number, checkpoint ID and ID', passing time and other information are output. The SSSP (Single Source Shortest Path) algorithm for calculating the shortest distance between graph vertices in the above calculation logic is already a known graph analysis algorithm in the industry, and the details will not be repeated here.

[0101] 4) Output discrete index data : The key of the discrete index is "license plate + checkpoint ID + checkpoint passing time" of the suspected fake license plate vehicle (please see the relevant description below for the data structure definition of the analysis index, the same below), and each key corresponds to the calculation logic fragment - query related attributes (such as photos, passing time) according to the key. Other discrete index information also includes the data island location information corresponding to the key, object identification, etc.

[0102] 2. Dynamic topology and asynchronous data combination conditions

[0103] A typical example of this combination of conditions is community discovery. The scenario is described as follows: Assume that there is call data from multiple telecom operators (corresponding to multiple data islands), and a social network based on citizen ID numbers is formed based on the call history records. Now it is necessary to find a community at a certain time based on the social network, and output the community member information (such as age, gender, etc.) based on the ID number at that time.

[0104] For community discovery, the process of establishing discrete index data in this embodiment is as follows:

[0105] 1) Define input variables And instantiate: According to this example, the problem needs to establish a graphical model ; Then establish the projection calculation logic:

[0106] ① Virtual object calculation logic: This instance does not need to establish a virtual object;

[0107] ② Projection method logic: This example uses the instantiation projection method logic to establish the graph model vertex object (instance), that is, to project the citizen ID numbers of all users in the underlying data island onto the graph model in a 1:1 manner to establish the graph model vertex instance; the graph model edge object (instance) uses the abstraction projection method logic, that is, to merge and abstract the communication history between citizen A and citizen B in the underlying data island (no matter how many times they have communicated with each other), and the edge object projected onto the graph model is just "there is a communication relationship between citizen A and citizen B";

[0108] ③ Security calculation logic: If information security is required, the above citizen ID number and the communication relationship identifier between citizens can be hashed. This processing step does not affect the subsequent The operation of the computational logic;

[0109] ④Synchronous calculation logic: that is, the input variables mentioned later Definition of;

[0110] After establishing the projection calculation logic, it is also necessary to establish the calculation logic of the data island association relationship: In this example, the method of establishing the calculation logic of the data island association relationship is mainly to find the situation where the vertex object instances (i.e. citizens) of the graph model have a communication relationship across operators (i.e. data islands), such as citizen A who uses the service of operator 1 has a communication relationship with citizen B who uses the service of operator 2, and then establish the relevant calculation logic;

[0111] Based on the established projection calculation logic and the calculation logic of data island association relationship, the graph model The vertex objects and edge objects can be created. During runtime, the graph model The topology is constantly changing (see below for details), so it belongs to the dynamic topology situation. The instantiation method is to extract data from the underlying data islands, and at the same time call the above-mentioned projection calculation logic and the calculation logic of the data island association relationship, establish a set of vertex object instances and edge object instances on the graph model, and complete the graph model instantiation.

[0112] 2) Define input variables : In this example scenario, since the problem being analyzed is the analysis of a social network (i.e., a graph model) at a certain time node, During the operation, there will be no input data, which belongs to the asynchronous data situation, that is, =0.

[0113] 3) Determine Calculation logic: There are many mature algorithms for community discovery based on social networks, such as the Girvan-Newman (GN) algorithm, which needs to calculate the betweenness for edge objects (instances) in the graph model; the so-called betweenness of an edge refers to the proportion of the shortest path passing through the edge in the network to all the shortest paths. Figure 8 As shown in Figure 2, the main steps of the GN algorithm include: a) calculating the betweenness of all edges in the graph model; b) finding the edge with the highest betweenness and removing it from the graph; c) repeating the above steps until each node is a community. Runtime graph model The topology keeps changing until The operation is completed, so it belongs to the dynamic topology situation.

[0114] 4) Output discrete index data : The key of the discrete index is the citizen ID number. The calculation logic fragment corresponding to each key is to query related attributes (such as name, age, etc.) according to the key. Other discrete index information also includes the location information of the data island corresponding to the key, object identifier, etc.

[0115] 3. Dynamic topology and synchronization data combination conditions

[0116] A typical example of this combination of constraints is group relationship discovery. The scenario is described as follows: This scenario requires a dynamic social network (such as Fig. 9 As shown in Figure 2-- Social relationships may change during the data analysis, and gang relationships may be discovered. The discovery rule is to satisfy the co-living relationship (obtained from the hotel data, i.e. Fig. 9 The relationship shown in 1), the same Internet access relationship (obtained based on Internet cafe data, that is, Fig. 9 The relationship shown in 2), communication relationship (obtained according to communication data, that is, Fig. 9 The relationship shown in 3) is that two of the three relationships can be identified as a pair of suspicious gang relationships; then further identification is made based on their gender, age and other information. It should be noted that the difference between this scenario and the "dynamic topology and asynchronous data combination conditions" is that there are multiple relationships between citizens, and the social network may change continuously during the entire analysis operation. The direct cause of this change comes from the data changes of the underlying data islands, which are not covered by the analysis algorithm of the "dynamic topology and asynchronous data combination conditions" scenario.

[0117] For gang relationship discovery, the process of establishing discrete index data in this embodiment is as follows:

[0118] 1) Define input variables And instantiate: According to this example, the problem needs to establish a graphical model ; Then establish the projection calculation logic:

[0119] ① Virtual object calculation logic: This instance does not need to establish a virtual object;

[0120] ② Projection method logic: This instance uses the instantiation projection method logic to establish the graph model vertex object (instance), that is, to project the citizen ID numbers of all users in the underlying data island onto the graph model in a 1:1 manner to establish the graph model vertex instance; the graph model edge object (instance) uses the abstraction projection method logic, that is, to merge and abstract the relationship between citizen A and citizen B (one of the above three relationships) in the underlying data island (no matter how many times they have relationships), and the edge object projected onto the graph model is just "Citizen A and Citizen B have relationship X" (X = 1, 2, 3), and the relationship is identified by a unique ID (such as Fig. 9 shown);

[0121] ③ Security calculation logic: If information security requires, the above citizen ID numbers and the relationship identifiers between citizens can be hashed. This processing step does not affect subsequent The operation of the computational logic;

[0122] ④Synchronous calculation logic: that is, the input variables mentioned later Definition of;

[0123] After establishing the projection calculation logic, you also need to establish the calculation logic of the data island association relationship: In this example, since the data island association relationship is reflected by the overlap of entity object instances (that is, the hotel data island, communication data island, and Internet cafe data island all have the same citizen entity object instance projection on the graph model), there is no need to establish the calculation logic of the data island association relationship (that is, the edge object);

[0124] Based on the established projection calculation logic and data island association relationship calculation logic, the graph model The vertex objects and edge objects can be created. During operation, social relationships are constantly changing, resulting in the continuous input of new social relationship data, which in turn makes the graph model The topology is constantly changing, so it belongs to the dynamic topology situation. The instantiation method is to extract data from the underlying data islands, and at the same time call the above-mentioned projection calculation logic and the calculation logic of the data island association relationship, establish a set of vertex object instances and edge object instances on the graph model, and complete the graph model instantiation.

[0125] 2) Define input variables : In this example scenario, as the backend data island data is updated, the relationship between people may change at any time (for example Fig. 9 The relationship between Li Si and Wang Wu in the analysis may be established later as time goes by). All these change data are synchronously input to the analysis subject according to the analysis subject requirements - the suspicious gang relationship is continuously output during the data analysis period, within the pre-defined delay range (for example, 30 seconds). , to support real-time detection of suspicious gang relationships.

[0126] 3) Determine The calculation logic is: The analysis logic is based on the graph analysis method, that is, for the graph topology at any time (considering that the changed data constantly changes the graph topology structure), run the graph traversal algorithm (such as BFS). If the number of edges between any pair of vertex instances at that time is greater than or equal to 2, then the vertex instance pair (i.e., the citizen ID number pair) is a candidate gang relationship. Since the relevant graph analysis algorithm is already a known technology in the industry, the details will not be repeated here.

[0127] 4) Output discrete index data : The key of the discrete index is the citizen ID number. The calculation logic fragment corresponding to each key is to query related attributes (such as name, gender, age, etc.) according to the key. Other discrete index information also includes the location information of the data island corresponding to the key, object identifier, etc.

[0128] 4. Static topology and asynchronous data combination conditions

[0129] A typical example of this combination of constraints is a "discrete" data warehouse analysis report. The scenario is described as follows: Assume that there are data objects (i.e. tables) with associated relationships coming from multiple data islands (represented by different databases in domain 1, domain 2, and domain 3), such as Fig.10 However, due to data security, transmission bandwidth, or data real-time constraints, it is impossible to use traditional technology to concentrate them in a data warehouse or similar storage media for joint analysis. Therefore, it is necessary to establish a "discrete" data warehouse (related data objects are distributed on isolated data islands) to output analysis reports across data islands.

[0130] For the "discrete" data warehouse analysis report, the process of establishing discrete index data in this embodiment is as follows:

[0131] 1) Define input variables And instantiate: According to the data model containing the association relationship between data objects across data silos (such as Fig.10 Build a graph model (like Fig.11 Then establish the projection calculation logic:

[0132] ① Virtual object calculation logic: This instance needs to establish a virtual relationship object for the relationship between objects within the data island. The establishment method is based on the foreign key relationship between two table objects using the "JOIN" calculation logic, such as Fig.10 The relationship between the "CourseChoosing" and "Course" table objects needs to call the JOIN calculation logic to establish a virtual relationship object so that it can correspond to Fig.11 The edge object of the "subordinate" relationship between "Courses" and "Selected Courses";

[0133] ② Projection method logic: This instance adopts the instantiation projection method logic. The graph model vertex instance corresponds to the table object instance; the graph model edge instance corresponds to the existing association table object in the data model (such as Fig.10 "TeachingClass" table object in the table) or a virtual relationship object instance;

[0134] ③ Security calculation logic: If information security is required, the primary key of the above entity or virtual table object instance can be hashed. This processing step does not affect the subsequent The operation of the computational logic;

[0135] ④Synchronous calculation logic: that is, the input variables mentioned later Definition of;

[0136] After establishing the projection calculation logic, you also need to establish the calculation logic of the data island association relationship: based on the association attributes of the vertex instances with association relationships, for example, based on the StudentNo. attribute of the "CourseChoosing" and "StudentLog-in" object instances. Fig.11 An example of an "elective" edge object in a data island association relationship across domains 2 and 3.

[0137] Based on the established projection calculation logic and the calculation logic of data island association relationship, the graph model The vertex objects and edge objects can be created. During runtime, the graph model It does not change, so it belongs to the static topology situation. The instantiation method is to extract data from the underlying data islands, and at the same time call the above-mentioned projection calculation logic and the calculation logic of the data island association relationship, establish a set of vertex object instances and edge object instances on the graph model, and complete the graph model instantiation.

[0138] 2) Define input variables : In this example scenario, general analysis reports are for analysis of a certain time node, so During operation, no change data will be entered, i.e. =0.

[0139] 3) Determine Calculation logic: The analysis logic of data reports is generally expressed in SQL. The analysis logic across heterogeneous table data objects (including table objects across data islands (databases)) is mainly expressed in JOIN operators. The JOIN operator is mainly based on inline calculation logic. Other calculation logics such as left and right joins are all relational algebra extensions based on inline calculations. Therefore, the following only takes inline calculations as an example. Other scenarios can refer to the industry common sense related to relational algebra calculations for expansion.

[0140] In this example scenario, see Fig.10 , the table objects "StudentLog-in" and "CourseChoosing" belong to different data islands (databases), and the two are established through the association attribute "StudentNo." Fig.11 If you create a "discrete" analysis report between the table objects "StudentLog-in" and "CourseChoosing", the corresponding The calculation logic is Fig.11 A graph traversal algorithm (such as BFS) is used to traverse all "elective" edge instances on the graph model shown, and the left and right vertex pair instances of the edge instances are output (each vertex ID is uniquely identified, and the vertex ID corresponds to the primary key of the table object row record to which it is projected).

[0141] 4) Output discrete index data : The vertex ID, that is, the primary key of the corresponding table object row record, is the key of the discrete index. The calculation logic fragment corresponding to each key is to query related attributes based on the key. Other discrete index information also includes the location information of the data island corresponding to the key, the (table) object identifier, etc.

[0142] In an embodiment of the present invention, the discrete index data structure includes at least four dimensions, namely (Key, DID, OID, CID), wherein Key is a unique identifier of a data object instance in the data object instance dimension, such as the primary key of the above-mentioned table object; DID is a unique identifier of a data object instance (corresponding data object) in the data island dimension, such as "domain name. database name. mode name", etc., wherein the domain name is a unique identifier of a network isolation area, which can be an enterprise or a department; the database name is a unique identifier of a database in the domain; the mode name is a unique identifier of the mode of the database; OID is a unique identifier of a data object instance (corresponding data object) in the data object dimension, such as a table object in the above-mentioned database mode (schema); CID is a unique identifier of a data object instance in the computing logic fragment dimension. There is an orthogonal relationship between the above four dimensions because: multiple data islands can contain a homogeneous data object, a homogeneous data object can contain multiple data object instances, and multiple instances of a homogeneous data object that may belong to different data islands can correspond to multiple different computing logic fragments. In practical applications, according to the needs of problem solving, the discrete index data structure can also include more other dimensions, such as user dimension, role dimension, etc., to control the user's access rights to data object instances.

[0143] It should be noted that a DID can point to a physical data island or a collection of data islands, i.e., a logical data island. In the embodiment of the present invention, a logical data island specifically refers to a collection of physical data islands in the problem domain in the form of a "logical" object abstract expression. The i-th logical data island vDID in the problem domain i The expression is as follows:

[0144] vDID i = {DID i1 , DID i2 , ..., DID in} (n represents the number of physical data islands contained)

[0145] Take the "search for a fake license plate vehicle" scenario in step S105 as an example. Fig.14 As shown, if the analysis subject needs to find cloned license plate vehicles used in cross-provincial crimes at the "department" level, then the data island "province" seen at the "department" level is actually a logical data island (corresponding to the fact that "province" is an administrative division at a logical level in reality), and only the "province" can see the truly independent data islands it contains - cities and prefectures.

[0146] In actual applications, since logical data islands are consistent with physical data islands in terms of semantic expression, they can also be mixed with independent data islands in the process of establishing discrete indexes. The specific method is to add a new namespace (namespace, generally the unique identifier of the logical data island in the problem domain) to the independent data islands in the logical data islands and the data objects they contain, without changing their properties and behaviors. If you need to instantiate the data object of the logical data island, it is actually a recursive call to the instantiation process of the data objects contained in the independent data islands involved in the logical data island. Since the relevant technology is already common knowledge in the industry, the details will not be repeated here.

[0147] S106: The discrete index data is persistently stored in a target storage medium according to a predefined traversal mechanism, and a read-write access interface is provided.

[0148] The traversal mechanism can be set according to dimensions such as Key or OID or DID or CID. According to the traversal mechanism set for the selected dimension, the discrete index data is sorted and then persistently stored in the target storage medium. The target storage medium can be a storage medium including a relational database, a Key-Value database, and a graph database. The discrete index data persisted on the target storage medium can also select other traversal dimensions to establish indexes to improve the traversal efficiency of the selected dimension; the method of establishing indexes based on discrete index data can adopt but is not limited to sorting, Hash and other methods. The relevant technology is mature and the details will not be repeated here. The read and write access interface includes but is not limited to interfaces that support query, update, delete and other operation steps.

[0149] The discrete indexing method provided by the embodiment of the present invention projects the problem domain data object to the graph model according to the projection calculation logic, and establishes the data island association relationship according to the attribute association relationship of the vertex object in the graph model data object, completes the graph model instantiation, and then realizes the decomposition calculation of the graph model to obtain the discrete index. The discrete index provided by the embodiment of the present invention is for entity objects with association relationships and located in different data islands, and records the association relationship between them, and then through the association relationship information, the data island can be decoupled, thereby supporting discrete data analysis.

[0150] See also Fig.13 , an embodiment of the present invention further provides a discrete indexing system, including:

[0151] The projection establishment module 501 is used to project the problem domain data object to the graph model according to the projection calculation logic, and establish the graph model data object; wherein the problem domain data object refers to the data object involved in the analysis subject described in the form of a data model, including entity objects and relationship objects;

[0152] The logic establishment module 502 is used to establish the calculation logic of the data island association relationship according to the attribute association relationship of the vertex objects in the graph model data object established by the projection establishment module 501;

[0153] An instantiation module 503, used to instantiate a graph model according to the projection calculation logic and the calculation logic established by the logic establishment module 502;

[0154] The computing and storage module 504 is used to perform decomposition calculations according to the graph model instantiated by the instantiation module 503, and output and store discrete index data.

[0155] The projection establishing module 501 further includes at least one of the following:

[0156] A virtual object computing logic unit, used to store virtual entity object computing logic and virtual relationship object computing logic;

[0157] A projection calculation logic unit, used to store instantiated projection method logic and abstracted projection method logic;

[0158] A virtual entity object projection unit, used to process and calculate the entity object attribute set actually defined and stored in the data model according to the virtual entity object calculation logic stored in the virtual object calculation logic unit to obtain a virtual entity data object;

[0159] A virtual relationship object projection unit is used to process and calculate the two entity object attribute sets actually defined and stored in the data model to obtain a virtual relationship data object according to the virtual relationship object calculation logic stored in the virtual object calculation logic unit;

[0160] A first instantiation projection unit is used to establish a vertex and edge object set with the same attribute set or part of the same attribute set in a 1:1 manner for the virtual entity data object and the virtual relationship data object on the graph model according to the instantiation projection method logic stored in the projection calculation logic unit;

[0161] A first abstract projection unit is used to establish a vertex and edge object set with the same attribute set or a part of the same attribute set on the graph model in N:1 for the virtual entity data object and the virtual relationship data object according to the abstract projection method logic stored in the projection calculation logic unit;

[0162] A second instantiation projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in a 1:1 ratio for the problem domain data object according to the instantiation projection method logic stored in the projection calculation logic unit;

[0163] The second abstract projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set in the graph model in N:1 for the problem domain data object according to the abstract projection method logic stored in the projection calculation logic unit;

[0164] A secure computing logic unit, used to store secure computing logic; or

[0165] A secure computing unit, configured to perform secure computing on at least one of the virtual entity data object, the virtual relationship data object, and the problem domain data object according to the secure computing logic stored in the secure computing logic unit;

[0166] Wherein N is a natural number greater than 1.

[0167] The computing and storage module 504 further includes:

[0168] A function building unit, used to build a decomposition calculation function between a discrete index and a graph model instantiated by an instantiated module;

[0169] a calculation output unit, used to calculate and output discrete index data according to the decomposition calculation function established by the function establishment unit; and

[0170] The storage unit is used to persistently store the discrete index data output by the calculation output unit to the target storage medium according to a predefined traversal mechanism, and provide a read-write access interface.

[0171] Furthermore, the computing and storage module 504 further includes:

[0172] The index establishment unit is used to persistently store the discrete index data of the storage unit on the target storage medium, and to traverse and establish the index again according to a dimension in the discrete index data structure.

[0173] The discrete indexing system provided by the embodiment of the present invention projects the problem domain data object to the graph model through the projection establishment module, establishes the data island association relationship through the logic establishment module, completes the graph model instantiation through the instantiation module, and then realizes the decomposition calculation of the graph model through the calculation storage module to obtain the discrete index. Through the discrete indexing system of the embodiment of the present invention, it is possible to effectively decouple the data island association relationship, realize the allocation and migration of the calculation logic to the data, and then significantly improve the overall data analysis efficiency through distributed computing.

[0174] Compared with the traditional index, the discrete index in the embodiment of the present invention has the following differences:

[0175] 1. Differences in technical positioning between traditional indexes and discrete indexes: Existing indexing technologies include relational database indexes, graph database indexes, and search engine indexes, all of which are query indexes used for query services; the discrete indexes proposed in the embodiments of the present invention are used for analysis services; the two are essentially different in technical positioning.

[0176] 2. Different establishment methods: Different from traditional index establishment methods such as sorting and hashing, discrete index establishment adopts a graph model method, which supports complex projection calculations, data island association relationship calculations, and decomposition calculations during the establishment process.

[0177] 3. Different calculation methods: Unlike the one-stage sorting calculation of traditional indexes, discrete indexes need to go through three stages of calculation: projection calculation, data island association relationship calculation, and decomposition calculation, serving the technical goals of each link in index establishment.

[0178] 4. Different scope: Traditional indexes can only be used to accelerate the traversal of one or more attributes of a homogeneous object in a data island, while discrete indexes can be used to accelerate the traversal of multiple attributes of multiple heterogeneous objects in multiple data islands.

[0179] 5. Different data structures: Traditional index data only needs to record two dimensions of data: keywords and logical pointers of positions, while discrete indexes include at least four dimensions of data, and more dimensional information can be expanded as needed.

[0180] The discrete index of the embodiment of the present invention is a data storage structure that stores a list of logical pointers to any data object instance and the corresponding information including but not limited to computing logic fragments, storage locations, etc. The discrete index of the embodiment of the present invention serves data analysis by decomposing the global data analysis computing logic into computing logic fragments for each data object instance, thereby significantly improving the overall data analysis efficiency through distributed computing.

[0181] In practical applications, each functional module and unit involved in this embodiment can be implemented by a computer program running on computer hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiments of the above methods. Among them, the hardware refers to a server or desktop computer, laptop computer, etc. containing one or more processors and storage media; the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM); the computer program is implemented by computer languages ​​including but not limited to C, C++, etc.

[0182] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a discrete index, characterized in that: include: According to the projection calculation logic, the problem domain data object is projected into the graph model to establish the graph model data object; The problem domain data object refers to the data object involved in the analysis subject described in the form of a data model, including entity objects and relationship objects; the projection calculation logic includes virtual object calculation logic and projection method logic, the virtual object calculation logic includes virtual entity object calculation logic and virtual relationship object calculation logic, the projection method logic includes instantiation projection method logic and abstraction projection method logic; the data model is a virtual data model aggregated from multiple data island local physical data models; According to the attribute association relationship of the vertex objects in the graph model data object, the calculation logic of the data island association relationship is established; Instantiate the graph model according to the projection calculation logic and the calculation logic of the data island association relationship; Perform decomposition calculation according to the instantiated graph model, and output and store discrete index data; The step of projecting the problem domain data object onto the graph model according to the projection calculation logic includes: According to the virtual entity object calculation logic, the entity object attribute set actually defined and saved in the data model is processed and calculated to obtain a virtual entity data object; the virtual entity object is a data object that does not exist in the data model but needs to be established in the graph model according to the problem analysis and has the same abstract semantics as the entity object; According to the virtual relationship object calculation logic, two entity object attribute sets actually defined and saved in the data model are processed and calculated to obtain a virtual relationship data object; According to the logic of the instantiation projection method, the virtual entity data object and the virtual relationship data object are used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in a 1:1 ratio; According to the logic of the abstract projection method, the virtual entity data object and the virtual relationship data object are used to establish a vertex and edge object set with the same attribute set or a partially identical attribute set on the graph model in an N:1 ratio; According to the logic of the instantiation projection method, the problem domain data objects are used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set in a 1:1 ratio on the graph model; According to the logic of the abstract projection method, the problem domain data objects are used to establish a set of vertices and edge objects with the same attribute set or partially the same attribute set in the graph model in an N:1 ratio; Where N is a natural number greater than 1; The data structure of the discrete index data includes at least four dimensions, namely, data object instance dimension, data island dimension, data object dimension and computing logic fragment dimension; the data island dimension includes independent data islands or logical data islands; the logical data island specifically refers to the physical data island set in the problem domain in the form of an abstract expression of a "logical" object.

2. The method for constructing a discrete index according to claim 1, wherein: The projection calculation logic also includes security calculation logic and synchronization calculation logic.

3. The method for constructing a discrete index according to claim 2, wherein: The attributes of the graph model data object include all or part of the attributes of the corresponding data object of the data model.

4. The method for constructing a discrete index according to claim 3, characterized in that: The attributes of the graph model data object also include custom attributes that need to be established in the graph model analysis.

5. The method for constructing a discrete index according to claim 1, wherein: The step of instantiating the graph model transmits data object instances through cross-domain, front-end system forwarding and / or encrypted transmission.

6. The method for constructing a discrete index according to claim 1, wherein: The step of performing decomposition calculation according to the instantiated graph model and outputting and storing discrete index data specifically includes: Establishing a decomposition calculation function between a discrete index and the instantiated graph model; Calculate and output discrete index data according to the decomposition calculation function; The discrete index data is persistently stored in the target storage medium according to a predefined traversal mechanism, and a read-write access interface is provided.

7. The method for constructing a discrete index according to claim 6, wherein: The mathematical expression of the decomposition calculation function is: in: It is discrete index data; To decompose the calculation function; It is an instantiated graph model, including the topological structure of the graph, the set of vertex and edge instances that constitute the topology, and the attributes of each element of the vertex and edge instance set; Update data for data islands during decomposition calculation runs.

8. The method for constructing a discrete index according to claim 7, characterized in that: The decomposition calculation function The existence form of converges to four intervals: under the constraints of static topology and synchronous data combination conditions, under the constraints of dynamic topology and asynchronous data combination conditions, under the constraints of dynamic topology and synchronous data combination conditions, and under the constraints of static topology and asynchronous data combination conditions.

9. The method for constructing a discrete index according to claim 8, characterized in that: The data structure of the discrete index data also includes a user dimension and / or a role dimension, which is used to control the user's access rights to the data object instance.

10. The method for constructing a discrete index according to claim 8, characterized in that: The target storage medium is a storage medium including a relational database, a Key-Value database, and a graph database.

11. The method for constructing a discrete index according to claim 8, wherein: The entity objects and relationship objects exist in a static form, a dynamic form, or a mixed form of static and dynamic forms; the data categories of the entity objects include structured data and unstructured data described in a structured form.

12. The method for constructing a discrete index according to claim 8, wherein: The step of instantiating the graph model is implemented through ETL, data cleaning or data synchronization methods, and is automatically started according to user-defined rules or manually started by the user.

13. The method for constructing a discrete index according to claim 8, wherein: After the step of persistently storing the discrete index data in the target storage medium according to the predefined traversal mechanism, the method further includes: traversing the discrete index data persistently stored in the target storage medium again according to one of the dimensions to establish an index.

14. A discrete index construction system, characterized in that: include: A projection establishment module is used to project the problem domain data object onto the graph model and establish the graph model data object according to the projection calculation logic; The problem domain data objects refer to the data objects involved in the analysis subject described in the form of a data model, including entity objects and relationship objects; the data model is a virtual data model aggregated from multiple local physical data models of data islands; A logic establishment module, used to establish a calculation logic of the data island association relationship according to the attribute association relationship of the vertex objects in the graph model data object established by the projection establishment module; An instantiation module, used to instantiate the graph model according to the projection calculation logic and the calculation logic established by the logic establishment module; A computing and storage module, used for performing decomposition calculations according to the graph model instantiated by the instantiation module, and outputting and storing discrete index data; The projection establishment module comprises: A virtual object computing logic unit, used to store virtual entity object computing logic and virtual relationship object computing logic; A projection calculation logic unit, used to store instantiated projection method logic and abstracted projection method logic; A virtual entity object projection unit, used to process and calculate the entity object attribute set actually defined and stored in the data model according to the virtual entity object calculation logic stored in the virtual object calculation logic unit to obtain a virtual entity data object; the virtual entity object is a data object that does not exist in the data model but needs to be established in the graph model according to the problem analysis and has the same abstract semantics as the entity object; A virtual relationship object projection unit, used to process and calculate the two entity object attribute sets actually defined and stored in the data model to obtain a virtual relationship data object according to the virtual relationship object calculation logic stored in the virtual object calculation logic unit; A first instantiation projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on a graph model in a 1:1 ratio for the virtual entity data object and the virtual relationship data object according to the instantiation projection method logic stored in the projection calculation logic unit; A first abstract projection unit is used to establish a vertex and edge object set with the same attribute set or a part of the same attribute set on the graph model in N:1 for the virtual entity data object and the virtual relationship data object according to the abstract projection method logic stored in the projection calculation logic unit; A second instantiation projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in a 1:1 ratio for the problem domain data object according to the instantiation projection method logic stored in the projection calculation logic unit; A second abstract projection unit is used to establish a set of vertex and edge objects with the same attribute set or partially the same attribute set on the graph model in N:1 for the problem domain data object according to the abstract projection method logic stored in the projection calculation logic unit; Where N is a natural number greater than 1; The data structure of the discrete index data includes at least four dimensions, namely, data object instance dimension, data island dimension, data object dimension and computing logic fragment dimension; the data island dimension includes independent data islands or logical data islands; the logical data island specifically refers to the physical data island set in the problem domain in the form of an abstract expression of a "logical" object.