Big data processing methods and models for data asset value-added

Abstract

The present invention discloses a method for the governance and organization of big data, GriDoc2, which includes a fusion model, an organization model and an operation model of big data, and supports users to process big data into data products that meet personalized needs through the fusion model and the organization model, thereby realizing the added value of big data. The fusion model constructs a dynamic data fusion network based on multiple bipartite graphs by abstracting data operations into four basic operations: G, T, S, A, and F, and realizes the strategic collection, processing, integration, assembly, and fusion of data in a multiple bipartite graph model. The organization model supports users to weave the data components in the fusion network into their own data and knowledge views through the "table inclusion" model. The use of this method simultaneously realizes the modeled operation governance of big data and the data and knowledge expression based on multiple bipartite graphs, and makes the application program and data governance highly independent.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to the development of computer application systems based on big data. Background Art

[0002] Data is a crucial factor of production, fundamental to the development of all industries. Big data refers to electronic information generated by computer systems, originating from heterogeneous sources, independently distributed, and with distinct structures. The use and management of big data requires addressing the unique characteristics of big data compared to general data generated by humans or a single system.

[0003] Before the concept of big data was coined, the concept of data modeling existed only in the context of relational databases and data warehouses. Driven by computer applications, the amount of data accumulated has become increasingly rich and massive, and the "secondary use" of this data has become a major issue. This has driven the emergence of the concept of big data and made the organization of big data an unavoidable issue.

[0004] Although different scholars have different understandings of the connotation of big data governance, its goal is clear, that is, it is mainly aimed at the secondary utilization of big data.

[0005] One of the primary goals of "secondary use" of big data is to increase its value. This has created new demands on the big data field from both management and technical perspectives. These demands revolve around how to rationally and conveniently reuse big data. To achieve this, we must address the model for secondary use of big data. Within this model, secondary use management based on big data organizations is a key technology.

[0006] Big data organization refers to the establishment and description of application-oriented logical relationships or models of big data, aiming to facilitate management and use and achieve independence from physical storage.

[0007] However, current technologies focus only on the physical organization of big data and simple retrieval-oriented logical organization, typically using the Key-Value model and the column model. The Key-Value model represents all data as "keyword-value" pairs. "Keyword-value" pairs are generally stored in hash tables (distributed hash tables), and the management system does not specify the specific usage of "values" or provide interpretation. The column model is similar to the Key-Value model, except that all data is represented as "columns" marked by "keywords," which is equivalent to "keyword-value" pairs with "values" being columns. This organizational method is generally query-oriented, making it difficult to describe specific application logic and unsuitable for knowledge-oriented management of big data, so its use is greatly limited.

[0008] On the other hand, the secondary utilization of big data based solely on big data organization technology is far from enough. This is why the current problems of big data application development being difficult and the lack of standardization arise. Therefore, it is imperative to seek other complementary models.

[0009] Summary of Invention Purpose and Content

[0010] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a new big data secondary utilization model and method, called GriDoc2, which supports users to conduct modeled collection and governance of distributed heterogeneous data for secondary utilization, realize the modeled operation and sharing of big data and applications for knowledge management.

[0011] GriDoc2 primarily encompasses big data fusion models, organization models, and access (operational access) models, as well as their implementation and support mechanisms. The fusion model is used to process big data, the organization model is used to generate and derive data views with specific structures, and the access model supports access to data views. The support mechanism defines how to implement these models and generate data nodes that meet the requirements.

[0012] The fusion model is a state-preserving data transformation model that transforms data to create new meaning, thereby increasing its value. Because the intermediate results of data transformation are preserved, this transformation process forms a network whose nodes are the output of the data transformation, known as a data fusion network. Nodes in the fusion network can be original data blocks or other nodes formed through fusion. Any node can serve as the "raw material" for the organizational model, generating the data view required by the user.

[0013] 2. Digital Segment Fusion Network Structure

[0014] The Data Grid Fusion Network describes the logic of data collection, processing, transformation, assembly, and fusion. We call these actions performed on data "data operations." Each node in the network represents a result of a data operation, called a "segment." Specifically, internal nodes (non-data source nodes) in the Segment Fusion Network are called segments. Segments are data structures that can be accessed using third-party tools, such as Office, PDF, images, videos, record sets, relational tables, XML / HTML / JSONS formatted files, or data output by software services. Segments are formed by GTSAF operations and are datasets directly accessible to users. Each segment is accompanied by a segment descriptor that describes the segment's semantic structure.

[0015] Data fusion operations are implemented through a combination of basic abstract operations. These operations are categorized as Get (G), Transform (T), Select (S), Assemble (A), and Fusion (Fusing), collectively known as GTSAF operations. They all generate data segments. Data segments are accessed using a segment operation language and can be either materialized data or virtual views. These five operations enable data to be transformed and fused, adding semantic meaning to the data, or in other words, adding value to the data.

[0016] The target of the collection operation is the initial data source, and the generated result is a standardized data segment. The initial data source is a data source generated or maintained by a third-party system, including files, relational data from relational database servers, and other server-type data. Server-type data refers to data provided through a service model, such as web service data and data on hardware devices accessed through MQTT or MODBUS protocols.

[0017] Except for the G operation, which uses an external data source as the operand, all other GTSAF operations have data segments as their operands. For each operation, the parameters other than the operands (operation command description, operation strategy) are defined by the corresponding data structure, which is called the operation (e.g., acquisition, processing, selection, etc.).

[0018] GTSAF operations can be nested to implement composite operations. The graph structure composed of the nodes generated during the composite process is called a segment fusion network.

[0019] They are called abstract operations because they only specify the operation model, while the specific operation semantics are determined by the user through the operation description language corresponding to the operation.

[0020] Each operation of GTSAF is a function, which is defined as follows.

[0021] (A) Collection operation G

[0022] Operation prototype: G(desc,s1,s2,…,sn)

[0023] Function: Collect data from the data source (s1, s2, ..., sn) according to the collection description desc, form a number segment as the return value.

[0024] Note: There can be any finite number of data sources s1, s2, ..., sn. Each data source can be an OS file or server-type data (including server-type relational database data). The collection description desc defines the collection behavior, including the collection target, extraction mode, synchronization strategy, access control strategy, and return value description. Operation A acts as a data adapter.

[0025] (B) Processing operation T

[0026] Operation prototype: T(desc,d1)

[0027] Function: Operate the number segment d1 according to the processing strategy to form a number segment as the return value.

[0028] Note: Segment d1 can be the return value of any valid operation in GTSAF. The processing operation description desc defines the parameters required for the processing, the processing semantics, and the description of the return value.

[0029] (C) Select operation S

[0030] Operation prototype: S(desc,d1,d2,…,dn)

[0031] Function: Select a number segment from the number segments d1, d2, ..., dn according to the selection strategy as the return value.

[0032] Note: The segments d1, d2, ..., dn can be the return value of any valid operation in GTSAF. The option desc defines the parameters required for the string selection and the description of the return value.

[0033] (D) Assembly Operation A

[0034] Operation prototype: A(desc,d1,d2,…,dn)

[0035] Function: Assemble the number segments d1, d2, ..., dn according to the assembly strategy to form a number segment as the return value.

[0036] Note: Segments d1, d2, ..., dn can be the return values ​​of any valid operation in GTSAF. The assembly operation desc defines the assembly model. The assembly model is a segment assembly framework consisting of segment instances and segment stubs. A segment stub is a segment definer that defines the location and content requirements of a segment. The result of instantiating the assembly model is that each segment stub is replaced with a segment instance according to the segment definer, making the entire model a segment instance.

[0037] (E) Fusion Operation F

[0038] Operation prototype: F(desc,d1,d2,…,dn)

[0039] Function: Merge the number segments d1, d2, ..., dn according to the fusion strategy to form a number segment as the return value.

[0040] Note: The segments d1, d2, ..., dn can be the return values ​​of any valid operation in GTSAF. The fusion operation desc defines the fusion function and the required parameters.

[0041] The formal definition of the segment fusion network is:

[0042] A segment fusion network is defined as a quadruple

[0043] DFN=(DS,DP,OP,OR)

[0044] DS: Data source set, which records the network access addresses and protocols of accessible data sources;

[0045] DP: a set of segments, which are generated by the operations in the operation set OP;

[0046] OP: Operation set, consisting of specific GTSAF operations, each of which carries a corresponding operation statement;

[0047] OR: Operation relationship set, each element describes an OP to DP relationship, the element form is:

[0048] {<GTSAF_Name,GTSAF_Desc,p1,p2,…,pn>}

[0049] Here, GTSAF_Name∈{G,T,S,A,F}; GTSAF_Desc is the operation description of GTSAF_Name; p1,p2,…,pn∈DP,1≤n<∞, are the operands in GTSAF.

[0050] Attachment Figure 1 Figure 1 is a schematic diagram of a data segment fusion network. In this example, 21 GTSAF operations are applied to 11 data sources to generate 21 data segments.

[0051] 3. Organizational Model

[0052] The Data Segment Fusion Network describes the data processing logic. Based on the data segments formed by this processing logic, users can define their own data views using a data grid organization model, called a "data grid," which is equivalent to their own data model. Data organization achieves value-added through data re-weaving.

[0053] A data grid is a structure composed of data segments, allowing users to define and access it in a standardized way. The description method for defining the data grid structure is called the data grid definition language, and the code-level description method for accessing the data grid is called the access interface.

[0054] The structure of the grid is a multi-bipartite graph. Two relationships are defined between the nodes of the graph: order and inclusion.

[0055] The formal definition of a grid is an ordered list:

[0056] DG=(b1,b2,...,bn);

[0057] The table element bi is either a number segment, i = 1, 2, ..., n, or another ordered list:

[0058] bi=(bi1,bi2,...,bim)

[0059] At this time, bi is said to include bi1, bi2, ..., bim; here, n and m are both non-zero finite positive integers.

[0060] Each table element must belong to one and only one table, and each table element node can "contain" a table.

[0061] In practical applications, a DG can be described in XML format or JSON format.

[0062] For JSON description, the "table" uses the JSON object model, that is, "(b1,b2,...,bn)" is a JSON object, bi is K / V, for the segment element, V directly indicates the corresponding segment; for other elements, V is another JSON object.

[0063] 4. Number Grid Operation Model

[0064] (4.1) Overview

[0065] Data grid operation access refers to access to data grids created by users, including creation, modification, and access of data grids.

[0066] Grid operations use a function model. Each operation function, except for the grid creation function, takes integers as its first two parameters. The first parameter is the status code that serves as the function's return value, the second parameter indicates the number of subsequent grid parameters, and the third and subsequent n parameters are all grid parameters, where n is the value of the second parameter. The last parameter is a user-defined string in JSON format.

[0067] The return value of each function is a grid. At the code level, this return value is a handle to the grid, which can be used in the code. The returned grid handle is specified in the last parameter. Exceptions in the operation function are handled through the exception handling mechanism. The general model of the grid operation function is as follows:

[0068] DG-Oper(rs,n,dg1,dg2,…,dgn,js)

[0069] If the actual return value is a number segment, it is also encapsulated as a number grid and returned, that is, a table consisting of number segments is returned.

[0070] (3.2) Creation of number grid

[0071] Before using a grid, you must first create it. After the grid is created, subsequent operations reference it using the name defined when it was created. The prototype of the grid creation command is as follows:

[0072] Model and Prototype: CreateDG(rs,DG)

[0073] Function: Create a grid according to the grid definition DG and return the handle of the created grid.

[0074] Description: GD is a string that meets the schema structure defined in Section 3, for example, a JSON string. If creation succeeds, the corresponding handle is returned, and a value of 0 is returned to rs. Otherwise, NULL is returned, an integer indicating an exception code is returned to rs, and an exception is thrown.

[0075] (3.3) Reference to number grid

[0076] A grid reference is a handle or ID obtained for a specific grid, which is used for subsequent grid operations.

[0077] The basic grid reference pattern is:

[0078] dg(g,s)

[0079] Here, dg represents the handle of each grid, and g is a positive or negative number with the following meanings:

[0080] When g is a positive number, g and s represent the level number and the sequence number within the level, respectively, of the level order traversal with dg as the base (root). dg is level 0, and its child node is level 1.

[0081] When g is negative, the absolute value m represents the mth ancestor of dg, where the father is the first-generation ancestor. Ancestor refers to a direct ancestor. s refers to the sth son of the specified ancestor.

[0082] (3.4) Reading the grid elements

[0083] The operation of the data grid is to read the specific content of the data grid and store it in the buffer for further reading by the user. The general mode of the read operation is:

[0084] DG-Read(dg_handle)

[0085] Its function is to return a reference to the value of the grid pointed to by the grid handle dg_hanndle. dg_hanndle is usually obtained through the dg() operation.

[0086] (3.4) Serialization of grid elements

[0087] Serialization of grid elements refers to outputting a bipartite graph grid into a persistent sequence in some way. This operation is also called traversal. Traversal can be performed in a variety of ways, including depth-first traversal, breadth-first traversal, and level-by-level traversal.

[0088] Since the basic elements in the grid are number segments and the amount of data is generally large, the traversal result is divided into two parts: element ID sequence and element index storage. The element ID sequence is the sequence of number segment IDs, and the number segment ID sequence acts as an index, indicating the storage location of the specific number segment value.

[0089] The general form of a serialization operation is:

[0090] DG-Serialize(dg_handle,mode,ss)

[0091] Its function is to serialize the grid pointed to by dg_handle (the grid with dg_handle as the root) according to the traversal mode, forming a serialized segment ID named ss and the corresponding segment storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of the segment fusion and grid structure in an embodiment of the present invention, wherein the upper figure is a schematic diagram of the segment fusion network, and the lower figure is a schematic diagram of the grid structure. DETAILED DESCRIPTION

[0093] The content and implementation methods of the present invention are further described below in conjunction with embodiments and drawings, but the implementation methods of the present invention are not limited thereto.

[0094] The most important part of the implementation method is the implementation system and method of the big data model provided by the present invention, namely the supporting environment and tools of the model, mainly from the perspective of the composition of the supporting environment and tools (system) and the functions of the main components.

[0095] A GriDoc2 support environment and tools, including the following components:

[0096] ■Data segment fusion device: used to realize the model fusion of data segments, generate new data segments, and is responsible for the modeling and service-based access of data segments.

[0097] The Segment Fusion Engine is the executor of GTSAF, responsible for executing each operation within the GTSAF. It can be divided into the G Fusion, T Fusion, S Fusion, A Fusion, and F Fusion. In particular, because the G operation acts as a data collector and publisher, and is also responsible for data synchronization and buffering, G functions are generally separated into a separate Fusion (called an adapter) to improve performance.

[0098] ■GriDoc2 Server: Main functions include the following aspects:

[0099] Execution of segment fusion: Automatically generate the corresponding segments according to the segment fusion description given by the user;

[0100] Grid definition and storage: Implement a virtual view of the grid according to the organizational model given by the user (define the grid), and otherwise manage the cache.

[0101] Service processing: accepts and processes service requests from user applications, equivalent to a command interpreter.

[0102] Grid Driver: This is an API that supports user operations to access grids. It implements the grid access model, enabling users to create, read, and serialize grids within applications. The grid driver is based on the GriDoc server described above.

[0103] ■ Grid description language interpreter: supports the implementation of grid creation operations.

[0104] The embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for organizing and operating access to big data for computer application system development is characterized by: Based on the GriDoc2 model, it supports users' governance, value-added and access to big data at three levels through three models; The first model is a dynamic fusion model, which implements the dynamic collection, processing, and fusion of big data according to the GTSAF model, forming a dynamic fusion view, called a data segment fusion network, to achieve data value-added. The second model is an organization model, which weaves the data segments in the dynamic fusion view into a data grid according to the "set-inclusion" model. The third model is a data grid operation model, which supports applications accessing the data grid based on composite access methods. The dynamic fusion model includes five operations, namely acquisition operation G, processing operation T, selection operation S, assembly operation A and fusion operation F; acquisition operation G collects data from the data source according to the strategy, packages it into segments, and returns it as a return value; the segment is the basic data unit in the GriDoc2 model and is a data structure that can be accessed using third-party tools; the processing operation T operates on the segment d1 according to the processing strategy to form a segment as a return value; the selection operation S selects a segment from the specified multiple segments according to the selection strategy as a return value; the assembly operation A assembles the specified multiple segments into a segment according to the assembly operation statement as a return value; the assembly operation statement defines the assembly model; the assembly model is a segment assembly framework, which consists of segment instances and segment piles. The segment pile is a segment definer that defines the location and content requirements of the segment; the instantiation result of the assembly model is to replace each segment pile with a segment instance according to the segment definer, so that the entire model becomes a segment instance; the fusion operation F merges the specified multiple segments according to the fusion strategy to form a segment as a return value; The fusion strategy is a user-defined function, the input is the specified number segments, and the output is the number segments.

2. The method for organizing and accessing big data for computer application system development according to claim 1, wherein the organizational model, referred to as a data lattice model, is composed of segments in a segment fusion network according to a multipartite graph pattern, and is used to describe a user data model. The resulting model instance is referred to as a data lattice. The multipartite graph of the data lattice is a specific graph structure that describes the combinatorial logic between segments using order and inclusion relationships. The formal definition of the data lattice is an ordered table: DG=(b1,b2,...,b n ); in, Table element b i Or a number range, i = 1, 2, ..., n, or another ordered list: b i =(b i1 ,b i2 ,...,b im ) At this time, b i Contains b i1 ,b i2 ,...,b im ; Here, n and m are both non-zero finite positive integers; each table element must belong to one and only one table, and each table element node "contains" a table.

3. The method for organizing and accessing big data for computer application system development as claimed in claim 2, wherein the grid operation model comprises two aspects: the first aspect is creating a grid, and the second aspect is accessing the created grid through operations; grid operations adopt a function model, and the return value of each function is a grid; when operating on a grid, a handle to a specified grid must be obtained through a grid reference operation; the general model of a grid operation function is as follows: DG-Oper(rs,n,dg1,dg2,…,dgn,js) Here, except for the grid creation function, their first two parameters are integers. The first parameter is the status code as the function return value, the second parameter indicates the number of subsequent grid parameters, and the third parameter and the next n parameters are all grid parameters, where n is the value of the second parameter; the last parameter is a string in JSON format, which is customized by the user; the return value of the function is a grid; at the code level, this return value is a grid handle, which is used in the code; the returned grid handle is indicated in the last parameter; the exception of the operation function is handled by the exception capture mechanism; the specific grid operation functions include CreateDG(rs,DG), DG-Read(dg_handle), DG-Seriali ze(dg_handle,mode,ss); Here, CreateDG(rs,DG) is used to create a grid according to the grid definition DG and return the handle of the created grid; the function of DG-Read(dg_handle) is to return the value of the grid pointed to by the grid handle dg_hanndle; dg_hanndle is generally obtained through the grid reference operation dg(); the function of DG-Serialize(dg_handle,mode,ss) is to serialize the grid pointed to by dg_handle (the grid with dg_handle as the root) according to the traversal mode, forming a serialized segment ID named ss and the corresponding segment storage.

4. The method for organizing and accessing big data for computer application system development according to claim 3, wherein the basic mode of the data grid reference operation is: dg(g,s) Here, dg represents the handle of each grid, and g is a positive or negative number with the following meanings: When g is a positive number, g and s represent the level number and the sequence number within the level, respectively, of the level order traversal with dg as the base (root); dg is level 0, and its child node is level 1; When g is negative, the absolute value m represents the mth ancestor of dg, where the father is the first-generation ancestor, the ancestor is the direct ancestor, and s is the sth son of the designated ancestor.

Images