Data grid system and method for data collaboration based on private message structure

Abstract

The present invention relates to a data grid system for realizing data collaboration based on a private message structure, which includes a control plane, a data plane, and data packets; the control plane, the data plane, and the data packets are connected in sequence; the system performs data transfer collaboration in a decentralized manner of the data grid. The present invention changes the concept that a large amount of data must be centralized in traditional centralized data collaboration for use, focuses on the domain attributes of data, and allows each domain to participate in data collaboration and transfer to achieve the goal of data joint governance and decentralization.

Description

Technical Field

[0001] The present invention relates to the field of data collaboration, and particularly to a data grid system and method for implementing data collaboration based on a private message structure. Background Art

[0002] With the rapid development of the mobile Internet, more and more data has become ubiquitous, and many enterprises regard data-driven business development as one of their strategic goals. As the current main data platform architecture mode of enterprises, the centralized mode also faces the following three problems that need to be solved urgently:

[0003] 1. Ubiquitous data and the spread of data sources.

[0004] 2. Using and coordinating all data under the control of one place and one platform.

[0005] 3. Management and control of local data permissions for a large amount of centralized data. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a data grid system and method for implementing data collaboration based on a private message structure, which changes the concept that traditional centralized data collaboration must centralize a large amount of data before it can be used, pays attention to the domain attributes of data, and realizes the goal of data joint governance and decentralization through the participation of each domain in data collaboration and circulation.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions:

[0008] A data grid system for implementing data collaboration based on a private message structure includes a control plane, a data plane, and data packets; the control plane, the data plane, and the data packets are connected in sequence; the system uses a decentralized data grid method for data transfer and collaboration.

[0009] Further, the control plane includes a data node management module, a node monitoring and management module, and a node control center;

[0010] The data node management module is mainly responsible for maintaining the basic information of the data plane nodes;

[0011] The node monitoring and management module is used to monitor the status of the data plane nodes and alarm and monitor abnormal nodes.

[0012] The node control center is responsible for basic data interaction with each data plane, access authentication, and data permission planning work.

[0013] Further, the data plane includes a data capture connector, a data processor, a data aggregator, and a data filler;

[0014] The data capture connector, as the entry of the data plane, automatically performs encoding / decoding and related packet legality verification when a data packet arrives.

[0015] The data processor is used to perform business processing on the data decoded by the data capture connector, and can obtain local source data from a locally customized data filler for data filling processing.

[0016] The data aggregator is used to packetize the business data processed by the data processor according to the data packet format, and deliver the data to the next data plane according to the routing table.

[0017] The data filler is a customizable component for each data plane, which realizes local data source data processing and imports it into the processor.

[0018] Furthermore, the data packet message structure is defined as follows:

[0019] Packet length Len: Records the length of the entire data packet. The data capture connector receives and determines whether the data is completely received based on the packet size, occupying the first four bytes of the data packet header.

[0020] Check field Check: Verifies the integrity of the data, with a length of 32 characters.

[0021] Data function number FunNo: Distinguishes the service attributes of the data packet to facilitate using the appropriate data processor for business processing.

[0022] Access control list AclList: Records the data permission list that needs to be authorized to access in the data field.

[0023] Data flag DF: The flag bit for data that needs to be authorized to access, so that when the data processor renders the data, it can determine whether to display the corresponding content based on this flag bit.

[0024] Access permission group AG: Records the set of data planes that can access this data identifier, and each data plane has a corresponding access permission identifier.

[0025] Routing table Route: Records the data packet transfer path, including the addresses and ports of the previous data plane and the next data plane. The corresponding key can be found through the previous data plane to complete the generation of the checksum of the check field; the data packet can be transferred to the next data plane after the data processing is completed through the address of the next data plane.

[0026] Data field Data: The actual data packet content, which consists of two parts: general data GeneralData and controlled data ControlledData.

[0027] GeneralData: non-controlled data, data accessible to all data planes;

[0028] ControlledData: Data that requires authorization to access.

[0029] Furthermore, the entire data content in the data domain Data is encrypted using a private encryption algorithm, and only the built-in data processor component can decrypt it, while other local programs cannot decrypt it.

[0030] A data collaboration method for a data grid system based on a private message structure to achieve data collaboration includes the following steps:

[0031] Step S1: The system administrator logs in to the control plane to manage the data nodes;

[0032] Step S2: Install the deployment data plane program on the server where the data plane needs to be deployed, and configure basic parameters such as the control plane IP and port;

[0033] In step S3, the data plane communicates with the control plane, completes necessary authorization and authentication, and downloads basic parameters. At this point, the entire data grid base is initialized and constructed.

[0034] Step S4: The data packet arrives at the data plane data capture connector, and the connector receives the data according to the "data packet length Len"; after the data is received, the "routing table Route" information is obtained, the corresponding key is determined according to the previous data plane address, and the data signature is calculated and compared with the check field Check in the data packet for signature verification.

[0035] Step S5: After the digital verification is passed, obtain the data function number FunNo, according to the function number, the received data content is forwarded to the corresponding data processor;

[0036] Step S6: The data processor obtains the permission control table AclList and the data domain Data, according to the permission list to obtain all data identifiers DF that have permission for this data surface;

[0037] Step S7: Get the general data GeneralData in the data domain, and obtain the restricted data content DC with permission to access the data identifier DF obtained in the previous step, and render the data through data integration and general data to obtain complete data;

[0038] Step S8: Get local data source data and integrated data packet data through custom data populator for business data collaboration processing, ultimately forming aggregated data;

[0039] Step S9: The data processor transfers the processed service data to the data aggregator. The aggregator updates the access control list AclList and the routing table Route according to the permission configuration requirements, data processing flow path, etc., calculates the verification field verification value, and finally completes the packet assembly of the data packet according to the data packet message structure;

[0040] Step S10: The data aggregator transfers the data packet assembled in Step S9 to the next data plane for data aggregation processing according to the routing table information.

[0041] Furthermore, the data node management includes maintaining the data plane name, ip, port, affiliated authorization group, and secret key.

[0042] Furthermore, in Step S8, the local data plane performs local permission division on the processed data according to the actual service requirements, performs data identification DF on the data that needs authorization, and stores the corresponding data in the controlled data domain of the data domain.

[0043] The present invention has the following beneficial effects compared with the prior art:

[0044] The present invention uses the data grid decentralized method for data flow collaboration. Through the data plane component, a large amount of data does not need to be centrally transmitted, and data aggregation processing is performed in place. Based on means such as custom access control, local permission control of data flow is realized, and a data ecosystem is realized with a unified standard specification. Data joint governance is realized based on the data grid to meet scenarios such as data aggregation analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the system structure of the present invention;

[0046] Figure 2 is a schematic diagram of the control plane structure in an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of the data plane structure in an embodiment of the present invention;

[0048] Figure 4 is a data packet message structure in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] The present invention will be further described below with reference to the drawings and embodiments.

[0050] Refer to Figure 1 , the present invention provides a data grid system for realizing data collaboration based on a private message structure, including a control plane, a data plane, and a data packet; the control plane, the data plane, and the data packet are connected in sequence; the system uses the data grid decentralized method for data flow collaboration.

[0051] In this embodiment, if Figure 2 As shown, the control plane includes a data node management module, a node monitoring management module, and a node control center;

[0052] The data node management module is mainly responsible for maintaining the basic information of data plane nodes;

[0053] The node monitoring and management module is used to monitor the status of data plane nodes and provide alarm monitoring for abnormal nodes.

[0054] The node control center is responsible for basic data interaction with each data plane, access authentication, and data permission planning.

[0055] In this embodiment, if Figure 3 As shown, the data plane includes a data capture connector, a data processor, a data consolidator, and a data filler;

[0056] The data capture connector, as the entry point to the data plane, automatically performs encoding and decoding and verifies the legitimacy of relevant data packets when a data packet arrives;

[0057] The data processor is used to perform business processing on the data decoded by the data capture connector, and can obtain local source data from the local customized data filler for data filling processing;

[0058] The data consolidator is used to package the business data processed by the data processor according to the data packet format and deliver the data to the next data plane according to the routing table;

[0059] Data populators are used for customizable components on each data plane to process data from local data sources and import them into the processor.

[0060] In this embodiment, the data packet message structure is defined as follows: Figure 3 As shown, the details are as follows:

[0061] 1. Data packet length Len: records the length of the entire data packet. The data capture connector receives and determines whether the data is received completely through the data packet size. It occupies four bytes of the data packet header.

[0062] 2. Check: Verifies data integrity. 32 characters in length. Check value sign = Sha256 (data packet content, previous data plane key). Note: The key mapping relationship can be found in the basic data through the data plane.

[0063] 3. Data function number FunNo: distinguishes the service attributes of the data packet so that the appropriate data processor can be used for service processing. FunNo encoding rule: FunNo = system number + module number + service function point number + 6-digit serial number. For example: YKT-FKMK-BK-000022

[0064] 4. Permission Control Table AclList: Records the data permission list that requires authorization to access in the data domain, mainly including Data Identifier DF and Access Permission Group AG.

[0065] Data Identifier DF: The identifier bit of the data that requires authorization to access, so that when the data processor renders the data, it can judge whether the corresponding content can be displayed according to this identifier bit. For example: CD_NF01

[0066] Access Permission Group AG: Records the set of data planes that can access this data identifier, and each data plane will have a corresponding access permission identifier.

[0067] 5. Routing Table Route: Records the packet transfer path, including the addresses and ports of the previous data plane and the next data plane. The corresponding key can be found through the previous data plane to complete the generation of the checksum in the verification field; the packet can be transferred to the next data plane after data processing through the address of the next data plane.

[0068] 6. Data Domain Data: The real packet content, mainly composed of General Data and Controlled Data. Note: The entire data content is encrypted using a private encryption algorithm, and only the built-in data processor component can decrypt it, and other local programs cannot decrypt it.

[0069] General Data: Uncontrolled data, data that can be accessed by all data planes. The data may contain the Data Identifier DF.

[0070] Controlled Data: Data that requires authorization to access. Note: It is composed of Data Identifier DF and Data Content DC.

[0071] In this embodiment, referring to Figure 4 , a data collaboration method for a data grid system that realizes data collaboration based on a private message structure is also provided, including the following steps:

[0072] Step S1: The system administrator logs in to the control plane to manage data nodes, mainly maintaining basic information such as data plane name, ip, port, affiliated authorization group, key, etc.;

[0073] Step S2: Install and deploy the data plane program on the server where the data plane needs to be deployed, and configure basic parameters such as the control plane IP and port; Preferably, the data plane can be deployed in different product lines and subsidiaries of the same company; it can also be deployed across regions or even across countries;

[0074] In step S3, the data plane communicates with the control plane, completes necessary authorization and authentication, and downloads basic parameters. At this point, the entire data grid base is initialized and constructed.

[0075] Step S4: The data packet arrives at the data plane data capture connector, which receives the data based on the "packet length Len". After receiving the data, the "routing table Route" information is obtained. The corresponding key is determined based on the previous data plane address (the mapping relationship comes from the basic parameters downloaded in step S3). The data signature is calculated using sign = Sha256 (packet content, secret key) and compared with the check field Check in the packet for verification.

[0076] Step S5: After the digital verification is passed, obtain the data function number FunNo, according to the function number, the received data content is forwarded to the corresponding data processor;

[0077] Step S6: The data processor obtains the permission control table AclList and the data domain Data, according to the permission list to obtain all data identifiers DF that have permission for this data surface;

[0078] Step S7: Get the general data GeneralData in the data domain, and obtain the restricted data content DC with permission to access the data identifier DF obtained in the previous step, and render the data through data integration and general data to obtain complete data;

[0079] Step S8: The local data source data is obtained through a custom data filler and then processed with the integrated data packet data for business data collaboration, ultimately forming aggregated data. Preferably, the local data plane can perform local authority division on the processed data according to actual business needs, assign a data identifier DF to the data requiring authorization, and store the corresponding data in the controlled data domain of the data domain.

[0080] Step S9: The data processor passes the processed business data to the data consolidator, which updates the permission control table AclList and the routing table Route according to the permission configuration requirements, data processing flow path, etc. and calculates the check field check value, and finally completes the packet group work according to the packet message structure;

[0081] Step S10: The data consolidator transmits the data packet assembled in step S9 to the next data plane for data aggregation processing according to the routing table information.

[0082] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0084] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0086] As described above, it is only the preferred embodiments of the present invention, and it is not a limitation to the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A data grid system for realizing data collaboration based on a private message structure, characterized in that: It includes a control plane, a data plane and data packets; the control plane, data plane and data packets are connected in sequence; the system adopts a data grid decentralized approach to data flow collaboration; The following steps are involved: Step S1: The system administrator logs in to the control plane to manage data nodes; Step S2: Install the data plane deployment program on the server where the data plane needs to be deployed, and configure the basic parameters of the control plane IP and port; Step S3: The data plane communicates with the control plane, completes necessary authorization and authentication, and downloads basic parameters. At this point, the entire data grid base is initialized and constructed. Step S4: The data packet arrives at the data plane data capture connector, which receives the data based on the "packet length Len"; after data reception is complete, the "routing table Route" information is obtained, the corresponding key is determined based on the previous data plane address, and the data signature is calculated and compared with the check field Check in the data packet for signature verification; Step S5: After the digital verification is passed, the data function number FunNo is obtained, and the received data content is forwarded to the corresponding data processor according to the function number; Step S6: The data processor obtains the authority control table AclList and the data field Data, and obtains all data identifiers DF that have authority on this data plane according to the authority list; Step S7: Obtain the general data GeneralData in the data domain, and obtain the restricted data content DC with permission through the data identifier DF obtained in the previous step, and render the data by integrating the data with the general data to obtain complete data; Step S8: Using a custom data filler, the local data source data is obtained and combined with the integrated data packet data for business data collaborative processing, ultimately forming aggregated data; Step S9: The data processor passes the processed business data to the data consolidator. The consolidator updates the permission control table AclList and the routing table Route according to the permission configuration requirements and the data processing flow path, calculates the checksum field check value, and finally completes the data packet assembly work according to the data packet message structure. Step S10: The data consolidator transmits the data packet assembled in step S9 to the next data plane for data aggregation processing according to the routing table information.

2. The data grid system for data collaboration implemented based on a private message structure according to claim 1, characterized in that, The control plane includes a data node management module, a node monitoring management module and a node control center; The data node management module is mainly responsible for maintaining the basic information of data plane nodes; The node monitoring and management module is used to monitor the status of data plane nodes and perform alarm monitoring on abnormal nodes; The node control center is responsible for basic data interaction with each data plane, access authentication, and data permission planning.

3. The data grid system for data collaboration implemented based on a private message structure according to claim 1, characterized in that, The data plane includes a data capture connector, a data processor, a data consolidator and a data filler; The data capture connector, as the entrance to the data plane, automatically performs encoding and decoding and verifies the legitimacy of relevant data packets when a data packet arrives; The data processor is used to perform business processing on the data decoded by the data capture connector, and can obtain local source data from a local custom data filler to perform data filling processing; The data aggregator is used to packetize the service data processed by the data processor in the format of data packets and deliver the data to the next data plane according to the routing table. The data filler is used for each data plane to customize components to implement local data source data processing and import it into the processor.

4. The data grid system for realizing data collaboration based on a private message structure according to claim 1, wherein The data packet message structure is defined as follows: Packet length Len: Records the length of the entire data packet. The data capture connector receives and determines whether the data is completely received based on the packet size, occupying the first four bytes of the data packet header. Check field Check: Verifies the integrity of the data, with a length of 32 characters. Data function number FunNo: Distinguishes the service attributes of the data packet for business processing using the data processor. Access control list AclList: Records the data permission list that requires authorization to access in the data field. Data flag DF: The flag bit for data that requires authorization to access, so that when the data processor renders the data, it can determine whether the corresponding content can be displayed based on this flag bit. Access permission group AG: Records the set of data planes that can access this data flag, and each data plane has a corresponding access permission identifier. Routing table Route: Records the data packet transfer path, including the addresses and ports of the previous data plane and the next data plane. The corresponding key can be found through the previous data plane to complete the generation of the checksum in the check field. The data packet can be transferred to the next data plane after data processing through the address of the next data plane. Data field Data: The actual data packet content, consisting of two parts: general data GeneralData and controlled data ControlledData. General data GeneralData: Uncontrolled data that can be accessed by all data planes. Controlled data ControlledData: Data that requires authorization to access.

5. The data grid system for realizing data collaboration based on the private message structure according to claim 1, characterized in that, The entire data content in the data field Data is encrypted using a private encryption algorithm, and only the built-in data processor component can decrypt it. Other local programs cannot decrypt it.

6. The data grid system for data collaboration implemented based on the private message structure according to claim 5, wherein, The data node management includes maintaining the data plane name, ip, port, affiliated authorization group, and key.

7. The data grid system for data collaboration implemented based on the private message structure according to claim 5, characterized in that, In step S8, the local data plane makes a partial permission division of the processed data according to the actual business requirements, marks the data that requires authorization with the data flag DF, and stores the corresponding data in the controlled data field of the data field.

Images