Real-time data network system and method for querying and processing distributed data

Abstract

The application discloses a real-time data network system and a method for querying and processing distributed data, and relates to the technical field of distributed data, and aims at solving the problem of real-time querying and processing of large-scale distributed real-time data with multiple structures. The system comprises one or more calling systems and one or more access execution systems, wherein each calling system is connected with one or more access execution systems, each access execution system is connected with one or more data storage systems, each calling system is used for providing an interface for accessing and processing distributed data to an external application program, receiving an instruction for accessing and processing distributed data from the external application program through the interface for accessing and processing distributed data, and accessing and processing distributed data stored in the data storage system through one or more access execution systems connected therewith.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed data processing, and in particular to a real-time data network system and a method for querying and processing large-scale distributed multi-structured real-time data (including unstructured data, semi-structured data and structured data). BACKGROUND

[0002] Information existing in an information society can be divided into structured data, semi-structured data and unstructured data. Structured data is organized data stored in a structured query language (SQL) database, having a certain logical structure and physical structure, and is generally implemented as data logically expressed in a two-dimensional table structure; semi-structured data is data having a certain degree of structure and stored in a non-relational (noSQL) database or a document (such as a spreadsheet). The semi-structured data structure varies greatly and cannot be completely analyzed and processed according to the processing method of unstructured or structured data; unstructured data has no predictable structure and is stored in any document without a specific structure, such as images, audio / video information, etc.

[0003] It is an urgent need in the information society to realize sharing and processing of multi-structured data (structured, semi-structured and / or unstructured data). Many organizations separated by geographical location or companies that need to cooperate require sharing of real-time data. For example, within a large enterprise group, the subsidiary needs to allow the headquarters to retrieve and review its financial data and other documents; a large bank needs to collect data from branch offices and synchronize the data to a central unit; a healthcare organization needs to share patient medical data and documents; business partners share certain data in a B2B manner in real time according to the contract requirements, etc. These information that needs to be shared not only includes structured data, but also more and more unstructured data and semi-structured data, and these multi-structured data need to be shared and processed in real time.

[0004] However, the current data sharing solutions cannot well solve the problem of sharing multi-structured real-time data, and in many cases, the needs cannot be met. The main difficulty lies in how to process data and documents stored using different technologies and different data structure formats, eliminate differences on a standard network platform, and realize seamless connection of data. Another challenge is the limitation of information technology (IT) resources. In an ideal case, if network bandwidth, storage and computing resources are unlimited, distributed real-time data can be collected when needed and processed at a central unit. However, in the real world, this is not the case. For example, when searching and extracting information from a large number of documents containing semi-structured and unstructured data, it is actually not feasible to collect all the documents to the central unit. Instead, the request needs to be processed at the edge where the data resides.

[0005] Data sharing is an important part of the management work between enterprises and within large enterprise groups, and therefore some scattered solutions have been designed for these institutions, which share data in semi-real-time or non-real-time manner, only to meet specific usage requirements. Users eventually have to customize and build multiple solutions to handle different usage scenarios. None of these solutions can solve the problem of real-time sharing and processing of large-scale multi-structured data. The following are several typical solutions currently used by large organizations:

[0006] Non-real-time electronic way: In some large enterprise groups, in order to assess risks and aggregate financial data, the large enterprise groups design forms and provide them to their subsidiaries to fill in regularly. The enterprise or subsidiary fills in the data into the form and submits it to the management authorities or the parent company regularly. The data submission frequency can be daily, weekly, monthly, quarterly or annually, and the enterprise can also submit the filled-in forms through a website or FTP website. In some cases, online data input can be performed using electronic forms.

[0007] This solution has many defects, one of which is the inability to access real-time data, and it is only suitable for the way data producers push data to data consumers for regular data submission, and does not support the usage scenario where data consumers autonomously obtain data from data producers. Another defect is that this solution causes enterprises to spend a lot of manual labor to prepare data and ensure successful data submission. In many cases, data must be manually entered and verified, and when data file packages arrive at the central unit, they must first be decompressed, and then the data must be cleaned and entered into the central database. Because data can only be submitted regularly, and due to network and centralized storage resource limitations, the amount of data is usually small. Therefore, the data collected in this way cannot be shared in real time, and is usually not suitable for complex analysis.

[0008] Real-time sharing of structured data through REST SQL: In B2B environments, enterprises may also reach agreements to allow one party to use the data of another party. This type of usage scenario requires data producers to allow data consumers to automatically obtain data through queries or downloads.

[0009] In the above usage scenarios, institutions and organizations that produce data usually provide some data through REST SQL application programming interfaces (APIs) to share with the public or business partners. In this way, the public and B2B partners can spontaneously issue data query requests as needed.

[0010] However, this solution has a major technical limitation: it generally works well only when there are very few data producers. For example, when there is only one data producer providing data to the public, or when two partners share data with each other.

[0011] If there are many data producers, sharing data through REST SQL API can be very difficult. This is because of the lack of a single standard REST SQL API, and all database providers have to create their own SQL access interface.

[0012] As Figure 1 shown in FIG. 1, an example of sharing structured and semi-structured data in a traditional way, for example, a data owner generally synchronizes a subset of its data into a PG or MySQL database, and then opens sharing of its replicated data using a REST SQL API. The drawbacks of this approach are obvious, first, the shared data is not real-time, and the open source software REST SQL API used by PG and MySQL has not been developed since 2018, and does not support orderby, groupby, etc. Commercial REST SQL API is completed independently by different database vendors, and does not need to follow a specific standard. In a multi-partner B2B scenario, unless all data producers use exactly the same commercial product, the collaboration network will stop working because the REST SQL data query APIs created by different vendors are not the same. This solution does not support sharing of semi-structured and unstructured data.

[0013] Other existing database tools also face similar problems when dealing with data sharing, for example, the APIs developed by data producers using Oracle are all non-standard, self-developed, and therefore cannot support unified queries for real-time multi-structured data.

[0014] Combination of structured data aggregation and real-time document access: Most traditional data sharing solutions are created for sharing structured data. These solutions cannot support sharing of multi-structured data. Multi-structured data includes structured, semi-structured, and unstructured text and media content, where the addresses of unstructured content files are stored in structured and semi-structured databases.

[0015] Figure 2This example illustrates a traditional approach to accessing distributed structured and unstructured data. In this solution, structured and semi-structured data are integrated into a central unit, but unstructured data files remain in distributed storage and are not integrated into the central unit. The reason for not integrating unstructured data files into the central unit is due to cost and technical considerations. Integrating a large number of files requires a significant investment of storage and network resources. Furthermore, since document links are stored as structured data in the database, once the files are relocated to the central unit, these links stored in the database must be updated in the new central unit. Updating the database is not only a massive task but could also corrupt it.

[0016] In this solution, structured and semi-structured data 0224-a,b,c are periodically integrated into a central database—fusion library 0232—using various methods. One data integration method involves each data generation node 0220-a,b,c preparing incremental data packets 0226-a,b,c according to a specific format. These incremental data packets are then submitted to the centralized server 0230 via pre-integrated structured data 0228. At the centralized server 0230, these data packets undergo collection, cleaning, and fusion processing before being finally integrated into the central database—fusion library 0232. Figure 2 As shown, application 0210 can call structured and semi-structured data 0212 from fusion library 0232, including structured information, document metadata, and links. Because the addresses of distributed documents are stored in the structured database, the addresses of unstructured data files can be extracted from fusion library 0232. As long as these files can be accessed through some file-sharing service 0222-a, b, c (such as WebDAV, HTTP, or FTP), application 0210 can retrieve and call unstructured files stored in the distributed file storage file system 0225-a, b, c, using the address of the unstructured data file as a guide.

[0017] A significant problem with this solution lies in its semi-real-time data sharing nature. Because data in structured and semi-structured databases is collected periodically, while unstructured data files are processed in real-time, structured and semi-structured data often become out of sync with unstructured data files. When files are relocated in the distributed data storage file systems 0225-a,b,c, their links in the central database—the fusion library 0232—are broken. Since unstructured data changes in real-time, the content and addresses of distributed documents also change. The distributed document addresses originally stored in the structured database, being collected periodically, cannot directly correspond to these real-time changes, making it impossible to access unstructured data using this incorrect document address information.

[0018] This solution also frequently places a heavy workload on IT staff for both data producers and central data consumers. When database integration fails, the process of rebuilding the central database can lead to prolonged data downtime. Summary of the Invention

[0019] The embodiments of this disclosure provide a real-time data network system and a method for querying and processing distributed data, relating to the field of distributed data technology, and can solve the problem of real-time querying and processing of large-scale distributed multi-structured real-time data.

[0020] The real-time data network system described in this disclosure may include:

[0021] One or more calling systems; and

[0022] One or more access execution systems; wherein,

[0023] Each calling system is connected to one or more access execution systems;

[0024] Each access execution system is connected to one or more data storage systems;

[0025] Each calling system provides an interface for external applications to access and process distributed data. Through this interface, it receives instructions from the external applications to access and process distributed data, and accesses and processes the distributed data stored in the data storage system through one or more access execution systems connected to it.

[0026] In some embodiments of this disclosure, each calling system is equipped with a first plug-in service program and a second plug-in service program; wherein, the calling system is configured to: receive a first application programming interface (API) instruction issued by the external application through the interface for accessing and processing distributed data; determine the execution location of the first API instruction; in response to determining that the execution location is an access execution system, start the first plug-in service program associated with the first API instruction, so that the calling system sends a second API instruction to one or more access execution systems associated with it, and accesses the distributed data stored in the data storage system through the one or more access execution systems; and in response to determining that the execution location is the calling system, start the second plug-in service program associated with the first API instruction, acquire data according to the data processing logic set by the second plug-in service program, process the acquired data, and feed back the processed data to the external application through the interface for accessing and processing distributed data.

[0027] In some embodiments of the present disclosure, each access execution system is installed with a third plug-in service program; wherein the access execution system receives a second application programming interface instruction issued by the calling system, starts the third plug-in service program related to the second application programming interface instruction, so that the access execution system reads the distributed data from the data storage system and processes the distributed data according to the data processing logic set by the third plug-in service program to obtain first response data.

[0028] In some embodiments of the present disclosure, each calling system is installed with a fourth plug-in service program; the calling system is further configured to receive the first response data from the access execution system, start the fourth plug-in service program related to the first application programming interface instruction, process the first response data to obtain second response data, and feed back the second response data to the external application program through the interface for accessing and processing distributed data.

[0029] In some embodiments of the present disclosure, the access execution system is configured with a data adapter; the access execution system matches the distributed data and reads the distributed data from the data storage system through the data adapter.

[0030] In some embodiments of the present disclosure, the access execution system is further installed with a data integration program; the access execution system converts the data model of the distributed data into the data model required by the third plug-in service program by starting the data integration program.

[0031] In some embodiments of the present disclosure, each access execution system is installed with a third plug-in service program and a fifth plug-in service program; wherein the access execution system receives a second application programming interface instruction issued by the calling system, starts the fifth plug-in service program related to the second application programming interface instruction; the fifth plug-in service program calls the third plug-in service program, so that the access execution system reads the distributed data from the data storage system and processes the distributed data according to the data processing logic set by the third plug-in service program to obtain intermediate data; the fifth plug-in service program processes the intermediate data according to the data processing logic set by the fifth plug-in service program to obtain first response data.

[0032] In some embodiments of the present disclosure, the system further comprises: an intermediate processing node and an intermediate storage system; wherein the intermediate processing node is connected to an invocation system and one or more access execution systems, for receiving the access and processing distributed data instructions from the invocation system, sending the access and processing distributed data instructions to the one or more access execution systems, receiving response data from the one or more access execution systems, and sending the response data to the invocation system after processing the response data; and the intermediate storage system is used to store the response data processed by the intermediate processing node.

[0033] In some embodiments of the present disclosure, the system further comprises:

[0034] an access execution system of an intermediate layer;

[0035] an invocation system of an intermediate layer; and

[0036] a data storage system of an intermediate layer; wherein,

[0037] the access execution system of the intermediate layer is connected to an invocation system, the invocation system of the intermediate layer and the data storage system of the intermediate layer;

[0038] the invocation system of the intermediate layer is connected to one or more access execution systems.

[0039] In some embodiments of the present disclosure, the system further comprises: a data network plug-in service management system for managing all plug-in service programs installed in the invocation system and the access execution system; wherein the data network plug-in service management system comprises: a plug-in service programming framework for providing a programming framework for developers to develop plug-in service program packages; and a program management module for managing and publishing the plug-in service program packages developed by the developers, so that the invocation system and the access execution system install the plug-in service program packages developed by the developers.

[0040] In some embodiments of the present disclosure, the access execution system comprises: a data integration programming framework for providing a programming framework for developers to develop data integration programs; and a data adapter configuration framework for providing an interface for developers to input an adapter name field, select and reference a data adapter microservice program, and input an address and access information field of a data storage system.

[0041] In some embodiments of the present disclosure, the system further comprises a data network management system for managing each system in the real-time data network; wherein the data network management system comprises a calling system and an access execution system account management module for creating or deleting calling systems and access execution systems; and a calling system and an access execution system network topology management module for adding or deleting calling systems and access execution systems in the real-time data network system, and managing the network connection relationship between the calling systems and the access execution systems.

[0042] In some embodiments of the present disclosure, the system further comprises a plug-in service programming framework for providing a programming template for developers to develop plug-in service packages; wherein the programming template allows the developers to define the application program interface (API) of each plug-in service program in the plug-in service package, define the execution location of each plug-in service program in the plug-in service package, and create the code of each plug-in service program executed in the calling system and / or the access execution system; wherein the API is a programming interface instruction used by the external application program.

[0043] Corresponding to the above real-time data network system, the embodiments of the present disclosure also disclose a method for querying and processing distributed data, applied to a real-time data network system, wherein the real-time data network system comprises one or more calling systems and one or more access execution systems; wherein the method for querying and processing distributed data comprises: each calling system respectively receives an instruction for accessing and processing distributed data from an external application program; and each calling system accesses and processes distributed data stored in the data storage system through one or more access execution systems connected thereto.

[0044] In some embodiments of the present disclosure, the receiving, by each calling system, an instruction from an external application to access and process distributed data comprises: the calling system receiving a first application programming interface instruction issued by the external application through an interface for accessing and processing distributed data; and the accessing and processing, by each calling system, distributed data stored by the data storage system through one or more access execution systems connected thereto comprises: the calling system determining an execution location of the first application programming interface instruction; in response to determining that the execution location is an access execution system, the calling system starting a first plug-in service program related to the first application programming interface instruction, sending a second application programming interface instruction to one or more access execution systems associated with the first plug-in service program, and accessing distributed data stored by the data storage system through the one or more access execution systems; and in response to determining that the execution location is a calling system, starting a second plug-in service program related to the first application programming interface instruction, obtaining data according to data processing logic set by the second plug-in service program, processing the obtained data, and feeding back the processed data to the external application through the interface for accessing and processing distributed data.

[0045] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, distributed data stored by the data storage system comprises: an access execution system receiving a second application programming interface instruction issued by the calling system; starting a third plug-in service program related to the second application programming interface instruction; reading the distributed data from the data storage system; and processing the distributed data according to data processing logic set by the third plug-in service program to obtain first response data.

[0046] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, distributed data stored by the data storage system further comprises: the calling system receiving the first response data from the access execution system, starting a fourth plug-in service program related to the first application programming interface instruction, processing the first response data according to data processing logic set by the fourth plug-in service program to obtain second response data, and feeding back the second response data to the external application through the interface for accessing and processing distributed data.

[0047] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, distributed data stored by the data storage system further comprises: the third plug-in service program of the one or more access execution systems converting a data model of the distributed data into a data model required by the third plug-in service program by starting the data integration program.

[0048] In some embodiments of the present disclosure, the accessing the distributed data stored in the data storage system by the one or more access execution systems comprises: receiving, by the access execution system, a second application programming interface instruction issued by the calling system; starting a fifth plug-in service program associated with the second application programming interface instruction; calling, by the fifth plug-in service program, a third plug-in service program; reading the distributed data from the data storage system; processing the distributed data according to data processing logic set by the third plug-in service program to obtain intermediate data; and processing the intermediate data according to data processing logic set by the fifth plug-in service program to obtain first response data.

[0049] In some embodiments of the present disclosure, the method further comprises: obtaining a calling system account and a topology position of the calling system in the real-time data network system; registering a calling system in the real-time data network management system according to the calling system account and the topology position of the calling system in the real-time data network system; receiving a first security token corresponding to the calling system from the real-time data network management system; determining whether the calling system has an associated access execution system according to the topology position of the calling system in the real-time data network system; and in response to determining that there is an associated access execution system, notifying all associated access execution systems that they are associated with the calling system and establishing a communication connection between the access execution system and the calling system.

[0050] In some embodiments of the present disclosure, the method further comprises: obtaining an access execution system account and a topology position of the access execution system in the data network system; registering an access execution system in the real-time data network management system according to the access execution system account and the topology position of the access execution system in the data network system; receiving a second security token corresponding to the access execution system from the real-time data network management system; and based on the topology position of the access execution system in the data network system, associating the access execution system to the calling system associated therewith and establishing a communication connection between the access execution system and the calling system.

[0051] In some embodiments of the present disclosure, establishing the communication connection between the access execution system and the calling system comprises: the access execution system providing the second security token to the calling system; the calling system providing a first access token to the access execution system; the access execution system providing a second access token to the upper calling system; wherein the first access token and the second access token are used for communication between the calling system and the access execution system.

[0052] In some embodiments of the present disclosure, the method further comprises: sending an installation request to the calling system to install the plug-in service package; the calling system installs all or part of the plug-in service programs in the plug-in service package into the pre-installed library; the calling system notifies all associated access execution systems to install the plug-in service package; and each access execution system installs all or part of the plug-in service programs in the plug-in service package into its own installation library; the each access execution system configures a data adapter for the plug-in service program to access distributed data stored in the data storage system connected to the access execution system, and associates the plug-in service program with the data adapter.

[0053] In some embodiments of the present disclosure, the method further comprises: each access execution system provides a programming framework of a data integration program, and obtains a data integration program from the programming framework of the data integration program; wherein the data integration program is used to convert a data model of the distributed data into a data model required by the third plug-in service program; and the third plug-in service program is associated with the data integration program.

[0054] In some embodiments of the present disclosure, the method further comprises: providing a programming template for a developer to develop a plug-in service package; wherein the programming template allows the developer to define an application program interface (API) of each plug-in service program in the plug-in service package, define an execution location of each plug-in service program in the plug-in service package, and write a code of each plug-in service program executed in the calling system and / or the access execution system; wherein the API is a programming interface instruction used by the external application program.

[0055] The technical scheme provided by the embodiments of the present disclosure realizes the unification of multi-structure data formats and the matching of regional data storage modes through the joint debugging of plug-in service programs, integration programs and data adapters, and realizes the purpose of real-time access and processing of large-scale distributed data through the cooperation of the calling system and the access execution system, and solves the problem that the existing data sharing solutions cannot well solve the problem of sharing multi-structure distributed real-time data. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical schemes in the present disclosure or related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0057] Figure 1Example of traditional structured and semi-structured data sharing approach;

[0058] Figure 2 Example of traditional approach to accessing distributed structured and unstructured data;

[0059] Figure 3 System components for one of the architecture implementations of a real-time data network;

[0060] Figure 4 First embodiment of system components layout for a real-time data network;

[0061] Figure 5 Second embodiment of system components layout for a real-time data network;

[0062] Figure 6 Third embodiment of system components layout for a real-time data network;

[0063] Figure 7 Fourth embodiment of system components layout for a real-time data network, containing two data sharing networks and one service management system;

[0064] Figure 8 Fifth embodiment of system components layout for a real-time data network, containing multiple data sharing networks with service management system;

[0065] Figure 9 Embodiment of system components layout for a real-time data network with multiple layers of data sharing networks;

[0066] Figure 10 One of the functional implementations of a real-time data network management system;

[0067] Figure 11 One of the functional implementations of a real-time data network plug-in service management system;

[0068] Figure 12 One of the functional implementations of a real-time data network invocation system and access execution system architecture;

[0069] Figure 13 One of the implementations of a real-time data network plug-in service programming framework;

[0070] Figure 14 Second implementation of a real-time data network plug-in service programming framework;

[0071] Figure 15 Scenario example for real-time invocation of distributed multi-structured data;

[0072] Figure 16 Scenario example for real-time processing of distributed multi-structured data;

[0073] Figure 17 Method embodiments for real-time data network system component communication;

[0074] Figure 18a and 18b Method embodiments for forming a real-time data network by adding a calling system and an access executing system, respectively;

[0075] Figure 19a and 19b Method embodiments for managing a real-time data network by deleting a calling system and an access executing system, respectively;

[0076] Figure 20a and 20b Method embodiments for installing a plug-in service program and integrating it into a real-time data network, respectively;

[0077] Figure 21a , 21b and 21c are method embodiments for executing a plug-in service program in a real-time data network, respectively;

[0078] Figure 22 is an example of a real-time data network applied to a large organization;

[0079] Figure 23 is an example of a network configuration when a real-time data network is applied to a large organization;

[0080] Figure 24 is an example of a real-time data calling network composition applied to a large organization;

[0081] Figure 25 is an example of performing data calling and analysis processing through a real-time data network;

[0082] Figure 26 is an example of a real-time data network applied to the field of the Internet of Things. DETAILED DESCRIPTION

[0083] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure is further described in detail below with reference to specific embodiments and with reference to the accompanying drawings.

[0084] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0085] Example 1

[0086] like Figure 3 As shown, this invention provides an embodiment of a system for collaboratively querying or processing distributed data. It should be noted that, in the embodiments of this disclosure, the aforementioned distributed data may refer to distributed data stored in one or more geographical locations.

[0087] In the above embodiments, the system can be referred to as real-time data network 0310. Specifically, the real-time data network 0310 can be composed of the following four system modules: data network management system 0311, invocation system 0313 (hereinafter referred to as GDDS), access execution system 0314 (hereinafter referred to as DSS), and data network plug-in service management system 0315.

[0088] The aforementioned data network plug-in service management system 0315 manages plug-in service programs, allowing developers to publish and install plug-in service programs on the data access and processing network 0312, which consists of the calling system 0313 and the access execution system 0314.

[0089] It should be noted that the above embodiments organize these functions into a system consisting of the four systems described above. In other embodiments of this disclosure, the above functions can also be reorganized into more or fewer than four systems. For example, a real-time data network 0310 can be constructed, consisting of three systems: a call-only system 0313, an access execution system 0314, and a data network plug-in service management system 0315.

[0090] The nodes distributed in various geographic locations can include data storage systems 0320 connected to the access execution systems 0314, which perform data access functions. The multi-structured data including structured data 0321, unstructured data (e.g., files) 0322, and semi-structured data 0323 are stored in the data storage systems 0320 distributed above. The access execution systems 0314 can be multiple in number and deployed in different geographic locations. The data adapters and / or data integration programs are configured in the access execution systems 0314 to implement real-time access functions of the distributed multi-structured data. The invocation system 0313 provides a unified interface for the external application 0330 to query and access the real-time data. The interface can be an interface for accessing and processing the distributed data. The invocation system 0313 is connected to the access execution systems 0314 and transmits the query and access data instructions from the external application 0330 to the access execution systems 0314 located in one or more locations to invoke and process the multi-structured data in the data storage systems 0320 connected to the access execution systems 0314 in real time. That is, the invocation system 0313 can receive the instructions for accessing and processing the distributed data from the external application 0330 through the interface for accessing and processing the distributed data and access and process the distributed data stored in the data storage systems 0320 through one or more access execution systems 0314 connected thereto.

[0091] The developers can use the plug-in service programming framework 0318 in the data network plug-in service management system 0315 to write plug-in service programs 0317-a, b, c and install them into the data access and processing network 0312 for real-time data access and processing. The data access and processing network 0312 is composed of the invocation system 0313 and the access execution system 0314. The plug-in service programs 0317-a, b, c can be installed into the invocation system 0313 and the access execution system 0314, or only into the invocation system 0313. The plug-in service programs 0317-a, b, c can also be installed partially into the invocation system 0313 and partially into the access execution system 0314. The service programs written by the developers are called plug-in service programs 0317-a, b, c, which provide APIs for the external application 0330, so that the external application 0330 can trigger the real-time execution functions of the installed plug-in service programs 0317-a, b, c to query, access, and process the real-time data.

[0092] The data network management system 0311 is responsible for managing all other modules or components in the real-time data network 0310.

[0093] The above Figure 3This is only one embodiment of the present disclosure. In addition, system components can be flexibly organized, integrated or divided to form different real-time data sharing and processing networks. Figures 4 to 9 Further embodiments are provided to illustrate how different network configurations can be implemented.

[0094] The technical solution provided by the embodiments of the present application can flexibly process data files of multiple structures by providing a data network plug-in module, realize real-time access and processing of distributed multi-structure data, and solve the problem that large-scale distributed multi-structure data cannot be accessed and processed in real time in the prior art.

[0095] Embodiment two

[0096] As shown in Figure 4 , the present application provides a first embodiment of the layout of a system for querying or processing distributed data in a collaborative manner. This embodiment demonstrates a layout of the real-time data network system described in Figure 3 . As shown in Figure 4 , the above real-time data network has three access execution system nodes. Of course, any number of access execution system nodes can be set according to actual conditions. Each access execution system 0314-a, b, c can be deployed in different geographical locations. At each access execution system node, each access execution system 0314-a, b, c is connected to a local data storage system 0320-a, b, c that stores multi-structure data. In this embodiment, the above real-time data network has two calling systems 0313-a, b. Of course, a single calling system can also be clustered, or more calling systems can be set. Each calling system can provide an API interface to an external application program 0420 to achieve the purpose of real-time calling and processing multi-structure data. The application program 0420 can call and remotely process distributed data connected to the access execution system 0314-a, b, c through the calling system 0313-a, b.

[0097] In the layout shown in this embodiment, the data network plug-in service management system 0315 is a separate system. Developers can download and use the plug-in service programming framework 0318 and program according to the framework to create plug-in service programs 0317-d. The plug-in service programs 0317-d can be uploaded to the data network plug-in service management system 0315. The plug-in service programs 0317-a, b, c uploaded in the data network plug-in service management system 0315 can publish and install complete plug-in service programs or subsets 0317-a, b, c of them to all calling systems 0313-a, b and access execution systems 0314-a, b, c in the data access and processing network 0312 through any calling system 0313-a, b. After installation, the application program can call real-time data.

[0098] The data access and processing network 0312 consists of calling systems 0313-a, b and access execution systems 0314-a, b, c. All real-time data network system components 0315, 0313-a, 0313-b, 0314-a, 0314-b, 0314-c are managed by the real-time data network management system 0311.

[0099] The technical solution provided by this invention provides developers with plugin program management and uploading through an independent data network plugin service management system. It allows plugin service programs to be published to some or all calling systems and access execution systems for installation, making access to and processing of real-time multi-structured data more flexible and elastic. Simultaneously, by establishing multiple independent calling systems, it solves the problem of different users needing to access different data separately.

[0100] Example 3

[0101] like Figure 5 As shown, this invention provides a second embodiment of the layout of a system for collaboratively querying or processing distributed data. This embodiment is similar to... Figure 4 The layout of the various components in the real-time data network system is very similar. In this embodiment, the aforementioned real-time data network has only one calling system. Furthermore, it is similar to... Figure 4 Unlike the calling system 0313 in this embodiment, the calling system 0510 in this embodiment embeds the functionality of the data network plug-in service management system 0315. Developers can use the plug-in service programming framework within the data network plug-in service management system 0315 to create plug-in service programs 0317-d, upload and install them to some or all of the calling systems 0510 and access execution systems 0314-a,b,c. After installation, the application 0520 sends calling and data processing instructions to the calling system 0510 to access the data storage systems 0320-a,b,c and process the data.

[0102] The technical solution provided by this invention simplifies the structure by embedding the functions of the data network plug-in service management system into the calling system.

[0103] Example 4

[0104] like Figure 6 As shown, this invention provides a third embodiment of the layout of a system for collaboratively querying or processing distributed data. This embodiment is similar to... Figure 5 The layouts of the various components in the real-time data network system are very similar. In this embodiment, the aforementioned real-time data network has only one calling system, and the functions of the data network plug-in service management system 0315 and the real-time data network management system 0311 are embedded in the calling system 0610.

[0105] The technical scheme provided by the embodiment of the present application integrates the functions of the data network plug-in service management system and the real-time data network management system into the calling system, so that the structure is simpler.

[0106] Embodiment five

[0107] As Figure 7 shown in FIG. 6, the present application provides a fourth embodiment of the layout of a system for querying or processing distributed data in a cooperative manner. This embodiment shows a layout embodiment of a real-time data network system composed of multiple data access and processing networks. In this embodiment, the real-time data network system contains two data access and processing networks 0312-a, b, one real-time data network management system 0311, and one plug-in service management system 0315. Of course, this embodiment is also suitable for the case of any multiple data access and processing networks. Figure 7 Data access and processing network 0312-a and data access and processing network 0312-b are two independent networks, each of which has its own independent calling system and access execution system. According to the selection of the target and embodiment, the same calling system and the same access execution system can be registered to both networks or can be limited to participate in only one network. In this embodiment, only one real-time data network management system 0311 manages the two data access and processing networks 0312-a, b. There is also a data network plug-in service management system 0315 to manage the plug-in service programs of the two networks.

[0108] Embodiment six

[0109] As Figure 8 shown in FIG. 6, the present application provides a fourth embodiment of the layout of a system for querying or processing distributed data in a cooperative manner. This embodiment shows a layout embodiment of a real-time data network system composed of multiple data access and processing networks. In this embodiment, the real-time data network system contains two data access and processing networks 0312-a, b, one real-time data network management system 0311, and one plug-in service management system 0315. Of course, this embodiment is also suitable for the case of any multiple data access and processing networks. Figure 7 The component layout in this embodiment is similar to that in

[0110] The above embodiments all refer to a two-layer data network, in which the access execution system 0314-a,b,c serves as the underlying connection area data storage, while the calling system 0313-a,b serves as the upper layer to provide data to the application in a unified manner.

[0111] While two-tier data networks address many data sharing and processing needs, multi-tier networks may be necessary in certain special circumstances. In data networks with two or more tiers, data storage is connected to the lower-level system, data is processed in the lower-level system, the processing results are collected in the middle-tier system, and the upper-level calling system can access and process data from the middle-tier system. Figure 9 The layout example in the document aims to illustrate that this goal can be achieved by constructing a multi-layered data network.

[0112] Example 7

[0113] like Figure 9 As shown, this invention provides a sixth embodiment of the layout of a system for collaboratively querying or processing distributed data. This embodiment illustrates a three-layer data access and processing network 0900. Regional data storage systems 0320-a, b, c, d, e are connected to the underlying access execution systems 0314-a, b, c, d, e, forming the underlying system. In this embodiment, there are two intermediate layers 0910 and 0920. One intermediate layer 0910 includes an intermediate processing node 0912 and an intermediate storage system 0320-i. The intermediate processing node 0912 is connected to the underlying access execution systems 0314-a, b. The other intermediate layer 0920 includes a back-to-back intermediate layer access execution system 0314-j, an intermediate layer calling system 0313-g, and an intermediate layer data storage system 0320-j. The calling system 0313-g is connected to the underlying access execution systems 0314-c, d, e, and the top layer includes a top-level calling system 0313-e. A real-time data network management system 0311 and a data network plug-in service management system 0315 provide network and plug-in service program management services for data access and processing network 0900.

[0114] When application 0950 issues an instruction to the top-level calling system 0313-e to execute a pre-installed plug-in program, the request can be sent to intermediate processing node 0912 and intermediate access execution system 0314-j. In one scenario, the intermediate-layer calling system 0313-g can collaborate with the lower-level access execution systems 0314-c, d, and e to pre-fetch data from regional data storage systems 0320-c, d, and e and store it in the intermediate-layer data storage system 0320-j. The intermediate-layer access execution system 0314-j can access this data. In another scenario, intermediate processing node 0912 can process requests from the top-level calling system 0313-e and then issue requests to all the lower-level access execution systems 0314-a and 0314-b connected to it. Data retrieved by the lower-level access execution systems 0314-a and 0314-b from regional data storage systems 0320-a and 0314-b can be further processed by intermediate processing node 0912. Intermediate processing node 0912 can also store data in the intermediate-layer data storage system 0320-i. Figure 14 This demonstrates a plug-in service programming framework that can be applied to this processing scenario. More details about the programming framework will follow. Figure 13 and Figure 14 Provided by China.

[0115] Although Figure 9 Only a three-layer data network example is given, but other layers can be added to the data access and processing network. For example, in another layout configuration, two data access and processing networks can be stacked to form a multi-layer network. It can also be similar to... Figure 9 As shown in the embodiment, the top-level calling system 0313-e is connected to the intermediate-layer access execution system 0314-j. The intermediate-layer access execution system 0314-j is then connected to the intermediate-layer calling system 0313-g, and the intermediate-layer calling system 0313-g is then connected to the bottom-level access execution systems 0314-c, d, and e. In this optional configuration, the top-level calling system 0313-e can connect to multiple intermediate-layer access execution systems, and each of these multiple intermediate-layer access execution systems can then connect to an intermediate-layer calling system, which in turn connects to multiple bottom-level access execution systems, and so on, thus forming a multi-layer real-time data access and processing network with any number of layers (three or more). Multiple plug-in service management systems 0315 can also be included in this multi-layer data network.

[0116] The technical solution provided by the embodiments of the present invention generates and stores intermediate processing results by establishing a multi-layered data access and processing network for use by users at relevant levels.

[0117] Example 8

[0118] like Figure 10As shown, the present application provides an embodiment of the functional modules of the real-time data network management system 0311. In this embodiment, the real-time data network management system 0311 is composed of four functional modules, namely the real-time data network management module 1010, the invocation system and access execution system account management module 1020 (hereinafter referred to as the GDDS and DSS account management module), the invocation system and access execution system network topology management module 1030 (hereinafter referred to as the GDDS and DSS network topology management module), and the plug-in service system management module 1040.

[0119] The real-time data network management module 1010 tracks and manages each data access and processing network 0312. For example, in Figure 4 , Figure 5 , Figure 6 and Figure 9 , there is only one data access and processing network 0312 or 0900. While in Figure 7 and Figure 8 , there are two data access and processing networks 0312-a, b, and the functions provided by the data network management module 1010 include, for example, creating and deleting network management resources (such as creating or deleting a data network) and maintaining data access and processing network 0312 deployment configurations. Each data access and processing network 0312 has a unique identifier (referred to as network id) and a network name.

[0120] The GDDS and DSS account management module 1020 in this embodiment provides functions including managing information for each invocation system and access execution system. Each invocation system 0313 and access execution system 0314 can create an account in the GDDS and DSS account management module 1020 and be managed by the GDDS and DSS account management module 1020. The account information of all invocation systems 0313 and access execution systems 0314 is saved in their resources. For example, Figure 10As shown, the account of a calling system 0313 and an access execution system 0314 can contain the following information: network id, used to indicate which data access and processing network 0312 the calling system 0313 or the access execution system 0314 is configured to; node id, used to generate a unique identifier for the calling system 0313 or the access execution system 0314; node type, used to indicate whether it is a calling system, an access execution system, or an intermediate processing node; node address, used to indicate the network address of the system; password, used to log in the GDDS and DSS account management module 1020; security token, used for communication security, etc. The system administrator can also create an account of a calling system 0313 or an access execution system 0314 (i.e. create a node) through the functions of the GDDS and DSS account management module 1020, and provide credentials for the node to join the network. When the calling system 0313 or the access execution system 0314 stops using, the account can delete the corresponding account (i.e. delete the node) through the functions of the GDDS and DSS account management module 1020. When configuring the calling system 0313 or the access execution system 0314, the system administrator inputs the provided credentials of the calling system 0313 or the access execution system 0314 to join the network into the calling system 0313 or the access execution system 0314, and the calling system 0313 or the access execution system 0314 performs registration to the GDDS and DSS account management module 1020. When the calling system 0313 or the access execution system 0314 needs to refresh its security token, it can obtain a new token through the functions of the GDDS and DSS account management module 1020. When a system attempts to join the network or refresh the token, the GDDS and DSS account management module 1020 uses its authentication function (verify node) to verify the credentials provided by the calling system 0313 or the access execution system 0314. The adding and deleting process of the calling system 0313 or the access execution system 0314 is shown in FIG. 18 and FIG. 19. It needs to be mentioned that although these figures only show the management of the calling system 0313 or the access execution system 0314, the system can increase the intermediate processing node 0912 by adjusting the embodiments.

[0121] The GDDS and DSS network topology management module 1030 in this embodiment manages the network link organization of the invocation system 0313 or access execution system 0314 and possible intermediate system 0912 in the data access and processing network 0312. In the resource, the GDDS and DSS network topology management module 1030 tracks the interrelation between the invocation system 0313 or access execution system 0314 (e.g. parent node id and child node id) to form the network hierarchy of the specific data access and processing network 0312 identified by the network id. The GDDS and DSS network topology management module 1030 provides the following functions: insert node, to specify the location of a new invocation system 0313 or access execution system 0314; delete node, to remove the invocation system 0313 or access execution system 0314 from the data access and processing network 0312; get topology function, to retrieve the topology or sub-topology of the data access and processing network 0312; get child node function, to get the child system of the invocation system 0313, which can be an access execution system or an intermediate layer node; get parent node function, to get the parent system of the specific access execution system 0314, which can be an invocation system 0313; and other functions.

[0122] The plug-in service system management module 1040 tracks and manages the plug-in service management system 0315. As shown in FIGS. 4 to Figure 9 The data access and processing network 0312 relies on the plug-in service management system 0315, and the developer can build plug-in service programs and install the programs into the invocation system 0313 or access execution system 0314. When there are multiple data access and processing networks 0312, these data access and processing networks 0312 can share one plug-in service management system 0315. Alternatively, each data access and processing network 0312 can have its own separate plug-in service management system 0315. The resource of the plug-in service system management module 1040 contains the information of the plug-in service management system 0315 in the network, including the network id, indicating the data access and processing network 0312 that the plug-in service management system 0315 is serving; the service id, being the unique identifier of the plug-in service management system 0315; the server address, being the network address of the plug-in service management system 0315; the server name, being the name of the plug-in service management system 0315; and the password, being the security password for the plug-in service management system 0315 to connect the real-time data network management system 0311. The plug-in service system management module 1040 also includes the following functions: adding or deleting the plug-in service management system 0315 (i.e. adding or deleting the service node); the plug-in service management system 0315 using the registration function to join the network; and other system components of the real-time data network can use the get service system address function to retrieve the network information of the plug-in service management system 0315. Figure 5 andFigure 6 As shown, if the retrieval plug-in service management system 0315 is embedded in the calling system 0313, then the plug-in service system management module 1040 does not need to manage it.

[0123] Figure 10 This is merely an example of real-time data network system management. This example uses a data network management system 0311 to manage the real-time data network system. Alternative embodiments may divide the management functions into multiple modules, or some functional modules may be combined into one module. For example, the GDDS and DSS account management module 1020 may be combined with the GDDS and DSS network topology management module 1030 into a subsystem. Another embodiment may integrate some or all of these functional modules into other system components of the real-time data network. For example, as... Figure 6 As shown, the functions of the data network management system 0311 are embedded in the calling system 0313. Thus, in Figure 6 In the given embodiments, it may not be necessary to invoke the system registration.

[0124] Example 9

[0125] like Figure 11 As shown, the present invention provides an embodiment of a data network plug-in service management system 0315 module. The system has three functional modules, namely a data network client module 1110, a plug-in service programming framework service module 1112, and a program management module 1116.

[0126] like Figure 10 As shown, the system administrator can create an account for the plug-in service management system 0315 from the plug-in service system management module 1040. In this embodiment, when the system administrator enters the created account into the newly deployed plug-in service management system 0315, the new system connects the real-time data network to the plug-in service system management module 1040 through registration 1121. The data network client module 1110 is responsible for registering, maintaining, and communicating with the plug-in service system management module 1040 1121.

[0127] The Plug-in Service Programming Framework Service Module 1112 manages the Plug-in Service Programming Framework 0318. Through this module 1112, system administrators can maintain the latest version of the programming framework, and developers can download the Plug-in Service Programming Framework 0318 to develop plug-in service programs 0317-a,b,c.

[0128] The program management module 1116 allows uploading and managing 1124 versions of the plug-in service programs 0317-a, b, c, and developers also rely on this module to manage program libraries. When a developer decides to install a program into the data access and processing network 0312, this module instructs the invocation system 0313 of the data access and processing network 0312 to distribute and install 1123 the plug-in service programs 0317-a, b, c into all invocation systems 0313 and access execution systems 0314-a, b, c in the network.

[0129] This embodiment is only one of the examples of how a real-time data network creates and manages plug-in service programs 0317-a, b, c, and alternative embodiments also include dividing or merging functional modules and integrating these modules into other system components of the real-time data network. For example, as shown in Figure 5 and Figure 6 The functions of the plug-in service management system 0315 are embedded into the invocation system 0313, as shown in Figure 6 In this embodiment, the data network client module 1110 does not need to register and connect to the real-time data network management system 0311, because the modules of the data network management system 0311 are also merged into the invocation system 0313-a.

[0130] Embodiment Ten

[0131] As shown in Figure 12 , the present application provides a detailed component layout embodiment of the system architecture for querying or processing distributed data in a collaborative manner. The implementation example of the invocation system 0313 and the access execution system 0314 of the real-time data network is shown. In the invocation system 0313 and the access execution system 0314, there are four main functions: the first is the formation of the data network; the second is the secure communication between the invocation system 0313 and the access execution system 0314-a, b, c; the third is the installation management of the service program; and the last is the execution management of the service program.

[0132] In this implementation example, the management and maintenance of the data network involve the invocation system network management module 1220, the DSS network management module 1270, and the real-time data network management system 0311. When a new invocation system 0313 is added to the real-time data network, an invocation system account is first created in the GDDS and DSS account management module 1020, and the new invocation system is located (i.e., inserted as a node) in the GDDS and DSS network topology management module 1030 of the real-time data network management system 0311, as shown in Figure 10 Then, the invocation system account information is provided to the invocation system 0313 through the invocation system network management module 1220. Subsequently, the invocation system network management module 1220 registers 1230 with the real-time data network management system 0311 and obtains a security token, as shown inFigure 10 As shown.

[0133] Accordingly, when adding a new access execution system 0314 to the real-time data network, an access execution system account must first be created in the GDDS and DSS account management module 1020, and the new access execution system must be located (i.e., inserted as a node) in the GDDS and DSS network topology management module 1030 of the real-time data network management system 0311, such as... Figure 10 As shown in Figures 18 and 19. Then, the access execution system account information is provided to the access execution system 0314 via the access execution system network management module 1270. Afterwards, the access execution system network management module 1270 registers 1257 in the real-time data network management system 0311. The data network formation process includes adding and deleting calling system 0313 and access execution system 0314 in the data network topology.

[0134] The secure communication channel 1208 between the calling system 0313 and the access execution systems 0314-a,b,c is handled by the security protocol service module 1222 of the calling system 0313 and the security protocol service module 1272 of the access execution systems 0314-a,b,c. For illustration, the service program installation communication 1207 and the service program execution communication are drawn outside the secure communication channel 1208. In practice, all communication is conducted through the secure communication channel 1208.

[0135] In this implementation example, the installation and management function module in the calling system 0313 is the plug-in service installation and management module 1210. Developers use the plug-in service programming framework 0318 to write plug-in service programs. Plug-in service programs 0317-a, b, and c can be uploaded to the plug-in service management system 0315. When the developer installs the program 1201, the plug-in service installation and management module 1210 in the calling system 0313 installs the plug-in service programs 0317-a, b, and c 1202 into the plug-in service program library 1212, forming plug-in service programs 1213-a, b, and c. It should be noted that, depending on the embodiment, plug-in service programs 0317-a,b,c and 1213-a,b,c can be the same program, or plug-in service programs 1213-a,b,c can be a subset of plug-in service programs 0317-a,b,c. For example, plug-in service programs 1213-a,b,c can only contain plug-in service programs 0317-a,b,c written in program attribute template 1310 and calling system program template 1320. Different labels are used in the figure to show that plug-in service programs 1213-a,b,c are installed in calling system 0313.

[0136] As part of the installation, the plugin service installation management module 1210 further sends the plugin service programs 0317-a,b,c along with the installation instructions 1207 to all the access execution systems 0314-a,b,c, communication 1207 being through the secure protocol service channel 1222, 1272. Upon receiving the installation instructions and programs, the plugin service installation management module 1260 in the access execution systems 0314-a,b,c installs 1256 the plugin service programs 0317-a,b,c to the plugin service program library 1281, forming plugin service programs 1282-a,b,c. It is noted that the plugin service programs 0317-a,b,c and the plugin service programs 1282-a,b,c can be the same programs, or the plugin service programs 1282-a,b,c can be a subset of the plugin service programs 0317-a,b,c, depending on the embodiment, for example, the plugin service programs 1213-a,b,c can only contain the plugin service programs 0317-a,b,c that are written in the program property template 1310 and the access execution system program template 1330. Different labels are used in the figure to show that the plugin service programs 1282-a,b,c are installed into the access execution systems 0314.

[0137] In order for the plugin service programs 1282-a,b,c to access the local data storage systems 0320-a,b,c, the local access execution systems 0314-a,b,c are to select appropriate data adapter microservices 1286-a,b,c, and configure the selected data adapter microservice programs 1286-a,b,c with the address and other connection information 1258 of the local data storage systems 0320-a,b,c using the data adapter configuration framework 1288, creating data adapters 1287-x,y,z to connect to the local data storage systems 0320-a,b,c. For example, if the local storage contains an Oracle database and an NFS file server, then an Oracle adapter microservice and an NFS client microservice are to be selected. Through configuration on the microservices, they are allowed to connect to the local NFS and Oracle servers. The association between the plugin service programs 1282-a,b,c and their data adapters 1287-x,y,z are created in the access execution system plugin program local access service module 1280.

[0138] Further, if the structured data in the local data storage system 0320-a,b,c does not have the same data model required by the plug-in service program 1282-a,b,c, the local access execution system 0314-a,b,c can add data format conversion logic through the access execution system integration programming service module 1283 to create a data integration program 1285-a,b,c using the data integration programming framework 1284. The association between the plug-in service program 1282-a,b,c and its data integration program 1285-a,b,c is managed and implemented by the access execution system plug-in program local access service module 1280. Not all plug-in service programs 1282-a,b,c need to be associated with a data adapter 1287-x,y,z or a data integration program 1285-a,b,c, if the data model of the structured data required by the plug-in service program 1282-a,b,c is the same as the data model in the local data storage system 0320-a,b,c, then a data integration program 1285-a,b,c is not needed. Further, the plug-in service program 1282-a,b,c can request data from other plug-in service programs 1282-a,b,c instead of from the local data storage system 0320-a,b,c, in which case a data adapter 1287-x,y,z is not needed. The process of installing and configuring a plug-in service program is shown in FIG. 20.

[0139] In this embodiment, the APIs 1312 of the installed plug-in service programs 1213-a, b, c, 1282-a, b, c are activated by invoking the system plug-in service API interface module 1216 and accessing the execution system plug-in service API interface module 1261, and the plug-in service programs 1213-a, b, c, 1282-a, b, c are ready for execution. The execution management of the plug-in service programs involves invoking the system plug-in service API interface module 1216, invoking the system plug-in service execution engine 1214, accessing the execution system plug-in service API interface module 1261, and accessing the execution system plug-in service execution engine 1262. When an external application 1299 issues an API instruction to a plug-in service program 1213-a, b, c by invoking the system plug-in service API interface module 1216, the interface module verifies the API. Once the verification is successful, the request is forwarded to the system plug-in service execution engine 1214, which is responsible for starting the plug-in service program 1213-a, b, c and executing according to the API instruction. Depending on the implementation of the plug-in service program 1213-a, b, c, the system plug-in service execution engine 1214 can simply execute the program 1213-a, b, c or it can make a request to all or selected access execution systems 0314-a, b, c to execute the relevant plug-in service program 1282-a, b, c, the request being transmitted through the secure communication channel between the security protocol service modules 1222, 1272 of the invocation system 0313 and the access execution systems 0314-a, b, c. For more detailed plug-in service programs 1213-a, b, c; 1282-a, b, c, please refer to Figure 13 and Figure 14 .

[0140] As shown in Figure 12 , in the access execution system 0314-a, b, c, upon receiving the access execution system request, the access execution system plug-in service API interface module 1261 verifies the request to ensure that the requested plug-in service program is installed in the plug-in service database 1281, the necessary integration has been completed 1285-a, b, c, and the necessary data adapter 1287-x, y, z has been configured. Once the verification is complete, the access execution system plug-in service execution engine 1262 starts 1255 the associated plug-in service program 1282-a, b, c to execute the request. The plug-in service program 1282-a, b, c executes its associated data integration program 1285-a, b, c if necessary, and uses its associated data adapter 1287-x, y, z to access local data if necessary.

[0141] Although Figure 12As shown, the present application provides an embodiment of a real-time data network programming framework. The programming framework includes a plug-in service programming framework 0318, a data integration programming framework 1284, and a data adapter configuration framework 1288. Figure 14 An embodiment of a plug-in service programming framework is shown that allows plug-in service programs to be created, installed, and executed in a network of more than two tiers.

[0142] Embodiment Eleven

[0143] As Figure 13 shown, the present application provides an embodiment of a real-time data network programming framework. The programming framework includes a plug-in service programming framework 0318, a data integration programming framework 1284, and a data adapter configuration framework 1288.

[0144] Referring to Figure 13 , a developer can download the plug-in service programming framework 0318 to create 1302 plug-in service programs 0317, 1213, 1282. Of these, plug-in service programs 0317-a, b, c are defined as programs uploaded to the plug-in service management module 0315 that have not yet been installed; plug-in service programs 1213-a, b, c are described as programs installed in the invocation system 0313; and plug-in service programs 1282-a, b, c are described as programs installed in the access execution system 0314.

[0145] Depending on the embodiment, these plug-in service programs 0317, 1213, 1282 can be the same programs, or the programs 1213 installed in the invocation system 0313 and the programs 1282 installed in the access execution system 0314 can be only a subset of the plug-in service programs 0317. For example, plug-in service programs 1213-a, b, c can contain only those plug-in service programs 0317 that are written in the program property template 1310 and the invocation system program template 1320 that are relevant to the invocation system 0313. Plug-in service programs 1282-a, b, c can contain only those plug-in service programs 0317 that are written in the program property template 1310 and the access execution system program template 1330 that are relevant to the access execution system 0314.

[0146] In this embodiment, the plug-in service programming framework is composed of three parts, the program property template 1310, the invocation system program template 1320, and the access execution system program template 1330.

[0147] In the invocation system program template 1320 section, there can be one or more function programming templates 1322, 1324, 1325, 1326, in which the developer can fill in the logic that he or she wants to be executed in the invocation system 0313. For example, the GDDS authentication template 1322 (GDDS_Auth) allows the developer to enter code that checks the credentials of the requesting application 1299 and its end user; the GDDS pre-processing template 1324 (GDDS_Pre_Process), the GDDS post-processing template 1325 (GDDS_Post_Process), and the GDSS execution template 1326 (GDSS_Execute) allow the developer to enter code that is executed in the invocation system 0313, and the behavior of the execution can be defined in the execution location 1314 attribute. The developer can also add more sub-programs to the invocation system program template 1320 section.

[0148] In the access execution system program template 1330 section, there can be one or more function programming templates 1332, in which the developer can fill in the logic that he or she wants to be executed in the access execution system 0314. In addition, the developer can also add more sub-programs to the access execution system program template 1330 section. In the access execution system program template 1330 section, there can also be query programming API interfaces of the access execution system integration program 1334 and the access execution system adapter 1336 provided by the access execution system plug-in program local access service module 1280. These API interfaces allow the plug-in service program 0317, 1213, 1282 to query 1308 the relevant data integration program 1285 and data adapter 1287, and allow the plug-in service program 1282 installed in the access execution system 0314 to connect to the local storage to extract data using the data adapter 1287 and perform data model conversion through the data integration program 1285 when necessary.

[0149] In the program attribute template 1310 section, the developer can define and fill in the program attributes for the plug-in service program 0317 being developed: such as the API name 1312, the execution location 1314, the data model 1316, and other attributes 1318. The developer can fill in the API name for the plug-in service program 0317, 1213, 1282 in the API name template 1312, and the fields in the API name 1312 are used as API instructions for the external application 1299 to issue instructions to the invocation system 0313 to trigger the execution of the relevant plug-in service program 1213, 1282. It must be declared that the API name of the plug-in service program 0317, 1213, 1282 can be the name of the API plug-in program service 0317, 1213, 1282 package, etc., and does not necessarily have to be filled in the framework program attribute template 1310.

[0150] In the program attribute template 1310 section, there can also be a definition template for the execution site 1314, which is filled in the plug-in service program 1213, 1282, indicating the calling system 0313 and the access execution system 0314 to execute the program written in the calling system program template 1320 and the access execution system program template 1330, for example, if the definition of the plug-in service program 1213, 1282 in the execution site 1314 is "GDDS", when the calling system 0313 receives the API instruction, the calling system plug-in service execution engine 1214 will start the plug-in service program 1213 to execute the program filled in the GDDS execution template 1326. If "DSS" is input in the execution site 1314, then the calling system plug-in service execution engine 1214 will start the plug-in service program 1213 to execute the program filled in the GDDS pre-processing template 1324, send a request to the access execution system 0314-a, b, c, the access execution system plug-in service execution engine 1262 starts the plug-in service program 1282 to execute the program filled in the DSS execution template 1332, forms a response, and sends the response to the calling system 0313. After receiving the access execution system response, the calling system plug-in service execution engine 1214 starts the plug-in service program 1213 to execute the program filled in the GDDS post-processing template 1325. It must be pointed out that the function programming templates 1322, 1324, 1325, 1326, 1332 in the calling system program template 1320 and the access execution system program template 1330 are only implementation examples, and the plug-in service program 0317, 1213, 1282 fills in the program in the function programming templates 1322, 1324, 1325, 1326, 1332 as needed, not all function programming templates 1322, 1324, 1325, 1326, 1332 must be filled in the program.

[0151] In the program attribute template 1310 section, the data model definition template 1316 is where developers input the data model definitions required by plug-in service programs 0317, 1213, and 1282. The data model definition contains the data structure format and some codes representing specific meanings; for example, the data model can define "1" as "male" and "2" as "female". If the data model definition differs from the locally stored data model, the local system administrator of the access execution system can use the data integration framework 1284 to create a data integration program 1285 to convert the local data's data model into the data model defined in the data model field 1316. Through the access execution system data integration programming service module 1283, the local system administrator inputs the data transformation logic into the user-defined function implementation (UDF_Imp) template 1342 section of the data integration framework 1284 to create an integration program 1285. The data integration program 1285 is also given a name, which is entered into the integration name template 1340 of the framework 1284. Using the Access Execution System Plugin Program Local Access Service Module 1280, the system administrator associates the data integration program 1285 with the plugin service programs 0317, 1213, and 1282. As shown in the Access Execution System Program 1330 section, the Access Execution System Plugin Program Local Access Service Module 1280 provides the Access Execution System Integration Program API 1334 for the plugin service programs 0317, 1213, and 1282, so that when the plugin service program 1282 executes in the Access Execution System 0314, it can query 1308 its related data integration program 1285. It must be noted that the above description is only one embodiment. For example, the data model does not necessarily have to be entered in the data model definition template 1316; the data model definition can be entered in the Access Execution System Program template 1330 section using a function programming template. In addition, the name of the data integration program 1285 does not necessarily need to be filled in the integration name 1340 template; for example, the name of the data integration program 1285 can be its software name.

[0152] Apart from Figure 13 In addition to the specific attributes 1312, 1314, and 1316 shown, there can be more attributes 1318. For example, as replacements for the fixed program tags 1322, 1324, 1325, 1326, and 1332 of the calling system program template 1320 and the access execution system program template 1330, the calling system and access execution system program names can be dynamically created by the developer and specified as additional attributes in attribute 1318. Figure 13As shown, if the plug-in service program 0317, 1213, 1282 is to access the data storage system 0320a, b, c attached to the access execution system 0314a, b, c, a local data adapter 1287 for the plug-in service program 0317, 1213, 1282 needs to be created, and the system administrator local to the access execution system can use the data adapter configuration framework 1288 to create 1306 the data adapter 1287.

[0153] Figure 13 The data adapter configuration framework 1288 in the embodiment shown allows the system administrator to enter the adapter name 1350 field for the adapter, select and reference the data adapter microservice program 1286-a, b, c, and enter the address and access information field for the data storage system in the configuration template 1356. When the data adapter microservice program 1286-a, b, c is selected, the address of the microservice is entered into the microservice program field 1352. The system administrator uses the access execution system plug-in program local access service module 1280 to associate the plug-in service program 0317, 1213, 1282 with the data adapter 1287. The access execution system plug-in program local access service module 1280 provides the plug-in service program 0317, 1213, 1282 with the access execution system adapter API 1336, so that the plug-in service program 1282, when executed in the access execution system 0314, can use the access execution system adapter 1336, API to query 1308 the relevant data adapter 1287 through the access execution system plug-in program local access service module 1280. The execution process of the plug-in service program is as shown in Figure 21a

[0154] It should be noted that Figure 13 The frameworks 0318, 1284 and 1288 shown in FIGS. 31, 1284 and 1288 are only one embodiment. In another embodiment, a more complex framework can be created to implement more complex execution control, for example, execution control attributes such as configuring execution mode, timing and scheduling of certain executions, etc. For example, the execution mode can specify that once an external application publishes an API of a plug-in service program, the access execution system set must first execute certain programs in their programs, then call the system set must execute certain programs, then another set of access execution systems or calling systems execute another set of programs, and so on.

[0155] ​In another example, the execution mode can be defined as follows: when the external application publishes the API of the plug-in service program, some programs in the invocation system and access execution system are executed first, then based on the dispatch configuration, another group of programs in the invocation system and access execution system are executed at a specific time, and so on. In a multi-layer network, the plug-in service programming framework permits the following execution control definition: some programs in the lowest layer access execution system are executed first; based on the execution result, the programs in the middle layer system are executed; and based on the result again, the programs in the upper layer network system are executed. In addition, in a different embodiment, all the program names in the invocation system program template 1320 and the access execution system program template 1330 can be dynamically defined by the developer.

[0156] In an embodiment of the present disclosure, the plug-in service program framework described above is essentially a number of programming templates used in the software development process, which can be downloaded by the developer to the local computer for programming. Specifically, in the process of using the access execution system execution template to write the service plug-in program, the developer can use the access execution system integration program API 1334, 1336 to access the data integration program and data adapter associated with the service plug-in program being written in the access execution system where the service plug-in program will be finally installed. Once the developer completes the programming and definition on each programming template, the code obtained by programming will be uploaded to the invocation system as a plug-in service program package. Next, the plug-in service program therein will be distributed and installed in all invocation systems and access execution systems.

[0157] In each access execution system, the local system personnel or developer can use the data integration program framework 1284 to write a data integration program 1285 for data conversion. In each access execution system, the local system administrator can associate the data integration program 1285 with the installed plug-in service program 1282. And through this association with the specific plug-in service program 1282, the access execution system data integration program API 1334 of the plug-in service program 1282 can access the associated data integration program 1285.

[0158] Furthermore, in each access execution system, local system personnel can create a data adapter 1287 by selecting a data adaptation microservice. Specifically, the data adapter can be configured first, such as configuring an IP address, identifier (ID), and password in local storage; then, the configured data adapter is associated with the already installed plug-in service program 1282. Through this association with a specific plug-in service program 1282, the access execution system data adapter API 1336 of the plug-in service program 1282 can access the associated data adapter 1287. For example, in embodiments of this disclosure, each access execution system can provide an Oracle database data adapter, an SAP data adapter, a NAS data adapter, a MongoDB database data adapter, and a MySQL data adapter, etc.

[0159] Example 12

[0160] like Figure 14 As shown, this invention provides an embodiment of a multi-layered plug-in service programming framework. In this embodiment, the program attribute template 1310, the calling system program template 1320, and the access execution system program template 1330 are partially integrated with... Figure 13 Similar to that in [the context]. In [the context] Figure 14 In this example, new intermediate system node programming templates 1420-a, b, and c are added between the system program calling template 1320 and the system program access execution template 1330. These intermediate system node programming templates can have one or more function programming templates, such as intermediate system node preprocessing template 1424 (Pre_Process), intermediate system node postprocessing template 1425 (Post_Process), and intermediate system node execution template 1426 (Execute). Developers can input code; in intermediate system node preprocessing template 1424-a, pre-execution commands are entered, and in intermediate system node postprocessing template 1425-a, intermediate execution results are entered. This is just another example demonstrating how the plug-in service programming framework can be adapted to execute in a multi-tiered data network.

[0161] Example 13

[0162] like Figure 15As shown, the present application provides an embodiment of a method for querying or processing distributed data in a coordinated fashion for real-time access to distributed multi-structured data. The real-time data network in this embodiment includes a calling system 0313 and three access execution systems 0314-x, y, z. At each location x, y, z there is a regional data store system 0320-x, y, z connected to its local access execution system 0314-x, y, z. A plug-in service program 0317-a is installed 1201 into the real-time data network, where the plug-in service program installed in the calling system 0313 is labeled 1213-a and the plug-in service programs installed in the access execution systems 0314-x, y, z are labeled 1282-a. In this embodiment, the plug-in service program 0317-a, 1213-a, 1282-a needs to access the regional data store systems 0320-x, y, z, but the data model required by the program does not match the data model in all of the regional data store systems 0320-x, y, z. Therefore, in each access execution system 0314-x, y, z there is a data integration program 1285-x, y, z and a data adapter 1287-x, y, z that are associated with the plug-in service program 1282-a by reference.

[0163] When the external application 1299 issues an API command for the plug-in service program 1213-a, the request is forwarded to the three access execution systems 0314-x, y, z assuming that the execution location 1314 of the plug-in service program 1213-a is "DSS". The plug-in service program 1282-a is executed in each access execution system 0314-x, y, z according to its implementation in the access execution system program template 1330. The complete access execution system program in the access execution system 0314-x location includes the installed plug-in service program 1282-a and the data integration program 1285-x, with data accessed through the data adapter 1287-x. The complete access execution program in the access execution systems 0314-y, z locations is similar to the program in the access execution system 0314-x. Once the requested data is retrieved by the access execution systems 0314-x, y, z, a response is generated and sent back to the calling system 0313. Depending on its implementation in the calling system program template 1320, the plug-in service program 1213-a can simply forward the response back to the external application 1299, or it can aggregate and process the responses from the access execution systems and send the final result back to the external application 1299. This figure illustrates real-time access to real-time distributed multi-structured data through a real-time data network.

[0164] Embodiment Fourteen

[0165] As Figure 16As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 16 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 15 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data.

[0166] Embodiment Fifteen

[0167] As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 17 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 10 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 17 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data.

[0168] As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data. Figure 17 As shown, the present application provides another embodiment of a method for querying or processing distributed data in a synergistic manner for real-time processing of distributed multi-structured data.

[0169] The data network administrator then provides the calling system account and its topology location to the calling system 0313, which uses this information to register 1712 in the real-time data network management system 0311. Once the calling system account information is verified, the real-time data network management system 0311 provides 1714 a security token (ST1) to the newly added calling system 0313.

[0170] Similarly, such as Figure 17 As shown, when a new access execution system 0314 is added, the data network administrator needs to first obtain an access execution system account through the real-time data network management system 0311, specifying the topological location of access execution system 0314 in the data access and processing network 0312. In the real-time data network management system 0311, the GDDS and DSS account management module 1020 is responsible for issuing and managing access execution system accounts, and the GDDS and DSS network topology management module 1030 is responsible for managing the locations of calling system 0313 and access execution system 0314 in the data network topology of data access and processing network 0312. The data network administrator then provides the access execution system account and its topological location to access execution system 0314, which uses this information to register in the real-time data network management system 0311. Once the access execution system account information is verified, the real-time data network management system 0311 provides the newly added access execution system 0314 with a security token (ST2). Then, based on the topology location provided by the GDDS and DSS network topology management modules 1030, the access execution system 0314 contacts and connects to the associated upper-layer calling system 0313.

[0171] Once the calling system 0313 and the access execution system 0314 are connected to the data access and processing network 0312 in the real-time data network, the two systems can begin their secure communication. For example... Figure 17 As shown, the access execution system 0314 can connect to its upper-layer calling system 0313 (1730) and provide a security token (ST2) to the calling system 0313. The calling system 0313 can verify the access execution system 0314 and its security token (ST2) with the real-time data network management system 0311. After successful verification, the calling system 0313 provides the access execution system 0314 with a unique access token (AT1) (1734), which the access execution system 0314 can use to communicate with the calling system 0313. The access execution system 0314 also provides the calling system 0313 with a unique access token (AT2) (1736) (1736) for communication between the calling system and the access execution system 0314.

[0172] Figure 17The communication process shown in FIG. 16 is an embodiment of many commonly used secure network communication protocols to illustrate how secure communication is conducted between system components in a real-time data network.

[0173] Embodiment 16

[0174] Figure 18a The embodiment of the method of constructing data access and processing network communication in the real-time data network by adding an access execution system 0314 in the existing embodiment is provided in the present disclosure. Figure 18b The embodiment of the method of constructing data access and processing network communication in the real-time data network by adding an access execution system 0314 in the existing embodiment is provided in the present disclosure. Figure 18a and Figure 18b Reference is made to Figure 17 , as it repeats some of the figures in Figure 17 .

[0175] In the embodiment of the present disclosure, when a new access execution system 0314-x is requested to be added, the following steps shown in FIG. 18 will be performed: Figure 18a

[0176] Step 1810: Add an account of the access execution system 0314-x in the GDDS and DSS account management module 1020 of the real-time network management system 0311.

[0177] Step 1812: Locate the access execution system 0314-x in the topology of the data access and processing network 0312 through the GDDS and DSS network topology management module 1030.

[0178] The above steps are also described in Figure 17 , and can also be referred to the description in Figure 17 .

[0179] Step 1814: Start the registration of the access execution system 0314-x in the real-time network management system 0311.

[0180] Specifically, the above steps will include: inputting the access execution system account information into the access execution system 0314-x, and starting the registration of the access execution system 0314-x in the GDDS and DSS account management module 1020 of the real-time network management system 0311.

[0181] After the above registration is completed, the access execution system 0314-x receives a security token (ST1) and the topology of the data network from the real-time network management system 0311.

[0182] ​Step 1816: The GDDS and DSS network topology management module 1030 checks the location of the calling system 0313-x in the network topology to determine whether the calling system 0313-x has an associated access execution system.

[0183] If system 0313-x has no associated access execution system, the process will terminate.

[0184] Step 1818: In response to the associated access execution systems 0314-i,j,k of calling system 0313-x, the GDDS and DSS network topology management module 1030 notifies all associated access execution systems 0314-i,j,k that they are associated with calling system 0313-x.

[0185] Step 1820: Each associated access execution system 0314-i,j,k connects with the calling system 0313-x.

[0186] In embodiments of this disclosure, each associated access execution system 0314-i,j,k first connects to the calling system 0313-x using its previously obtained security token (ST2) from the real-time network management system 0311. Then, the calling system 0313-x authenticates the access execution system 0314-i,j,k using the real-time network management system 0311. If authentication is successful, the calling system 0313-x and the access execution systems 0314-i,j,k exchange access tokens (AT1 and AT2).

[0187] In embodiments of this disclosure, when a request is made to add a new access execution system 0314-x, execution will be performed. Figure 18b The following steps are shown:

[0188] Step 1830: Add an account for access execution system 0314-x in the GDDS and DSS account management module 1020 of the real-time network management system 0311.

[0189] Step 1832: Locate the access execution system 0314-x in the topology of the data access and processing network 0312 using the GDDS and DSS network topology management module 1030.

[0190] Step 1834: Associate the access execution system 0314-x with one or more existing calling systems 0313-e,f.

[0191] The above steps are in Figure 17 It is also described in the text, which you can refer to. Figure 17 The description in the text.

[0192] Step 1836: Start the registration of the access execution system 0314-x with the real-time network management system 0311.

[0193] In particular, the above steps will include entering the access enforcement system account information into the access enforcement system 0314-x and initiating the registration of the access enforcement system 0314-x with the GDDS and DSS account management module 1020 of the real-time network management system 0311.

[0194] Upon completion of the above registration, the access enforcement system 0314-x can receive a security token (ST2) and the topology of the data network from the real-time network management system 0311. The network topology can include the associated invocation systems 0313-e,f from the real-time network management system 0311.

[0195] Step 1838: The access enforcement system 0314-x establishes a connection with each associated invocation system 0313-e,f.

[0196] In particular, the access enforcement system 0314-x can connect to each associated invocation system 0313-e,f with its security token (ST2). The invocation system 0313-e,f then authenticates the access enforcement system 0314-x with the real-time network management system 0311. Once authenticated, each invocation system 0313-e,f and the access enforcement system 0314-x system exchange access tokens (AT1 and AT2) with each other.

[0197] Embodiment Seventeen

[0198] Figure 19a An embodiment of a method for managing data access and processing network 0312 in a real-time data network by removing invocation systems in existing embodiments is shown, provided by the present disclosure; Figure 19b An embodiment of a method for managing data access and processing network 0312 in a real-time data network by removing invocation systems in existing embodiments is shown, provided by the present disclosure;

[0199] Wherein the removal of invocation systems 0313-x can refer to Figure 19a and mainly includes the following steps.

[0200] Step 1910: Initiating the removal of invocation systems 0313-x on the real-time data network management system 0311.

[0201] Step 1912: Checking in the GDDS and DSS network topology management module 1030 of the real-time data network management system 0311 whether the invocation system 0313-x has an associated access enforcement system.

[0202] If the calling system 0313-x has an associated access execution system, proceed to step 1916; if the calling system 0313-x has no associated access execution system, proceed to step 1914.

[0203] Step 1914: Remove calling system 0313-x from the real-time data network management system 0311.

[0204] The above steps may specifically include: the GDDS and DSS account management module 1020 deleting the account that calls system 0313-x; and the GDDS and DSS network topology management module 1030 deleting the account that calls system 0313-x from the network topology.

[0205] Step 1916: Real-time data network management system 0311 refuses to delete call system 0313-x.

[0206] For deleting access execution system 0314-x, please refer to [the relevant documentation]. Figure 19b The main steps include the following.

[0207] Step 1930: Start the deletion access execution system 0314-x in the real-time data network management system 0311.

[0208] Step 1932: The GDDS and DSS network topology management module 1030 in the real-time data network management system 0311 notifies the calling system 0313-e,f associated with the access execution system 0314-x that the access execution system 0314-x is being deleted.

[0209] Step 1934: Call system 0313-e,f to remove access execution system 0314-x from its topology.

[0210] Step 1936: Remove access execution system 0314-x from real-time data network management system 0311.

[0211] The above steps may specifically include: the GDDS and DSS account management module 1020 deleting the access execution system 0314-x account; and the GDDS and DSS network topology management module 1030 deleting the access execution system 0314-x from the data network topology.

[0212] Example 18

[0213] Figure 20a This illustration shows an embodiment of installing a plug-in service program in the current embodiment, as provided in this disclosure. Figure 20a This disclosure illustrates an embodiment of a method for integrating a plug-in service program into a real-time data network, as provided in the present embodiment. Figure 20a and Figure 20b Operations in the flowchartFigure 12 and Figure 13 are also described in Figure 20a and Figure 20b is a simple example among numerous embodiments, alternative embodiments can have different system components, different modules, and different protocols to achieve the same results.

[0214] In embodiments of the present disclosure, before installing the plug-in service program, the plug-in service program needs to be created and uploaded. Specifically, for the creation of the plug-in service program, the plug-in service programming framework 0318 can be downloaded through the plug-in programming framework service module 1112 in the data network plug-in service management system 0315; then, the plug-in service program 0317-a, b, c is created using the plug-in service programming framework 0318. Specific about how to create the plug-in service program 0317-a, b, c is described in Figure 13 , which can be referred to Figure 13 . For the upload of the plug-in service program, the plug-in service program 0317-a, b, c can be uploaded to the data network plug-in service management system 0315 through the program management module 1116. Among them, the above-mentioned program management module 1116 can also manage the version of the plug-in service program 0317-a, b, c.

[0215] After the plug-in service program is created and uploaded, the installation of the plug-in service program 0317-a, b, c can be started according to the steps shown in Figure 20a .

[0216] Step 2010: instruct the program management module 1116 of the data network plug-in service management system 0315 to install the plug-in service program 0317-a, b, c.

[0217] Step 2012: the program management module 1116 sends an installation request to the calling system 0313 to install the plug-in service program 0317-a, b, c.

[0218] Step 2014: after receiving the request, the calling system 0313 calls the plug-in service installation management module 1210 to install the plug-in service program 0317-a, b, c into the pre-installed program library 1212.

[0219] Step 2016: the plug-in service installation management module 1210 notifies all associated access execution systems 0314-a, b, c to install the plug-in service program 0317-a, b, c.

[0220] The plug-in services pre-installed on the calling system 0313 are described as 1213-a, b, c, the plug-in services 0317-a, b, c and the installed plug-in services 1213-a, b, c can be the same, or the plug-in services 1213-a, b, c can be a subset of the plug-in services 0317-a, b, c related to the calling system 0313.

[0221] Step 2018: Each access execution system 0314-a, b, c installs the plug-in service program 0317-a, b, c into its own installation library 1264 by the plug-in service installation management module 1260 of the system.

[0222] The plug-in service programs installed on the access execution system 0314-a, b, c are described as 1282-a, b, c. Among them, the plug-in service programs 0317-a, b, c and the installed plug-in service programs 1282-a, b, c can be the same, or the plug-in service programs 1282-a, b, c can be a subset of the plug-in service programs 0317-a, b, c related to the access execution system 0314.

[0223] After the plug-in service program is installed, the integration of the plug-in service program into the real-time data network can be started according to the steps shown in Figure 20b

[0224] Step 2020: The access execution system receives the data integration program created by the system administrator through the data integration programming framework provided by the access execution system.

[0225] In an embodiment of the present disclosure, at each access execution system, the local system administrator can use the data integration programming framework 1284 provided by the access execution system to create a data integration program 1285. Specifically, the local system administrator can input the data model conversion logic into the user-defined implementation field 1342 of the data integration programming framework 1284 to create the data integration program 1285. Among them, the above-mentioned data integration program 1285 can convert the local data model into the model required by the plug-in service program 1213-a, b, c.

[0226] Step 2022: The access execution system associates the data integration program created by the system administrator through the operation of the access execution system plug-in program local access service module with the plug-in service program.

[0227] ​That is, in embodiments of the present disclosure, after the data integration program 1285 is created, the local system administrator can further associate the data integration program 1285 with the plug-in service program 1282 through the access execution system plug-in program local access service module 1280. Once the above association is established, the access execution system data integration program API 1334 in the plug-in service program 1282 can access the data integration program 1285 of the access execution system to perform data model conversion.

[0228] In step 2024: the access execution system receives the data adapter created by the system administrator through the data adapter configuration framework provided by the access execution system.

[0229] In embodiments of the present disclosure, the local system administrator can also use the data adapter configuration framework 1288 provided by the access execution system to create a data adapter 1287 on each access execution system. Specifically, the local system administrator can provide a data adapter name 1350, select and reference the adapter microservice 1286-a, b, c to the microservice program field 1352, and provide a data storage address and access credentials to create the data adapter 1287.

[0230] In step 2026: the access execution system associates the created data adapter with the plug-in service program according to the operation of the system administrator through the access execution system plug-in program local access service module.

[0231] That is, in embodiments of the present disclosure, after the data adapter 1287 is created, the local system administrator can further associate the data adapter 1287 with the plug-in service program 1282 through the access execution system plug-in program local access service module 1280. Once the above association is established, the access execution system data adapter API 1336 in the plug-in service program 1282 can access the local data storage system through the data adapter 1287 of the access execution system.

[0232] The above process of integrating the plug-in service program plug-in service program 1282-a, b, c into the real-time data network will be described in detail below. Figure 12 and Figure 13 The above process of integrating the plug-in service program plug-in service program 1282-a, b, c into the real-time data network will be described in detail below. Figure 13As shown, when the required data model filled in by the plug-in service program 0317-a,b,c in its program or attribute 1316 is inconsistent with the data model in the local data storage system 0320, the access execution system needs to create a data integration program 1285-a,b,c to input the data model conversion logic into the user-defined function template 1342 of the data integration programming framework 1284. Through the local access service module 1280 of the access execution system plug-in program, in... Figure 12 and Figure 13 In the process, the system administrator accessing the execution system associates the plug-in service programs 1282-a,b,c with the data integration programs 1285-a,b,c. For example... Figure 12 As shown, the system administrator accessing the execution system also configures data adapters 1287-x,y,z for the plug-in service programs 1282-a,b,c to access local data. For example... Figure 13 As shown, the configuration includes providing a data adapter name 1350 using the data adapter configuration framework 1288, selecting and referencing the adapter microservice 1286 in the microservice application field template 1352, and providing the data storage address and access credentials in the storage system settings template 1356. Figure 12 As shown, by accessing the local access service module 1280 of the execution system plug-in program, the local system administrator associates the plug-in service programs 1282-a,b,c and the data adapters 1287-x,y,z.

[0233] Example 19

[0234] like Figure 21a As shown, the present invention provides an embodiment of the process of executing a plug-in service program in a real-time data network. Figure 21a The illustrated embodiment is a simple example among many embodiments, and an alternative embodiment may achieve the same result with different system components, different modules, and different protocols.

[0235] The flowchart in this embodiment can be referred to. Figure 12 Calling System 0313 and Access Execution System 0314 Module and Figure 13 Various programming frameworks. Figure 21a The flowchart shown uses Figure 15 and Figure 16 The example in the document illustrates the execution of the plugin service program.

[0236] Step 2110: An external application requests the execution of an installed plug-in service program.

[0237] Specifically, in the embodiments of this disclosure, an external application 1299 can request the execution of the installed plug-in service program 1213-b through API instructions of the published plug-in service program 1213-b.

[0238] Step 2120: Call the plugin service program that requests system startup.

[0239] Specifically, in the embodiments of this disclosure, upon receiving the aforementioned instruction, the system plug-in service interface 1216 is invoked to verify the API. After successful verification, the plug-in service execution engine 1214 is triggered to start the requested plug-in service program 1213-b. Specifically, the plug-in service execution engine 1214 first instructs the requested plug-in service program 1213-b to execute the program in its GDDS verification template 1322 to verify the requester.

[0240] Step 2130: Call the system to verify the application's identity.

[0241] Specifically, in the embodiments of this disclosure, the plug-in service execution engine 1214 can check the verification result of the program in the GDDS verification template 1322. If the authentication fails, the process ends; if the authentication succeeds, step 2124 continues.

[0242] Step 2140: Determine the execution location of the plug-in service program.

[0243] Specifically, in the embodiments of this disclosure, the plug-in service execution engine 1214 can check the execution location of the configured plug-in service program 1213-b. This execution location can be on the calling system 0313 or on the access execution system 0314.

[0244] Step 2150: Execute the plug-in service program at the determined execution location and obtain the execution result.

[0245] Specifically, in the embodiments of this disclosure, step 2150 may include:

[0246] In response to the determination that the plug-in service program 1213-b is configured to execute on the calling system 0313, the plug-in service execution engine 1214 executes the program in the GDDS execution template 1326 of the requested plug-in service program 1213-b and obtains the execution result.

[0247] In response to the determination that plug-in service program 1213-b is configured to execute on access execution system 0314, the following steps are performed:

[0248] A, the plugin service execution engine 1214 executes the program in the GDDS preprocessing template 1324 of the plugin service program 1213-b that requested the plugin service program.

[0249] B. The calling system sends an instruction to the access execution system 0314-x,y,z selected by the application to execute the plug-in service program 1282-b associated with the plug-in service program 1213-b.

[0250] Note that the plug-in service program 1213-b of the calling system and the plug-in service program 1282-b of the access execution system come from the same service program 0317-b.

[0251] C. Upon receiving the instruction, the access execution system plug-in service interface 1261 of each access execution system 0314-x,y,z triggers the plug-in service execution engine 1262 to start the requested plug-in service program 1282-b.

[0252] D. The plug-in service execution engine 1262 instructs the requested plug-in service program 1282-b to execute the program in its DSS execution template 1332.

[0253] In the above program, the plug-in service program 1282-b can request the plug-in service execution engine 1262 to execute another plug-in service program 1282-a. If the plug-in service program 1282-b or the plug-in service program 1282-a needs to call data in the local data storage system 0320, its program can use the query link 1334 of the access execution system integration program and the query link 1336 of the access execution system adapter to access the data integration program 1285 associated with it and its related data adapter 1287 through the access execution system plug-in program local access service module 1280, and access the access execution system regional data storage system 0320 using its related data adapter 1287. If there is a data integration program 1285, the plug-in service program 1282-a executes the data integration program 1285 to achieve data model conversion.

[0254] E. In each access execution system node 0314-x,y,z, the access execution system plug-in service program 1282-b generates and sends back the result data to the calling system plug-in service execution engine 1214 of the calling system 0313.

[0255] F. When the calling system 0313 receives the result from the access execution system 0314-x,y,z, the calling system plug-in service execution engine 1214 instructs the plug-in service program 1213-b to execute the program in its GDDS post-processing template 1325 and obtains the execution result.

[0256] Step 2160, return the execution result to the application 1299 that made the request.

[0257] So far, the above process ends.

[0258] Figure 21b A specific example of the process of executing a plug-in service program in a real-time data network is shown. As shown in Figure 21b The process of executing a plug-in service program can include the following steps:

[0259] In step 2110b, the external application 1299 sends an API instruction request by calling the system plug-in service API interface 1216.

[0260] In step 2120b, the plug-in service execution engine 1214 of the system call itself is called to load and execute the requested plug-in service program 1213-b.

[0261] In step 2130b, it is determined whether the code in the GDDS verification template 1322 is filled in, and if the code in the GDDS verification template 1322 is filled in, the code in the GDDS verification template 1322 is executed to verify the received API instruction request. If the execution result of the code is not verified, the process ends; if the code in the GDDS verification template 1322 is not filled in or the execution result of the code is verified, step 2131b is performed.

[0262] In step 2131b, the execution location of the plug-in service program 1213-b is determined, and if it is a system call, step 2132b is performed; if it is an access execution system, step 2133b is performed.

[0263] In step 2132b, in the system call, the code in the system call execution template 1326 is executed, and the result is returned to the external application 1299, and then the process ends.

[0264] In step 2133b, in the system call, if the code in the system call preprocessing template 1324 is filled in, the code in the system call preprocessing template 1324 is executed, and then a request is sent to the selected access execution system to execute the plug-in service program 1282b therein; otherwise, a request is directly sent to the selected access execution system to execute the plug-in service program 1282b therein.

[0265] It should be noted that the plug-in service program in the system call and the plug-in service program in the access execution system come from the same plug-in service program package. Among them, the plug-in service program 1213-b in the system call contains the code created by the developer through the system program template 1320 when developing the plug-in service program package 0317-b; and the plug-in service program 1282-b in the access execution system contains the code created by the developer through the access execution system program template 1330 when developing the plug-in service program package 0317-b.

[0266] In addition, as mentioned above, the calling system program template 1320 can include: a calling system verification template, a calling system preprocessing template, a calling system execution template, and a calling system post-processing template. Among them, the calling system verification template, the calling system preprocessing template, and the calling system post-processing template are optional templates for developers, that is, the developers can create corresponding codes in the calling system verification template, the calling system preprocessing template, and the calling system post-processing template, or can not fill in the codes in these templates.

[0267] In step 2134b, after receiving the above request through the plug-in service program API interface 1261, each access execution system respectively starts the plug-in service execution engine of itself to load the requested plug-in service program 1282-b, and executes the code in the access execution system execution template 1332.

[0268] In the embodiment of the present disclosure, the code in the above-mentioned access execution system execution template 1332 can request the execution of another plug-in service program 1282a.

[0269] In the embodiment of the present disclosure, the code in the above-mentioned access execution system execution template 1332 can request the access execution system plug-in program local access module 1280 to start the data integration program 1285 and the data adapter 1287 associated therewith. The code in the above-mentioned access execution system execution template 1332 extracts data from the local data storage system 0320 through the data adapter 1287, and can further execute the data integration program 1285 to complete the data model conversion.

[0270] In the embodiment of the present disclosure, the code in the above-mentioned access execution system execution template 1332 can generate response data, at which time the access execution system feeds back the response data to the calling system.

[0271] In step 2135b, in the calling system, if the code in the calling system post-processing template 1325 is filled in, the code in the calling system post-processing template 1325 is executed, and the finally integrated processing result is returned to the external application program 1299, and then the present flow ends; otherwise, the response data is directly returned to the external application program 1299, and then the present flow ends.

[0272] At this point, the above-mentioned flow ends.

[0273] Figure 21c A specific example of the process of executing a plug-in service program in a real-time data network is shown. In the present example, the calling system and the access execution system further include an intermediate processing node. As shown in FIG. 21c, the process of executing a plug-in service program can include:

[0274] In step 2110c, the external application 1299 sends an API instruction request by calling the system plug-in service API interface 1216.

[0275] In step 2120c, the plug-in service execution engine 1214 of the calling system itself is called to load and execute the requested plug-in service program 1213-b.

[0276] In step 2130c, it is confirmed whether the code is filled in the GDDS verification template 1322, and when the code is filled in the GDDS verification template 1322, the code in the GDDS verification template 1322 is executed to verify the received API instruction request. If the execution result of the code is not passed, the flow ends; if the code is not filled in the GDDS verification template 1322 or the execution result of the code is passed, step 2131c is implemented.

[0277] In step 2131c, the execution place of the plug-in service program 1213-b is determined, and if it is a calling system, step 2132c is executed; if it is an intermediate processing node or an access execution system, step 2133c is executed.

[0278] In step 2132c, in the calling system, the code in the calling system execution template 1326 is executed, and the result is returned to the external application 1299, and then the flow ends.

[0279] In step 2133c, in the calling system, if the code is filled in the calling system preprocessing template 1324, the code in the calling system preprocessing template 1324 is executed, and a request is sent to the intermediate processing node; otherwise, the request is directly sent to the intermediate processing node.

[0280] In step 2134c, the execution place of the plug-in service program 1213-b is determined, and if it is an intermediate processing node, step 2135c is executed; if it is an access execution system, step 2136c is executed.

[0281] In step 2135c, the intermediate processing node executes the code in the intermediate processing node execution template 1426, and returns the processing result to the calling system, and then step 2139c is executed.

[0282] In step 2136c, if the code is filled in the intermediate processing node preprocessing template 1424, the intermediate processing node executes the code in the intermediate processing node preprocessing template 1424, and sends a request to the selected access execution system. Otherwise, the intermediate processing node directly sends a request to the selected access execution system.

[0283] In step 2137c, after receiving the above request through the plug-in service program API interface 1261, each access execution system respectively starts its plug-in service execution engine to load the requested plug-in service program 1282-b, and executes the code in the access execution system execution template 1332.

[0284] In embodiments of the present disclosure, the code in the above access execution system execution template 1332 can request the execution of another plug-in service program 1282a.

[0285] In embodiments of the present disclosure, the code in the above access execution system execution template 1332 can request the access execution system plug-in program local access module 1280 to start the data integration program 1285 and the data adapter 1287 associated therewith. The code in the above access execution system execution template 1332 extracts data from the local data storage system 0320 through the data adapter 1287, and can further execute the data integration program 1285 to complete the data model conversion.

[0286] In embodiments of the present disclosure, the code in the above access execution system execution template 1332 can generate response data, at which time the access execution system feeds back the response data to the intermediate processing node.

[0287] In step 2138c, in the intermediate processing node, if the code in the intermediate processing node post-processing template 1425 is executed, the code in the intermediate processing node post-processing template 1425 is executed, and the aggregated processing result is returned to the calling system; otherwise, the response data is directly returned to the calling system.

[0288] In step 2139c, in the calling system, if the code in the calling system post-processing template 1325 is executed, the code in the calling system post-processing template 1325 is executed, and the final aggregated processing result is returned to the external application program 1299, and then the present flow is ended; otherwise, the response data is directly returned to the external application program 1299, and then the present flow is ended.

[0289] At this point, the above flow is ended.

[0290] Figure 20a 、 20b and Figure 21a 、 21b The flowcharts in 21a, 21b and 21c illustrate how to use the real-time data network programming framework (refer to Figure 13 and Figure 14 ) to build installable plug-in service programs, and how the modules of the calling system 0313 and the access execution system 0314 work together (refer to Figure 12 ) to collaboratively access (refer to Figure 15 ) and processFigure 16 ) is different or the organization of system modules of system 0313 and access execution system 0314 is different, the above-mentioned flow can also be different. In addition, the developer does not necessarily need to fill in program codes in all function templates of plug-in service framework (refer to Figure 13 ) such as GDDS verification template 1322, GDDS execution template 1326, GDDS preprocessing template 1324, GDDS post-processing template 1325 and DSS execution template 1332 when writing plug-in service program 0317. If these functions do not contain program codes, these functions do not need to be executed. Figure 13

[0291] The application of the real-time data network described above will be illustrated below in combination with two specific examples.

[0292] Example 1: Application of real-time data network in large-scale organization

[0293] Figure 22 An example of application of the real-time data network described in some embodiments of the present disclosure in a large-scale organization is shown.

[0294] It is assumed that the above-mentioned large-scale organization contains four levels of organizations, i.e. first-level organization 2210, second-level organization 2220, third-level organization 2230 and fourth-level organization 2240. Among them, first-level organization 2210 is the highest level organization and fourth-level organization 2240 is the lowest level organization. Figure 22 In FIG. 22, a first-level organization 2211, three second-level organizations 2221-a, b, c, six third-level organizations 2231-a, b, c, d, e, f and 12 fourth-level organizations 2241-a, b, …, k, 1 are shown exemplarily.

[0295] As shown in Figure 22 , in actual application, the upper-level organization should be able to obtain data and files from the lower-level organization. In addition, the first-level organization should also have the right to access the data and files of the organizations at all levels.

[0296] In Figure 22 , each organization itself has storage of structured data, semi-structured data and unstructured data 2212, 2222-a, b, c, 2232-a, …, f, 2242-a, …, 1. It should be noted that although the case that each organization manages its own data is described in FIG. 22, in actual application, in some areas, the data and files of the lower-level organization can be centrally managed by the upper-level organization to which the lower-level organization belongs. In addition, there can also be a case that the unstructured data is separately hosted and managed by a third-party supplier.

[0297] ​Figure 22 It is also shown that each of the agencies is deployed with an access enforcement system 0314. For example, the first level agency is deployed with an access enforcement system 2218; the second level agencies are deployed with access enforcement systems 2228-a, b, c; the third level agencies are deployed with access enforcement systems 2238-a, b, c, d, e, f; and the fourth level agencies are deployed with access enforcement systems 2248-a, b,..., k, 1. The data 2212, 2222-a, b, c, 2232-a, b,..., e, f, 2242-a, b,..., k, 1 stored in these agencies will be connected to the access enforcement systems 2218, 2228-a, b, c, 2238-a, b, c, d, e, f, 2248-a, b,..., k, 1 deployed at the local agencies, respectively. Data adapters (e.g., 1287-x, y, z as shown in FIG. 12) are also configured in these access enforcement systems 2218, 2228-a, b, c, 2238-a, b, c, d, e, f, 2248-a, b,..., k, 1 to ensure that the access enforcement systems can access the data stored locally. For example, the data 2212 stored in the first level agency can be connected to the access enforcement system 2218 through the data adapter deployed in the access enforcement system 2218. Figure 12 It is also shown that each of the agencies is deployed with an access enforcement system 0314. For example, the first level agency is deployed with an access enforcement system 2218; the second level agencies are deployed with access enforcement systems 2228-a, b, c; the third level agencies are deployed with access enforcement systems 2238-a, b, c, d, e, f; and the fourth level agencies are deployed with access enforcement systems 2248-a, b,..., k, 1. The data 2212, 2222-a, b, c, 2232-a, b,..., e, f, 2242-a, b,..., k, 1 stored in these agencies will be connected to the access enforcement systems 2218, 2228-a, b, c, 2238-a, b, c, d, e, f, 2248-a, b,..., k, 1 deployed at the local agencies, respectively. Data adapters (e.g., 1287-x, y, z as shown in FIG. 12) are also configured in these access enforcement systems 2218, 2228-a, b, c, 2238-a, b, c, d, e, f, 2248-a, b,..., k, 1 to ensure that the access enforcement systems can access the data stored locally. For example, the data 2212 stored in the first level agency can be connected to the access enforcement system 2218 through the data adapter deployed in the access enforcement system 2218.

[0298] In the above example, the first level agency 2211 can also be deployed with a call system 2216; the second level agencies 2221-a, b, c can also be deployed with call systems 2226-a, b, c; and the third level agencies 2231-a can also be deployed with a call system 2236-a. As described earlier, the call system allows external applications to query and process the data stored in each of the agencies through the access enforcement systems in a distributed manner. The data collected from the access enforcement systems can also be further processed in the call system before being passed to the external applications. In this example, some of the second level agencies, for example, the second level agencies 2231-b, c, d, e, f, can not be deployed with a call system. The staff of these agencies can access the data through the call system of their respective third level agencies after obtaining the appropriate security authorization.

[0299] Figure 23The example shows that the first-level institution 2211 deploys the centralized plug-in management system 2214 alone. Through the plug-in management system 2214 described above, the system administrator of the first-level institution 2211 can upload and publish plug-in service programs to the calling system and the access execution system of all institutions or selected institutions nationwide. In addition, in each second-level institution 2221-a, b, c and third-level institution 2231-a, the calling system 2226-a, b, c, 2236-a can be bound with its plug-in management system. The purpose of such deployment is to enable these second-level institutions 2221-a, b, c and third-level institutions 2231-a to create their own plug-in service programs, respectively, and publish and install them into their own calling system and the access execution system to which they belong.

[0300] In this example, the first-level institution 2211 is also deployed with a real-time data network management system 2213 for managing all calling systems and access execution systems deployed nationwide.

[0301] Figure 23 The network configuration of the calling system and the access execution system in the embodiment of the present disclosure is shown. For the convenience of description, Figure 23 The network configuration of the real-time data network of the entire large institution is shown only with several institutions as representatives.

[0302] In this example, according to the requirements of the first-level institution 2211, all access execution systems deployed nationwide have been registered and connected to the calling system 2216 of the first-level institution 2211, so that the staff can access the real-time data and files of the institutions at all levels as needed. In Figure 22 In the fourth-level institution, the access execution systems 2248-a, b, …, k, l deployed can be connected to the calling system 2216 described above; in the third-level institution, the access execution systems 2238-a, b, …, e, f deployed can be connected to the calling system 2216 described above; in the second-level institution, the access execution systems 2228-a, b, c deployed can be connected to the calling system 2216 described above; in the first-level institution, the access execution systems 2218 deployed can also be connected to the calling system 2216 described above. External applications, such as the work system 2310 of the first-level institution, are applications used by staff to access data and files of all institutions. The application 2310 can send an API request to the calling system 2216 described above to trigger the execution of the plug-in service program installed in the calling system 2216 and all access execution systems.

[0303] Staff at each level of organization typically need access to data and files within their own organization and those of the subordinate organizations they manage. Therefore, all access execution systems within themselves and their subordinate organizations need to register and connect to their own calling system. For example, access execution systems 2248-a,b,c,d deployed in the fourth-level organizations 2241-a,b,c,d, access execution systems 2238-a,b deployed in the third-level organizations 2231-a,b, and access execution system 2228-a deployed in the second-level organization should register and connect to the calling system 2226-a of their respective second-level organization 2221-a. Similarly, access execution systems 2248-e,f,g,h deployed in the fourth-level organization 2241-e,f,g,h, access execution systems 2238-c,d deployed in the third-level organization 2231-c,d, and access execution system 2228-b deployed in the second-level organization should also register and connect to the calling system 2226-b of their respective second-level organization 2221-b. Access execution systems 2248-i,j,k,l deployed in the fourth-level organization 2241-i,j,k,l, access execution systems 2238-e,f deployed in the third-level organization 2231-e,f, and access execution system 2228-c deployed in the second-level organization should register and connect to the calling system 2226-c of their respective second-level organization 2221-c.

[0304] exist Figure 23 and Figure 22 In the example shown, the third-level organization 2231-a also deploys a calling system 2236-a. At this time, the access execution system 2238-a deployed in the third-level organization 2231-a itself, and the access execution systems 2248-a,b of the fourth-level organizations 2241-a,b under its jurisdiction, should register and connect to the calling system 2236-a deployed in the third-level organization 2231-a.

[0305] The invocation system 2216 deployed at the first-level organization 2211 allows staff to use local applications, such as office system 2310, to read data and files through all access execution systems deployed across the network. The invocation system 2226-a,b,c deployed at the second-level organizations 2221-a,b,c allows staff to use local applications 2320-a,b,c to read data and files through all access execution systems deployed within their management scope.

[0306] Since the third level institution 2231-a deploys its own invocation system 2236-a, which allows its local application 2330-a to read its own data and files and the data and files of the fourth level institutions 2241-a,b within its administrative scope by accessing the execution system 2238-a, 2248-a,b. In this example, other third level institutions 2231-b,c,d,e,f do not deploy their own invocation system, then their applications 2330-b,c,d,e,f can read the data and files of the fourth level institutions 2241-c,d,e,f,g,h,i,j,k,l within their administrative scope through the invocation system 2226-a,b,c deployed by their superior second level institutions 2221-a,b,c. As shown in Figure 23 the third level institution 2231-b has the fourth level institution 2241-c and the fourth level institution 2241-d as its subordinate institutions; the third level institution 2231-c has the fourth level institution 2241-e and the fourth level institution 2241-f as its subordinate institutions; the third level institution 2231-d has the fourth level institution 2241-g and the fourth level institution 2241-h as its subordinate institutions; the third level institution 2231-e has the fourth level institution 2241-i and the fourth level institution 2241-j as its subordinate institutions; and the third level institution 2231-f has the fourth level institution 2241-k and the fourth level institution 2241-l as its subordinate institutions. The data of these fourth level institutions are connected to the invocation system 2226-a,b,c of the second level institution.

[0307] In Figure 24 the access execution system 2218 of the first level institution can only register and connect to one invocation system 2216, while other access execution systems can register and connect to multiple invocation systems. For example, the access execution systems 2248-a,b deployed in the fourth level institutions 2241-a,b can both register and connect to the invocation system 2216, the invocation system 2226-a, and the invocation system 2236-a.

[0308] Figure 24 Another example of deploying a real-time data network in a large institution is shown, which is described in another embodiment of the present disclosure. Figure 24 The case where the real-time data network management system 0311 manages five sub-networks 2400, 2410, 2412, 2414, 2420 is clearly described. Each sub-network is associated with the invocation system 2216, 2226-c, 2226-b, 2226-a, 2236-a, respectively. Figure 25It is also clear that the access enforcement systems 2248-a,b deployed at the fourth level 2241-a,b have registered and connected to the invocation system 2216 in the subnetwork 2400, the invocation system 2226-a in the subnetwork 2414, and the invocation system 2236-a in the subnetwork 2420.

[0309] Figure 12 A real-time data network 2400 of the first level organization and a list of plug-in service programs 2500 created by the first level organization 2210 are shown as an example. Among them, the plug-in service programs 2500 are marked as 0317-a,b,c in the previous figures. As shown in Figure 25 Once the plug-in service programs 0317-a,b,c are uploaded and published on the network, the programs installed in the invocation system 0313 will be marked as 1213-a,b,c; and the programs installed in the access enforcement system 0314 are marked as 1282-a,b,c.

[0310] Figure 25 An implementation example of three plug-in service programs is shown, namely the first plug-in service program 2510 (GetAllLiveCases), the second plug-in service program 2512 (ReadFile), and the third plug-in service program 2514 (ExtractAllLiveCaseDocData). The API names of these plug-in service programs are defined under their API names. As described earlier, the plug-in service programs 0317 are implemented by filling in the plug-in service programming framework 0138, which is essentially a programming template. The above Figure 13 The example of the plug-in service programs shown in Figure 25 is only a subset of the plug-in service programming framework 0138 shown in

[0311] As described earlier, the invocation system program 1320 part of the programming framework 0318 is a template for filling in the code that needs to be executed in the invocation system 0313. The access enforcement system program 1330 part is used to fill in the code that needs to be executed in the access enforcement system 0314. In Figure 25 , the program code executed by the invocation system 0313 in the above example implementation is in the invocation system program. The program code executed by the access enforcement system 0314 in the above example implementation is in the access enforcement system program. In order to simplify the description, Figure 12 The data integration program and data adapter of each access enforcement system are not shown.

[0312] The aforementioned first plug-in service program 2510 demonstrates that when the office system 2310 sends a GetAllLiveCases API request to the calling system 2216, the calling system 2216 will execute the code filled in by its developers in the GDDS preprocessing template 1324 of the aforementioned first plug-in service program 2510 to send the aforementioned GetAllLiveCases API request to the access execution system of the designated organization. Subsequently, these access execution systems will execute the code in the DSS execution template 1332 of the aforementioned first plug-in service program 2510.

[0313] As shown in the code of the DSS execution template 1332 of the first plug-in service program 2510 above, if there exists... Figure 26 The data integration program 1285 shown above modifies the database query statement using the data integration program 1285. This statement maps the data model of the local data storage to the data model required by the first plug-in service program 2510; otherwise, a standard database query statement is required. The code in the DSS execution template 1332 of the first plug-in service program 2510 uses the data adapter 1287 to query the local database using a standard or modified database query statement. The query result is sent back to the calling system 2216. Then, the calling system 2216 executes the code in the GDDS post-processing template 1325 of the first plug-in service program 2510, formats the query result, and feeds it back as a response to the office system application 2310.

[0314] To read a particular file, the office system 2310 can issue a ReadFile API request to the calling system 2216 with the URL of the file as a parameter of the ReadFile API request. Note that the office system 2310 can send other API requests to obtain the URL of the file before issuing the ReadFile API request. Further, the ReadFile API request can also include a data range of the file to be retrieved as a parameter. Upon receiving the ReadFile API request, the calling system 2216 will load the second plug-in service program 2512 and execute the code in the GDDS pre-processing template 1324 of the second plug-in service program 2512. The code in the GDDS pre-processing template 1324 of the second plug-in service program 2512 can use the URL in the ReadFile API request to identify the access enforcement system associated with the agency where the file is located and forward the ReadFile API request to the responding access enforcement system. Upon receiving the ReadFile API request, the access enforcement system will execute the code in the DSS execution template 1332 of the second plug-in service program 2512 to perform link validation, retrieval of the file content within the specified range, and send the retrieved data as a response to the calling system 2216. Finally, the calling system 2216 forwards the response to the application office system 2310.

[0315] As shown in the above examples, the staff of the first level agencies can access the latest real-time data from any agency through the office system 2310 and other applications. Most importantly, the above scheme does not require the collection of a large number of files and data from various agencies to a centralized location.

[0316] The implementation of the third plug-in service program 2514 demonstrates another important function of the present application. The first level agencies can need to extract meaningful data from the unstructured data in the documents for use. For example, the first level agencies want to create an index of the files to facilitate the staff to perform a search of the file content; analyze the quality of the file content; and analyze and identify the content of interest, etc. Since it is not practical to collect all the files of all the agencies to a central location to extract the information, a plug-in service program, such as the third plug-in service program 2514, can be created and uploaded to the calling system 2216. Then, it can be published and installed to all the access enforcement systems.

[0317] In this way, data can be extracted from documents in a distributed manner at the network edge using the distributed computing resources in the access execution system. Upon receiving an ExtractAllLiveCaseDocData API request, the calling system 2216 will load the third plug-in service program 2514 and execute the code in the GDDS pre-processing template 1324 of the third plug-in service program 2514 to send the ExtractAllLiveCaseDocData API request to all or selected access execution systems of the institutions. Upon receiving the ExtractAllLiveCaseDocData API request, the access execution systems execute the code in the DSS execution template 1332 of the third plug-in service program 2514. The above implementation can make direct requests to other plug-in service programs to extract document information and document content, then convert the content to text using OCR logic, and extract data according to application requirements. The extracted data can then be fed back as a response to the calling system 2216 and ultimately forwarded to the application 2310 that sent the request.

[0318] Through the above first plug-in service program 2510, second plug-in service program 2512 and third plug-in service program 2514, distributed data retrieval and data processing logic can be very easily implemented using the service plug-in programming framework 0318. Moreover, these distributed data retrieval and data processing logic can be installed on all systems in the real-time data network as needed.

[0319] The above application scenarios show that using the real-time data network can meet the requirements of large institutions to automatically access and process distributed data without the need to concentrate a large amount of system resources in a central location.

[0320] Example 2: Application of Real-Time Data Network in Internet of Things

[0321] Figure 26 An example of an Internet of Things application implemented using the real-time data network as described in some embodiments of the present disclosure is shown. In this example, a four-tier network structure consisting of a calling system 0313, intermediate processing nodes 0912, access execution systems 0314 and devices / sensors is applied. The above four-tier network includes a device layer 2640, an edge computing layer 2630, a regional computing layer 2620 and a central computing layer 2610.

[0322] The device layer 2640 is located at the network edge, where devices and sensors a…p for detecting microclimate are deployed.

[0323] The edge computing layer 2630 consists of Internet of Things gateways 2632-a,…,g for computing data streams received from devices and sensors in the device layer 2640. Access execution systems 2634-a,…,g will be embedded in Internet of Things gateways 2632-a,…,g respectively. As shown inFigure 26 As shown, sensor a and sensor b are connected to the IoT gateway 2632-a; sensor c and sensor d are connected to the IoT gateway 2632-b; sensor e, f, g, h are connected to the IoT gateway 2632-c; sensor i and sensor j are connected to the IoT gateway 2632-d; sensor k, l, m are connected to the IoT gateway 2632-e; sensor n and sensor o are connected to the IoT gateway 2632-f; and finally, sensor p is connected to the IoT gateway 2632-g. When the IoT gateways 2632-a,...,g process, analyze and generate meaningful data, the processed data will be forwarded to the regional computing layer 2620.

[0324] The regional computing layer 2620 is composed of IoT servers 2622-a,b,c for computing the data generated by the IoT gateways 2632-a,...,g. Real-time data network intermediate processing nodes 2624-a,b,c will be embedded in the IoT servers 2622-a,b,c respectively. As shown, Figure 26 As shown, the IoT gateways 2632-a,b,c are connected to the IoT server 2622-a; the IoT gateways 2632-d,e are connected to the IoT server 2622-b; and finally, the IoT gateways 2632-f,g are connected to the IoT server 2622-c. The data generated by the IoT servers 2622-a,b,c will be forwarded to the central computing layer 2610.

[0325] The central computing layer 2610 is located at the top of the network. Although Figure 26 Only one centralized IoT server 2613 is shown, but in fact there can be multiple such servers. As shown, Figure 26 As shown, all regional IoT servers 2622-a,b,c are connected to the centralized IoT server 2613. The invocation system 2614 will be embedded in the centralized IoT server 2613.

[0326] In addition, as shown, Figure 26 The embedded invocation system 2614, intermediate processing nodes 2624-a,b,c and access execution systems 2634-a,...,g can be managed by a centralized real-time data network management system 2611. In addition, the aforementioned real-time data network is also deployed with a plug-in management system 2612 for managing all plug-in service programs 0317-a,b,c uploaded by developers to the real-time data network, and then publishing and installing them into the invocation system 2614, intermediate processing nodes 2624-a,b,c and access execution systems 2634-a,...,g through the invocation system 2614.

[0327] Figure 14An example of a fourth plug-in service program 2650 (Flood_Alert) is shown. The fourth plug-in service program 2650 is created by the developer using the multi-tier service plug-in programming framework 1410 as shown above. As described previously, the program code filled in by the developer into the portion of the calling system program 1320 will be executed by the embedded execution calling system 2614. The program code filled in by the developer into the portion of the programming template 1420-a of the intermediate system node will be executed by the intermediate processing node 2624-a,b,c. The program code filled in by the developer into the portion of the access execution system program 1330 will be executed by the access execution system 2634-a,...,g. ​

[0328] ​The fourth plug-in service program 2650 allows a user to start or stop its analytics service. As shown in the code in the GDDS pre-processing template 1324 and the intermediate system node pre-processing template 1424-a of the fourth plug-in service program 2650, upon receiving a flood alert API command (start / stop,... parameters), the call system 2614 is invoked to propagate the flood alert API command down to all intermediate processing nodes 2624-a,b,c. The intermediate processing nodes 2624-a,b,c load and execute the code in the intermediate system node pre-processing template 1424-a of the fourth plug-in service program 2650, which will be further forwarded to all access execution systems 2634-a,...,g in the Internet of Things gateways 2632-a,...,g. Upon receiving the flood alert API command, the access execution systems 2634-a,...,g load and execute the code in the DSS execution template 1332 of the fourth plug-in service program 2650. Depending on the command, the fourth plug-in service program 2650 will start or stop the computation process at the access execution systems 2634-a,...,g. After the fourth plug-in service program 2650 is started at the access execution systems 2634-a,...,g, the code continuously processes the data stream from the devices and sensors 2640-a,...,p in a distributed manner to extract meaningful data and send the extracted information to the relevant intermediate processing nodes 2624-a,b,c. In addition, the developer can also program the code in the DSS execution template 1332 of the fourth plug-in service program 2650 to detect abnormal events or thresholds in the data stream it receives. When an abnormal event or threshold is detected, an alert can be sent to some external emergency management application. When the embedded intermediate processing nodes 2624-a,b,c receive the response from their relevant access execution systems 2634-a,...,g, the code in the intermediate processing node post-processing template 1425-a of the fourth plug-in service program 2650 can aggregate the data from multiple access execution systems and further analyze them, and then send the analysis results to the call system 2614. Meanwhile, the intermediate processing nodes 2624-a,b,c can also send an alert to some external emergency management application. When the call system 2614 receives the results from the intermediate processing nodes 2624-a,b,c, the code in the GDDS post-processing template 1325 of the fourth plug-in service program 2650 further analyzes the data and generates decision support information for the external emergency management application. The code in the GDDS post-processing template 1325 of the fourth plug-in service program 2650 can also generate necessary reports.

[0329] The fourth plug-in service program 2650 is only an example and is intended to demonstrate that distributed data processing and analysis programs can be easily created, published, installed, and started on a large-scale Internet of Things through a real-time data network. Without the real-time data network, Internet of Things data processing and analysis software is usually static. In general, software updates on all Internet of Things nodes must be done manually. Through the real-time data network, not only can software be quickly published and installed, but also multiple plug-in service programs can be published and executed simultaneously in the network. In addition, software updates can also be automatically executed. For example, after viewing the data results, the developer may want to add more logic in the fourth plug-in service program 2650, or the developer may want to add more emergency management applications as their alarm targets. Through the real-time data network, software updates can be completed very quickly and published on the Internet of Things network.

[0330] The order of the above embodiments is only for the convenience of description, and does not represent the advantages and disadvantages of the embodiments.

[0331] As can be seen from the above embodiments, the embodiments of the present disclosure provide a real-time data network system and a method for querying and processing distributed data, relate to the field of distributed data technology, and can solve the problem of real-time querying and processing of large-scale distributed real-time data of multiple structures.

[0332] The real-time data network system according to the embodiments of the present disclosure can include:

[0333] one or more calling systems; and

[0334] one or more access execution systems; wherein

[0335] Each calling system is connected to one or more access execution systems;

[0336] Each access execution system is connected to one or more data storage systems;

[0337] Each calling system is configured to provide an external application with an interface for accessing and processing distributed data, receive an instruction for accessing and processing distributed data from the external application through the interface for accessing and processing distributed data, and access and process distributed data stored in the data storage system through one or more access execution systems connected thereto.

[0338] In some embodiments of the present disclosure, each calling system is installed with a first plug-in service program and a second plug-in service program; wherein the calling system is configured to receive a first application programming interface instruction issued by the external application program through the interface for accessing and processing distributed data; determine an execution location of the first application programming interface instruction; in response to determining that the execution location is an access execution system, start a first plug-in service program related to the first application programming interface instruction, so that the calling system sends a second application programming interface instruction to one or more access execution systems associated therewith, and accesses distributed data stored in the data storage system through the one or more access execution systems; and in response to determining that the execution location is a calling system, start a second plug-in service program related to the first application programming interface instruction, obtain data according to a data processing logic set by the second plug-in service program, process the obtained data, and feed back the processed data to the external application program through the interface for accessing and processing distributed data.

[0339] In some embodiments of the present disclosure, each access execution system is installed with a third plug-in service program; wherein the access execution system receives the second application programming interface instruction issued by the calling system, starts a third plug-in service program related to the second application programming interface instruction, so that the access execution system reads the distributed data from the data storage system, and processes the distributed data according to a data processing logic set by the third plug-in service program to obtain first response data.

[0340] In some embodiments of the present disclosure, each calling system is installed with a fourth plug-in service program; the calling system is further configured to receive the first response data from the access execution system, start a fourth plug-in service program related to the first application programming interface instruction, process the first response data to obtain second response data, and feed back the second response data to the external application program through the interface for accessing and processing distributed data.

[0341] In some embodiments of the present disclosure, the access execution system is configured with a data adapter; the access execution system matches the distributed data and reads the distributed data from the data storage system through the data adapter.

[0342] In some embodiments of the present disclosure, the access execution system is further installed with a data integration program; the access execution system converts a data model of the distributed data into a data model required by the third plug-in service program by starting the data integration program.

[0343] In some embodiments of the present disclosure, each access execution system is installed with a third plug-in service program and a fifth plug-in service program; wherein the access execution system receives a second application programming interface instruction issued by the calling system, starts a fifth plug-in service program related to the second application programming interface instruction; the fifth plug-in service program calls the third plug-in service program, so that the access execution system reads the distributed data from the data storage system, and processes the distributed data according to the data processing logic set by the third plug-in service program to obtain intermediate data; the intermediate data is processed according to the data processing logic set by the fifth plug-in service program to obtain first response data.

[0344] In some embodiments of the present disclosure, the distributed data stored by the data storage system comprises one or any combination of structured data, semi-structured data and unstructured data.

[0345] In some embodiments of the present disclosure, the system further comprises an intermediate processing node and an intermediate storage system; wherein the intermediate processing node is connected to one calling system and one or more access execution systems, for receiving the access and processing distributed data instructions from the calling system, sending the access and processing distributed data instructions to the one or more access execution systems, and receiving response data from the one or more access execution systems and processing the response data before sending to the calling system; the intermediate storage system is used to store the response data processed by the intermediate processing node.

[0346] In some embodiments of the present disclosure, the system further comprises:

[0347] an access execution system of an intermediate layer;

[0348] a calling system of an intermediate layer; and

[0349] a data storage system of an intermediate layer; wherein,

[0350] the access execution system of the intermediate layer is connected to a calling system and the calling system of the intermediate layer and the data storage system of the intermediate layer;

[0351] the calling system of the intermediate layer is connected to one or more access execution systems.

[0352] In some embodiments of the present disclosure, the system further comprises a data network plug-in service management system for managing all plug-in service programs installed in the calling system and the access execution system; wherein the data network plug-in service management system comprises a plug-in service programming framework for providing a programming framework for developers to develop plug-in service program packages; and a program management module for managing and publishing the plug-in service program packages developed by the developers, so that the calling system and the access execution system install the plug-in service program packages developed by the developers.

[0353] In some embodiments of the present disclosure, the access execution system comprises a data integration programming framework for providing a programming framework for developers to develop data integration programs; and a data adapter configuration framework for providing an interface for developers to input an adapter name field, select and reference a data adapter microservice program, and input an address and access information field of a data storage system.

[0354] In some embodiments of the present disclosure, the system further comprises a data network management system for managing various systems in a real-time data network; wherein the data network management system comprises a calling system and access execution system account management module for creating or deleting calling systems and access execution systems; and a calling system and access execution system network topology management module for adding or deleting calling systems and access execution systems in the real-time data network system, and managing network connection relationships between calling systems and access execution systems.

[0355] In some embodiments of the present disclosure, the system further comprises a plug-in service programming framework for providing a programming template for developers to develop plug-in service program packages; wherein the programming template allows the developers to define application program interfaces (APIs) of various plug-in service programs in the plug-in service program packages, define execution locations of various plug-in service programs in the plug-in service program packages, and create codes of various plug-in service programs executed in the calling system and / or the access execution system; wherein the APIs are programming interface instructions used by the external application programs.

[0356] In some embodiments of the present disclosure, the execution location comprises one of the calling system and the access execution system.

[0357] In some embodiments of the present disclosure, the execution location comprises one of the calling system, an intermediate processing node, and the access execution system.

[0358] In some embodiments of the present disclosure, the programming template comprises a calling system verification template, a calling system preprocessing template, a calling system execution template, a calling system post-processing template, and an access execution system execution template.

[0359] In some embodiments of the present disclosure, the programming template further comprises: an intermediate processing node pre-processing template, an intermediate processing node execution template, and an intermediate processing node post-processing template.

[0360] In some embodiments of the present disclosure, the plug-in service programming framework further provides the developer with access to an access execution system API, through which the developer can start and execute its associated data integration program and data adapter in each of the access execution systems in the plug-in service program.

[0361] Corresponding to the real-time data network system described above, embodiments of the present disclosure also disclose a method for querying and processing distributed data, applied to a real-time data network system, wherein the real-time data network system comprises: one or more calling systems; and one or more access execution systems; wherein the method for querying and processing distributed data comprises: each calling system respectively receiving an instruction for accessing and processing distributed data from an external application program; and each calling system accessing and processing distributed data stored in a data storage system through one or more access execution systems connected thereto.

[0362] In some embodiments of the present disclosure, the calling system receiving an instruction for accessing and processing distributed data from an external application program comprises: the calling system receiving a first application programming interface instruction issued by the external application program through an interface for accessing and processing distributed data; and the calling system accessing and processing distributed data stored in the data storage system through one or more access execution systems connected thereto comprises: the calling system determining an execution location of the first application programming interface instruction; in response to determining that the execution location is an access execution system, the calling system starting a first plug-in service program associated with the first application programming interface instruction, sending a second application programming interface instruction to one or more access execution systems associated with the first plug-in service program, and accessing distributed data stored in the data storage system through the one or more access execution systems; in response to determining that the execution location is a calling system, starting a second plug-in service program associated with the first application programming interface instruction, obtaining data according to a data processing logic set by the second plug-in service program, processing the obtained data, and feeding back the processed data to the external application program through the interface for accessing and processing distributed data.

[0363] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system comprises: receiving, by the access execution system, a second application programming interface instruction issued by the calling system; starting a third plug-in service program associated with the second application programming interface instruction; reading the distributed data from the data storage system; and processing the distributed data according to data processing logic set by the third plug-in service program to obtain first response data.

[0364] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system further comprises: receiving, by the calling system, the first response data from the access execution system, starting a fourth plug-in service program associated with the first application programming interface instruction, processing the first response data according to data processing logic set by the fourth plug-in service program to obtain second response data, and feeding back the second response data to the external application program through the interface for accessing and processing the distributed data.

[0365] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system further comprises: the third plug-in service program of the one or more access execution systems converting a data model of the distributed data into a data model required by the third plug-in service program by starting the data integration program.

[0366] In some embodiments of the present disclosure, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system comprises: receiving, by the access execution system, a second application programming interface instruction issued by the calling system; starting a fifth plug-in service program associated with the second application programming interface instruction; the fifth plug-in service program calling a third plug-in service program; reading the distributed data from the data storage system; and processing the distributed data according to data processing logic set by the third plug-in service program to obtain intermediate data; and processing the intermediate data according to data processing logic set by the fifth plug-in service program to obtain first response data.

[0367] In some embodiments of the present disclosure, the method further comprises: obtaining a calling system account and a topology position of the calling system in the real-time data network system; registering a calling system in the real-time data network management system according to the calling system account and the topology position of the calling system in the real-time data network system; and receiving a first security token corresponding to the calling system from the real-time data network management system; determining whether there is an associated access execution system of the calling system according to the topology position of the calling system in the real-time data network system; and in response to determining that there is an associated access execution system, notifying all associated access execution systems of the association with the calling system and establishing a communication connection between the access execution system and the calling system.

[0368] In some embodiments of the present disclosure, the method further comprises: obtaining an access execution system account and a topology position of the access execution system in the data network system; registering an access execution system in the real-time data network management system according to the access execution system account and the topology position of the access execution system in the data network system; receiving a second security token corresponding to the access execution system from the real-time data network management system; and associating the access execution system to the calling system associated therewith and establishing a communication connection between the access execution system and the calling system based on the topology position of the access execution system in the data network system.

[0369] In some embodiments of the present disclosure, establishing the communication connection between the access execution system and the calling system comprises: the access execution system providing the second security token to the calling system; the calling system providing a first access token to the access execution system; and the access execution system providing a second access token to the upper calling system; wherein the first access token and the second access token are used for communication between the calling system and the access execution system.

[0370] In some embodiments of the present disclosure, the method further comprises: sending an installation request to the calling system to install a plug-in service package; the calling system installing all or part of the plug-in service programs in the plug-in service package into a pre-installed library; the calling system notifying all associated access execution systems to install the plug-in service package; and each access execution system installing all or part of the plug-in service programs in the plug-in service package into its own installation library; the each access execution system configuring a data adapter for the plug-in service program to access distributed data stored in a data storage system connected to the access execution system and associating the plug-in service program with the data adapter.

[0371] In some embodiments of the present disclosure, the method further comprises: each access execution system provides a programming framework of a data integration program, and obtains the data integration program from the programming framework of the data integration program; wherein the data integration program is used to convert a data model of the distributed data into a data model required by the third plug-in service program; and the third plug-in service program is associated with the data integration program.

[0372] In some embodiments of the present disclosure, the method further comprises: providing a programming template for a developer to develop a plug-in service program package; wherein the programming template allows the developer to define an application program interface (API) of each plug-in service program in the plug-in service program package, define an execution location of each plug-in service program in the plug-in service program package, and write a code of each plug-in service program executed in the calling system and / or the access execution system; wherein the API is a programming interface instruction used by the external application program.

[0373] In some embodiments of the present disclosure, the programming template comprises: a calling system verification template, a calling system preprocessing template, a calling system execution template, a calling system post-processing template, and an access execution system execution template.

[0374] In some embodiments of the present disclosure, the access and processing of the distributed data stored in the data storage system by each calling system through one or more access execution systems connected thereto comprises:

[0375] In response to determining that the execution location is the calling system, the calling system executes the code in the calling system execution template, and returns a first execution result to the external application program;

[0376] In response to determining that the execution location is the access execution system, the calling system executes the code in the calling system preprocessing template, and sends the processed access and instructions for processing the distributed data to the access execution system; the access execution system executes the code in the access execution system execution template, and feeds back a second execution result to the calling system; the calling system receives the second execution result from the access execution system, executes the code in the calling system post-processing template, and returns a third execution result to the external application program.

[0377] In some embodiments of the present disclosure, the programming template further comprises: an intermediate processing node preprocessing template, an intermediate processing node execution template, and an intermediate processing node post-processing template.

[0378] In some embodiments of the present disclosure, the access and processing of the distributed data stored in the data storage system by each calling system through one or more access execution systems connected thereto comprises:

[0379] In response to determining that the execution site is the calling system, the calling system executes the code in the calling system execution template and returns a fourth execution result to the external application program;

[0380] In response to determining that the execution site is the intermediate processing node, the calling system executes the code in the calling system preprocessing template and sends the processed access and processing distributed data instructions to the intermediate processing node; the intermediate processing node executes the code in the intermediate processing node execution template and feeds back a fifth execution result to the calling system; the calling system receives the fifth execution result from the intermediate processing node, executes the code in the calling system post-processing template, and returns a sixth execution result to the external application program;

[0381] In response to determining that the execution site is the access execution system, the calling system executes the code in the calling system preprocessing template and sends the processed access and processing distributed data instructions to the intermediate processing node; the intermediate processing node executes the code in the intermediate processing node preprocessing template and sends the processed access and processing distributed data instructions to the access execution system; the access execution system executes the code in the access execution system execution template and feeds back a seventh execution result to the intermediate processing node; the intermediate processing node receives the seventh execution result from the access execution system, executes the code in the intermediate processing node post-processing template, and feeds back an eighth execution result to the calling system; the calling system receives the eighth execution result from the intermediate processing node, executes the code in the calling system post-processing template, and returns a ninth execution result to the external application program.

[0382] In some embodiments of the present disclosure, the above method further comprises: in response to determining that the code in the calling system verification template is filled in, the calling system executes the code in the calling system verification template before executing other plug-in service programs.

[0383] In some embodiments of the present disclosure, the execution of the code in the calling system preprocessing template by the calling system comprises: in response to the code in the calling system preprocessing template being empty, the calling system sends the received access and processing distributed data instruction to the access execution system; in response to the code in the calling system preprocessing template being not empty, the calling system executes the code in the calling system preprocessing template first, and sends the processed access and processing distributed data instruction to the access execution system; the execution of the code in the calling system post-processing template comprises: in response to the code in the calling system post-processing template being empty, the calling system returns the received execution result to the external application program; in response to the code in the calling system post-processing template being not empty, the calling system executes the code in the calling system post-processing template, and returns the processed execution result to the external application program.

[0384] In some embodiments of the present disclosure, the execution of the code in the intermediate processing node preprocessing template by the intermediate processing node comprises: in response to the code in the intermediate processing node preprocessing template being empty, the intermediate processing node sends the received access and processing distributed data instruction to the access execution system; in response to the code in the intermediate processing node preprocessing template being not empty, the intermediate processing node executes the code in the intermediate processing node preprocessing template first, and sends the processed access and processing distributed data instruction to the access execution system; the execution of the code in the intermediate processing node post-processing template comprises: in response to the code in the intermediate processing node post-processing template being empty, the intermediate processing node returns the received execution result to the calling system; in response to the code in the intermediate processing node post-processing template being not empty, the intermediate processing node executes the code in the intermediate processing node post-processing template, and returns the processed execution result to the calling system.

[0385] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time data network system, comprising: one or more calling systems; and one or more access execution systems; wherein each calling system is connected to one or more access execution systems respectively; each access execution system is connected to one or more data storage systems respectively; each calling system is configured to provide an external application with an interface for accessing and processing distributed data, receive an instruction for accessing and processing distributed data from the external application through the interface for accessing and processing distributed data, and access and process distributed data stored in the data storage system through one or more access execution systems connected thereto; wherein each calling system is installed with a first plug-in service program and a second plug-in service program; wherein the calling system is configured to: receive a first application programming interface instruction issued by the external application through the interface for accessing and processing distributed data; determine an execution location of the first application programming interface instruction; in response to determining that the execution location is an access execution system, start a first plug-in service program related to the first application programming interface instruction, so that the calling system sends a second application programming interface instruction to one or more access execution systems associated therewith, and accesses distributed data stored in the data storage system through the one or more access execution systems; and in response to determining that the execution location is a calling system, start a second plug-in service program related to the first application programming interface instruction, obtain data according to a data processing logic set in the second plug-in service program, process the obtained data, and feed back the processed data to the external application through the interface for accessing and processing distributed data; each access execution system is installed with a third plug-in service program; wherein the access execution system receives the second application programming interface instruction issued by the calling system, starts a third plug-in service program related to the second application programming interface instruction, so that the access execution system reads the distributed data from the data storage system, and processes the distributed data according to a data processing logic set in the third plug-in service program to obtain first response data; the access execution system is configured with a data adapter; the access execution system matches the distributed data and reads the distributed data from the data storage system through the data adapter; the access execution system is further installed with a data integration program; and the access execution system converts a data model of the distributed data into a data model required by the third plug-in service program by starting the data integration program. each calling system is installed with a fourth plug-in service program; 2. The system of claim 1, wherein, the calling system is further configured to receive the first response data from the access execution system, start a fourth plug-in service program related to the first application programming interface instruction, process the first response data to obtain second response data, and feed back the second response data to the external application through the interface for accessing and processing distributed data. ​ 3. The system of claim 1, wherein, Each access execution system is installed with a third plug-in service program and a fifth plug-in service program; wherein, The access execution system receives the second application programming interface instruction issued by the calling system, starts the fifth plug-in service program related to the second application programming interface instruction; the fifth plug-in service program calls the third plug-in service program, so that the access execution system reads the distributed data from the data storage system, processes the distributed data according to the data processing logic set by the third plug-in service program, obtains intermediate data, processes the intermediate data according to the data processing logic set by the fifth plug-in service program, and obtains first response data.

4. The system of claim 1, wherein, The distributed data stored by the data storage system includes one or any combination of structured data, semi-structured data and unstructured data.

5. The system of claim 1, further comprising: An intermediate processing node and an intermediate storage system; wherein, The intermediate processing node is connected with a calling system and one or more access execution systems, for receiving the access and processing distributed data instructions from the calling system, sending the access and processing distributed data instructions to the one or more access execution systems, receiving response data from the one or more access execution systems, and processing and sending the response data to the calling system after processing; The intermediate storage system is used to store the response data processed by the intermediate processing node.

6. The system of claim 1, further comprising: an intermediate layer access execution system; an intermediate layer calling system; and an intermediate layer data storage system; wherein, The intermediate layer access execution system is connected with a calling system and the intermediate layer calling system and the intermediate layer data storage system; The intermediate layer calling system is connected with one or more access execution systems. A data network plug-in service management system is used to manage all plug-in service programs installed in the calling system and the access execution system; wherein, 7. The system of any of claims 2-6, further comprising: The data network plug-in service management system comprises: a plug-in service programming framework for providing a programming framework for developers to develop plug-in service program packages; and a program management module for managing and publishing the plug-in service program packages developed by the developers, so that the calling system and the access execution system install the plug-in service program packages developed by the developers. The access execution system comprises:

8. The system of claim 1, wherein, a data integration programming framework for providing a programming framework for developers to develop data integration programs; and a data adapter configuration framework for providing an interface for developers to input an adapter name field, select and reference a data adapter microservice program, and input an address and access information field of a data storage system. A data network management system is used to manage various systems in a real-time data network; wherein, 9. The system of claim 1, further comprising: The data network management system comprises: a calling system and access execution system account management module for creating or deleting calling systems and access execution systems; and ​ The calling system and the access execution system network topology management module is used to add or delete the calling system and the access execution system in the real-time data network system, and manage the network connection relationship between the calling system and the access execution system.

10. The system of claim 1, further comprising: The plug-in service programming framework is used to provide a programming template for the developer to develop a plug-in service program package; wherein, the programming template allows the developer to define the application program interface (API) of each plug-in service program in the plug-in service program package, define the execution location of each plug-in service program in the plug-in service program package, and create the code of each plug-in service program executed in the calling system and / or the access execution system; wherein, the API is the programming interface instruction used by the external application program.

11. The system of claim 10, wherein, The execution location includes one of the calling system and the access execution system.

12. The system of claim 10, wherein, The execution location includes one of the calling system, the intermediate processing node, and the access execution system.

13. The system of claim 10, wherein, The programming template includes a calling system verification template, a calling system preprocessing template, a calling system execution template, a calling system post-processing template, and an access execution system execution template.

14. The system of claim 10, wherein, The programming template further includes an intermediate processing node preprocessing template, an intermediate processing node execution template, and an intermediate processing node post-processing template.

15. The system of claim 10, wherein, The plug-in service programming framework further provides the developer with an access execution system API, through which the developer can start and execute the associated data integration program and data adapter in each access execution system in the plug-in service program.

16. A method of querying and processing distributed data, applied to a real-time data network system, wherein, The real-time data network system includes one or more calling systems and one or more access execution systems; wherein, the method for querying and processing distributed data includes: Each calling system respectively receives the access and processing distributed data instructions from the external application program; and Each calling system accesses and processes the distributed data stored in the data storage system through one or more access execution systems connected thereto; wherein, Each calling system receiving the access and processing distributed data instructions from the external application program includes: the calling system receiving the first application programming interface instruction issued by the external application program through the interface for accessing and processing distributed data; and Each calling system accessing and processing the distributed data stored in the data storage system through one or more access execution systems connected thereto includes: The calling system determines the execution location of the first application programming interface instruction; In response to determining that the execution location is an access execution system, the calling system starts a first plug-in service program associated with the first application programming interface instruction, sends a second application programming interface instruction to one or more access execution systems associated with the first plug-in service program, and accesses the distributed data stored in the data storage system through the one or more access execution systems; in response to determining that the execution site is the calling system, starting a second plug-in service program related to the first application programming interface instruction, obtaining data according to data processing logic set by the second plug-in service program, processing the obtained data, and feeding back the processed data to the external application program through the interface for accessing and processing distributed data; the accessing, by the one or more access execution systems, the distributed data stored in the data storage system comprises: the access execution system receives a second application programming interface instruction issued by the calling system; starting a third plug-in service program related to the second application programming interface instruction; reading the distributed data from the data storage system; and processing the distributed data according to data processing logic set by the third plug-in service program to obtain first response data; the access execution system matches the distributed data through a data adapter and reads the distributed data from the data storage system; and the accessing, by the one or more access execution systems, the distributed data stored in the data storage system further comprises: the third plug-in service program of the one or more access execution systems converts a data model of the distributed data into a data model required by the third plug-in service program by starting a data integration program.

17. The method of claim 16, wherein, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system further comprises: the calling system receives the first response data from the access execution system, starts a fourth plug-in service program related to the first application programming interface instruction, processes the first response data according to data processing logic set by the fourth plug-in service program to obtain second response data, and feeds back the second response data to the external application program through the interface for accessing and processing distributed data.

18. The method of claim 16, wherein, the accessing, by the one or more access execution systems, the distributed data stored in the data storage system comprises: the access execution system receives a second application programming interface instruction issued by the calling system; starting a fifth plug-in service program related to the second application programming interface instruction; the fifth plug-in service program calls a third plug-in service program; reading the distributed data from the data storage system; processing the distributed data according to data processing logic set by the third plug-in service program to obtain intermediate data; and processing the intermediate data according to data processing logic set by the fifth plug-in service program to obtain first response data.

19. The method of claim 16, further comprising: obtaining a calling system account and a topology position of the calling system in the real-time data network system; registering a calling system in a real-time data network management system according to the calling system account and the topology position of the calling system in the real-time data network system; and receiving a first security token corresponding to the calling system from the real-time data network management system; and ​ determining whether the calling system has an associated access execution system based on a topology location of the calling system in the real-time data network system; and in response to determining that there is an associated access execution system, notifying all associated access execution systems of the association with the calling system and establishing a communication connection between the access execution system and the calling system.

20. The method of claim 16, further comprising: obtaining an access execution system account and a topology location of the access execution system in the data network system; registering an access execution system in the real-time data network management system based on the access execution system account and the topology location of the access execution system in the data network system; receiving a second security token corresponding to the access execution system from the real-time data network management system; and associating the access execution system to a calling system associated therewith and establishing a communication connection between the access execution system and the calling system based on the topology location of the access execution system in the data network system.

21. The method of claim 20, wherein, establishing the communication connection between the access execution system and the calling system comprises: the access execution system providing the second security token to the calling system; the calling system providing a first access token to the access execution system; the access execution system providing a second access token to the calling system; wherein the first and second access tokens are used for communication between the calling system and the access execution system.

22. The method of claim 16, further comprising: sending an installation request to the calling system to install a plug-in service package; the calling system installing all or part of the plug-in service programs in the plug-in service package into a pre-installed library; the calling system notifying all associated access execution systems of the installation of the plug-in service package; and each access execution system installing all or part of the plug-in service programs in the plug-in service package into its own installation library; each access execution system configuring the data adapter for the plug-in service program to access distributed data stored in a data storage system connected to the access execution system and associating the plug-in service program with the data adapter.

23. The method of claim 22, further comprising: each access execution system providing a programming framework of the data integration program and obtaining the data integration program from the programming framework of the data integration program; and associating the third plug-in service program with the data integration program. providing a programming template for a developer to develop a plug-in service package; wherein the programming template allows the developer to define an application program interface (API) for each plug-in service program in the plug-in service package, define an execution location for each plug-in service program in the plug-in service package, and write code for each plug-in service program to be executed in the calling system and / or the access execution system; wherein the API is a programming interface instruction for use by the external application.

24. The method of claim 22, further comprising: ​ 25. The method of claim 24, wherein, The programming templates comprise: a system call verification template; a system call preprocessing template; a system call execution template; a system call post-processing template; and an access execution system execution template.

26. The method of claim 25, wherein, Each of the system calls accesses and processes the distributed data stored by the data storage system through one or more access execution systems connected thereto, and comprises: In response to determining that the execution site is a system call, the system call executes the code in the system call execution template and returns a first execution result to the external application program; In response to determining that the execution site is an access execution system, the system call executes the code in the system call preprocessing template, sends the processed access and processing distributed data instructions to the access execution system; the access execution system executes the code in the access execution system execution template and feeds back a second execution result to the system call; the system call receives the second execution result from the access execution system, executes the code in the system call post-processing template, and returns a third execution result to the external application program.

27. The method of claim 25, wherein, The programming templates further comprise: an intermediate processing node preprocessing template; an intermediate processing node execution template; and an intermediate processing node post-processing template.

28. The method of claim 27, wherein, Each of the system calls accesses and processes the distributed data stored by the data storage system through one or more access execution systems connected thereto, and comprises: In response to determining that the execution site is a system call, the system call executes the code in the system call execution template and returns a first execution result to the external application program; In response to determining that the execution site is an intermediate processing node, the system call executes the code in the system call preprocessing template, sends the processed access and processing distributed data instructions to the intermediate processing node; the intermediate processing node executes the code in the intermediate processing node execution template and feeds back a fifth execution result to the system call; the system call receives the fifth execution result from the intermediate processing node, executes the code in the system call post-processing template, and returns a sixth execution result to the external application program; In response to determining that the execution site is an access execution system, the system call executes the code in the system call preprocessing template, sends the processed access and processing distributed data instructions to the intermediate processing node; the intermediate processing node executes the code in the intermediate processing node preprocessing template, sends the processed access and processing distributed data instructions to the access execution system; the access execution system executes the code in the access execution system execution template and feeds back a seventh execution result to the intermediate processing node; the intermediate processing node receives the seventh execution result from the access execution system, executes the code in the intermediate processing node post-processing template, and feeds back an eighth execution result to the system call; the system call receives the eighth execution result from the intermediate processing node, executes the code in the system call post-processing template, and returns a ninth execution result to the external application program.

29. The method of claim 25, further comprising: In response to determining that the filled-in code in the calling system verification template, the calling system executes the code in the calling system verification template before executing other plug-in service programs.

30. The method of claim 25 or 28, wherein, The calling system executing the code in the calling system preprocessing template includes: in response to the code in the calling system preprocessing template being empty, the calling system sending the received access and processing distributed data instructions to an access execution system; in response to the code in the calling system preprocessing template not being empty, the calling system executing the code in the calling system preprocessing template first, and sending the processed access and processing distributed data instructions to the access execution system. The executing the code in the calling system post-processing template includes: in response to the code in the calling system post-processing template being empty, the calling system returning the received execution result to the external application program; in response to the code in the calling system post-processing template not being empty, the calling system executing the code in the calling system post-processing template, and returning the processed execution result to the external application program.

31. The method of claim 28, wherein, The intermediate processing node executing the code in the intermediate processing node preprocessing template includes: in response to the code in the intermediate processing node preprocessing template being empty, the intermediate processing node sending the received access and processing distributed data instructions to an access execution system; in response to the code in the intermediate processing node preprocessing template not being empty, the intermediate processing node executing the code in the intermediate processing node preprocessing template first, and sending the processed access and processing distributed data instructions to the access execution system. The executing the code in the intermediate processing node post-processing template includes: in response to the code in the intermediate processing node post-processing template being empty, the intermediate processing node returning the received execution result to the calling system; in response to the code in the intermediate processing node post-processing template not being empty, the intermediate processing node executing the code in the intermediate processing node post-processing template, and returning the processed execution result to the calling system.

Images