Data access method, system and storage medium

Abstract

The present disclosure proposes a data access method, system and storage medium, which relate to the field of database technology. A data access system disclosed in the present disclosure includes: a data platform, configured to receive data requests and send them to a data engine; receive request execution results fed back by the data engine; a data engine, configured to split data requests, obtain multiple request subtasks, and send request subtasks to each data probe; aggregate the subtask execution results from each data probe, generate request execution results, and feed back to the data platform; and multiple data probes, configured to obtain request subtasks; call the corresponding data node to execute the request subtask, generate subtask execution results and feed back to the data engine. Such a data access system can achieve non-invasive access to data nodes, and does not require data synchronization operations, thereby improving the real-time nature of data requests and improving request efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to a data access method, system, and storage medium. Background Art

[0002] In modern enterprises, there are multiple information systems, and the business data information in the systems is stored in databases. Among them, relational databases are the most common databases. There are many types of relational databases, and there are some differences between them. Figure 1 As shown, the ETL data warehouse can realize data sharing between different databases to achieve the purpose of supporting data warehousing service 100. Its operation process includes:

[0003] (1) Extraction: Extract data from data nodes (city 131-134, province 111-112, top level 101) and store them. Except for the top-level data nodes, all other data nodes need to support incremental extraction.

[0004] (2) Transform: transform the data extracted from the leaf nodes (131-134) into the data structure defined by the current data node;

[0005] (3) Load: Write the data converted in step (2) into the database of the current data node. Summary of the Invention

[0006] One purpose of the present disclosure is to propose a data access architecture to improve the efficiency of data access in the case of multiple data nodes.

[0007] According to one aspect of some embodiments of the present disclosure, a data access system is proposed, including: a data platform, configured to receive data requests and send them to a data engine; receive request execution results fed back by the data engine; the data engine, configured to split the data request, obtain multiple request subtasks, and send the request subtasks to each data probe; aggregate the subtask execution results from each data probe, generate a request execution result, and feed it back to the data platform; and multiple data probes, configured to obtain request subtasks; call corresponding data nodes to execute request subtasks, generate subtask execution results, and feed back the subtask execution results to the data engine.

[0008] In some embodiments, the data probes are deployed in a distributed manner, and each database's intranet environment corresponds to a data probe.

[0009] In some embodiments, at least one of the data platform or the data engine is deployed in a distributed cluster.

[0010] In some embodiments, the data probe is configured to: receive feedback data from the data node; and generate subtask execution results based on the data model of the data platform according to the feedback data.

[0011] In some embodiments, the data engine is configured to publish the message queue of the request subtask to the subtask topic; the data probe is configured to filter the message queue from the subtask topic based on preset message filtering rules to obtain its own request subtask.

[0012] In some embodiments, the data probe is also configured to publish the subtask execution results to the result topic; the data engine is also configured to: obtain the subtask execution results of the request subtask belonging to the same data request according to the message queue of the result topic; and aggregate the obtained subtask execution results into request execution results that conform to the data model of the data platform.

[0013] In some embodiments, the data platform is further configured to perform at least one of the following: establishing a standard model and sending it to the data engine; generating a correspondence between each data probe and data node, as well as rules for each corresponding data node, and sending them to the corresponding data probe; receiving data requests from external applications; after receiving the request execution result from the data engine, feeding back the request execution result to the external application; or after receiving the data request, performing data request verification, and sending the data request to the data engine if the verification passes.

[0014] In some embodiments, at least one of the data platform, the data engine, or the data probe is deployed in a virtualized manner.

[0015] In such a data access system, the data engine can split data requests, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-intrusive access to the data nodes and eliminating the need for data synchronization operations, thereby improving the real-time nature of data requests and improving request efficiency.

[0016] According to one aspect of some embodiments of the present disclosure, a data access method is proposed, including: the data platform sends a data request to a data engine; the data engine splits the data request, obtains multiple request subtasks, and sends the request subtasks to each data probe; the data probe obtains the request subtask, calls the corresponding data node to execute the request subtask, generates a subtask execution result, and feeds back the subtask execution result to the data engine; the data engine summarizes the subtask execution results from each data probe, generates a request execution result, and feeds back to the data platform.

[0017] In some embodiments, generating the subtask execution result includes: receiving feedback data from a data node; and generating the subtask execution result based on a data model of a data platform according to the feedback data.

[0018] In some embodiments, sending the request subtask to each data probe includes: the data engine publishes the message queue of the request subtask to the subtask topic; the data probe obtains the request subtask including: the data probe filters the message queue from the subtask topic based on preset message filtering rules to obtain its own request subtask.

[0019] In some embodiments, feeding back the subtask execution results to the data engine includes: the data probe publishes the subtask execution results to the result topic; the data engine aggregates the subtask execution results from each data probe, and generates a request execution result including: the data engine obtains the subtask execution results of the request subtasks belonging to the same data request according to the message queue of the result topic; and aggregates the obtained subtask execution results into request execution results that conform to the data model of the data platform.

[0020] In some embodiments, at least one of the following items is also included: the data platform establishes a standard model and sends it to the data engine; the data platform generates a correspondence between each data probe and the data node, as well as the rules of each corresponding data node, and sends them to the corresponding data probe; the data platform receives data requests from external applications; after receiving the request execution result from the data engine, the request execution result is fed back to the external application; after receiving the data request, the data platform performs data request verification, and if the verification passes, the data request is sent to the data engine.

[0021] Through this method, the data engine can split the data request, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-intrusive access to the data nodes and eliminating the need for data synchronization operations, thereby improving the real-time performance of data requests and improving request efficiency.

[0022] According to one aspect of some embodiments of the present disclosure, a data access system is proposed, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the above data access methods based on instructions stored in the memory.

[0023] In such a data access system, the data engine can split data requests, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-intrusive access to the data nodes and eliminating the need for data synchronization operations, thereby improving the real-time nature of data requests and improving request efficiency.

[0024] According to one aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the instructions are executed by a processor, the steps of any one of the above data access methods are implemented.

[0025] By executing instructions on such storage media, the data engine can split data requests, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-intrusive access to the data nodes and eliminating the need for data synchronization operations, thereby improving the real-time nature of data requests and improving request efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0027] Figure 1 This is a schematic diagram of database-level deployment in related technologies.

[0028] Figure 2 Schematic diagram of some embodiments of the data access system disclosed herein.

[0029] Figure 3 Schematic diagrams of other embodiments of the data access system disclosed herein.

[0030] Figures 4A to 4C The diagram is a schematic diagram of the operation of some embodiments of the data access system disclosed herein.

[0031] Figure 5 Schematic diagrams of some further embodiments of the data access system disclosed herein.

[0032] Figure 6 Schematic diagrams of some further embodiments of the data access system disclosed herein.

[0033] Figure 7 The following is a flowchart of some embodiments of the data access method disclosed herein.

[0034] Figure 8 Flowcharts of other embodiments of the data access method disclosed herein. DETAILED DESCRIPTION

[0035] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples.

[0036] The inventors discovered that in the data sharing methods used in related technologies, all data nodes except the top-level data nodes need to support incremental data extraction, which is somewhat invasive to the database. With the exception of leaf data nodes, all other nodes need to establish independent data warehouses to store all data from lower-level data nodes, which requires a large amount of storage resources. Full data transmission also increases network bandwidth usage. Furthermore, the more data node levels there are, the more redundant data storage is required. As the amount of data increases, storage costs will increase, leading to poor database and application performance.

[0037] Furthermore, when the data structure of a leaf data node changes, the leaf data node and all its parent data structures must be modified. Furthermore, data extraction, conversion, and loading are asynchronous operations, resulting in certain delays, which are more pronounced at multiple levels, making real-time performance less reliable. Furthermore, if any of these steps fail, the top-level data warehouse service will experience incomplete or incorrect data. For example, if a data node crashes, its leaf data nodes will be unable to synchronize data with their parent nodes.

[0038] Schematic diagrams of some embodiments of the data access system disclosed herein are as follows: Figure 2 shown.

[0039] The data platform 21 can receive data requests and send them to the data engine, and receive feedback from the data engine on the request execution results. In some embodiments, the data platform 21 can be centrally deployed and open user access interfaces and external application call interfaces for users to initiate requests and accept calls from external applications.

[0040] In some embodiments, the data platform 21 may be a separate server, or may be deployed in a virtual manner to reduce physical resource requirements and deployment costs.

[0041] In some embodiments, the data platform 21 may be deployed in a distributed cluster manner to improve service capabilities and system robustness. In some embodiments, data requests may include data query (call) requests, data statistics requests, and the like.

[0042] The data engine 22 can split the data request, obtain multiple request subtasks, and send the request subtasks to each data probe; summarize the subtask execution results from each data probe, generate the request execution results, and feed them back to the data platform.

[0043] In some embodiments, the data engine 22 stores database call statement rules and can split tasks through statement analysis to obtain multiple request subtasks. In some embodiments, data requests can be split based on the target data node of the query, or based on the data probe corresponding to the target data node of the query, so that the same request subtask is only targeted at the same data node, or only at data nodes accessible by the same data probe, thereby ensuring that the data probe can smoothly and completely execute the obtained request subtasks.

[0044] In some embodiments, the data engine 22 may be a separate server, or may be deployed in a virtual manner to reduce physical resource requirements and deployment costs.

[0045] In some embodiments, the data engine 22 may be deployed in a distributed cluster manner to improve service capabilities and system robustness.

[0046] Data probes 231-23n (n is a positive integer greater than 1) can obtain requested subtasks, call corresponding data nodes to execute the requested subtasks, generate subtask execution results, and feed the subtask execution results back to the data engine. In some embodiments, data probes are deployed in a distributed manner, for example, they can be deployed close to the data node deployment location to improve access efficiency to the data node.

[0047] In some embodiments, each data probe may preset message filtering rules, and the data engine 22 may provide all request subtasks to each data probe. The data probe filters the request subtasks according to its own message filtering rules to obtain the request subtasks that need to be executed by itself.

[0048] In some embodiments, the data probes 231 - 23 n may be separate servers, or may be deployed in a virtual manner to reduce physical resource requirements and deployment costs.

[0049] In some embodiments, the intranet environment of each database can correspond to a data probe, thereby breaking through the access restrictions of the intranet environment and expanding the scope of data calls.

[0050] In this data access system, the data engine can split data requests, with probes associated with data nodes executing the corresponding subtasks. This enables non-invasive access to data nodes and eliminates the need for data synchronization, improving the real-time nature of data requests and increasing request efficiency. Furthermore, in this data access system, data nodes do not interact with each other and do not affect each other. Even if an intermediate data node experiences an anomaly or failure, the leaf data nodes can still provide feedback, maximizing task execution reliability and feedback data integrity.

[0051] In some embodiments, since the data structure and data completeness of each data node are different, the data platform can generate a standard model as needed, and the data probe generates as rich sub-task execution results as possible that conform to the standard model based on the data feedback from the data node. For example, for the same information, there are 100 parameters under the standard model, data node A has 50 parameters, 46 of which are included in the standard model; data node B has 80 parameters, 70 of which are included in the standard model. Then, the data probe that queries data node A will feedback the sub-task execution results with 46 parameters that conform to the standard model, and the data probe that queries data node B will feedback the sub-task execution results with 70 parameters that conform to the standard model, thereby ensuring smooth service execution even when the data structure is different and the data is incomplete, improving the success rate, and reducing the requirements for the similarity and completeness of the data nodes.

[0052] Schematic diagrams of other embodiments of the data access system disclosed in the present invention are as follows: Figure 3 shown.

[0053] Data platform cluster 31 includes multiple data platforms 311-31p, where p is a positive integer greater than 1. Each data platform in data platform cluster 31 operates in a distributed logical manner. In some embodiments, data platform cluster 31 includes functions for business standard model management, virtual network probe topology management and monitoring, database instance management and monitoring, and virtual data service management and query management.

[0054] In some embodiments, the administrator can create a new standard model through the data platform cluster 31, and establish a mapping relationship between the standard model and the data node data structure for each data probe, so that the data probe can perform data conversion between the standard model and the calling rules of the data node; establish a data service, establish a data request standard model for the data service, and publish the data service for calling, thereby improving the scalability of the system.

[0055] In some embodiments, the data platform cluster 31 may send the correspondence between each data probe and the data node, as well as the rules (data structure) of each corresponding data node, to the corresponding data probe.

[0056] Data engine cluster 32 includes multiple data engines 321-32p, where q is a positive integer greater than 1. Each data platform in data engine cluster 32 operates in a distributed logical manner. In some embodiments, data engine cluster 32 is capable of parsing database statements (e.g., V-SQL), breaking down data requests, distributing subquery tasks, and aggregating subquery results.

[0057] Data probes 331-33n, where n is a positive integer greater than 1. In some embodiments, each data probe may include a data node's internal and external network proxy engine, a subquery execution engine, and a database instance information collection probe. In some embodiments, data probes 331-33n can receive configuration information from the data platform cluster 31 as the basis for subsequently executing request subtasks and providing feedback on subtask execution results.

[0058] In some embodiments, the data nodes accessed by the data probes 331-33n may not be connected to each other, thereby expanding the scope of data calls. In some embodiments, the data nodes accessed by the data probes 331-33n may have a hierarchical relationship, such as Figure 1 As shown, there is no need for data nodes to call each other, avoiding intrusion and impact on intermediate data nodes, and also avoiding task failure due to the performance of a certain intermediate data node, thereby improving the reliability of task execution and the integrity of feedback data.

[0059] This data access system supports standard model configurations based on the data platform, improving user-friendliness and scalability. Cluster deployment also enhances the system's load capacity and parallel processing capabilities. When the data structure of a leaf node changes, other data nodes do not need to be notified or modified, and the data warehouse service remains operational. Data synchronization is also unnecessary, ensuring the real-time performance of data queries and ensuring the reliability and efficiency of data requests.

[0060] In some embodiments, ZooKeeper can be used to implement distributed coordination among the data platform, data engine, and data probes. ZooKeeper is software that provides consistency services for distributed applications. Its capabilities include configuration maintenance, domain name services, distributed synchronization, and group services, simplifying the implementation of data access systems and improving their efficiency.

[0061] In some embodiments, each data engine can publish tasks based on MQ (Message Queue) rules, such as publishing the message queue of a requested subtask to a subtask Topic based on MQ's Topic mode (called "topic mode," also known as "publisher-subscriber mode"). Each data probe can filter tasks based on MQ rules, such as filtering the message queue from a subtask Topic based on preset message filtering rules to obtain its own requested subtask.

[0062] Such a system can use message queues to implement asynchronous task processing and system decoupling, thereby reducing service pressure while ensuring the reliability of task execution.

[0063] In some embodiments, each data engine can also collect execution results based on MQ rules. For example, based on the message queue of the result topic, the subtask execution results of the request subtask belonging to the same data request are obtained; based on the obtained subtask execution results, the request execution results are aggregated into a data platform data model. Each data probe can provide task execution result feedback based on MQ rules, such as publishing the subtask execution results to the result topic for collection by each data engine.

[0064] Such a system can use message queues to implement asynchronous data feedback and system decoupling, thereby reducing service pressure while ensuring the reliability of feedback data.

[0065] Schematic diagrams of some embodiments of various units in the data access system of the present disclosure are as follows: Figures 4A to 4C The data access system includes a data platform 41, a data engine 42 and a data probe 43.

[0066] The data platform 41 and the information connection between the data platform 41 and other units are as follows Figure 4A As shown:

[0067] The data platform 41 (e.g., virtual management platform v-ms) mainly includes configuration management 411, service engine 412, and network topology management 413. v-ms uses distributed cluster deployment to improve data service performance and reliability.

[0068] Configuration Management 411 provides management functions for standard models and mapping relationships (e.g., creation, update, decommissioning, and deployment). Through ZooKeeper distributed deployment, it ensures the configuration and coordination between V-MS and multiple V-Agents (virtual data probes). Configuration Management 411 also distributes standard models and mapping relationships to V-Engine (virtual data engine).

[0069] The service engine 412 provides data service management functions, including service registration, service monitoring, service launch, service shutdown, service processing, and a virtual data service query manager (supporting V-SQL queries written on the page). The service engine 412 receives service call requests and responses from external applications; issues query tasks to the V-engine; and receives query results returned by the V-engine.

[0070] The network topology management 413 can provide data node network topology management, visualization, database node health and data monitoring; receive monitoring data from all v-agents and store it in a relational database for statistical analysis and visualization.

[0071] The data engine 42 and the information connection between the data engine 42 and other units are as follows Figure 4B As shown:

[0072] The data engine (virtual data engine v-engine) 42 is deployed in a clustered distributed manner. In some embodiments, the number of appropriate v-engine instances in the cluster can be defined based on the query volume of the data service and the performance indicators of the actual service.

[0073] The v-engine primarily consists of a task receiver 421, a parser 422, and a data collector 423. Modules communicate with each other and with the v-agent via MQ. Each module operates in a multi-threaded manner, improving system throughput and utilization.

[0074] The task receiver 421 can receive the query task of v-ms, perform business verification of the task, and send the task MQ to the subtask topic after success.

[0075] Parser 422 can asynchronously consume the subtask MQ in the task Topic. The task is composed of V-SQL. Refer to the syntax rules of V-SQL. According to the parsing rules, the task MQ is parsed into multiple subtasks. Each subtask sends the subtask MQ to the subtask Topic for each v-agent node. The v-agent uses the MQ message filter rules to consume its own subtask MQ messages.

[0076] Data collector 423 consumes and parses MQ messages on the result topic. The v-agent sends the results of subtask execution to the result topic. A query task is split into multiple subtasks because the data collector needs to collect the results returned by multiple subtasks. The results are then aggregated into a data structure that conforms to a common standard model and returned to the v-ms.

[0077] The data probe 43 and the information connection between the data probe 43 and other units are as follows Figure 4C As shown:

[0078] The data probe (virtual data probe v-agent) 43 mainly functions as an intranet and intranet agent engine, a subquery execution engine, and a database instance information collection probe. The data probe 43 includes a configuration engine 431, a task engine 432, and a probe module 433.

[0079] The number of v-agents is consistent with the number of databases in the data network. A v-agent needs to be deployed in the intranet environment of each database.

[0080] The configuration engine 431 can receive the configuration information of the standard model and mapping relationship distributed by the v-ms in a monitoring mode.

[0081] The task engine 432 includes three important logic modules: task configuration parser, task processor, and data encapsulator.

[0082] The task configuration parser is primarily responsible for parsing v-SQL, a crucial component of subtasks. V-SQL has a detailed set of syntax rules to support diverse queries. The parser supports all v-SQL syntax rules, reads the configuration engine's general standard model and mapping relationships, parses and maps v-SQL (using the general model's data structure) into database node data structures, and passes it to the task processor.

[0083] The task processor can receive the database query instructions passed by the task configuration parser, access the data node data in the intranet, and return the query results to the data encapsulator;

[0084] The data encapsulator can receive the data returned by the task processor, read the general standard model and mapping relationship of the configuration engine, map the data of the data node into the data of the general standard model, and return it to the v-engine.

[0085] Since enterprises typically have internal and external networks, and business databases are typically deployed on the intranet, Probe 433 can act as an agent for both internal and external networks. Through task polling, it regularly checks the health status and data status of database nodes and feeds the results back to the V-MS. The probe can interact with the V-MS through asynchronous MQ, reducing the pressure on the V-MS server while ensuring data reliability.

[0086] This data access system, through a unique architectural design, distributed deployment, big data platform and other technical combinations, has changed the traditional database data sharing implementation method, greatly reducing the implementation and maintenance costs of human and physical resources, and improving the security of business databases and the real-time performance of data services.

[0087] In some embodiments, the data access system can be deployed through a virtual architecture, for example, using ZooKeeper scheduling, thereby avoiding the burden of physical device configuration and reducing maintenance costs.

[0088] A structural diagram of an embodiment of the data access system disclosed in the present invention is shown in FIG. Figure 5 As shown. The implementation body of each part in the data access system includes a memory 501 and a processor 502. Among them: the memory 501 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiment of the data access method executed by the corresponding part below. The processor 502 is coupled to the memory 501 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 502 is used to execute the instructions stored in the memory, which can improve the real-time performance of data requests and improve the efficiency of requests.

[0089] In one embodiment, it is also possible to Figure 6 As shown, each component of data access system 600 includes a memory 601 and a processor 602. Processor 602 is coupled to memory 601 via a BUS 603. Data access system 600 can also be connected to an external storage device 605 via a storage interface 604 to access external data, and can also be connected to a network or another computer system (not shown) via a network interface 606. A detailed description is omitted here.

[0090] In this embodiment, by storing data instructions in a memory and then processing the instructions through a processor, the real-time performance of data requests and the efficiency of requests can be improved.

[0091] The flowcharts of some embodiments of the data access method disclosed in the present invention are as follows: Figure 7 shown.

[0092] In step 701, the data platform sends a data request to the data engine. In some embodiments, the data request can be initiated by a user on the data platform or sent to the data platform through an external application.

[0093] In some embodiments, upon receiving a data request, the data platform performs data request validation; if validation passes, the data request is sent to the data engine. This validation may include analyzing the business type corresponding to the request, which can be achieved by determining whether the request complies with at least one standard business template of the data platform. In some embodiments, permissions verification may also be performed to determine whether the user and application corresponding to the request have permission to request the corresponding data.

[0094] In step 702, the data engine splits the data request, obtains multiple request subtasks, and sends the request subtasks to each data probe. In some embodiments, the data request can be split based on the target data node of the query, or based on the data probe corresponding to the target data node of the query, so that the same request subtask is only targeted at the same data node, or only at data nodes accessible by the same data probe, thereby ensuring that the data probe can smoothly and completely execute the obtained request subtasks.

[0095] In step 703, the data probe obtains the requested subtask, calls the corresponding data node to execute the requested subtask, generates the subtask execution result, and feeds the subtask execution result back to the data engine. In some embodiments, each data probe can preset message filtering rules. The data engine can provide all request subtasks to each data probe. The data probe then filters the request subtasks according to its own message filtering rules to obtain the request subtasks it needs to execute.

[0096] In step 704, the data engine aggregates the subtask execution results from each data probe, generates a request execution result, and feeds it back to the data platform. In some embodiments, the request execution result must conform to the data platform's standard model to facilitate its recognition. In some embodiments, the data platform can feed the request execution result back to external applications.

[0097] Through this method, the data engine can split the data request, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-intrusive access to the data nodes and eliminating the need for data synchronization operations, thereby improving the real-time performance of data requests and improving request efficiency.

[0098] In some embodiments, after receiving feedback data from the data node, the data probe can generate subtask execution results based on the feedback data and the data model of the data platform, thereby overcoming the inconsistencies in the data structure, language rules, etc. of each data node, ensuring that the data platform can obtain unified standard execution results, and improving the adaptability and application scope of the system.

[0099] In some embodiments, the data engine can publish the message queue of the request subtasks to a subtask topic. Based on pre-set message filtering rules, the data probe filters the message queue from the subtask topic and retrieves its own request subtasks. This method enables asynchronous communication between the data engine and data probes, reduces server load, achieves traffic peak shaving, and improves operational performance.

[0100] In some embodiments, after generating the subtask execution results, the data probe can publish the subtask execution results to the result Topic; the data engine aggregates the subtask execution results from each data probe to generate a request execution result, including: the data engine obtains the subtask execution results of the request subtasks belonging to the same data request according to the message queue of the result topic; based on the obtained subtask execution results, aggregates them into request execution results that conform to the data model of the data platform.

[0101] In some embodiments, the data platform can establish a standard model and send it to the data engine; the data platform can also generate the correspondence between each data probe and the data node, as well as the rules of each corresponding data node, and send them to the corresponding data probe; the data platform receives data requests from external applications.

[0102] Through this method, the data platform can support users to execute standard model configurations and synchronize them to relevant nodes. It can timely update data node rules and the correspondence with data probes, improve user-friendliness and scalability, and further improve the reliability of data request task execution. In addition, since multiple data probes can execute request subtasks in parallel, the execution efficiency of the task can be further improved.

[0103] Flowcharts of other embodiments of the data access method disclosed herein are as follows: Figure 8 shown.

[0104] In step 801, the data platform establishes a standard model and distributes it to the data engine. The standard model can include syntax standards, data structure standards, and identifier standards for the same meaning. The standard model can also be distributed to data probes so that they can generate subtask execution results that conform to the standard model. For example, if multiple data nodes use "1, 0" and "Y, N" to represent "yes" and "no," respectively, a unified standard, such as "1, 0," can be established.

[0105] In step 802, the data platform generates a correspondence between each data probe and data node, as well as rules for each corresponding data node, and distributes them to the corresponding data probe. In some embodiments, the rules for a data node may include the node's syntax rules, data structure rules, and the meaning of each identifier.

[0106] In step 803, the data platform receives a data request from an external application and performs data request validation. In some embodiments, this validation may include analyzing the business type corresponding to the request, which may be achieved by determining whether it complies with at least one standard business template of the data platform. In some embodiments, a permissions check may also be performed to determine whether the user and application corresponding to the request have permission to request the corresponding data.

[0107] In step 804, it is determined whether the verification is passed. If the verification is passed, step 805 is executed. Otherwise, the execution of the current request can be terminated and an error is fed back.

[0108] In step 805 , the data platform routes the data request to the data engine.

[0109] In step 806 , the data engine splits the data request, obtains multiple request subtasks, and publishes the message queue of the request subtask to the subtask Topic based on the Topic mode of MQ.

[0110] In step 807, the data probe filters the message queue from the subtask topic based on the preset message filtering rules and obtains its own request subtask.

[0111] In step 808 , the data probe calls the corresponding data node to execute the requested subtask and receives feedback data from the data node.

[0112] In step 809, the data probe generates a subtask execution result based on the feedback data and the data platform's data model (i.e., the standard model). In some embodiments, the data probe may perform result rule conversion on the feedback data based on the rules of the current data node and the data platform's standard model to generate a subtask execution result that conforms to the standard model.

[0113] In step 810 , based on the Topic mode of MQ, the data probe publishes the subtask execution result to the result Topic.

[0114] In step 811 , the data engine obtains the subtask execution results of the request subtask belonging to the same data request according to the message queue of the result Topic.

[0115] In step 812, the data engine aggregates the subtask execution results into a request execution result that conforms to the data platform's data model. In some embodiments, if the data request is a data query, the data in the subtask execution results can be aggregated to generate a request execution result that conforms to the data platform's standard model. In some embodiments, if the data request is data statistics, the statistical information in each subtask execution result is further aggregated to generate a request execution result that conforms to the data platform's standard model.

[0116] In step 813, after receiving the request execution result from the data engine, the data platform feeds back the request execution result to the external application.

[0117] Through this method, the data engine can split data requests, and the probes associated with the data nodes execute the corresponding request subtasks, thereby achieving non-invasive access to the data nodes and eliminating the need for data synchronization operations, improving the real-time nature of data requests and increasing request efficiency. In addition, in such a data access system, data nodes do not need to interact with each other and will not affect each other. When an intermediate data node experiences an exception or failure, the leaf data node can also feedback data, ensuring the reliability of task execution and the integrity of feedback data to the greatest extent possible. When the data structure of a leaf node changes, there is no need to notify other data nodes, and other data nodes do not need to make any modifications, which does not affect the use of data warehouse services. Message queues are used to implement asynchronous task processing and system decoupling, which reduces service pressure while ensuring the reliability of task execution.

[0118] In one embodiment, the present disclosure further proposes a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the method in the corresponding embodiment of the data access method. Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] The present disclosure has been described in detail so far. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0123] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and not to limit it. Although the present disclosure has been described in detail with reference to the preferred embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present disclosure can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solutions of the present disclosure, which should all be included in the scope of the technical solutions requested for protection in the present disclosure.

Claims

1. A data access system comprising: The data platform is configured to build data models and send them to the data engine and data probes; Generate the corresponding relationship between each data probe and data node, as well as the rules of each corresponding data node, and send them to the corresponding data probe; Receive data requests and send them to the data engine; receiving a request execution result fed back by the data engine that conforms to the data model; A data engine is configured to split the data request, obtain multiple request subtasks, and send the request subtasks to each data probe; Aggregating the subtask execution results from each data probe, generating a request execution result that conforms to the data model, and feeding it back to the data platform, wherein splitting the data request includes: splitting the data request according to the data probe corresponding to the target data node of the query, so that the same request subtask targets the data node that can be accessed by the same data probe; and Multiple data probes are configured to obtain the requested subtask; call the corresponding data node to execute the requested subtask, receive feedback data from the data node, perform result rule conversion on the feedback data based on the rules of the data node and the data model, generate a subtask execution result that conforms to the data model, and feed back the subtask execution result to the data engine.

2. The system according to claim 1, wherein: The data probes are deployed in a distributed manner, and each database's intranet environment corresponds to one data probe.

3. The system according to claim 1, wherein: At least one of the data platform or the data engine is deployed in a distributed cluster.

4. The system according to claim 1, wherein: The data probe is configured to: receiving feedback data from the data node; According to the feedback data, the subtask execution result is generated based on the data model of the data platform.

5. The system according to claim 1, wherein: The data engine is configured to: publish the message queue of the request subtask to the subtask topic; The data probe is configured to filter the message queue from the subtask topic based on a preset message filtering rule to obtain the request subtask belonging to itself.

6. The system according to claim 5, wherein: The data probe is further configured to publish the subtask execution result to a result topic; The data engine is further configured to: Obtaining, according to the message queue of the result topic, subtask execution results of the request subtask belonging to the same data request; According to the obtained subtask execution results, a request execution result that conforms to the data model of the data platform is aggregated.

7. The system according to any one of claims 1 to 6, wherein: The data platform is further configured to perform at least one of the following: Establishing a standard model and sending it to the data engine; Generate the corresponding relationship between each data probe and data node, as well as the rules of each corresponding data node, and send them to the corresponding relationship with the probe; Receive data requests from external applications; After receiving the request execution result from the data engine, feeding back the request execution result to the external application; or After receiving the data request, performing data request verification; If the verification is passed, the data request is sent to the data engine.

8. The system according to any one of claims 1 to 6, wherein: At least one of the data platform, the data engine, or the data probe is deployed in a virtualized manner.

9. A data access method comprising: The data platform establishes a data model and sends it to the data engine and data probes. It generates the correspondence between each data probe and data node, as well as the rules for each corresponding data node, and sends them to the corresponding data probes, sending data requests to the data engine. The data engine splits the data request, obtains multiple request subtasks, and sends the request subtasks to each data probe, wherein splitting the data request includes: splitting the data request according to the data probe corresponding to the target data node of the query, so that the same request subtask is targeted at the data node that can be accessed by the same data probe; The data probe obtains the requested subtask, calls the corresponding data node to execute the requested subtask, receives feedback data from the data node, performs result rule conversion on the feedback data based on the rules of the data node and the data model, generates a subtask execution result that conforms to the data model, and feeds back the subtask execution result to the data engine; The data engine aggregates the subtask execution results from each data probe, generates a request execution result that conforms to the data model, and feeds it back to the data platform.

10. The method according to claim 9, wherein: The generating subtask execution result includes: receiving feedback data from the data node; According to the feedback data, the subtask execution result is generated based on the data model of the data platform.

11. The method according to claim 9, wherein The sending of the request subtask to each data probe comprises: the data engine publishing the message queue of the request subtask to the subtask topic; The data probe acquiring the request subtask includes: the data probe filtering the message queue from the subtask topic based on a preset message filtering rule to acquire the request subtask belonging to itself.

12. The method according to claim 11, wherein Feeding back the subtask execution result to the data engine includes: the data probe publishing the subtask execution result to a result topic; The data engine aggregates the subtask execution results from each data probe and generates a request execution result, including: The data engine obtains the subtask execution results of the request subtask belonging to the same data request according to the message queue of the result topic; According to the obtained subtask execution results, a request execution result that conforms to the data model of the data platform is aggregated.

13. The method according to any one of claims 9 to 12, further comprising at least one of the following: The data platform establishes a standard model and sends it to the data engine; The data platform generates a correspondence between each data probe and data node, as well as rules for each corresponding data node, and sends them to the corresponding data probe; The data platform receives a data request from an external application; after receiving the request execution result from the data engine, feeds back the request execution result to the external application; or After receiving the data request, the data platform performs data request verification; if the verification passes, the data request is sent to the data engine.

14. A data access system comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 9 to 13 based on instructions stored in the memory.

15. A computer-readable storage medium having computer program instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 9 to 13 are implemented.

Images