Analysis method, apparatus and system for a database with shared analytical resources

Abstract

This invention discloses an analysis method, apparatus, and system for a database with shared analytical resources. Its features include: generating a task identifier corresponding to the analytical statement text input by a data analysis user based on at least one transactional data engine; querying the subset of data to be analyzed corresponding to the analytical statement text for each transactional data engine; storing the subset of data to be analyzed in a transaction result pool; generating a data analysis request based on the data storage address and task identifier of the transaction result pool for each transactional data engine, and sending the data analysis request to the analytical data cluster; receiving the data analysis results returned by the analytical data cluster and displaying the data analysis results to the data analysis user. This invention supports separate deployment of transactional data and analytical resources, sharing of analytical resources among multiple transactional data clusters, and supports individual scaling; it dynamically allocates analytical resources to tasks, improving resource utilization.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and in particular to an analysis method, apparatus, and system for a database with shared analytical resources. Background Technology

[0002] In database systems, transactional operations and analytical operations are two different types of data operations. Transactional operations primarily focus on processing and managing individual transactions. They typically require fast read and write capabilities to ensure high throughput and low latency. Analytical operations, on the other hand, focus on complex analysis, summarization, and reporting of data to support decision-making and business analysis. Analytical operations generally do not require low latency but need to be able to handle large datasets for efficient data analysis. Because the implementation principles of transactional and analytical operations differ significantly at the engine level, traditional database engines often excel at only one type of operation, or they may perform a simple combination that fails to achieve efficient data interoperability and resource isolation between the two. In existing technologies, if the transactional data engine and the analytical data engine are completely isolated at the engine level, the transactional data engine cannot efficiently submit data to the analytical data engine, or the analytical data engine may bypass the transactional data engine and load data directly from storage. This can lead to problems such as imprecise data filtering and potential data inconsistencies. If the transactional and analytical data engines are tightly coupled, the resource isolation between them is not high. The execution of jobs in the analytical data engine can significantly impact the operation of the transactional data engine. Furthermore, both the transactional and analytical data engines need to scale up and down simultaneously, which can easily lead to resource contention or waste. A traditional hybrid structure of transactional and analytical data engines offers high resource isolation but low resource utilization. Summary of the Invention

[0003] This invention provides an analysis method, apparatus, and system for a database with shared analytical resources, to solve the technical problems in the prior art where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up or down the two types of resources individually.

[0004] According to one aspect of the present invention, an analysis method for a database sharing analytical resources is provided, applied to at least one transactional data cluster; comprising:

[0005] Based on the analytical statement text input by the user, which is analyzed by at least one transaction data engine response data analysis, a task identifier corresponding to the analytical statement text is generated.

[0006] For each of the transaction data engines, query the subset of data to be analyzed corresponding to the analytical statement text; store the subset of data to be analyzed in the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed;

[0007] A data analysis request is generated based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and the data analysis request is sent to the analysis data cluster;

[0008] The system receives the data analysis results returned by the data analysis cluster in response to the data analysis request and displays the data analysis results to the data analysis user.

[0009] According to another aspect of the present invention, an analysis method for a database sharing analytical resources is provided, applied to an analytical data cluster; comprising:

[0010] The analysis coordinator based on the analysis data cluster receives and processes data analysis requests, and allocates at least one analysis data engine to the data analysis request.

[0011] A data analysis task is generated based on at least one data storage address and a task identifier, and the data analysis task is issued to each of the data analysis engines respectively.

[0012] For each of the analysis data engines, a data retrieval request is generated based on the work number of the analysis data engine and the number of engine allocations for the analysis data engine corresponding to the data analysis request, and the data retrieval request is sent to each of the data storage addresses.

[0013] According to one aspect of the present invention, an analysis apparatus for a database sharing analytical resources is provided, deployed in at least one transactional data cluster; comprising:

[0014] The task response module is used to generate a task identifier corresponding to the analytical statement text input by the user based on at least one transaction data engine response data analysis.

[0015] The query module is used to query the subset of data to be analyzed corresponding to the analytical statement text for each of the transaction data engines; and to store the subset of data to be analyzed into the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed.

[0016] The communication module is used to generate a data analysis request based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and send the data analysis request to the analysis data cluster;

[0017] The display module is used to receive the data analysis results returned by the analysis data cluster from the data analysis request, and to display the data analysis results to the data analysis user.

[0018] According to one aspect of the present invention, an analysis apparatus for a database sharing analytical resources is provided, deployed in an analytical data cluster; comprising:

[0019] An analysis coordination module is used to receive and process data analysis requests based on the analysis coordinator of the analysis data cluster, and to allocate at least one analysis data engine for the data analysis request.

[0020] The allocation module is used to generate data analysis tasks based on at least one data storage address and a task identifier, and to distribute the data analysis tasks to each of the data analysis engines respectively.

[0021] The analysis execution module is used to generate a data retrieval request for each of the analysis data engines based on the work number of the analysis data engine and the engine allocation quantity of the analysis data engine corresponding to the data analysis request, and send the data retrieval request to each of the data storage addresses.

[0022] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of an analysis method for a database with shared analysis resources is provided as an embodiment of the present invention;

[0026] Figure 2 A flowchart of an analysis method for a database with shared analysis resources is provided as an embodiment of the present invention;

[0027] Figure 3 A flowchart of an analysis method for a database with shared analysis resources is provided as an embodiment of the present invention;

[0028] Figure 4 A flowchart illustrating an analysis method for a database with shared analytical resources, provided as an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of the structure of an analysis device for a database with shared analysis resources provided in an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of the structure of an analysis device for a database sharing analysis resources, provided in an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram of the structure of an analysis system for a database with shared analysis resources, provided as an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Figure 1 This invention provides a flowchart of an analysis method for a database with shared analysis resources. This embodiment is applicable to situations where a transactional data cluster responds to the data analysis needs of data analysis users. This embodiment can be executed within a transactional data cluster. The analysis device for the database with shared analysis resources can be implemented in hardware and / or software, and can be configured within the transactional data cluster. Figure 1 As shown, the method includes:

[0035] S110. Based on the analytical statement text input by the user through at least one transaction data engine response data analysis, generate a task identifier corresponding to the analytical statement text.

[0036] Among them, the transactional data engine can be a database that processes and manages individual transactions or transactions, supporting transactional operations (OLTP - Online Transaction Processing, TP) for daily business activities.

[0037] Optionally, the components related to transactional operations in the transactional data engine may include a hybrid transactional analysis query plan parser, a transaction processing executor, and a transaction result pool. The hybrid transactional analysis query plan parser is used to parse the data requests from data analysis users and compile them into execution plans corresponding to the data requests. The transaction processing executor is responsible for transactional queries and produces the data corresponding to the data requests. The transaction result pool is responsible for caching the data corresponding to the data requests. If the data request is an analytical statement text, the transaction result pool caches the data and waits for the analysis data engine to pull the data.

[0038] Analytical text can be text containing statements that perform complex analysis on data; for example, analytical text can be used for social network analysis, recommendation systems, and path analysis.

[0039] Optionally, transactional statement text can be statement text that performs transactional processing on data. For example, transactional statement text can be used to process the addition, deletion, and modification operations of nodes and relationships in a graph.

[0040] In this context, data analysis users can be users of the database client or driver corresponding to the transactional data cluster. It should be noted that data analysis users can send analytical statement text and / or transactional statement text to the transactional data cluster. The transactional data cluster can parse the analytical statement text and transactional statement text, compile them into execution plans corresponding to the analytical statement text and transactional statement text, and generate corresponding task IDs as task identifiers. The task identifier corresponding to the analytical statement text is called the analysis task identifier; the task identifier corresponding to the transactional statement text is called the transaction task identifier.

[0041] Optionally, a transactional data cluster can be a cluster of at least one transactional data engine capable of collectively processing transactional operations.

[0042] Optionally, data analysis users can input analytical statement text into the transactional data cluster through the database client page provided by the database client, and then send the analytical statement text to the transactional data cluster.

[0043] Specifically, after the transaction data cluster receives the analytical statement text, at least one transaction data engine in the transaction data cluster responds to the analytical statement text input by the data analysis user, processes the analytical statement text, compiles it into an execution plan corresponding to the analytical statement text, and generates a task identifier corresponding to the analytical statement text.

[0044] S120. For each of the transaction data engines, query the subset of data to be analyzed corresponding to the analytical statement text; store the subset of data to be analyzed in the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed.

[0045] The subset of data to be analyzed can be the query results constructed from data that conforms to the analytical statement text within the data stored in the transactional data engine. For example, the data stored in the transactional data engine could be user profiles of a video website; the analytical statement text could be analyzing the age distribution of users of the video website in a certain region. In this case, querying the transactional data engine for all users and their ages in that region would yield the query results for all users and their ages in that region; and all users and their ages in that region would constitute the subset of data to be analyzed within the analytical statement text.

[0046] The transaction result pool can be set up by the transactional data engine to cache a subset of data to be analyzed. It should be noted that each transactional data engine sets up its own transaction result pool.

[0047] The data to be analyzed can be obtained from analytical statement text queries within the transactional data engine. It's important to note that during an analytical statement text query, the transactional data engine retrieves each data point matching the analytical statement text one by one, storing each data point into the transaction result pool to form a subset of the data to be analyzed.

[0048] Optionally, the subset of data to be analyzed includes at least one set of data to be analyzed. For example, the data stored in the transactional data engine could be user profiles of a video website; the analytical statement text could be analyzing the age distribution of users of the video website in a certain region. In this case, querying the transactional data engine for all users and their ages in that region, and finding a single user and their age in that region, would constitute the data to be analyzed. After the query is completed, the query result would be all users and their ages in that region; all users and their ages in that region would be the subset of data to be analyzed in the analytical statement text.

[0049] Optionally, when the transactional data cluster receives the analytical statement text, since the data corresponding to the analytical statement text may be stored in different transactional data engines of the transactional data cluster, each transactional data engine performs a data query based on the analytical statement text and stores the query results in the transaction result pool of each transactional data engine.

[0050] Specifically, each transaction data engine receives the analytical statement text, queries the subset of data to be analyzed corresponding to the analytical statement text, and stores the subset of data to be analyzed into the transaction result pool.

[0051] Optionally, the transaction result pool includes at least one storage data slot. This slot can be used to store various data to be analyzed. After retrieving the data to be analyzed, it is stored in a storage data slot. It should be noted that the transaction result pool can be divided into any number of storage data slots. These slots can be numbered, and each slot has a corresponding slot number. Each piece of data to be analyzed can be placed in any storage data slot, or in a specific assigned slot. Each storage data slot will only be retrieved by one analysis data engine. For example, the transaction result pool can be divided into 1024 or 65526 ​​storage data slots, numbered from 0, 1, 2, up to the total number of slots minus one, i.e., 1023 or 65525.

[0052] Optionally, in another optional embodiment of the present invention, storing the subset of data to be analyzed into a transaction result pool includes:

[0053] For each piece of data to be analyzed, identify the target data slot corresponding to the data to be analyzed in the storage data slot; and sequentially store each piece of data to be analyzed into the target data slot of each piece of data to be analyzed.

[0054] The target data slot can be the storage slot where the data to be analyzed needs to be stored. Optionally, when storing or caching the data to be analyzed, a target data slot can be matched, and the data to be analyzed can be stored in that target data slot.

[0055] Optionally, each piece of data to be analyzed has a target data slot, but multiple pieces of data to be analyzed can exist in the same target data slot.

[0056] Optionally, after the transaction data engine retrieves each piece of data to be analyzed, it identifies each piece of data to be analyzed, matches each piece of data to be analyzed with a storage data slot as the target data slot, and stores each piece of data to be analyzed into the target data slot of each piece of data to be analyzed in turn.

[0057] Specifically, for each piece of data to be analyzed, the target data slot corresponding to the data to be analyzed is identified in the storage data slot; each piece of data to be analyzed is then stored into the target data slot of each piece of data to be analyzed in turn.

[0058] Optionally, in another optional embodiment of the present invention, identifying the target data slot corresponding to the data to be analyzed in the data storage slot includes:

[0059] The data to be analyzed is hashed using a preset hash function to obtain the hash value corresponding to the data to be analyzed.

[0060] Obtain the data slot number of each of the storage data slots, and determine the storage data slots whose data slot numbers are the same as the hash calculation value as the target data slots of the data to be analyzed.

[0061] The preset hash function can be a consistent hash function.

[0062] The hash value can be the hash value calculated from the data to be analyzed using a hash function.

[0063] Optionally, for each piece of data to be analyzed, a hash calculation is performed on the data to be analyzed using a preset hash function to obtain the hash calculation value corresponding to the data to be analyzed, and the data slot number with the same hash calculation value is used as the target data slot.

[0064] Specifically, a hash calculation is performed on the data to be analyzed using a preset hash function to obtain the hash calculation value corresponding to the data to be analyzed; the data slot number of each storage data slot is obtained, and the storage data slot with the same data slot number and hash calculation value is determined as the target data slot of the data to be analyzed.

[0065] S130. Generate a data analysis request based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and send the data analysis request to the analysis data cluster.

[0066] The data storage address can be the storage address of the subset of data to be analyzed stored in the transaction result pool of the transaction data engine. It should be noted that the data storage address of the transaction data engine points to the transaction result pool, and the analysis data engine can obtain the subset of data to be analyzed stored in the transaction result pool by accessing the data storage address.

[0067] Specifically, a data analysis request can be a request notification sent from the transactional data cluster to the analytical data cluster. A data analysis request consists of the data storage address of each transactional data engine responding to the analytical statement text, the task identifier, and the analysis requirements corresponding to the analytical statement text.

[0068] Optionally, the transactional data cluster can also assess the amount of data to be analyzed in the analytical statement text, and include the amount of data to be analyzed in the data analysis request to notify the analytical data cluster.

[0069] Optionally, the transaction data cluster parses the analytical statement text to obtain the analytical requirements of the analytical statement text.

[0070] Optionally, the transactional data cluster can also set a data return strategy for each transactional data engine, and include this strategy in the data analysis request to notify the analysis data cluster. Specifically, the data return strategy is the strategy for returning the data analysis results to the transactional data cluster.

[0071] Specifically, the transaction data cluster generates a data analysis request based on the data storage address and task identifier of the transaction result pool of each transaction data engine, and sends the data analysis request to the analysis data cluster.

[0072] S140. Receive the data analysis results returned by the analysis data cluster from the data analysis request, and display the data analysis results to the data analysis user.

[0073] The data analysis results can be derived from the analytical statement text and the data to be analyzed by the analytical data cluster. It should be noted that after the analytical data cluster obtains the data to be analyzed from each transaction data engine in the transaction data cluster, it performs joint analysis on the data to be analyzed based on the analytical statement text to obtain the data analysis results.

[0074] Optionally, after the analytical data cluster analyzes the data to be analyzed and obtains the data analysis results, it returns the data analysis results to the transactional data cluster, which then displays the data analysis results in the database client of the data analysis user.

[0075] Optionally, if the data analysis results are returned synchronously, the transactional data cluster displays a waiting animation in the database client of the data analysis user while waiting for the data analysis results. After the data analysis results are obtained, the waiting animation ends and the data analysis results are displayed.

[0076] Optionally, if the data analysis results are returned asynchronously, the transactional data cluster does not display the waiting animation for analysis in the database client of the data analysis user. Instead, it loads the data analysis results gradually in a streaming manner until the data cluster completes all analysis and loads all the data analysis results.

[0077] Specifically, the transactional data cluster receives the data analysis results returned by the analysis data cluster from the data analysis request and displays the data analysis results to the data analysis user.

[0078] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0079] Figure 2This invention provides a flowchart of an analysis method for a database with shared analysis resources. This embodiment is applicable to situations where an analysis data cluster responds to data analysis requests from various transaction data clusters. This embodiment can be executed within an analysis data cluster. The analysis device for the database with shared analysis resources can be implemented in hardware and / or software, and can be configured within the analysis data cluster. Figure 2 As shown, the method includes:

[0080] S210. The analysis coordinator based on the analysis data cluster receives and processes data analysis requests, and allocates at least one analysis data engine for the data analysis request.

[0081] Optionally, the analysis data cluster can be used to perform large-scale data analysis tasks. The analysis data cluster consists of at least one analysis data engine and an analysis coordinator. The analysis coordinator can be a data analysis request in the transaction data cluster within the analysis data cluster. The analysis coordinator can be a specific computing device with computing capabilities; or any analysis data engine can be specified as the analysis coordinator. The analysis data engine can pull data, execute the analysis tasks corresponding to the analytical statement text, and write the data analysis results back to a specified location in the transaction data cluster.

[0082] Optionally, the analytics coordinator of the analytics data cluster can communicate with multiple transactional data clusters to enable the analytics data cluster to be shared across multiple transactional data clusters.

[0083] Optionally, the analytical data engine can consist of an input controller, a task executor, and a result controller. The input controller can be used to pull data, the task executor can be used to execute analytical tasks corresponding to analytical statement texts, and the result controller can be used to write the data analysis results back to a specified location in the transactional data cluster.

[0084] Specifically, after receiving a data analysis request, the analysis coordinator of the analysis data cluster processes the data analysis request, assesses the number of analysis data engines required to process the data analysis request, and allocates at least one analysis data engine to the data analysis request.

[0085] S220. Generate a data analysis communication message based on at least one data storage address and task identifier, and send the data analysis communication message to each of the data analysis engines respectively.

[0086] Specifically, data analysis communication messages can be used to notify the data analysis engine to process data analysis requests. It should be noted that a data analysis communication message consists of a task identifier, the number of data analysis engines assigned to handle the data analysis request, the job number of each data analysis engine handling the request, and the data storage address of each data analysis engine.

[0087] Optionally, when the analysis coordinator of the analysis data cluster allocates more than one analysis data engine to a data analysis request, the analysis coordinator will assign a work number to each allocated analysis data engine and record the number of engine allocations for each analysis data engine. The work number can be the identifier corresponding to the data analysis request when the analysis data engine processes it, and the number of engine allocations can be the total number of analysis data engines allocated to the data analysis request by the analysis coordinator of the analysis data cluster. It should be noted that within the same data analysis request, the work number uniquely corresponds to one analysis data engine; however, the work number of the analysis data engine can be different in different data analysis requests.

[0088] Specifically, a data analysis communication message is generated based on at least one data storage address and a task identifier, and the data analysis communication message is sent to each of the data analysis engines.

[0089] S230. For each of the analysis data engines, a data retrieval request is generated based on the work number of the analysis data engine and the engine allocation quantity of the analysis data engine corresponding to the data analysis request, and the data retrieval request is sent to each of the data storage addresses.

[0090] Specifically, a data retrieval request can be a communication request from an analytical data engine to a transactional data engine to retrieve the data to be analyzed from the transaction result pool of that transactional data engine. It should be noted that the analytical data engine generates the data retrieval request based on the work number assigned to the data analysis request by the analytical data cluster's analytical coordinator and the number of engines allocated to it.

[0091] Optionally, when the analysis data engine generates a data retrieval request, a task identifier can be added to the data retrieval request to inform the transaction data engine which data to be analyzed needs to be retrieved.

[0092] Optionally, since the analytics data engine can synchronously pull data from at least one transactional data engine when pulling data, the analytics data engine can generate a data pull request for each transactional data engine and send the data pull request to each transactional data engine that needs to pull data.

[0093] Optionally, the fractional data engine can also generate only one data fetch request and send it to each transactional data engine that needs to fetch data.

[0094] Specifically, for each analysis data engine, a data retrieval request is generated based on the analysis data engine's job number and the number of engine allocations for the analysis data engine corresponding to the data analysis request, and the data retrieval request is sent to each data storage address.

[0095] Optionally, in another optional embodiment of the present invention, each of the analysis data engines receives at least one piece of data to be analyzed sent by at least one transaction data engine; each of the analysis data engines receives a data transmission completion identifier sent by all the transaction data engines; data analysis is performed on at least one piece of data to be analyzed based on the data analysis task to obtain data analysis results; and the data analysis results are returned to at least one transaction data cluster corresponding to the data analysis request.

[0096] The data sending completion flag can be generated by the transactional data engine after it has queried all the data to be analyzed that matches the analytical statement text. This completion flag informs the analytical data engine that no new data to be analyzed will be added to the transaction result pool.

[0097] Optionally, since the analysis data engine pulls the data to be analyzed from the transaction result pool of the transaction data engine in a streaming manner, it stops pulling data from the transaction data engine after receiving a completion flag from the transaction data engine.

[0098] Optionally, after each analysis data engine receives the data transmission completion flag sent by each transaction data engine, it is considered that each analysis data engine has finished pulling all the data to be analyzed, and each analysis data engine needs to perform data analysis on all the pulled data to be analyzed.

[0099] Specifically, each analysis data engine receives at least one piece of data to be analyzed from at least one transaction data engine; each analysis data engine receives data transmission completion identifiers from all transaction data engines; performs data analysis on at least one piece of data to be analyzed based on the data analysis task to obtain data analysis results; and returns the data analysis results to at least one transaction data cluster corresponding to the data analysis request.

[0100] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0101] Figure 3 This invention provides a flowchart of an analysis method for a database with shared analytical resources, applicable to the entire process of a transactional data cluster and an analytical data cluster processing an analytical statement text. Figure 3 As shown, the method includes:

[0102] S310. Based on the analytical statement text input by the user through at least one transaction data engine response data analysis, generate a task identifier corresponding to the analytical statement text.

[0103] S320. For each of the transaction data engines, query the subset of data to be analyzed corresponding to the analytical statement text; store the subset of data to be analyzed in the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed.

[0104] S330. Generate a data analysis request based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and send the data analysis request to the analysis data cluster.

[0105] S340. The analysis coordinator based on the analysis data cluster receives and processes data analysis requests, and allocates at least one analysis data engine to the data analysis request.

[0106] S350. Generate a data analysis communication message based on at least one data storage address and task identifier, and send the data analysis communication message to each of the data analysis engines respectively.

[0107] S360. For each of the analysis data engines, a data retrieval request is generated based on the work number of the analysis data engine and the number of engine allocations for the analysis data engine corresponding to the data analysis request, and the data retrieval request is sent to each of the data storage addresses.

[0108] S370. For each of the transaction data engines, respond to and process at least one received data retrieval request, and determine the job number and engine allocation quantity of the analysis data engine corresponding to each data retrieval request.

[0109] Optionally, when the transaction data engine of the transaction data cluster receives data retrieval requests from various analysis data engines, if a task identifier exists in the data retrieval request, it identifies the task identifier and determines the data to be scored corresponding to the task identifier.

[0110] Specifically, when the transaction data engine of the transaction data cluster receives data retrieval requests from various analysis data engines, it responds to and processes the data retrieval requests from each analysis data engine, determines the work number of the analysis data engine corresponding to each data analysis request, and the total number of engine allocations for all analysis data engines.

[0111] S380. Based on the working number and the number of engines allocated to the analysis data engine, an analysis data engine is allocated to each of the storage data slots, and the data to be analyzed in the storage data slots is sent to the analysis data engine corresponding to the storage data slots in the form of a data stream.

[0112] Optionally, when the analysis data engine pulls the data to be analyzed from each storage data slot, a unique analysis data engine is assigned to each storage data slot based on the analysis data engine's job number and the number of engines allocated, thereby improving the efficiency of data retrieval and preventing data retrieval conflicts.

[0113] Optionally, during the data retrieval process, the transactional data engine continues to query the data to be analyzed corresponding to the analytical statement text, places the data to be analyzed into the target data slot corresponding to the data to be analyzed, and the analytical data engine retrieves the data to be analyzed from the storage data slot in the form of a data stream, so as to achieve synchronous query and retrieval.

[0114] Optionally, if the analysis data engine finds that there is no data to be analyzed in a storage data slot, it will wait for the transaction data engine to store the data to be analyzed in the storage data slot until the transaction data engine notifies the analysis data engine that the data sending is complete, at which point it will stop pulling data from the storage data slot.

[0115] Specifically, an analysis data engine is assigned to each storage data slot based on the work number and engine allocation quantity of the analysis data engine, and the data to be analyzed in the storage data slot is sent to the corresponding analysis data engine in the form of a data stream.

[0116] Optionally, in another optional embodiment of the present invention, the allocation of an analysis data engine to each of the storage data slots based on the job number and the number of engines allocated by the analysis data engine includes:

[0117] The modulo operation is performed on each of the data slot numbers and the engine allocation quantity to determine the data retrieval number corresponding to each of the storage data slots;

[0118] An analysis data engine is assigned to each of the storage data slots based on the data retrieval number and the job number.

[0119] Optionally, in this invention, the number of storage data slots is greater than the number of engines allocated to the analysis data engine. The data retrieval number corresponding to each storage data slot is determined by performing a modulo calculation on each data slot number and the number of engines allocated. Each storage data slot is then evenly allocated to each analysis data engine. Each analysis data engine can be allocated to multiple storage data slots, and the difference in the number of storage data slots allocated to each analysis data engine is minimal.

[0120] Optionally, after assigning a data fetch number to the analysis data engine, the transaction data engine finds an analysis data engine with the same data fetch number and restores the data to be analyzed from the storage data slot to that analysis data engine.

[0121] Specifically, the modulo operation is performed on each data slot number and the number of engines allocated to determine the data retrieval number corresponding to each storage data slot; and an analysis data engine is allocated to each storage data slot based on the data retrieval number and the working number.

[0122] S390, Receive at least one piece of data to be analyzed from at least one transaction data engine through each of the analysis data engines.

[0123] S3100: Each of the analysis data engines receives a data transmission completion identifier sent by all the transaction data engines, and performs data analysis on at least one of the data to be analyzed based on the data analysis task to obtain data analysis results;

[0124] S3110. Return the data analysis results to at least one transaction data cluster corresponding to the data analysis request.

[0125] S3120: Receive the data analysis results returned by the analysis data cluster from the data analysis request, and display the data analysis results to the data analysis user.

[0126] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0127] Figure 4 A flowchart illustrating a database analysis method for shared analysis resources provided in an embodiment of the present invention; as shown below. Figure 4 As shown, the database client sends analytical statement text to the transactional data cluster. The transactional data engine 401 and transactional data engine 402 of the transactional data cluster query the stored subset of data to be analyzed based on the analytical statement text. The transactional data cluster notifies the analytical data cluster to retrieve the data and sends a data analysis request to the analytical data cluster. The analytical data cluster's analytical coordinator receives and processes the data analysis request and notifies the analytical data engine 403. The analytical data engine 403 pulls the data of the subset of data to be analyzed from the transactional data engine 401 and transactional data engine 402 respectively, performs data analysis, and returns the data analysis results to the transactional data cluster.

[0128] Figure 5 This is a schematic diagram of the structure of an analysis device for a database with shared analysis resources provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes: a task response module 510, a query module 520, a communication module 530, and a display module 540; wherein,

[0129] The task response module 510 is used to generate a task identifier corresponding to the analytical statement text based on the analytical statement text input by the user, which is analyzed by at least one transaction data engine response data analysis.

[0130] The query module 520 is used to query the subset of data to be analyzed corresponding to the analytical statement text for each of the transaction data engines; and to store the subset of data to be analyzed into the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed.

[0131] Communication module 530 is used to generate a data analysis request based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and send the data analysis request to the analysis data cluster;

[0132] The display module 540 is used to receive the data analysis results returned by the analysis data cluster from the data analysis request, and to display the data analysis results to the data analysis user.

[0133] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0134] Optionally, the query module 520 is specifically used for: the transaction result pool including at least one storage data slot;

[0135] For each piece of data to be analyzed, identify the target data slot corresponding to the data to be analyzed in the storage data slot; and sequentially store each piece of data to be analyzed into the target data slot of each piece of data to be analyzed.

[0136] Optionally, the query module 520 is further configured to: perform hash calculation on the data to be analyzed using a preset hash function to obtain the hash calculation value corresponding to the data to be analyzed;

[0137] Obtain the data slot number of each of the storage data slots, and determine the storage data slots whose data slot numbers are the same as the hash calculation value as the target data slots of the data to be analyzed.

[0138] Optionally, the device further includes: a data slot module and a data retrieval module; wherein,

[0139] The allocation module is used to respond to and process at least one received data retrieval request for each of the transaction data engines, and determine the work number and engine allocation quantity of the analysis data engine corresponding to each data retrieval request;

[0140] The data retrieval module is used to allocate an analysis data engine to each storage data slot based on the work number and the number of engines allocated by the analysis data engine, and to send the data to be analyzed in the storage data slot to the analysis data engine corresponding to the storage data slot in the form of a data stream.

[0141] The data slot module is specifically used for:

[0142] Modulo calculation is performed on each of the data slot numbers and the number of engines allocated to determine the data retrieval number corresponding to each of the storage data slots; an analysis data engine is allocated to each of the storage data slots based on the data retrieval number and the work number.

[0143] The database analysis device for shared analysis resources provided in the embodiments of the present invention can execute the database analysis method for shared analysis resources provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0144] Figure 6 A schematic diagram of the structure of an analysis device for a database with shared analysis resources provided in an embodiment of the present invention is shown below. Figure 6 As shown, the device includes: an analysis and coordination module 610, an allocation module 620, and an analysis and execution module 630; wherein,

[0145] The analysis coordination module 610 is used to receive and process data analysis requests based on the analysis coordinator of the analysis data cluster, and to allocate at least one analysis data engine for the data analysis request.

[0146] The allocation module 620 is used to generate a data analysis communication message based on at least one data storage address and a task identifier, and to send the data analysis communication message to each of the data analysis engines respectively.

[0147] The analysis execution module 630 is used to generate a data retrieval request for each of the analysis data engines based on the working number of the analysis data engine and the engine allocation quantity of the analysis data engine corresponding to the data analysis request, and send the data retrieval request to each of the data storage addresses.

[0148] The technical solution of this invention reduces the resource idle rate of the analysis data cluster and improves the utilization rate of analytical resources by setting up a shared analysis data cluster shared among multiple transaction data clusters. By setting an analysis coordinator for the analysis data engine in the analysis data cluster, the analysis data cluster and the transaction data cluster can be allocated and scaled up / down independently. This allows for matching analysis data engine resources during analysis jobs and not allocating them when no analysis jobs are needed, thus decoupling resource utilization. Furthermore, the data query results of the transaction data cluster are cached using consistent hashing, waiting for the analysis data engine to pull them. The analysis data engine can concurrently pull non-duplicate data, and due to the consistent hashing algorithm, it can ensure that the data can be analyzed directly without repositioning after being pulled, greatly improving analysis efficiency. This solves the technical problems in existing technologies where analytical computing resources and transaction processing computing resources in database engines are too highly coupled, analytical resources cannot be shared, and it is difficult to scale up and down the two types of resources independently. This invention supports the separate deployment of transaction data and analytical resources, with analytical resources shared among multiple transaction data clusters and supporting individual scaling up and down; it also dynamically allocates analytical resources to tasks, improving resource utilization.

[0149] Optionally, the device further includes: a data receiving module, a data analysis module, and a data response module; wherein,

[0150] The data receiving module is used to receive at least one piece of data to be analyzed sent by at least one transaction data engine through each of the analysis data engines;

[0151] The data analysis module is configured to receive a data transmission completion identifier sent by all the transaction data engines in each of the analysis data engines, perform data analysis on at least one of the data to be analyzed based on the data analysis task, and obtain data analysis results.

[0152] The data response module is used to return the data analysis results to at least one transaction data cluster corresponding to the data analysis request.

[0153] The database analysis device for shared analysis resources provided in the embodiments of the present invention can execute the database analysis method for shared analysis resources provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0154] Figure 7 This is a schematic diagram of the structure of an analysis system for a database with shared analysis resources, provided as an embodiment of the present invention. For example... Figure 7 As shown: It includes at least one transactional data cluster and one analytical data cluster; the transactional data cluster includes at least one transactional data engine, and the analytical data cluster includes at least one analytical coordinator and at least one analytical data engine; wherein,

[0155] The transaction analysis cluster includes transaction analysis cluster 701, transaction analysis cluster 702 and transaction analysis cluster 703; transaction analysis cluster 1 includes 5 transaction analysis engines, transaction analysis cluster 2 includes 3 transaction analysis engines and transaction analysis cluster 3 includes 2 transaction analysis engines; each transaction data cluster can execute the database access method provided in any embodiment of the present invention.

[0156] The analysis data cluster includes an analysis coordinator and four analysis data engines; the analysis data cluster is capable of executing the database access methods provided in any embodiment of the present invention.

[0157] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An analysis method for a database with shared analytical resources, characterized in that, Applicable to at least one transactional data cluster; including: Based on the analytical statement text input by the user, which is analyzed by at least one transaction data engine response data analysis, a task identifier corresponding to the analytical statement text is generated. For each of the transaction data engines, query the subset of data to be analyzed corresponding to the analytical statement text; store the subset of data to be analyzed in the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed; A data analysis request is generated based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and the data analysis request is sent to the analysis data cluster. Receive the data analysis results returned by the data analysis cluster from the data analysis request, and display the data analysis results to the data analysis user; The transaction result pool includes at least one data storage slot; The step of storing the subset of data to be analyzed into the transaction result pool includes: For each piece of data to be analyzed, identify the target data slot corresponding to the data to be analyzed in the storage data slot; and sequentially store each piece of data to be analyzed into the target data slot of each piece of data to be analyzed. Before receiving the data analysis results returned by the analysis data cluster based on the data analysis request and displaying the data analysis results to the data analysis user, the process includes: For each of the transaction data engines, respond to and process at least one received data fetch request, and determine the job number and engine allocation quantity of the analysis data engine corresponding to each data fetch request; Based on the job number and the number of engines allocated to each storage data slot, an analysis data engine is assigned, and the data to be analyzed in the storage data slot is sent to the analysis data engine corresponding to the storage data slot in the form of a data stream.

2. The method according to claim 1, characterized in that, The step of identifying the target data slot corresponding to the data to be analyzed in the storage data slot includes: The data to be analyzed is hashed using a preset hash function to obtain the hash value corresponding to the data to be analyzed. Obtain the data slot number of each of the storage data slots, and determine the storage data slots whose data slot numbers are the same as the hash calculation value as the target data slots of the data to be analyzed.

3. The method according to claim 1, characterized in that, The allocation of an analysis data engine to each storage slot based on the job number and the number of engines allocated by the analysis data engine includes: The modulo operation is performed on each data slot number and the number allocated by the engine to determine the data retrieval number corresponding to each storage data slot; An analysis data engine is assigned to each of the storage data slots based on the data retrieval number and the job number.

4. An analysis method for a database with shared analytical resources, characterized in that, Applied to analyzing data clusters; including: The analysis coordinator based on the analysis data cluster receives and processes data analysis requests, and allocates at least one analysis data engine to the data analysis request. Data analysis communication messages are generated based on at least one data storage address and a task identifier, and the data analysis communication messages are sent to each of the data analysis engines respectively. For each of the analysis data engines, a data retrieval request is generated based on the work number of the analysis data engine and the number of engine allocations for the analysis data engine corresponding to the data analysis request, and the data retrieval request is sent to each of the data storage addresses; Each of the aforementioned analysis data engines receives at least one piece of data to be analyzed sent by at least one transaction data engine; Each of the analytical data engines receives a data transmission completion identifier sent by all the transaction data engines, and performs data analysis on at least one of the data to be analyzed based on the data analysis task to obtain data analysis results; The data analysis results are returned to at least one transactional data cluster corresponding to the data analysis request; The step of receiving at least one piece of data to be analyzed from at least one transaction data engine through each of the analysis data engines includes: The analysis data engine pulls at least one of the data to be analyzed from the storage data slot in the form of a data stream; If the analysis data engine finds that the data to be analyzed is not in the storage data slot, it chooses to wait for the transaction data engine to store the data to be analyzed in the storage data slot until the transaction data engine notifies the analysis data engine that the data sending is complete, then it stops pulling data from the storage data slot.

5. An analysis device for a database with shared analytical resources, characterized in that, Deployed in at least one transactional data cluster; including: The task response module is used to generate a task identifier corresponding to the analytical statement text input by the user based on at least one transaction data engine response data analysis. The query module is used to query the subset of data to be analyzed corresponding to the analytical statement text for each of the transaction data engines; and to store the subset of data to be analyzed into the transaction result pool; the subset of data to be analyzed includes at least one set of data to be analyzed. The communication module is used to generate a data analysis request based on the data storage address of the transaction result pool of each transaction data engine and the task identifier, and send the data analysis request to the analysis data cluster; The display module is used to receive the data analysis results returned by the analysis data cluster from the data analysis request, and to display the data analysis results to the data analysis user; The transaction result pool includes at least one data storage slot; The query module is specifically used to, for each piece of data to be analyzed, identify the target data slot corresponding to the data to be analyzed in the storage data slot; and sequentially store each piece of data to be analyzed into the target data slot of each piece of data to be analyzed. The device further includes an allocation module and a data retrieval module; wherein... The allocation module is used to respond to and process at least one received data retrieval request for each of the transaction data engines, and determine the work number and engine allocation quantity of the analysis data engine corresponding to each data retrieval request; The data retrieval module is used to allocate an analysis data engine to each storage data slot based on the work number and the number of engines allocated by the analysis data engine, and to send the data to be analyzed in the storage data slot to the analysis data engine corresponding to the storage data slot in the form of a data stream.

6. An analysis apparatus for a database with shared analytical resources, characterized in that, Deployed in analytical data clusters; including: An analysis coordination module is used to receive and process data analysis requests based on the analysis coordinator of the analysis data cluster, and to allocate at least one analysis data engine for the data analysis request. The allocation module is used to generate a data analysis communication message based on at least one data storage address and a task identifier, and to send the data analysis communication message to each of the data analysis engines respectively. The analysis execution module is used to generate a data retrieval request for each of the analysis data engines based on the work number of the analysis data engine and the engine allocation quantity of the analysis data engine corresponding to the data analysis request, and send the data retrieval request to each of the data storage addresses. The device further includes: a data receiving module, a data analysis module, and a data response module; wherein... The data receiving module is used to receive at least one piece of data to be analyzed sent by at least one transaction data engine through each of the analysis data engines; The data analysis module is used to receive data transmission completion identifiers from all the transaction data engines in each of the analysis data engines, and to perform data analysis on at least one of the data to be analyzed based on the data analysis task to obtain data analysis results. The data response module is used to return the data analysis results to at least one transaction data cluster corresponding to the data analysis request; The step of receiving at least one piece of data to be analyzed from at least one transaction data engine through each of the analysis data engines includes: The analysis data engine pulls at least one of the data to be analyzed from the storage data slot in the form of a data stream; If the analysis data engine finds that the data to be analyzed is not in the storage data slot, it chooses to wait for the transaction data engine to store the data to be analyzed in the storage data slot until the transaction data engine notifies the analysis data engine that the data sending is complete, then it stops pulling data from the storage data slot.

7. An analysis system for a database with shared analytical resources, characterized in that, It includes at least one transactional data cluster and an analytical data cluster, wherein the transactional data cluster includes at least one transactional data engine, and the analytical data cluster includes at least one analytical coordinator and at least one analytical data engine; wherein, The transaction data cluster is used to execute the analysis method of the database with shared analysis resources as described in any one of claims 1-3; The analytical data cluster is used to execute the analytical method of the database with shared analytical resources as described in claim 4.

Images