Cloud view management method, device, system, equipment, medium and product

Abstract

The invention discloses a cloud view management method, device, system, equipment, medium and product. A driver node in a cloud view cluster receives a cloud view management request sent by a database; wherein the cloud view management request is used for requesting to execute a management operation on a cloud view; determining a target server node in response to the cloud view management request; sending a management operation instruction to the target server node; wherein the management operation instruction is used for triggering the target server node to execute a management operation on the clouded view. By adopting the method and the device, the cloud view positioned outside the database is managed by utilizing host resources on a network, so that the view performance is favorably improved, and the database resources are saved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, system, device, medium, and product for managing cloud-based views. Background Technology

[0002] In the field of computer science, a materialized view is a database object that structurally stores the result set of an SQL query. The process of creating a materialized view is called materialization, which is a form of pre-computation. Based on the number of original data tables in the materialized view, it can be divided into single-table and multi-table materialized views; based on whether there are set operations in the SQL query, it can be divided into detailed and aggregated materialized views; based on whether the result set of the materialized view needs to be fully refreshed during updates, it can be divided into full and incremental materialized views, the latter also known as partitioned materialized views, where incremental data is written to a new partition.

[0003] However, because all database objects reside within the space of the database storage nodes, resources such as CPU, memory, and disk are limited. For databases, especially OLAP databases, when the original data tables are large, the data volume of materialized views is usually also large. Therefore, the performance of materialized views is affected by the limitations of database storage resources. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, system, device, medium, and product for managing cloud-based views, which utilizes host resources on the network to manage cloud-based views located outside the database, thereby improving view performance and saving database resources.

[0005] To achieve the above objectives, embodiments of the present invention provide a cloud-based view management method applied to a driver node, the method comprising:

[0006] Receive a cloud-based view management request sent by the database; wherein the cloud-based view management request is used to request the execution of management operations on the cloud-based view;

[0007] In response to the cloud-based view management request, the target server node is determined;

[0008] Send a management operation instruction to the target server node; wherein the management operation instruction is used to trigger the target server node to perform management operations on the cloud view.

[0009] As an improvement to the above solution, before receiving the cloud-based view management request sent by the database, the method further includes:

[0010] A registration request is initiated to the database; wherein the registration request includes the name of the cloud-based view cluster and the service address of the driver node; the cloud-based view cluster includes driver nodes and server nodes;

[0011] Receive and store the service address of the database sent by the database.

[0012] As an improvement to the above scheme, the management operation includes at least one of creation, refresh, reading, and deregistration.

[0013] In a preferred embodiment, if the management operation is to create a cloud view, then the cloud view management request is a cloud view creation request, the cloud view creation request is used to request the creation of a cloud view, and the cloud view creation request includes cloud view data.

[0014] The management operation instruction is a data write instruction, which includes the cloud-based view data; therefore, sending the management operation instruction to the target server node includes:

[0015] Send a data write command to the target server node; wherein the data write command is used to trigger the target server node to write and store the cloud-based view data.

[0016] As an improvement to the above scheme, before determining the target server node, the method further includes:

[0017] The cloud-based view data is divided into blocks to obtain block view data;

[0018] Based on the size and quantity of the segmented view data, at least one target server node is determined for each segmented view data.

[0019] The step of sending a data write instruction to the target server node includes:

[0020] A data write instruction is sent to the target server node corresponding to each of the segmented view data; wherein, the data write instruction includes the segmented view data, and the data write instruction is specifically used to trigger the target server node to write and store the segmented view data.

[0021] As an improvement to the above scheme, after sending the data write instruction to the target server node, the method further includes:

[0022] Receive the write result returned by the target server node; wherein the write result is either a successful write or a failed write.

[0023] When the write result is a write failure, the target server node is re-determined or the view data is resent, depending on the reason for the write failure.

[0024] In a preferred embodiment, if the management operation is to refresh the cloud view, then the cloud view management request is a cloud view refresh request, which is used to request a refresh of the cloud view; the cloud view refresh request includes updated cloud view data and an update method, which is an incremental update method or a full update method.

[0025] The management operation instruction is a data update instruction, which includes the updated cloud view data. The data update instruction is used to trigger the target service node to update the data according to the update method and the updated cloud view data.

[0026] In a preferred embodiment, if the management operation is to read the cloud view, then the cloud view management request is a cloud view read request, and the cloud view read request is used to request to read the cloud view.

[0027] The management operation command is a data read command, which is used to trigger the target service node to query the target cloud view data and return it;

[0028] After sending a data read instruction to the target server node, the method further includes:

[0029] Receive the target cloud view data sent by the target server node, and send the target cloud view data to the database.

[0030] In a preferred embodiment, if the management operation is to deregister the cloud view, then the cloud view management request is a cloud view deregistration request, and the cloud view deregistration request is used to request to deregister the cloud view.

[0031] The management operation command is a data deregistration command, which is used to trigger the target service node to delete the cloud view data stored locally.

[0032] This invention also provides a method for managing cloud-based views, applied to server nodes, the method comprising:

[0033] Receive management operation instructions sent by the driver node; wherein the management operation instructions are sent by the driver node in response to the cloud view management request of the database;

[0034] According to the management operation instructions, perform management operations on the cloud view.

[0035] As an improvement to the above scheme, the management operation includes at least one of creation, refresh, reading, and deregistration.

[0036] This invention also provides a method for managing cloud-based views, applied to a cloud-based view cluster, the cloud-based view cluster including driver nodes and server nodes, the method comprising:

[0037] The driver node receives a cloud view management request sent by the database, and in response to the cloud view management request, determines the target server node and sends a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0038] The target server node executes management operations on the cloud view according to the management operation instructions.

[0039] This invention also provides a method for managing cloud-based views, applied to a database, the method comprising:

[0040] Generate a cloud-based view management request; wherein the cloud-based view management request is used to request the execution of management operations on the cloud-based view;

[0041] Identify the target cloud-based view cluster; wherein the cloud-based view cluster includes driver nodes and server nodes;

[0042] Send the cloud view management request to the target driver node in the target cloud view cluster; wherein, the cloud view management request is used to trigger the target driver node to determine the target server node and send a management operation instruction to the target server node, the management operation instruction is used to trigger the target server node to perform management operations on the cloud view.

[0043] This invention also provides a cloud-based view management device applied to a driver node, the device comprising:

[0044] The management request receiving module is used to receive cloud view management requests sent by the database; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0045] The server node determination module is used to determine the target server node in response to the cloud-based view management request;

[0046] The management operation instruction sending module is used to send management operation instructions to the target server node; wherein, the management operation instructions are used to trigger the target server node to perform management operations on the cloud view.

[0047] This invention also provides a cloud-based view management device applied to a server node, the device comprising:

[0048] A management operation instruction receiving module is used to receive management operation instructions sent by the driver node; wherein, the management operation instructions are sent by the driver node in response to the cloud-based view management request of the database;

[0049] The management operation execution module is used to perform management operations on the cloud view according to the management operation instructions.

[0050] This invention also provides a cloud-based view cluster, which includes driver nodes and server nodes;

[0051] The driver node is used to respond to the cloud view management request sent by the database, determine the target server node, and send management operation instructions to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0052] The server node is used to receive management operation instructions sent by the driver node, and to perform management operations on the cloud view according to the management operation instructions.

[0053] This invention also provides a cloud-based view management system, which includes a database and a cloud-based view cluster, wherein the cloud-based view cluster includes driver nodes and server nodes.

[0054] The database is used to send cloud-based view management requests to the driver node;

[0055] The driver node is configured to receive a cloud view management request sent by the database, and in response to the cloud view management request, determine a target server node and send a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of a management operation on the cloud view;

[0056] The server node is used to receive management operation instructions sent by the driver node, and to perform management operations on the cloud view according to the management operation instructions.

[0057] This invention also provides a cloud view management device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the cloud view management method as described in any of the above embodiments.

[0058] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the cloud-based view management method as described in any of the preceding embodiments.

[0059] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the cloud-based view management method as described in any of the above embodiments.

[0060] Compared to existing technologies, the cloud-based view management method, apparatus, system, device, medium, and product disclosed in this invention manages the materialized view structure located outside the database—that is, the cloud-based view—by constructing a cloud-based view cluster containing driver nodes and server nodes outside the database, and through interaction between the cloud-based view cluster and the database, utilizing host resources on the network. Compared to materialized views within the database, cloud-based views do not use or compete for internal database resources, indirectly improving database performance, saving database resources, and reducing database load. Cloud-based views utilize network resources, and their scalability is unlimited. Under ideal conditions with sufficient network resources, cloud-based views can be fully loaded into memory, effectively improving view performance. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a cloud-based view management method provided in an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the management framework for cloud-based views in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the process for creating a cloud-based view in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the process of refreshing the cloud view in an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram of the process of reading the cloud view in an embodiment of the present invention;

[0066] Figure 6 This is a schematic diagram of the process for registering the cloud view in an embodiment of the present invention;

[0067] Figure 7 This is a flowchart illustrating another cloud-based view management method provided in an embodiment of the present invention;

[0068] Figure 8This is a flowchart illustrating another cloud-based view management method provided in an embodiment of the present invention;

[0069] Figure 9 This is a schematic diagram of the structure of a cloud-based view management system provided in an embodiment of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0072] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0073] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0074] See Figure 1 This is a flowchart illustrating a cloud-based view management method provided by an embodiment of the present invention. The embodiment of the present invention provides a cloud-based view management method applied to a driver node, the method comprising steps S11 to S13:

[0075] S11. Receive a cloud-based view management request sent by the database; wherein the cloud-based view management request is used to request the execution of management operations on the cloud-based view;

[0076] S12. In response to the cloud-based view management request, determine the target server node;

[0077] S13. Send a management operation instruction to the target server node; wherein the management operation instruction is used to trigger the target server node to perform management operations on the cloud view.

[0078] It should be noted that the database mentioned is a basic data management system responsible for storing raw data tables, processing SQL queries, and providing metadata management capabilities. Materialized views are built-in objects in the database that structurally store the result sets of SQL queries. They are used to improve the performance of complex queries, but their resources (CPU, memory, disk) are limited by the database's own storage nodes and cannot utilize external resources.

[0079] See Figure 2 This is a schematic diagram of the management framework of the cloud view in an embodiment of the present invention. In order to solve the problem that materialized views occupy a large amount of internal database resources and their performance is affected by the limited database storage resources, an external cloud view cluster (CMVC, CMV Cluster) is designed in this embodiment of the present invention. The cloud view is managed through the interaction between the database and the cloud view cluster, so as to make up for the performance bottleneck of materialized views caused by the limited database resources.

[0080] The cloud-based view refers to the cloud materialized view (CMV) of the database. It is a network extension of the materialized view. It is a materialized view structure located outside the database, built using external computer resources on the network. Its metadata is managed by the database, but the actual data storage and computation depend on the external cloud-based view cluster (CMVC) and do not occupy internal database resources.

[0081] In this embodiment of the invention, the database is functionally modified by adding a new type of metadata called "cloud view". The metadata records the cloud view cluster information and the definition information of the cloud view.

[0082] The cloud-based view cluster comprises a driver sub-cluster and a server sub-cluster (client). The driver sub-cluster includes at least one driver node, and the server sub-cluster includes at least one server node. The framework uses ordinary virtual machines or physical machines, on which the cloud-based view driver software or server software is installed. Generally, the driver node and server node are installed and deployed close to the database to obtain better network bandwidth. By configuring the server node's list file, the driver node can identify and connect to the server node.

[0083] This invention implements management operations for cloud-based views through data interaction between a database and a cloud-based view cluster. These management operations include at least one of creation, refresh, reading, and deregistration. The driver node of each cloud-based view cluster is responsible for registering, refreshing, deregistering, querying, recording server status, and allocating servers for the cloud-based views. The server nodes are responsible for storing and reading / writing the folders and files corresponding to the cloud-based views.

[0084] During the cloud-based view management process, the database first indexes the cloud-based view cluster by name, and then routes the data to the server node through the cluster's driver node. The server node is responsible for reading and writing cloud-based view data and storing cloud-based view data or data blocks. Optionally, because the data of the cloud-based view originates from the base table, data loss will not occur due to cloud-based view corruption, so cloud-based views generally do not need to have replicas.

[0085] Specifically, after the database indexes the target cloud-based view cluster, it sends a cloud-based view management request to the driver node to request the execution of management operations on the cloud-based view. The driver node receives the cloud-based view management request from the database and, in response, identifies and connects to the target server node using a pre-configured list file of the driver node's server nodes. The driver node then sends management operation instructions to the target server node to trigger the target server node to execute management operations on the cloud-based view.

[0086] By employing the technical means of this invention, a cloud-based view cluster containing driver nodes and server nodes is constructed outside the database. Through interaction between the cloud-based view cluster and the database, host resources on the network are used to manage the materialized view structure located outside the database, i.e., the cloud-based view. The cloud-based view does not use or compete for internal database resources, indirectly improving database performance, saving database resources, and reducing database load. The cloud-based view utilizes network resources, and its scalability is unrestricted. Under ideal conditions with sufficient network resources, such as ample host memory and network bandwidth, the cloud-based view can be fully loaded into memory, effectively improving view performance.

[0087] As a preferred embodiment, this invention optimizes the registration process between the database and the cloud-based view cluster. In step S11, i.e., before receiving the cloud-based view management request sent by the database, the method further includes:

[0088] A registration request is initiated to the database; wherein the registration request includes the name of the cloud-based view cluster and the service address of the driver node; the cloud-based view cluster includes driver nodes and server nodes;

[0089] Receive and store the service address of the database sent by the database.

[0090] In this embodiment of the invention, the driver node reports to the database's metadata cluster to register the cloud-based view cluster. This includes reporting the cloud-based view cluster name and the driver service address of the cloud-based view cluster. The reported cloud-based view cluster name is `cloud_view_cluster_name`, which is configured and unique within the database. The database defaults to the first cluster to report, and the default cluster setting can be adjusted in the database management interface.

[0091] After reporting the service address of the cloud-based view cluster driver, the database returns its own service address to the driver node. This exchange of service addresses prepares the database for subsequent API calls.

[0092] Once each cloud-based view cluster reports to the database, the database metadata holds information about multiple cloud-based view clusters. Generally, it's agreed that a cloud-based view cluster can only report to one database.

[0093] Furthermore, to ensure high availability, the driver sub-cluster is typically a cluster with an odd number of nodes. The driver sub-cluster includes several driver nodes, divided into one Leader node and multiple Fellow nodes. The Leader node is elected using an algorithm, such as Paxos or Raft. During the registration process, the elected Leader node is responsible for reporting to the database metadata and registering this cloud-based view cluster.

[0094] In one specific implementation, the registration process of the Cloud View Cluster (CMVC) is as follows: The driver sub-cluster of the Cloud View Cluster elects a Leader node using algorithms such as Paxos and Raft. The Leader node actively initiates a registration request to the database metadata, submitting the cluster name (cloud_view_cluster_name) and the driver sub-cluster service address. The database receives the registration request from the Leader node, verifies the uniqueness of the cluster name to ensure no duplicates in the database, records the cluster registration information to its own metadata, and returns its own service address to the driver Leader node. The driver Leader node receives the service address returned by the database, stores the address, completes bidirectional address interaction, and uses it for subsequent interface calls with the database. After registration, it synchronizes the cluster information to other nodes (Fellow nodes) in the driver sub-cluster for unified management of the server sub-cluster. After the server sub-cluster nodes start, they connect to the driver sub-cluster through the configured driver list file, report their own resource status (such as storage capacity and memory) to the driver, and are in a standby state, waiting for the driver to assign tasks. They do not directly participate in the registration interaction with the database, but only accept the management and scheduling of the driver.

[0095] By employing the technical means of this invention, bidirectional connectivity between the database and the cloud-based view cluster is achieved, ensuring that the cluster can be identified and scheduled by the database, thus providing a foundation for subsequent operations such as the creation, reading, and writing of the cloud-based view. Unique cluster names and addresses are used for interaction, ensuring the uniqueness and standardization of cluster management and avoiding resource conflicts. Furthermore, electing a Leader node as a unified agent simplifies the interaction link between the database and the cluster, improving collaboration efficiency.

[0096] Furthermore, the embodiments of the present invention are further implemented based on any of the above embodiments to optimize the management process of cloud-based views.

[0097] In a preferred embodiment, if the management operation is to create a cloud view, then the cloud view management request is a cloud view creation request, which is used to request the creation of a cloud view and includes cloud view data.

[0098] The management operation instruction is a data write instruction, which includes the cloud-based view data. Step S13, namely sending the management operation instruction to the target server node, includes:

[0099] Send a data write command to the target server node; wherein the data write command is used to trigger the target server node to write and store the cloud-based view data.

[0100] It should be noted that the SQL syntax for creating a regular materialized view in a database is generally designed as follows:

[0101] CREATE MATERIALIZEDVIEW view_name AS query WITH[NO]DATA;

[0102] In this embodiment of the invention, there are two syntax implementations for creating cloud-based views from the database:

[0103] 1. Use the default cloud-based view cluster:

[0104] CREATE OUTER MATERIALIZEDVIEW view_name AS query WITH[NO]DATA;

[0105] The default cloud-based view cluster refers to the default cloud-based view cluster configured by the database.

[0106] 2. Use a specified cloud-based view cluster:

[0107] CREATE OUTER / *+cloud_view_cluster_name* / MATERIALIZEDVIEW view_name ASquery WITH[NO]DATA;

[0108] The above statement uses a hint mechanism to specify instructions for cloud view clusters in SQL queries. This syntax specifies that a cloud view named `view_name` should be created within a cloud view cluster named `cloud_view_cluster_name`.

[0109] It should be noted that, apart from the difference in the OUTER keyword during creation, the syntax of cloud-based views is the same as that of regular views.

[0110] In one specific implementation, see [link to implementation details]. Figure 3This is a schematic diagram of the process for creating a cloud-based view in this embodiment of the invention. The specific process for creating a cloud-based view is as follows: When the database receives a cloud-based view creation SQL command submitted by the user, that is, the default cluster syntax of CREATE OUTER MATERIALIZEDVIEW or the specified cluster syntax with a hint, the database parses the SQL command, determines the target cloud-based view cluster, which can be the default cluster or the hint-specified cluster. A cloud-based view creation request is initiated to the driver Leader node of the target cloud-based view cluster, the RegisteView interface is called, and information such as the view definition is passed. A write transaction is started through the BeginWrite interface, the transaction ID is obtained, and the cloud-based view data of the base table is sent to the driver in batches through the WriteBatch interface, carrying the transaction ID.

[0111] The driver node receives the cloud view creation request initiated by the database through the RegisteView interface, records the view definition in its own metadata, allocates the target server node according to the view data volume and server resource status, such as storage capacity and memory, confirms to the database that the write is ready through the BeginWrite interface and returns the transaction ID; and calls the WriteData interface with the server to send the view data transmitted by the database through the WriteBatch interface in batches to the allocated target server node, carrying the transaction ID and view definition.

[0112] The target server node receives the data write instruction sent by the driver node through the WriteData interface, which includes the transaction ID, view definition, and view data. It then stores the view data in a local folder and / or file and returns the write result to the driver node.

[0113] The driver node receives the write results returned by the server node. After all data has been written, it returns the transaction commit result to the database through the EndWrite interface, updating the mapping between view data and the server in the view metadata. The database calls the EndWrite interface to commit the transaction, receives the write results returned by the driver, and updates the state of the cloudified view in its own metadata.

[0114] In a preferred embodiment, where the management operation is creating a cloud view, the method further includes the following steps before determining the target server node:

[0115] The cloud-based view data is divided into blocks to obtain block view data;

[0116] Based on the size and quantity of the segmented view data, at least one target server node is determined for each segmented view data.

[0117] The step of sending a data write instruction to the target server node includes:

[0118] A data write instruction is sent to the target server node corresponding to each of the segmented view data; wherein, the data write instruction includes the segmented view data, and the data write instruction is specifically used to trigger the target server node to write and store the segmented view data.

[0119] In this embodiment of the invention, the cloud-based view supports partitioning, with further subdivision within each partition. Each block can correspond to N server folders and files, and index creation is supported. If the cloud-based view does not have explicit partitioning, it defaults to one partition. Users can explicitly define partitions by business dimensions (such as time, region, etc.). Partitioning of the cloud-based view helps optimize storage distribution and query performance. If the driver node finds that a server node cannot hold the data in a certain partition, the driver will divide the partition data into blocks and find a suitable server node to store the new blocks. The block size is dynamically adjusted by the driver based on server resources (such as memory and storage capacity) to ensure that a single block can be efficiently stored and read / written by the target server.

[0120] The storage hierarchy of the cloud-based view is as follows:

[0121] Drive view - (1:N) - Partitions - (1:N) - Blocks - Server folders and files.

[0122] Compared to HDFS, the driver is equivalent to the Namenode, responsible for managing the namespace of the view, maintaining the metadata of the view partitions, blocks within the partitions, blocks and their corresponding folders and files on the server, and recording all operations such as file creation, deletion, and renaming; while the server is equivalent to the Datanode, responsible for the storage and read / write operations of the folders and files corresponding to the view.

[0123] During the cloud view creation process, after the driver node receives information such as cloud view data, it partitions the cloud view data and then divides the partitioned cloud view data into blocks to obtain block view data. Based on the size and quantity of the block view data, several target server nodes are determined, and at least one target server node is assigned to each block view data. Upon receiving the data write instruction, the target driver node stores the block view data to local folders and files according to the partitioning and block division rules specified by the driver node.

[0124] In a preferred embodiment, after sending the data write instruction to the target server node, the method further includes:

[0125] Receive the write result returned by the target server node; wherein the write result is either a successful write or a failed write.

[0126] When the write result is a write failure, the target server node is re-determined or the view data is resent, depending on the reason for the write failure.

[0127] In this embodiment of the invention, after successfully writing and storing view data, the target server node returns a successful write result to the driver node, carrying the number of successful writes. If the target server node itself lacks resources, such as a full disk, it returns a failed write result, carrying the reason for the failure, and rejects subsequent writes. When the driver node detects that the server node lacks resources while writing view data, it marks the server node's status as "full," and the driver node rolls back the current write, continuing to manage view partitioning and coordinate data writing by other servers.

[0128] Preferably, when the driver node detects that a server node is damaged, the damaged server node is removed, and then the cloud view is recreated.

[0129] By employing the technical means of this invention, a new dedicated SQL syntax is added to achieve standardized creation of cloud-based views, ensuring compatibility with database operation habits and lowering the barrier to entry. Data is stored on an external server, without consuming internal database resources, thus reducing database load. Furthermore, the partitioning and block-based mechanism, combined with dynamic server allocation, enables distributed data storage, improving storage efficiency in high-volume scenarios.

[0130] In a preferred embodiment, if the management operation is to refresh the cloud view, then the cloud view management request is a cloud view refresh request, which is used to request a refresh of the cloud view; the cloud view refresh request includes updated view data and an update method, wherein the update method is an incremental update method or a full update method.

[0131] The management operation instruction is a data update instruction, which includes the updated cloud view data. The data update instruction is used to trigger the target service node to update the data according to the update method and the updated cloud view data.

[0132] In one specific implementation, see Figure 4 This is a schematic diagram of the process of refreshing the cloud view in an embodiment of the present invention. The specific process of refreshing the cloud view is as follows:

[0133] The database triggers view refresh conditions, such as after the base table data is updated. It then initiates a cloud-based view refresh request to the driver of the target cloud-based view cluster, starts a refresh transaction via the BeginWrite interface, obtains the transaction ID, and sends the updated data to the driver via the WriteBatch interface, carrying the transaction ID, in either incremental or full update mode. Incremental update mode refers to writing new view data, while full update mode refers to overwriting old view data.

[0134] The driver node receives the cloud view refresh request from the database, parses the scope of the view to be updated (e.g., whether it is incremental or full view data), and locates the target server node storing the cloud view data through the correspondence between partitions, blocks, and server nodes in the metadata. It calls the BeginWrite interface to confirm refresh readiness to the database, returns a transaction ID, and sends the updated cloud view data, carrying the transaction ID, to the server node through the WriteData interface.

[0135] The target server node receives data update instructions sent by the driver through the WriteData interface, including updated cloud view data, such as incremental view data or full view data, and a transaction ID. If it's an incremental refresh, the data is written to the file corresponding to the new partition; if it's a full refresh, the old partition file is overwritten. The node returns the write result to the driver node, such as successful writes and the number of successful writes, or write failures and the reason for the failure.

[0136] The driver node receives the write result returned by the server. If it fails, it reassigns a server and retryes. Upon completion, it returns the transaction commit result to the database via the EndWrite interface, updating the partition or block information of the view metadata. After the data is sent, the database calls the EndWrite interface to commit the transaction, receives the refresh result returned by the driver node, and updates the version information of the view metadata.

[0137] The technical methods employed in this invention support incremental or full refresh of cloud-based views, ensuring consistency between view data and the base table and guaranteeing the accuracy of query results. Furthermore, the driver node dynamically coordinates server node resources, automatically handling storage shortages and ensuring the stability of the refresh process.

[0138] In a preferred embodiment, if the management operation is to read the cloud view, then the cloud view management request is a cloud view read request, which is used to request to read the cloud view.

[0139] The management operation command is a data reading command, which is used to trigger the target service node to query the target cloud view data and return it.

[0140] After sending a data read instruction to the target server node, the method further includes:

[0141] Receive the target cloud view data sent by the target server node, and send the target cloud view data to the database.

[0142] In this embodiment of the invention, sending the view data read instruction to the target server node specifically involves: parsing the query conditions according to the cloud-based view read request; using operator pushdown technology to convert the query conditions into query rules executable by the target server node, generating the view data read instruction, and sending the view data read instruction to the target server node.

[0143] It should be noted that operator pushdown is a SQL tuning technique that aims to push computational operations in a query down to the data storage layer as much as possible, thereby reducing data transmission and improving query execution efficiency. This technique effectively reduces the overhead of the network layer and intermediate layers by pushing computational operations down to a lower-level data source. In this embodiment of the invention, the computational operation is reading a cloud-based view, which improves query performance.

[0144] In one specific implementation, see Figure 5 This is a schematic diagram of the process of reading the cloud view in an embodiment of the present invention. The specific process of reading the cloud view is as follows:

[0145] The database receives a query request for the cloud-based view, parses the query conditions (such as filtering and aggregation operators), initiates a cloud-based view read request to the driver node of the target cloud-based view cluster, calls the BeginRead interface, passes the query conditions, and obtains the transaction ID; and requests the driver node to read data in batches through the ReadBatch interface, specifying the quantity to be retrieved in each batch.

[0146] The driver node receives a read request from the database via the BeginRead interface, parses the query conditions, and uses operator pushdown technology to convert the conditions into filtering rules executable by the server node. Based on the partition-block-server node correspondence, it locates the server node storing the view data. It returns a transaction ID to the database, confirming read read read read is ready. It then calls the ReadData interface with the server node, passing the transaction ID, query conditions, and current read position, requesting to read data in batches.

[0147] The target server node receives a data read instruction sent by the driver node through the ReadData interface, which includes the transaction ID, query conditions, and current position. It filters the locally stored block view data according to the query conditions, starts reading from the specified position, and returns the read data and the last position to the driver for use in the next batch read.

[0148] The driver node receives batched data and the last read position returned by the server, and records the position information; it forwards the data to the database through the ReadBatch interface until the data reading is complete. The database receives batched data returned by the driver node; when the returned quantity is less than the quantity to be retrieved, it determines that the reading is complete, calls the EndRead interface to end the transaction, integrates the data, and returns it to the user. The driver node receives the end notification from the database's EndRead interface and completes the transaction.

[0149] The technical methods employed in this invention rely on a massive network memory matrix to store view data. When there is an application demand for the cloud-based view, it is read through a cloud-based view cluster, reducing disk I / O dependence and demonstrating significant performance advantages when bandwidth is ideal. Furthermore, operator pushdown technology pushes query conditions down to the server, reducing data transmission volume and significantly improving query speed. The batch read mechanism adapts to scenarios with large result sets, avoids memory overflow, and improves the reliability of the read process.

[0150] In a preferred embodiment, if the management operation is to deregister the cloud view, then the cloud view management request is a cloud view deregistration request, which is used to request the deregistration of the cloud view. The management operation instruction is a data deregistration instruction, which is used to trigger the target service node to delete the locally stored cloud view data.

[0151] It should be noted that deregistering a cloud view refers to deleting the cloud view.

[0152] In one specific implementation, see Figure 6 This is a schematic diagram of the process for deregistering a cloud-based view in an embodiment of the present invention. The specific process for deregistering a cloud-based view is as follows:

[0153] When the database receives a user's request to delete a cloud view, it calls the DeregisteView interface to send a request to the driver node of the target cloud view cluster to deregister the cloud view.

[0154] The driver node receives a cloud view registration request initiated by the database through the DeregisteView interface, deletes the definition, partition, block and server correspondence of the cloud view from its own metadata, and notifies the target server node corresponding to the storage of the view data to delete the locally stored cloud view and release resources.

[0155] The target server node receives data from the driver node to register the instruction, deletes all block files of the cloud view stored locally, releases the occupied storage resources, updates its own resource status, and reports it to the driver.

[0156] The driver node returns the deregistration result to the database, and the database receives the deregistration result returned by the driver and deletes the record of the cloud view from its own metadata.

[0157] By employing the technical means of this invention, server storage resources can be released in a timely manner, preventing invalid data from occupying space and optimizing resource utilization. The database and driver work together to delete metadata, ensuring data consistency and avoiding management chaos caused by residual information.

[0158] The following is a list of relevant functional interfaces for your reference.

[0159] Table 1 shows an example of the interface between the driver node and the database. In this table, "passive" means that the driver passively accepts requests from the database, while "active" means that the driver actively requests requests from the database.

[0160] Table 1. Examples of the interface between the driver node and the database.

[0161]

[0162]

[0163]

[0164] The transaction IDs mentioned above correspond to the system version numbers and are used for Multi-Version Concurrency Control (MVCC). MVCC is a method used in database management systems to provide concurrency control, particularly under Read Committed and Repeatable Read isolation levels. The core idea of ​​MVCC is to save a snapshot of the data at a certain point in time. Each transaction sees a consistent view of the data when reading it, even if other transactions are modifying that data simultaneously.

[0165] Table 2 shows an example of the interface between the driver node and the server node. In this table, "passive" means that the driver passively accepts the server's request, while "active" means that the driver actively requests the server.

[0166] Table 2 Examples of interfaces between driver nodes and server nodes

[0167]

[0168]

[0169] See Figure 7 This is a flowchart illustrating another cloud-based view management method provided by an embodiment of the present invention. The embodiment of the present invention provides another cloud-based view management method applied to server nodes, the method including steps S21 to S22:

[0170] S21. Receive management operation instructions sent by the driver node; wherein, the management operation instructions are sent by the driver node in response to the cloud view management request of the database;

[0171] S22. Perform management operations on the cloud view according to the management operation instructions.

[0172] Preferably, the management operation includes at least one of creating, refreshing, reading, and deregistering.

[0173] In a preferred embodiment, if the management operation is to create a cloud view, then the cloud view management request is a cloud view creation request, which is used to request the creation of a cloud view and includes cloud view data; the management operation instruction is a data writing instruction, which includes the cloud view data.

[0174] The step of performing management operations on the cloud-based view according to the management operation instructions includes:

[0175] According to the data writing instruction, the cloud-based view data is written and stored.

[0176] Furthermore, when the cloud-based view data is successfully written, the method also includes:

[0177] Send a successful write result to the driver node; wherein the write result carries the number of successful writes.

[0178] In a preferred embodiment, if the management operation is to refresh the cloud view, then the cloud view management request is a cloud view refresh request, which is used to request a refresh of the cloud view; the cloud view refresh request includes updated cloud view data and an update method, wherein the update method is an incremental update method or a full update method; and the management operation instruction is a data update instruction, which includes the updated cloud view data.

[0179] The step of performing management operations on the cloud view according to the management operation instructions includes:

[0180] According to the data update instruction, the data is updated in accordance with the update method and the updated cloud view data.

[0181] Furthermore, when the cloud-based view data update is complete, the method further includes:

[0182] Send a successful write result to the driver node; wherein the write result carries the number of successful writes.

[0183] In a preferred embodiment, if the management operation is to read the cloud view, then the cloud view management request is a cloud view read request, which is used to request to read the cloud view; and the management operation instruction is a data read instruction.

[0184] The step of performing management operations on the cloud view according to the management operation instructions includes:

[0185] Query the target cloud view data according to the data reading instruction;

[0186] The target cloud view data is sent to the driver node.

[0187] In a preferred embodiment, if the management operation is to deregister the cloud view, then the cloud view management request is a cloud view deregistration request, which is used to request the deregistration of the cloud view; and the management operation instruction is a data deregistration instruction.

[0188] The step of performing management operations on the cloud view according to the management operation instructions includes:

[0189] Based on the data, register the instruction and delete the locally stored cloud view data.

[0190] In a preferred embodiment, the method further includes:

[0191] When a data write instruction or data update instruction is received and the device's own resources are insufficient, a write failure result is returned to the driver node; wherein the write result carries the reason for the write failure.

[0192] It should be noted that the cloud view management method applied to the server node provided in this embodiment of the invention corresponds one-to-one with all the process steps of the cloud view management method applied to the driver node in the above embodiment. The working principle and beneficial effects of the two are the same, so they will not be described again.

[0193] See Figure 8 This is a flowchart illustrating another cloud-based view management method provided by an embodiment of the present invention. The embodiment of the present invention provides another cloud-based view management method applied to a cloud-based view cluster, wherein the cloud-based view cluster includes driver nodes and server nodes. The method includes steps S31 and S32:

[0194] S31. Receive a cloud view management request sent by the database through the driver node, and in response to the cloud view management request, determine the target server node and send a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0195] S32. The target server node performs management operations on the cloud view according to the management operation instructions.

[0196] Preferably, the management operation includes at least one of creating, refreshing, reading, and deregistering.

[0197] It should be noted that the cloud view management method for cloud view clusters provided in this embodiment includes all the process steps of the cloud view management method for driver nodes and server nodes in the above embodiments. The working principle and beneficial effects of the two are the same, so they will not be described again.

[0198] This invention provides yet another method for managing cloud-based views, applied to a database, the method comprising steps S41 and S42:

[0199] S41. Generate a cloud-based view management request; wherein the cloud-based view management request is used to request the execution of management operations on the cloud-based view;

[0200] S42. Determine the target cloud-based view cluster; wherein, the cloud-based view cluster includes driver nodes and server nodes;

[0201] S43. Send the cloud view management request to the target driver node in the target cloud view cluster; wherein, the cloud view management request is used to trigger the target driver node to determine the target server node and send a management operation instruction to the target server node, and the management operation instruction is used to trigger the target server node to perform management operations on the cloud view.

[0202] Preferably, the management operation includes at least one of creating, refreshing, reading, and deregistering.

[0203] Prior to step S41, the method further includes:

[0204] Receive a registration request sent by a driver node in the cloud-based view cluster; wherein the registration request includes the name of the cloud-based view cluster and the service address of the driver node;

[0205] Store the name of the cloud-based view cluster and the service address of the driver node;

[0206] Send its own service address to the driver node.

[0207] It should be noted that the cloud view management method applied to the database provided in this embodiment of the invention corresponds one-to-one with all the process steps of the cloud view management method applied to the driver node in the above embodiment. The working principle and beneficial effects of the two are the same, so they will not be described again.

[0208] This invention also provides a cloud-based view management device applied to a driver node, the device comprising:

[0209] The management request receiving module is used to receive cloud view management requests sent by the database; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0210] The server node determination module is used to determine the target server node in response to the cloud-based view management request;

[0211] The management operation instruction sending module is used to send management operation instructions to the target server node; wherein, the management operation instructions are used to trigger the target server node to perform management operations on the cloud view.

[0212] It should be noted that the cloud view management device for driver nodes provided in this embodiment of the invention is used to execute all the process steps of the cloud view management method for driver nodes in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0213] This invention also provides a cloud-based view management device applied to a server node, the device comprising:

[0214] A management operation instruction receiving module is used to receive management operation instructions sent by the driver node; wherein, the management operation instructions are sent by the driver node in response to the cloud-based view management request of the database;

[0215] The management operation execution module is used to perform management operations on the cloud view according to the management operation instructions.

[0216] It should be noted that the cloud view management device for server nodes provided in this embodiment of the invention is used to execute all the process steps of the cloud view management method for server nodes described in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0217] This invention provides a cloud-based view cluster, which includes n driver nodes and m server nodes; n≥1, m≥1.

[0218] The driver node is used to respond to the cloud view management request sent by the database, determine the target server node, and send management operation instructions to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0219] The server node is used to receive management operation instructions sent by the driver node, and to perform management operations on the cloud view according to the management operation instructions.

[0220] It should be noted that the cloud-based view cluster provided in this embodiment of the invention is used to execute all the process steps of the cloud-based view management method applied to the cloud-based view cluster in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0221] See Figure 9 This is a schematic diagram of the structure of a cloud-based view management system provided in an embodiment of the present invention. The embodiment of the present invention provides a cloud-based view management system 10, which includes a database 11 and N cloud-based view clusters 12. The cloud-based view clusters include n driver nodes 121 and m server nodes 122; N≥1, n≥1, m≥1.

[0222] The database 11 is used to send cloud-based view management requests to the driver node;

[0223] The driver node 121 is configured to receive a cloud view management request sent by the database, and in response to the cloud view management request, determine a target server node and send a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view;

[0224] The server node 122 is used to receive management operation instructions sent by the driver node, and to perform management operations on the cloud view according to the management operation instructions.

[0225] It should be noted that the process steps executed by the driver node, server node and database in the cloud-based view management system provided in this embodiment of the invention can be referred to in the previous embodiment. The working principles and beneficial effects of the two correspond one-to-one, so they will not be repeated here.

[0226] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the cloud view management method as described in any of the above embodiments.

[0227] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the cloud view management method as described in any of the above embodiments.

[0228] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0229] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for managing cloud-based views, characterized in that, Applied to a driver node, the method includes: Receive a cloud-based view management request sent by the database; wherein the cloud-based view management request is used to request the execution of management operations on the cloud-based view; In response to the cloud-based view management request, the target server node is determined; Send a management operation instruction to the target server node; wherein the management operation instruction is used to trigger the target server node to perform management operations on the cloud view.

2. The cloud-based view management method as described in claim 1, characterized in that, Before receiving the cloud-based view management request sent by the database, the method further includes: A registration request is initiated to the database; wherein the registration request includes the name of the cloud-based view cluster and the service address of the driver node; the cloud-based view cluster includes driver nodes and server nodes; Receive and store the service address of the database sent by the database.

3. The cloud-based view management method as described in claim 1, characterized in that, The management operations include at least one of creating, refreshing, reading, and deregistering.

4. The cloud-based view management method as described in claim 1, characterized in that, The management operation is to create a cloud view. The cloud view management request is a cloud view creation request. The cloud view creation request is used to request the creation of a cloud view. The cloud view creation request includes cloud view data. The management operation instruction is a data writing instruction, and the data writing instruction includes the cloud-based view data; The step of sending management operation instructions to the target server node includes: Send a data write command to the target server node; wherein the data write command is used to trigger the target server node to write and store the cloud-based view data.

5. The cloud-based view management method as described in claim 4, characterized in that, Before determining the target server node, the method further includes: The cloud-based view data is divided into blocks to obtain block view data; Based on the size and quantity of the segmented view data, at least one target server node is determined for each segmented view data. The step of sending a data write instruction to the target server node includes: A data write instruction is sent to the target server node corresponding to each of the segmented view data; wherein, the data write instruction includes the segmented view data, and the data write instruction is specifically used to trigger the target server node to write and store the segmented view data.

6. The cloud-based view management method as described in claim 4, characterized in that, After sending the data write instruction to the target server node, the method further includes: Receive the write result returned by the target server node; wherein the write result is either a successful write or a failed write. When the write result is a write failure, the target server node is re-determined or the view data is resent, depending on the reason for the write failure.

7. The cloud-based view management method as described in claim 1, characterized in that, The management operation is to refresh the cloud view. Therefore, the cloud view management request is a cloud view refresh request. The cloud view refresh request is used to request a refresh of the cloud view. The cloud view refresh request includes the updated cloud view data and the update method. The update method is an incremental update method or a full update method. The management operation instruction is a data update instruction, which includes the updated cloud view data. The data update instruction is used to trigger the target service node to update the data according to the update method and the updated cloud view data.

8. The cloud-based view management method as described in claim 1, characterized in that, If the management operation is to read the cloud view, then the cloud view management request is a cloud view read request, and the cloud view read request is used to request to read the cloud view. The management operation command is a data read command, which is used to trigger the target service node to query the target cloud view data and return it; After sending a data read instruction to the target server node, the method further includes: Receive the target cloud view data sent by the target server node, and send the target cloud view data to the database.

9. The cloud-based view management method as described in claim 1, characterized in that, If the management operation is to deregister the cloud view, then the cloud view management request is a cloud view deregistration request, and the cloud view deregistration request is used to request to deregister the cloud view. The management operation command is a data deregistration command, which is used to trigger the target service node to delete the cloud view data stored locally.

10. A method for managing cloud-based views, characterized in that, Applied to a server node, the method includes: Receive management operation instructions sent by the driver node; wherein the management operation instructions are sent by the driver node in response to the cloud view management request of the database; According to the management operation instructions, perform management operations on the cloud view.

11. The cloud-based view management method as described in claim 10, characterized in that, The management operations include at least one of creating, refreshing, reading, and deregistering.

12. A method for managing cloud-based views, characterized in that, Applied to a cloud-based view cluster, the cloud-based view cluster including driver nodes and server nodes, the method includes: The driver node receives a cloud view management request sent by the database, and in response to the cloud view management request, determines the target server node and sends a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of management operations on the cloud view; The target server node executes management operations on the cloud view according to the management operation instructions.

13. A cloud-based view management device, characterized in that, Applied to a driver node, the device includes: The management request receiving module is used to receive cloud view management requests sent by the database; wherein, the cloud view management request is used to request the execution of management operations on the cloud view; The server node determination module is used to determine the target server node in response to the cloud-based view management request; The management operation instruction sending module is used to send management operation instructions to the target server node; wherein, the management operation instructions are used to trigger the target server node to perform management operations on the cloud view.

14. A cloud-based view management device, characterized in that, Applied to a server node, the device includes: A management operation instruction receiving module is used to receive management operation instructions sent by the driver node; wherein, the management operation instructions are sent by the driver node in response to the cloud-based view management request of the database; The management operation execution module is used to perform management operations on the cloud view according to the management operation instructions.

15. A cloud-based view management system, characterized in that, The management system includes a database and a cloud-based view cluster, which includes driver nodes and server nodes. The database is used to send cloud-based view management requests to the driver node; The driver node is configured to receive a cloud view management request sent by the database, and in response to the cloud view management request, determine a target server node and send a management operation instruction to the target server node; wherein, the cloud view management request is used to request the execution of a management operation on the cloud view; The server node is used to receive management operation instructions sent by the driver node, and to perform management operations on the cloud view according to the management operation instructions.

16. A cloud-based view management device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the cloud view management method as described in any one of claims 1 to 12.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the cloud-based view management method as described in any one of claims 1 to 12.

18. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the cloud-based view management method as described in any one of claims 1 to 12.

Images