Business execution method, storage medium, device, and distributed system

Abstract

This disclosure provides a business execution method, apparatus, distributed system, and storage medium. The cloud service system includes at least one computing cluster and at least one storage cluster, wherein storage nodes in the storage cluster are accessible to computing nodes in the computing cluster. The method includes: obtaining a user's instance usage request; based on the instance usage request, determining a target storage node allocated to the user in the storage cluster and a target computing node allocated to the user in the computing cluster for the user's use; the target computing node runs a target virtual machine, which is used to communicate with the target storage node, the operating system of the target virtual machine provides a data access interface, and the target virtual machine runs an application based on a storage-compute integrated architecture; after receiving a data access request initiated by the application through the data access interface of the target virtual machine, the target storage node is accessed to provide data access services for the application.

Description

Technical Field

[0001] This disclosure relates to the field of cloud service technology, and in particular to business execution methods, storage media, devices and distributed systems. Background Technology

[0002] In a compute-in-memory architecture, compute nodes and storage nodes are highly integrated. The data that the computation needs to access resides on the local storage node, eliminating the need for remote access. This architecture reduces data transfer time and offers high compute and storage locality. Many big data analytics applications utilize compute-in-memory architectures.

[0003] In a compute-storage separation architecture, compute nodes and storage nodes are isolated and connected via a network. Compute nodes need to access remote storage nodes to read data. The advantage of this architecture is its high flexibility; compute or storage can be upgraded independently, and configuration and scale can be dynamically adjusted as needed.

[0004] The data involved in the aforementioned big data analytics applications can be stored using a data lake architecture. Data lake architectures can store structured, semi-structured, and unstructured data. In data lake scenarios, an increasing number of users are looking to transform their in-memory computing (IMC) applications into a compute-storage separation architecture to achieve flexible computing and storage capabilities and improve cost-effectiveness. Users typically modify the software architecture on the user side, separating the computing and storage modules within the software system and managing their resources separately. However, this transformation is often costly and requires significant human and material resources. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a business execution method, storage medium, device and distributed system.

[0006] According to a first aspect of the embodiments of this specification, a business execution method is provided, the method being applied to a cloud service system, the cloud service system including at least one computing cluster and at least one storage cluster, wherein storage nodes in the storage cluster are accessible to computing nodes of the computing cluster; the method includes:

[0007] Get the user's instance usage request;

[0008] Based on the instance usage request, a target storage node and a target compute node in the computing cluster are determined for the user to use; wherein, the target compute node runs a target virtual machine, the target virtual machine is used to communicate with the target storage node, the operating system of the target virtual machine provides a data access interface, and the target virtual machine runs an application based on a storage-compute architecture;

[0009] After receiving the data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application.

[0010] According to a second aspect of the embodiments of this specification, a business execution method is provided, the method being applied to a target virtual machine, the target virtual machine running in a target computing node of a computing cluster of a cloud service system, the cloud service system further including a storage cluster, and the storage nodes in the storage cluster being accessible to the computing nodes of the computing cluster;

[0011] The method includes:

[0012] Establish a communication connection with the target computing node;

[0013] The data access request is received through the operating system's data access interface; the data access request is initiated by an application running on the target virtual machine based on a storage-computing architecture.

[0014] Access the target computing node to provide data access services to the application.

[0015] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method embodiments described in the first aspect above.

[0016] According to a fourth aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method embodiments described in the first aspect above.

[0017] According to a fifth aspect of the embodiments of this specification, a distributed system is provided, the distributed system including a management node, at least one computing cluster and at least one storage cluster, wherein the storage nodes in the storage cluster are accessible to the computing nodes of the computing cluster; the management node is used to perform the steps of the method embodiments described in the first aspect above.

[0018] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0019] In the embodiments described in this specification, a solution is provided at the infrastructure level. The cloud service system includes at least one computing cluster and at least one storage cluster. Storage nodes in the storage cluster are accessible to computing nodes in the computing cluster. The cloud service system can determine the target storage node allocated to the user in the storage cluster and the target computing node allocated to the user in the computing cluster. Thus, the user uses the target virtual machine running on the target computing node. The target virtual machine is used to establish a communication connection with the target storage node. The target virtual machine allows the user to install applications based on the in-memory computing architecture. The operating system of the target virtual machine provides a data access interface to the application. After receiving the application's data access request through the data access interface, the application accesses the target storage node to provide data access services to the application. This allows for easy migration of the user's in-memory computing applications to a storage-compute separation architecture, and the way the user accesses data using the in-memory computing applications remains consistent, but the underlying storage adopts a storage-compute separation type.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this specification and, together with the specification, serve to explain the principles of this disclosure.

[0022] Figure 1 This is a schematic diagram illustrating a storage-compute separation architecture according to an exemplary embodiment of this specification.

[0023] Figure 2A This is a flowchart illustrating a business execution method according to an exemplary embodiment of this specification.

[0024] Figure 2B and 2C This is a schematic diagram of a data lake architecture illustrated in this specification according to an exemplary embodiment.

[0025] Figure 3 This is a flowchart illustrating another business execution method according to an exemplary embodiment of this specification.

[0026] Figure 4 This is a block diagram illustrating a service execution apparatus according to an exemplary embodiment.

[0027] Figure 5 This specification is a hardware structure diagram of a computer device containing a service execution apparatus, according to an exemplary embodiment. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0029] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] In a compute-in-memory architecture, compute nodes and storage nodes are highly integrated. The data that the computation needs to access resides on the local storage node, eliminating the need for remote access. This architecture reduces data transfer time and offers high compute and storage locality. Typical distributed applications such as Spark (a big data analytics engine developed by UC Berkeley) employ a compute-in-memory architecture.

[0032] In a compute-storage separation architecture, compute nodes and storage nodes are isolated and connected via a network. Compute nodes need to access remote storage nodes to read data. The advantage of this architecture is its high flexibility; compute or storage can be upgraded independently, and configuration and scale can be dynamically adjusted as needed.

[0033] Currently, mainstream big data / database systems, such as Spark or Clickhouse (a columnar database management system for online analytical processing), primarily employ an in-store compute architecture for big data analytics applications. The data involved in these applications can utilize a data lake architecture. A data lake can be understood as a centralized repository. Unlike traditional databases that store structured data, data lakes allow for the storage of all structured, semi-structured, or unstructured data at any scale. Data can be stored as is (without prior structuring) and various types of analytics can be run, ranging from dashboards and visualizations to big data processing, real-time analytics, and machine learning.

[0034] In data lake scenarios, driven by big data analytics needs, an increasing number of users are looking to transform their in-memory computing architectures into separate-memory architectures to achieve flexible computing and storage capabilities and improve cost-effectiveness. Current solutions primarily involve implementation at the software architecture level, requiring users to configure computing and storage resources themselves. Users also need to define how to modify the software architecture to separate the computing and storage modules within the software system and manage their resources separately. However, this approach is often costly, requiring significant investment of human and material resources from users.

[0035] Based on this, the embodiments of this specification provide a business execution solution. This solution provides a solution from the infrastructure level, which can easily migrate users' in-memory computing applications to a storage-compute separation architecture. The way users access data using in-memory computing applications remains the same, but the underlying storage adopts storage-compute separation.

[0036] like Figure 1 The diagram shown is a schematic representation of a compute-in-memory separation architecture according to an exemplary embodiment of this specification. The architecture uses compute clusters 11 and 12 as examples; in practical applications, the number of compute clusters can be arbitrarily configured as needed. Each compute cluster includes multiple physical compute devices. Each compute device can run at least one compute node, which is a program used to perform a specific compute task, such as a virtual machine.

[0037] Optionally, the computing cluster may also include a management service node. The management service node is a program that manages the various computing devices in the cluster. The management service node can be used to schedule computing devices and allocate computing nodes to computing devices.

[0038] Figure 1 Taking storage clusters 21 and 22 as examples, the number of storage clusters can be configured arbitrarily as needed in practical applications. Each storage cluster includes multiple storage devices, such as... Figure 1 The image shows multiple storage devices under two storage clusters.

[0039] Each storage cluster includes storage resources, such as storage resource 31 and storage resource 32 connected to storage cluster 21 in the figure. Specifically, storage resources can be multiple external storage devices (such as solid-state drives). Each storage device within a storage cluster can access any external storage device within its cluster. Furthermore, each storage device within a storage cluster can also access any external storage device in other storage clusters through storage devices within those clusters.

[0040] In this configuration, any computing cluster is connected to any storage cluster via one or more network devices. Specifically, at least one computing device is connected to at least one storage device via a network device. This connection can occur between some computing devices and some storage devices, or between any computing device and any storage device. Network devices include, but are not limited to, switches or routers. Figure 1 For ease of illustration, network devices are not shown.

[0041] The storage services in this embodiment include storage services such as Object Storage Service (OSS). Object storage is a technology for storing and managing data in an unstructured format (called objects). Each storage device can run at least one storage node (computer program), which is used to receive data read / write requests sent by computing devices and access the corresponding external storage according to the data read / write requests.

[0042] The system also includes management service nodes for managing each storage cluster. These management service nodes are computer programs that can run on any storage device within the storage cluster, or on a separate device. Each storage cluster may also include a management service node for that cluster, which can run on any storage device within the cluster, or on other physical devices within the cluster.

[0043] This embodiment can be understood as a solution implemented from the IaaS (Infrastructure as a Service) layer. Infrastructure as a Service provides consumers with processing, storage, networking, and various basic computing resources to deploy and execute operating systems or applications and other software. In IaaS, consumers can choose the type and quantity of infrastructure resources they need and pay according to actual usage. Consumers have complete control over these resources, managing and maintaining them themselves or entrusting them to cloud service providers. By using IaaS, consumers no longer need to purchase, deploy, and maintain expensive hardware and software; instead, they can directly use the infrastructure resources provided by cloud service providers, significantly reducing enterprise IT costs and operational burdens. Simultaneously, IaaS can elastically scale according to actual business needs, improving flexibility and responsiveness. Figure 1 As shown, the compute cluster and storage cluster are IaaS.

[0044] The cloud service system in this embodiment can be a system configured by the cloud service provider. In some examples, this embodiment can be applied to... Figure 1 In the management node shown. Optionally, the cloud service provider may also provide a user interface to the user, which may be a browser page or a client page, through which the user can access the services provided by the cloud service provider through the cloud service system.

[0045] like Figure 2A The diagram shown is a flowchart illustrating a business execution method according to an exemplary embodiment, comprising the following steps:

[0046] In step 202, the user's instance usage request is obtained.

[0047] In step 204, based on the instance usage request, the target storage node allocated to the user in the storage cluster and the target computing node allocated to the user in the computing cluster are determined for the user's use.

[0048] The target computing node runs a target virtual machine, which is used to communicate with the target storage node. The operating system of the target virtual machine provides a data access interface, and the target virtual machine runs an application based on a storage-compute architecture.

[0049] In step 205, after the data access interface of the target virtual machine receives the data access request initiated by the application, the target virtual machine accesses the target storage node to provide data access services for the application.

[0050] In some examples, users can initiate instance usage requests through the aforementioned user interface. In this embodiment, an instance refers to a virtual computer device provided to the user by the cloud service provider. This virtual machine's computer device may include computing resources (including a virtual CPU (Central Processing Unit) and memory, etc.) and remote storage resources (such as persistent storage like solid-state drives and storage services that manage these resources). In this embodiment, this instance is referred to as a data lake instance. The naming of this instance indicates that the data stored in this instance may include semi-structured or unstructured data, which can be used in data lake computing scenarios. In practical applications, it is optional to use it only for storing structured data. This naming does not constitute a limitation on the data type or the application scenario of this embodiment.

[0051] In some examples, when a user uses the instance for the first time, the instance usage request may be an instance creation request. This request can include the number of instances and / or instance specifications, etc. Instance specifications can include basic attributes such as computing performance, storage performance, and network performance. Based on the instance creation request, target storage nodes can be allocated to the user from the storage cluster, and target compute nodes can be allocated to the user from the compute cluster. The target compute nodes can run virtual machines and can be used for persistent storage of user data. Optionally, the user can create multiple instances, and in this case, multiple target compute nodes can be allocated; there can also be multiple target storage nodes. Here, multiple target storage nodes can be understood as multiple software nodes providing storage services, enabling distributed data access, but ultimately sharing access to the same user data.

[0052] In this embodiment, the target compute node runs the target virtual machine. From the user's perspective, the data lake instance is equivalent to a virtual machine, but at the underlying storage layer, it uses a compute-storage separation architecture. The target virtual machine can establish communication connections with the remote target storage node. The operating system of the target virtual machine provides a data access interface to applications. This interface receives data access requests from applications and accesses the target storage node, thereby providing applications with compute-storage separation remote data access services.

[0053] In practical applications, users can deploy any in-memory computing-based application on the target virtual machine as needed, such as Spark or Clickhouse mentioned above. The deployment process can be completed by the user, optionally in a single-machine or distributed manner. For example, a user can request the creation of multiple target virtual machines and distribute the application based on the in-memory computing architecture across these machines. The deployment process can involve the user installing the application on each target virtual machine or using a distributed deployment tool.

[0054] In related technologies, when users originally used applications based on in-memory computing architectures, some applications implemented their own distributed file systems, which operated local data. When an application accessed local data, it generated a call request, which invoked an interface provided by the distributed file system. The distributed file system then received the call request through this interface and accessed the local data. Other applications utilized the local file system of the operating system of the host computer device to access local data. The application generated a call request, which invoked an interface provided by the local file system. The local file system then received the call request through this interface and accessed the local data. In this embodiment, the application runs on a target virtual machine, and the operating system of the target virtual machine provides a data access interface for the application to invoke.

[0055] In some examples, the data access interface may include a distributed file system interface or a local file system interface. As an example, a distributed file system interface may include an interface providing the Hadoop Distributed File System (HDFS), which is a distributed file system designed to run on general-purpose hardware. A local file system may be a POSIX (Portable Operating System Interface of UNIX) interface, etc. The POSIX interface is a standard operating system interface used for operating file systems and process control. It defines a set of interfaces including file system operations, process management, signal handling, thread operations, etc. In this embodiment, the virtual machine's operating system can implement both of these interfaces to meet the needs of different types of user applications.

[0056] In some examples, the target virtual machine is configured with local storage devices for caching data. These local storage devices may include storage devices conforming to the Non-Volatile Memory Host Controller Interface (NVMExpress, NVME) specification, etc. After receiving a data access request from the application via the target virtual machine's data access interface, the target virtual machine accesses the target storage node to provide data access services to the application. This includes: receiving the data access request from the application via the target virtual machine's data access interface, first determining whether the local storage device can provide data access services to the application; if not, then accessing the target storage node to provide data access services to the application. Since there may be a certain delay in virtual machines accessing remote storage resources, this embodiment can provide fast caching for the target virtual machine. For example, during application runtime, it may need to access some data. In this embodiment, the local storage device configured on the target virtual machine can be used to cache all or part of the data, allowing the target virtual machine to prioritize accessing the local storage device. If the required data is not stored on the local storage device, then the remote storage node is accessed.

[0057] In some cases, when a user migrates an application to a virtual machine for the first time, some of their data may also need to be migrated to the cloud. The target virtual machine is configured with local storage devices, and data access requests include data storage requests. After the data access interface of the target virtual machine receives the data access request initiated by the application, the target virtual machine accesses the target storage node to provide data access services for the application. This includes: after receiving the data storage request initiated by the application, the target virtual machine first caches all or part of the data that the user needs to store on the local storage device, and then accesses the target storage node to persistently store the cached data on the target storage node. Since this embodiment first uses the local cache to provide services and then accesses the remote target storage node to persistently store the cached data, the efficiency of data migration can be improved.

[0058] In some examples, the data access request includes a data storage request for unstructured data. After the data access interface of the target virtual machine receives the data access request initiated by the application, the target virtual machine accesses the target storage node to provide data access services for the application. This includes: after receiving the data storage request initiated by the application, the data access interface of the target virtual machine obtains the unstructured data that the user needs to store; and persistently stores the unstructured data to the target storage node using an object-based storage method. This embodiment can be applied to data lake scenarios, where the data is specifically unstructured. In this embodiment, the unstructured data can be persistently stored to the target storage node using an object-based storage method, thereby providing users with an object-based data lake solution.

[0059] like Figure 2B The diagram shown is another example architecture diagram illustrated in this specification according to an exemplary embodiment. In the diagram, ① is a compute node. From the user's perspective, the compute node is a virtual machine. The data lake runtime is deployed on the virtual machine. Here, the data lake runtime is a running process running on the virtual machine's operating system.

[0060] Compute nodes can be configured with NVMe storage as a high-speed cache to accelerate data access; their data lake runtime provides a data access interface through which users can access data services, such as local caching services or remote object storage services. The data access interface can include HDFS interfaces and POSIX interfaces, etc.

[0061] Data lake runtime also includes FUSE (Filesystem in Userspace) runtime. FUSE is an implementation of a userspace file system. The FUSE runtime refers to the file system program running on FUSE, providing distributed file system interfaces and POSIX interfaces. FUSE allows non-privileged users to implement their own file systems without compromising system security, without modifying kernel code. The FUSE runtime can perform file system mounting, unmounting, and access operations through userspace processes, and also supports the development of custom file systems. In a data lake, the FUSE runtime can be used as a caching layer to improve file read and write performance.

[0062] The data lake runtime also provides AMS service, which refers to cloud storage services. In this embodiment, the storage service takes object storage as an example. The cloud can help users manage and maintain data, that is, the cloud can provide users with managed storage services for data.

[0063] In the diagram, ② represents object storage (managed and customized), which is a remote storage node separate from the compute nodes. Optionally, object storage in the cloud can be implemented using object storage buckets. An object storage bucket is a container in an object storage system used to store and manage multiple objects. Each object can be unstructured data, such as text, images, and videos. Object storage buckets have globally unique names, can store large amounts of data, and provide reliable data protection and high availability. Users can create, delete, upload, and download objects in object storage buckets through APIs or console management tools. In data lakes, object storage buckets are typically used to store raw data, analysis results, logs, etc. ① and ② in the diagram constitute a data lake instance.

[0064] In the diagram, ③ refers to the management node, which can be used to manage various instances, manage metadata services, and store data for each user, such as user identifiers, virtual machines created by users, and the correspondence between user storage nodes.

[0065] In the diagram, ④ refers to the user interface. The management node can provide user interface services through the user interface, and users can access the cloud through the user interface. The management node can centrally schedule and manage the user interface services. As an example, this user interface service can receive various requests initiated by users through the user interface and return the request results to the user interface for display.

[0066] Figure ⑤ illustrates two data lake instances as an example. The data lake instances in this embodiment can provide users with elastic storage services and accelerate the process using NVME caching (i.e., elastic storage acceleration in the figure).

[0067] The basic functionalities that the data lake instance in this embodiment can possess are:

[0068] (1) Power on the system and create a data lake instance;

[0069] For example, users can initiate instance creation requests through the user interface; the management node can create computing and storage resources for users, such as allocating target storage nodes from the storage cluster and target compute nodes from the compute cluster. A target compute node is equivalent to a virtual machine.

[0070] During creation, the management node can store the correspondence between the user, the target storage node, and the target virtual machine running on the target compute node.

[0071] (2) Shutdown: Shut down the computing resources of the data lake instance without affecting storage services;

[0072] As an example, the instance usage request includes an instance shutdown request; the method further includes: saving the correspondence between the user's user information, the target storage node, and the target virtual machine running on the target computing node, and stopping the operation of the target virtual machine after saving the running data of the target virtual machine.

[0073] For example, a user can request to shut down a virtual machine through the user interface, thus ceasing service on the target compute node allocated to the user. However, because the correspondence between the user's information, the target storage node, and the target virtual machine running on the target compute node is saved before shutdown, along with the target virtual machine's runtime data, the storage resources allocated to the user remain unaffected. A user may have multiple virtual machines allocated to them, and can choose to shut down one or more. The management node can back up the data of the stopped virtual machines to restore them upon restart. Existing computing resources can be reclaimed as needed. In the event of shutdown, user fees can be settled based on the shutdown mode, thereby reducing user resource usage costs while ensuring that the user's storage resources are not affected.

[0074] ③ Restart: Restart the virtual machine and bind it to the target storage node;

[0075] In some examples, the instance usage request includes an instance restart request; determining the target storage node allocated to the user in the storage cluster and the target compute node allocated to the user in the compute cluster for the user's use includes: based on the user's user information, using the correspondence between the saved user information, the target storage node, and the target virtual machine running on the target compute node, querying the running data of the target storage node and the target virtual machine corresponding to the user; restarting the target virtual machine based on the running data of the target virtual machine; and sending the information of the target storage node to the target virtual machine so that the target virtual machine establishes a communication connection with the target storage node.

[0076] For example, a user can request the restart of a stopped virtual machine through the user interface; the management node can restart the virtual machine based on the data of the stopped virtual machine by storing the correspondence between the user, the target storage node, and the target virtual machine running on the target compute node, and then bind the restarted virtual machine to the target storage node, thereby accessing the user's existing storage resources.

[0077] ④ Release, release the data lake instance;

[0078] For example, a user can request the restart of a stopped virtual machine through the user interface; in a release scenario, it indicates that the user no longer needs the instance service, and the user can choose whether to retain storage resources as needed. The management node can reclaim the storage and / or compute resources allocated to the user as needed.

[0079] ⑤ Binding to AMS means binding the compute instance to the storage service and registering it with the management node. For example, the management node stores the correspondence between the user, the target storage node, and the target virtual machine running on the target compute node. Only when the above relationship is registered can the virtual machine communicate and connect with the target storage node, access data, and obtain object storage services.

[0080] ⑥ Unbind AMS: Unbind the compute instance from the AMS service and delete it from the registered services on the management node; for example, the management node deletes the correspondence between the user, the target storage node, and the target virtual machine running on the target compute node.

[0081] The aforementioned AMS, or Elastic Data Storage Service, is a managed object storage service that separates storage and compute. This embodiment allows user-submitted data to the cloud for remote storage, managed and hosted by the cloud. The storage nodes hosting this remotely stored data can be bound to the user's virtual machine, which can accelerate object storage through NVMe caching. The virtual machine provides HDFS and POSIX file interfaces, facilitating direct migration of distributed file systems and local file systems. This is transparent to the user; the user will see normal file system services within the data lake instance.

[0082] For the storage side, the functions provided in this embodiment include: creating AMS services, adding AMS nodes (that is, creating one or more storage nodes according to user needs), and binding AMS services (that is, binding the target compute node allocated to the user with the target storage node).

[0083] Figure 2C This specification is a schematic diagram illustrating a data lake architecture according to an exemplary embodiment, such as... Figure 2C The diagram shown is a schematic of another data lake instance illustrated in an embodiment of this specification. The big data / database application can be selected by the user and may include applications such as Spark / ClickHouse or any other application currently running on a storage-compute architecture. Through the user interface, the user can configure the number of data lake instances and deploy applications on the opened data lake instances, smoothly migrating storage and computing resources to the data lake instances. During application operation, the user can add and remove computing resources in real time as needed without affecting storage, achieving elasticity in both computing and storage.

[0084] This embodiment supports storage-compute separation at the IaaS layer, achieving managed storage and accelerating remote storage via NVMe. Users do not need to manage compute and storage resources, and its usage is consistent with integrated storage-compute resources. This embodiment is an IaaS layer data lake solution, the industry's first of its kind, enabling big data systems to seamlessly implement a data lake storage-compute separation architecture. This embodiment separates and decouples compute and storage resources, enabling elastic scaling of compute and storage, and independently manages storage, providing HDFS and POSIX interfaces to meet diverse storage needs. This embodiment uses NVMe for caching acceleration, satisfying both elastic storage and user performance requirements.

[0085] like Figure 3 The diagram shown is a flowchart illustrating another business execution method according to an exemplary embodiment of this specification. This embodiment describes the method from the perspective of execution by a target virtual machine. The method is applied to a target virtual machine, which runs on a target computing node of a cloud service system's computing cluster. The cloud service system also includes a storage cluster, and the storage nodes in the storage cluster are accessible to the computing nodes of the computing cluster. The method includes the following steps:

[0086] In step 302, a communication connection is established with the target computing node.

[0087] In step 304, a data access request is received through the operating system's data access interface; the data access request is initiated by an application running on the target virtual machine based on a storage-computing architecture.

[0088] In step 306, the target computing node is accessed to provide data access services to the application.

[0089] The execution process of the target virtual machine can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0090] Corresponding to the embodiments of the aforementioned business execution method, this specification also provides embodiments of the business execution apparatus and the computer on which it is applied.

[0091] like Figure 4 As shown, Figure 4 This is a block diagram illustrating a service execution apparatus according to an exemplary embodiment of this specification, the apparatus comprising:

[0092] Module 41 is used to: obtain the user's instance usage request;

[0093] The determination module 42 is used to: determine, based on the instance usage request, the target storage node allocated to the user in the storage cluster and the target computing node allocated to the user in the computing cluster, for the user's use;

[0094] The target computing node runs a target virtual machine, which is used to communicate with the target storage node. The operating system of the target virtual machine provides a data access interface, and the target virtual machine runs an application based on a storage-computing architecture.

[0095] Service providing module 43 is used to: after receiving a data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application.

[0096] In some examples, the target virtual machine is configured with a local storage device, which is used for caching data by the target virtual machine;

[0097] The service providing module is further configured to: receive a data access request initiated by the application through the data access interface of the target virtual machine, first determine whether the local storage device can provide data access services to the application, and if it is determined that it cannot, then access the target storage node to provide data access services to the application.

[0098] In some examples, the target virtual machine is locally configured with local storage devices, and the data access request includes a data storage request;

[0099] The service providing module is further configured to: after receiving the data storage request initiated by the application through the data access interface of the target virtual machine, the target virtual machine first caches all or part of the data that the user needs to store to the local storage device, and then accesses the target storage node to persistently store the cached data to the target storage node.

[0100] In some examples, the instance uses a request that includes an instance creation request;

[0101] The determining module is used for:

[0102] The user is assigned a target storage node from the storage cluster and a target computing node from the computing cluster.

[0103] In some examples, the device also includes a shut-down module for:

[0104] In response to an instance shutdown request, the system saves the correspondence between the user's user information, the target storage node, and the target virtual machine running on the target compute node, as well as the running data of the target virtual machine, and then stops the running of the target virtual machine.

[0105] In some instances, the instance use request includes: an instance restart request;

[0106] The determining module is further configured to:

[0107] Based on the user's user information, and utilizing the correspondence between the saved user information, the target storage node, and the target virtual machine running on the target computing node, the running data of the target storage node and the target virtual machine corresponding to the user are queried.

[0108] The target virtual machine is restarted based on its running data, and the information of the target storage node is sent to the target virtual machine so that the target virtual machine and the target storage node can establish a communication connection.

[0109] In some examples, the data access interface includes a distributed file system interface or a local file system interface.

[0110] In some examples, the data access request includes a data storage request for unstructured data; the service module is also configured to:

[0111] After receiving the data storage request initiated by the application through the data access interface of the target virtual machine, the unstructured data that the user needs to store is obtained.

[0112] The unstructured data is persistently stored to the target storage node using an object-based storage method.

[0113] The specific implementation process of the functions and roles of each module in the above-mentioned business execution device can be found in the implementation process of the corresponding steps in the above-mentioned business execution method, and will not be repeated here.

[0114] The implementation examples described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of a computer device containing the business execution device described in this specification. (Except for...) Figure 5 In addition to the processor 510, memory 530, network interface 520, and non-volatile memory 540 shown, the computer device where the service execution device 531 is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0115] Accordingly, this specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned business execution method embodiment.

[0116] Accordingly, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the business execution method embodiment.

[0117] Accordingly, this specification also provides a distributed system, which includes a management node, at least one computing cluster and at least one storage cluster, wherein the storage nodes in the storage cluster are accessible to the computing nodes of the computing cluster; and the management node is used in the steps of the aforementioned business execution method embodiments.

[0118] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0119] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0120] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0121] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0122] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0123] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0124] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0125] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0126] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0127] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0128] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A business execution method, the method being applied to a cloud service system, the cloud service system comprising at least one computing cluster and at least one storage cluster independent of the computing cluster, wherein the storage nodes in the storage cluster comprise: A program for receiving data read / write requests sent by computing devices running computing nodes in the computing cluster, and accessing the memory in the storage cluster according to the data read / write requests; The method includes: Get the user's instance usage request; Based on the instance usage request, a target storage node and a target compute node in the computing cluster are determined for the user to use. The target compute node runs a target virtual machine, which communicates with the target storage node. The operating system of the target virtual machine runs a data lake runtime process, which provides a data access interface. The target virtual machine runs an application based on a storage-compute architecture. The data access interface includes a distributed file system interface and / or a local file system interface. The distributed file system interface is used by applications based on a storage-compute architecture that implement a distributed file system, and the local file system interface is used by applications based on a storage-compute architecture that use a local file system. After receiving the data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application.

2. The method according to claim 1, wherein the target virtual machine is locally configured with a local storage device, and the local storage device is used for caching data by the target virtual machine; After receiving the data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application, including: The data access interface of the target virtual machine receives the data access request initiated by the application, and first determines whether the local storage device can provide data access services to the application; If it is determined that it is not possible, then access the target storage node to provide data access services for the application.

3. The method according to claim 1, wherein the target virtual machine is locally configured with a local storage device, and the data access request includes a data storage request; After receiving the data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application, including: After receiving the data storage request initiated by the application through the data access interface of the target virtual machine, the target virtual machine first caches all or part of the data that the user needs to store to the local storage device, and then accesses the target storage node to persistently store the cached data to the target storage node.

4. The method according to claim 1, wherein the instance usage request includes an instance creation request; The step of determining the target storage node allocated to the user in the storage cluster and the target computing node allocated to the user in the computing cluster includes: The user is assigned a target storage node from the storage cluster and a target computing node from the computing cluster.

5. The method according to claim 4, further comprising: In response to an instance shutdown request, the system saves the correspondence between the user's information, the target storage node, and the target virtual machine running on the target compute node, as well as the running data of the target virtual machine, and then stops the running of the target virtual machine.

6. The method according to claim 5, wherein the instance usage request includes an instance restart request; The step of determining the target storage node allocated to the user in the storage cluster and the target computing node allocated to the user in the computing cluster for the user's use includes: Based on the user's user information, and utilizing the correspondence between the saved user information, the target storage node, and the target virtual machine running on the target computing node, the running data of the target storage node and the target virtual machine corresponding to the user are queried. The target virtual machine is restarted based on its running data, and the information of the target storage node is sent to the target virtual machine so that the target virtual machine and the target storage node can establish a communication connection.

7. The method according to claim 1, wherein the data access request includes a data storage request for unstructured data; After receiving the data access request initiated by the application through the data access interface of the target virtual machine, the target virtual machine accesses the target storage node to provide data access services for the application, including: After receiving the data storage request initiated by the application through the data access interface of the target virtual machine, the unstructured data that the user needs to store is obtained. The unstructured data is persistently stored to the target storage node using an object-based storage method.

8. A business execution method, the method being applied to a target virtual machine, the target virtual machine running on a target computing node of a computing cluster of a cloud service system, the cloud service system further comprising a storage cluster independent of the computing cluster, the storage nodes in the storage cluster comprising: A program for receiving data read / write requests sent by computing devices running computing nodes in the computing cluster, and accessing the memory in the storage cluster according to the data read / write requests; The method includes: Establish a communication connection with the target computing node; Data access requests are received through the data access interface provided by the data lake runtime process running in the operating system; the data access request is initiated by an application based on a storage-computing architecture running on the target virtual machine; the data access interface includes a distributed file system interface and / or a local file system interface, the distributed file system interface is used for applications based on a storage-computing architecture and implementing a distributed file system to call, and the local file system interface is used for applications based on a storage-computing architecture and using a local file system to call. Access the target computing node to provide data access services to the application.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A distributed system comprising a management node, at least one computing cluster and at least one storage cluster, wherein storage nodes in the storage cluster are accessible to computing nodes of the computing cluster; the management node is configured to perform the steps of the method according to any one of claims 1 to 8.

Images