Data processing methods, devices, chips and clusters

Abstract

A data processing method, apparatus, chip, and cluster are disclosed, relating to the field of cloud computing. The method includes, when multiple tenants' task data are stored in the memory of a first computing node, a storage resource management module stores the task data of the first tenant in the memory of the first computing node according to the virtual storage resources of the first tenant, and stores the task data of the second tenant in the memory of the first computing node according to the virtual storage resources of the second tenant; the virtual storage resources of the first tenant and the second tenant are isolated. The data processing method provided in this application ensures that each tenant has independent virtual storage resources, and the virtual storage resources of multiple tenants are isolated. Even when multiple tenants share memory, storing the task data of multiple tenants in memory according to their different virtual storage resources achieves data isolation among multiple tenants and improves data security.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more particularly to a data processing method, apparatus, chip, and cluster. Background Technology

[0002] Currently, multiple computing nodes handle tasks from multiple tenants, improving computing efficiency, enhancing data processing capabilities, and providing higher reliability and scalability. Distributed computing is widely used in many key technology areas such as modern high-performance computing, big data processing, and cloud computing. In this approach, computing nodes utilize shared memory mechanisms to achieve data flow, enabling inter-process shared memory access on the same computing node and achieving high-performance storage without copying. However, allowing any process to access the memory of a computing node makes data from different tenants vulnerable to leakage, resulting in lower data security. Summary of the Invention

[0003] This application provides a data processing method, apparatus, chip, and cluster that achieves data isolation among multiple tenants under a shared memory mechanism, thereby improving data security.

[0004] In a first aspect, a data processing method is provided. The cloud system includes multiple computing nodes, including a first computing node. The first computing node includes multiple working modules and a storage resource management module. The multiple working modules are used to process tasks from multiple tenants. The method includes: when storing the task data of multiple tenants in the memory of the first computing node, the storage resource management module stores the task data of the first tenant in the memory of the first computing node according to the virtual storage resources of the first tenant, and stores the task data of the second tenant in the memory of the first computing node according to the virtual storage resources of the second tenant; the virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant.

[0005] The data processing method provided in this application involves multiple working modules within a computing node sharing access to the node's memory. These modules process tasks from multiple tenants and store these tasks in the computing node's memory. With multiple working modules accessing the computing node's memory based on a shared memory mechanism, the task data from multiple tenants is stored in memory according to different virtual storage resources. Each tenant has independent virtual storage resources, thus isolating the virtual storage resources of multiple tenants and achieving data isolation, thereby improving data security.

[0006] In one possible implementation, the compute node is a node implemented using virtualization technology; for example, it can be a POD (Programmable Node), virtual machine, or container within a cloud system. When the compute node is a POD, the working modules it includes are virtual machines or containers. The memory of the compute node is the storage resources allocated to the compute node that it can access or that it has the right to use.

[0007] In one possible implementation, the storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources, including: allocating physical storage resources in the memory mapped by the first tenant's virtual storage resources; and storing the first tenant's task data into the physical storage resources mapped by the first tenant's virtual storage resources.

[0008] When it is necessary to store tenants' task data, physical storage resources are allocated to tenants, and virtual storage resources and physical storage resources are dynamically bound together. This decouples virtual storage resources from physical storage resources, thereby allowing for flexible allocation of physical storage resources to tenants and binding of virtual storage resources and physical storage resources as needed. This avoids waste of physical storage resources, improves storage resource utilization, and reduces fragmentation of physical storage resources.

[0009] In another possible implementation, allocating the physical storage resources in memory mapped to the virtual storage resources of the first tenant includes: allocating the physical storage resources mapped to the virtual storage resources of the first tenant according to the amount of data of the first tenant's task data.

[0010] Physical storage resources are allocated on demand based on the amount of task data that needs to be stored. That is, the size of the allocated physical storage resources is related to the amount of task data of the tenant, thereby reducing the allocation of excessive physical storage resources and improving the utilization rate of storage resources.

[0011] In another possible implementation, allocating physical storage resources in memory mapped by the virtual storage resources of the first tenant includes: allocating physical storage resources mapped by the virtual storage resources of the first tenant according to storage blocks, wherein the size of the physical storage resources mapped by the virtual storage resources of the first tenant is equal to the size of at least one storage block.

[0012] Allocating physical storage resources at the storage block level facilitates the management of physical storage resources. Dynamically binding virtual storage resources and physical storage resources decouples physical memory and virtual memory, enabling flexible allocation of physical storage resources to tenants.

[0013] In another possible implementation, the virtual storage resources of the first tenant include a first virtual storage resource and a second virtual storage resource, and the task data of the first tenant includes first task data and second task data; storing the task data of the first tenant into the physical storage resources mapped by the virtual storage resources of the first tenant includes: storing the first task data of the first tenant into the first physical storage resources mapped by the first virtual storage resources of the first tenant; storing the second task data of the first tenant into the second physical storage resources mapped by the second virtual storage resources of the first tenant.

[0014] A tenant can own multiple virtual storage resources, and store smaller amounts of data on each virtual storage resource to improve storage resource utilization and reduce fragmentation of physical storage resources.

[0015] In another possible implementation, the size of the first tenant's virtual storage resource is greater than or less than the size of the memory.

[0016] In another possible implementation, before the storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources, the method further includes: the storage resource management module allocating virtual storage resources for the first tenant.

[0017] Allocate sufficient virtual storage resources to tenants, with the same virtual storage resource assigned to a single tenant. There is no need to statically bind the virtual storage resources and physical storage resources of a tenant, which enables flexible allocation of physical storage resources to tenants, improves storage resource utilization, and reduces fragmentation of physical storage resources.

[0018] In another possible implementation, the storage resource management module allocates virtual storage resources to the first tenant, including: when the first tenant's authentication is successful, the storage resource management module allocates virtual storage resources to the first tenant.

[0019] When tenants access the system, they are authenticated and authorized to improve the system's security.

[0020] In another possible implementation, the method further includes: the storage resource management module reading the task data of the first tenant from the memory of the first computing node according to the virtual storage resources of the first tenant; and the storage resource management module reading the task data of the second tenant from the memory of the first computing node according to the virtual storage resources of the second tenant.

[0021] Because the virtual storage resources of multiple tenants are isolated, when multiple tenants share memory, the task data of that tenant can be read according to the virtual storage resources of that tenant, and the task data of other tenants cannot be read, thus achieving data isolation of multiple tenants and improving data security.

[0022] In another possible implementation, the virtual storage resources of the first tenant include a first virtual storage resource and a second virtual storage resource, and the task data of the first tenant includes first task data and second task data; the storage resource management module reads the task data of the first tenant from the memory of the first computing node according to the virtual storage resources of the first tenant, including: reading the first task data of the first tenant from the first physical storage resource mapped from the first virtual storage resource of the first tenant; and reading the second task data of the first tenant from the second physical storage resource mapped from the second virtual storage resource of the first tenant.

[0023] In another possible implementation, the method further includes: the storage resource management module unbinding the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant.

[0024] Timely unbinding of the mapping relationship between virtual storage resources and physical storage resources can improve storage resource utilization and reduce fragmentation of physical storage resources.

[0025] In another possible implementation, the storage resource management module unbinds the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant, including: unbinding the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant according to the unbinding timing determined by the available physical storage resources in memory.

[0026] Based on the timing of unbinding determined by the available physical storage resources in memory, the mapping relationship between virtual storage resources and physical storage resources is unbound, reducing the frequency of binding and unbinding the mapping relationship between virtual storage resources and physical storage resources, thus saving computing resources and performance costs.

[0027] In another possible implementation, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound according to the unbinding timing determined by the available physical storage resources in memory. This includes: when there are sufficient physical storage resources available in memory, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound after a first period of time, improving unbinding performance and efficiency through asynchronous or delayed unbinding; when there are insufficient physical storage resources available in memory, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound in a timely manner to ensure resource availability.

[0028] In a second aspect, a data processing apparatus is provided, comprising modules for performing the methods of the first aspect or any possible design of the first aspect. For example, the data processing apparatus includes a working module and a storage resource management module.

[0029] The module is used to process tasks from multiple tenants; the storage resource management module is used to store the task data of the first tenant into the memory of the first compute node according to the virtual storage resources of the first tenant; the storage resource management module is also used to store the task data of the second tenant into the memory of the first compute node according to the virtual storage resources of the second tenant; the virtual storage resources of the first tenant and the virtual storage resources of the second tenant are isolated.

[0030] In one possible implementation, when the storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources, it is specifically used to allocate physical storage resources in the memory mapped by the first tenant's virtual storage resources; and to store the first tenant's task data into the physical storage resources mapped by the first tenant's virtual storage resources.

[0031] In another possible implementation, when the storage resource management module allocates the physical storage resources in memory mapped to the virtual storage resources of the first tenant, it specifically allocates the physical storage resources mapped to the virtual storage resources of the first tenant based on the amount of data in the first tenant's task data.

[0032] In another possible implementation, when the storage resource management module allocates the physical storage resources in memory mapped by the virtual storage resources of the first tenant, it specifically allocates the physical storage resources mapped by the virtual storage resources of the first tenant according to the storage blocks, and the size of the physical storage resources mapped by the virtual storage resources of the first tenant is equal to the size of at least one storage block.

[0033] In another possible implementation, the virtual storage resources of the first tenant include a first virtual storage resource and a second virtual storage resource, and the task data of the first tenant includes first task data and second task data; when the storage resource management module stores the task data of the first tenant into the physical storage resource mapped by the virtual storage resource of the first tenant, it is specifically used to store the first task data of the first tenant into the first physical storage resource mapped by the first virtual storage resource of the first tenant; and store the second task data of the first tenant into the second physical storage resource mapped by the second virtual storage resource of the first tenant.

[0034] In another possible implementation, the virtual storage resources of the first tenant are greater than or less than the size of the memory.

[0035] In another possible implementation, the storage resource management module is also used to allocate virtual storage resources for the first tenant.

[0036] In another possible implementation, the storage resource management module is also used to read the task data of the first tenant from the memory of the first computing node according to the virtual storage resources of the first tenant; and to read the task data of the second tenant from the memory of the first computing node according to the virtual storage resources of the second tenant.

[0037] In another possible implementation, the virtual storage resources of the first tenant include a first virtual storage resource and a second virtual storage resource, and the task data of the first tenant includes first task data and second task data. When the storage resource management module reads the task data of the first tenant from the memory of the first computing node according to the virtual storage resources of the first tenant, it is specifically used to read the first task data of the first tenant from the first physical storage resource mapped by the first virtual storage resource of the first tenant; and to read the second task data of the first tenant from the second physical storage resource mapped by the second virtual storage resource of the first tenant.

[0038] In another possible implementation, the storage resource management module is also used to unbind the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant.

[0039] In another possible implementation, when the storage resource management module unbinds the mapping relationship between the virtual storage resource of the first tenant and the physical storage resource mapped to the virtual storage resource of the first tenant, it is specifically used to unbind the mapping relationship between the virtual storage resource of the first tenant and the physical storage resource mapped to the virtual storage resource of the first tenant according to the unbinding timing determined by the available physical storage resources in memory.

[0040] In another possible implementation, the storage resource management module determines the unbinding timing based on the available physical storage resources in memory. Specifically, when there are sufficient available physical storage resources in memory, the module unbinds the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant after a first duration; when there are insufficient available physical storage resources in memory, the module unbinds the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant.

[0041] In another possible implementation, when the storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources, it is specifically used to store the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources after the first tenant's authentication is passed.

[0042] Thirdly, a chip system is provided, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the operational steps of the method in the first aspect or any possible implementation of the first aspect.

[0043] Fourthly, a computing device is provided, the computing device including a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the computing device is caused to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0044] Fifthly, a computing device cluster is provided, comprising at least one computing device, each computing device including a processor and a memory; the processor of the computing device is configured to execute instructions stored in the memory of the computing device to cause the computing device cluster to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0045] A sixth aspect provides a computer-readable storage medium comprising: computer software instructions; which, when executed in a computing device cluster, cause the computing device clusterer to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0046] In a seventh aspect, a computer program product containing instructions is provided, which, when run on a cluster of computing devices, causes the cluster of computing devices to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.

[0047] The technical effects of any of the design methods in aspects two through seven can be found in aspect one or in different design methods in aspect one.

[0048] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0049] Figure 1 A schematic diagram of a Ray framework provided in this application;

[0050] Figure 2 A schematic diagram of the architecture of a data processing system provided in this application;

[0051] Figure 3 A schematic diagram of the architecture of another data processing system provided in this application;

[0052] Figure 4 This application provides a schematic diagram of the deployment of a storage resource management function;

[0053] Figure 5 A flowchart illustrating a data processing method provided in this application;

[0054] Figure 6A flowchart illustrating another data processing method provided in this application;

[0055] Figure 7 A flowchart illustrating another data processing method provided in this application;

[0056] Figure 8 A schematic diagram of the structure of a data processing device provided in this application;

[0057] Figure 9 A schematic diagram of the structure of a computing device provided in this application;

[0058] Figure 10 This application provides a schematic diagram of the structure of a computing device cluster;

[0059] Figure 11 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0060] To facilitate understanding, the main terms used in this application will be explained first.

[0061] With the development of cloud computing technology, computing has evolved from virtual machines to containers, and eventually to the serverless trend.

[0062] Serverless is a cloud computing model. For example, it provides a micro-architecture for serverless computing based on Platform as a Service (PaaS). Users do not need to deploy, configure, or manage servers; the cloud platform provides the computing resources required for code execution. Serverless frees users from worrying about infrastructure, allowing them to focus solely on business development. Serverless computing is also known as Function-as-a-Service (FaaS).

[0063] Currently, data transfer between Serverless functions is achieved using cloud storage services, meaning data exchange between functions occurs between compute nodes and external storage systems. While storage resources can be scaled up as business load increases, the repeated cross-node and cross-service data copying, serialization, and deserialization processes often account for more than half of the end-to-end business latency. As Serverless becomes more complex, end-to-end business latency becomes even more severe, placing higher demands on function orchestration and data transfer between functions.

[0064] Ray is a high-performance distributed computing framework that supports traditional parallel tasks as well as distributed training of artificial intelligence models. For example, ... Figure 1As shown in (a), the Ray framework comprises an application layer and a system layer. The application layer provides the Ray interface for users to write and deploy distributed applications. The application layer includes drivers, workers, and actors. A driver is a process that executes a user program. A worker is a stateless process that executes a task (e.g., a remote function) called by a driver or other worker. When an actor is invoked, the methods exposed by the actor are executed.

[0065] The system layer ensures Ray's high scalability and fault tolerance. It includes the Global Control Store, the Global Scheduler, and the Object Store. The Global Control Store records the state information of various objects in Ray, guaranteeing Ray's fault tolerance. The Global Scheduler is responsible for allocating tasks to achieve parallel execution. The Object Store allows communication and data sharing between workers. It utilizes shared memory mechanisms to achieve efficient data transfer between functions, enabling high performance without copying.

[0066] like Figure 1 As shown in (b), 1. Object storage creates an anonymous memory file and obtains the file descriptor fd; 2. Object storage calls the mmap function to map the anonymous memory file to the object storage memory space and allocate physical memory; 3. Object storage sends the anonymous memory file descriptor to the worker process through a public Unix domain socket; 4. The worker process receives the anonymous memory file descriptor through a public domain interface and maps it to the worker process's memory space; 5. Functions access shared memory through object storage, thereby realizing data flow between functions based on shared memory.

[0067] While shared memory mechanisms solve the problem of efficient data transfer between functions, they have the following drawbacks. First, data security is relatively low. For example... Figure 1 As shown in (c), task data from different tenants may be allocated to the same virtual storage resource, making data leakage between different tenants easy. Secondly, as... Figure 1 As shown in (d), virtual memory is directly bound to physical memory. Memory fragmentation generated by virtual memory will lead to physical memory fragmentation, resulting in inefficient use of physical memory and poor flexibility.

[0068] To address the issue of data isolation among multiple tenants under a shared memory mechanism, this application provides a data processing method applied to a cloud system. The cloud system includes multiple computing nodes, including a first computing node. The first computing node includes multiple working modules and a storage resource management module. The working modules are used to process tasks from multiple tenants. When storing the task data of multiple tenants in the memory of the first computing node, the storage resource management module stores the task data of the first tenant in the memory of the first computing node according to the virtual storage resources of the first tenant, and stores the task data of the second tenant in the memory of the first computing node according to the virtual storage resources of the second tenant; the virtual storage resources of the first tenant and the second tenant are isolated.

[0069] The data processing method provided in this application involves multiple working modules within a computing node sharing access to the node's memory. These modules process tasks from multiple tenants and store these tasks in the computing node's memory. With multiple working modules accessing the computing node's memory based on a shared memory mechanism, the task data from multiple tenants is stored in memory according to different virtual storage resources. Each tenant has independent virtual storage resources, thus isolating the virtual storage resources of multiple tenants and achieving data isolation, thereby improving data security.

[0070] Shared memory is a mechanism that allows multiple processes or threads to access the same memory region. It is one form of inter-process communication (IPC). For example, a shared region is created in memory, and multiple applications read and write data through this shared region. When one application modifies the shared region, other applications can immediately see the modified data. This sharing method does not require data copying and transfer, therefore storage speed is very fast.

[0071] The implementation methods of the data processing method provided in this application will now be described in detail with reference to the accompanying drawings.

[0072] Figure 2 This is a schematic diagram of the architecture of a data processing system provided in this application. Figure 2 As shown, the data processing system 200 includes a client 210 and a computing node 220.

[0073] One possible implementation is that the compute node 220 is implemented in software. For example, the compute node 220 can be implemented using a virtual machine (VM), container, software model, etc. For instance, the compute node 220 can be a Point of View (POD) in a cloud system. Multiple compute nodes 220 are deployed on one or more computer devices. For ease of understanding, the following embodiments use compute nodes as an example. Figure 2 The following explanation uses computing node 220 as an example.

[0074] Optionally, the data processing system 200 includes infrastructure such as computing devices, storage devices, and network devices. Virtualization technology is used to pool and virtualize the resources of this infrastructure (such as computing devices, storage devices, and network devices), providing virtual resources to upper-layer applications and achieving decoupling between software and hardware. When developing new services, it is not necessary to deploy separate hardware devices; instead, virtualization technology can be used to virtualize the service as an independent application. For example, multiple computing nodes can be deployed in the system, and tenant tasks can be deployed on these nodes, allowing the computing nodes to process the tenant's tasks. These tasks include big data processing tasks, model training tasks, or model inference tasks.

[0075] In some embodiments, the storage device provides storage resources for compute nodes, and a compute node occupies all or part of the storage resources in the storage device. The storage resources of a compute node are storage resources allocated to the compute node that it can call upon or that it has the right to use. For example, the storage resources of a compute node include the compute node's memory, which is either storage resources allocated to the compute node that it can call upon or that it has the right to use. The compute node's memory includes volatile memory, such as random access memory (RAM). The compute node's memory is used to store tenant task data.

[0076] Computing devices provide computing resources to computing nodes, and a computing node occupies all or part of the computing resources in the computing device. The computing resources of a computing node are the computing resources allocated to the computing node that it can call upon or that the computing node has the right to use. The computing resources of a computing node include those provided by computing units with computing capabilities, such as the central processing unit (CPU), graphics processing unit (GPU), data processing unit (DPU), neural processing unit (NPU), and embedded neural-network processing unit (NPU).

[0077] Another possible implementation is that the solution provided in this application can be applied to network systems that include clients and servers. For example, a network system that uses a client / server (C / S) architecture for network communication. The communication method between the client and the server does not constitute a limitation on the applicable scenarios of this application.

[0078] For example, a client / server (C / S) architecture network may include a single server or a server cluster. This application does not limit the type and function of the servers described. For instance, server types include blade servers, tower servers, rack servers, and platform servers. Furthermore, servers may include storage servers with storage capabilities or computing servers with computing capabilities, and computing servers may also have storage capabilities. Computing node 220 is a computer device, such as a server, desktop computer, or a controller for a storage array.

[0079] Optionally, compute node 220 is a heterogeneous computing architecture to provide high-performance computing. For example, compute node 220 includes computing units with computing capabilities such as CPU, GPU, DPU, NPU, and NPU to provide high-performance computing.

[0080] The compute node 220 also includes memory, including volatile memory such as RAM. The memory is used to store tenant task data.

[0081] In some embodiments, a compute node runs multiple worker processes and a storage resource management process. For example, a compute node is a POD in a cloud system, and the containers or virtual machines running on the compute node can refer to worker processes or storage resource management processes.

[0082] Multiple worker processes are used to handle tasks from multiple tenants. One worker process is used to handle tasks from one or more tenants. Multiple worker processes handle tasks from a single tenant. For example, a tenant's task may be divided into multiple subtasks, which are then handled by one or more worker processes.

[0083] The storage resource management process manages the storage resources of compute nodes. For example, it allocates virtual storage resources to different tenants, ensuring isolation between these virtual storage resources. It also allocates and releases physical storage resources in the compute node's memory while handling tenant tasks. Furthermore, it dynamically binds virtual and physical storage resources.

[0084] like Figure 3 As shown, the worker processes running on compute node 1 handle tasks from tenant 1 and tenant 2. The storage resource management process allocates virtual storage resources to tenant 1. The storage resource management process allocates virtual storage resources to tenant 2. The virtual storage resources of tenant 1 and tenant 2 are isolated. The storage resource management process stores tenant 1's task data in the memory of compute node 1 based on tenant 1's virtual storage resources. The storage resource management process stores tenant 1's task data in the memory of compute node 2 based on tenant 2's virtual storage resources.

[0085] Compute node 2 runs worker process 1 to handle tenant 1's tasks, and worker process 2 to handle tenant 3's tasks. The storage resource management process allocates virtual storage resources to tenant 1. The storage resource management process allocates virtual storage resources to tenant 3. The virtual storage resources of tenant 1 and tenant 3 are isolated.

[0086] The worker processes running on compute node 1 and compute node 2 handle the tasks of tenant 1.

[0087] Optionally, the data processing system 200 includes multiple computing nodes 220 that communicate with each other. The multiple computing nodes 220 execute tasks for multiple tenants. One or more computing nodes process multiple subtasks contained in a tenant's task.

[0088] In some embodiments, client 210 communicates with computing node 220 via network 230. For example, client 210 sends a data processing request to computing node 220 via network 230, requesting computing node 220 to obtain data to be processed according to the data processing request, and to process the data to obtain the data processing result. For example, data processing includes big data processing, model training, or model inference. Network 230 refers to an enterprise internal network (such as a local area network (LAN)) or the Internet. Client 210 refers to a computer connected to network 230, which can also be called a workstation. Different clients share network resources (such as computing resources and storage resources).

[0089] In some embodiments, user 240 operates the user interface (UI) displayed by client 210 to submit data processing requests. For example, user 240 operates the user interface to submit multiple document processing requests. After receiving the document processing requests, computing node 220 obtains the first set of fragments of the first document, processes the first set of fragments, and obtains the summary result of the first document.

[0090] Optionally, the system administrator 250 may configure system information, such as virtual storage resources configured for the computing node 120, by calling the application platform interface (API) or command-line interface (CLI) through the client 210.

[0091] It is worth noting that, Figure 2This is merely a schematic diagram of a data processing system to which the technical solution protected in this application applies. This application does not limit the connection method or the number of devices in the data processing system. For example, the data processing system includes multiple clients. One client connects to multiple computing nodes. Different clients establish connections with different computing nodes.

[0092] The data processing function provided in this application is implemented by a software system, a hardware device, or a combination of both.

[0093] In one possible implementation, the data processing system described above is a cloud system. For example... Figure 4 As shown, the cloud service provider abstracts the storage resource management function provided in this application into a cloud service and deploys this cloud service in the cloud environment. Tenants consult and purchase cloud services through the cloud platform. After a tenant purchases this cloud service, the cloud platform provides the cloud service to the tenant. When the cloud platform handles tasks for multiple tenants, data is stored according to the data processing method provided in this application, thereby achieving data isolation among multiple tenants.

[0094] In another possible implementation, the storage resource management module is deployed within the application, packaged as a software package by the software provider. The tenant purchases this package and deploys it on their own server, or on a cloud server. For example, the package is deployed by the tenant on computing resources (e.g., virtual machines) in a rented data center. The tenant purchases cloud computing resources from a cloud service provider through a cloud service platform and runs the package on the purchased computing resources, enabling the storage resource management module to perform the storage resource management functions provided in this application. Optionally, the storage resource management module also encrypts the data uploaded by the tenant and its file paths, avoiding direct access to the tenant-uploaded data without hindering the implementation effect, thus ensuring information security.

[0095] Next, the data processing procedure will be explained in detail with reference to the attached diagram.

[0096] First, the process of allocating virtual storage resources will be explained. Figure 5 This is a flowchart illustrating a data processing method provided in this application. Here, we will use... Figure 2 The computing node shown or Figure 4 The cloud environment shown below illustrates how the storage resource management module allocates virtual storage resources. Figure 5 As shown in (a) of the diagram, the method includes the following steps.

[0097] Step 510: The storage resource management module allocates virtual storage resources to the first tenant.

[0098] Step 520: The storage resource management module allocates virtual storage resources to the second tenant.

[0099] In practical applications, the storage resource management module allocates virtual storage resources to multiple tenants. Here, we take the allocation of virtual storage resources to the first tenant as an example.

[0100] In some embodiments, virtual storage resources are allocated to the first tenant in units of storage blocks (e.g., chunks). A virtual storage resource includes at least one storage block. Alternatively, the size of the virtual storage resource may be equal to the size of at least one storage block. For example, the size of the virtual storage resource may be equal to the size of one storage block. Or, for instance, the size of the virtual storage resource may be equal to the size of three storage blocks.

[0101] Virtual storage resources refer to a type of logical storage resource. Virtual storage resources are not the same as the memory storage space of a compute node. Virtual storage resources are not bound to the memory storage space of a compute node; that is, there is no mapping relationship between virtual storage resources and the memory storage space of a compute node.

[0102] Optionally, the virtual storage resource of the first tenant refers to the virtual storage space available to the first tenant. The size of the virtual storage resource of the first tenant refers to the size of the virtual storage space available to the first tenant or the size of the virtual storage capacity available to the first tenant. Virtual storage resources are represented by addresses. For example, a segment of virtual storage resource can be represented from a start address to an end address. Alternatively, virtual storage resources can be represented by a start address and an offset address.

[0103] In some embodiments, the size of the virtual storage resource of the first tenant is greater than, equal to, or less than the size of the memory.

[0104] Allocate sufficient virtual storage resources to tenants, with the same virtual storage resource assigned to a single tenant. There is no need to statically bind the virtual storage resources and physical storage resources of a tenant, which enables flexible allocation of physical storage resources to tenants, improves storage resource utilization, and reduces fragmentation of physical storage resources.

[0105] In some embodiments, if the first tenant is authenticated, the storage resource management module allocates virtual storage resources to the first tenant.

[0106] For example, the working module executing the first tenant's task sends an authentication request to the storage resource management module. This authentication request may include, for example, a verification code such as a username, password, or digital signature. The storage resource management module verifies the first tenant's verification code. If the first tenant's verification code authentication is successful, the storage resource management module allocates virtual storage resources to the first tenant. This application does not limit the specific authentication method.

[0107] When different tenants access the system, they are authenticated and authorized to improve the system's security.

[0108] Optionally, the first tenant's virtual storage resources consist of a contiguous segment of virtual storage resources. Alternatively, the first tenant's virtual storage resources consist of multiple non-contiguous segments of virtual storage resources, thereby improving storage resource utilization and reducing fragmentation of physical storage resources.

[0109] For example, such as Figure 5 As shown in (b), the virtual storage resources of the first tenant include a first virtual storage resource and a second virtual storage resource. The first virtual storage resource and the second virtual storage resource are not contiguous. That is, the addresses of the first virtual storage resource and the second virtual storage resource are not contiguous.

[0110] It should be noted that the virtual storage resources of multiple tenants are isolated from each other. The virtual storage resources of different tenants are isolated. For example, the virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant. The first tenant cannot access the task data of the second tenant, and the second tenant cannot access the task data of the first tenant. This application does not limit the technology for isolating the virtual storage resources of multiple tenants.

[0111] The same virtual storage resource can only be allocated to one tenant. The same segment of virtual storage resource can only be allocated to one tenant. Each tenant's virtual storage resource has a unique address.

[0112] This application does not limit the timing of allocating virtual storage resources to tenants. For example, when the first tenant is created, the storage resource management module allocates virtual storage resources to the first tenant. Alternatively, when the working module processes the first tenant's task data and needs to store the first tenant's task data, the storage resource management module, upon receiving the working module's instruction to store the first tenant's task data, allocates virtual storage resources to the first tenant.

[0113] The storage resource management module writes or reads tenant task data on the virtual storage resources allocated to the tenant, and task data of different tenants are not stored on the same virtual storage resources.

[0114] Figure 6 A flowchart illustrating another data processing method provided in this application. Here, it is shown... Figure 2 The computing node shown or Figure 4 The cloud environment shown illustrates the data storage process based on virtual storage resources. For example... Figure 6 As shown in (a) of the diagram, the method includes the following steps.

[0115] Step 610: The storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources.

[0116] When the storage resource management module stores the first tenant's task data based on the first tenant's virtual storage resources, it binds the first tenant's virtual storage resources with the physical storage resources in the memory of the first computing node, that is, it establishes a mapping relationship between the first tenant's virtual storage resources and the physical storage resources in the memory of the first computing node.

[0117] In some embodiments, the storage resource management module allocates physical storage resources in memory to the first tenant, that is, establishes a mapping relationship between the first tenant's virtual storage resources and the allocated physical storage resources. The storage resource management module stores the first tenant's task data in the physical storage resources mapped from the first tenant's virtual storage resources, that is, stores the first tenant's task data in the memory of the first computing node.

[0118] For example, the task processing module for the first tenant sends a first request to the storage resource management module, instructing that the task data of the first tenant be stored. The first request includes the identifier of the first tenant and the amount of task data. Correspondingly, the storage resource management module receives the first request from the task processing module and allocates physical storage resources mapped from the virtual storage resources of the first tenant based on the amount of task data. By allocating physical storage resources on demand according to the amount of task data to be stored, that is, the size of the allocated physical storage resources is proportional to the amount of task data, thus reducing the allocation of excessive physical storage resources and improving storage resource utilization.

[0119] For example, the first tenant's task data volume is 1G, and the physical storage resources mapped by the virtual storage resources allocated to the first tenant by the storage resource management module are 1G.

[0120] For example, the storage resource management module allocates physical storage resources mapped to the virtual storage resources of the first tenant based on storage blocks. The physical storage resources mapped to the virtual storage resources of the first tenant include at least one storage block. The size of the physical storage resources mapped to the virtual storage resources of the first tenant is equal to the size of at least one storage block. By allocating physical storage resources at the storage block level and dynamically binding virtual and physical storage resources, physical memory and virtual memory are decoupled, enabling elastic resource allocation for multiple tenants.

[0121] Optionally, the storage resource management module can call the fallocate command to dynamically bind the virtual storage resources of the first tenant to the physical storage resources in memory, thereby avoiding memory fragmentation.

[0122] Optionally, the storage resource management module binds a portion of the first tenant's virtual storage resources to physical storage resources; that is, it maps a portion of the first tenant's virtual storage resources to physical storage resources, but does not map all of the first tenant's virtual storage resources to physical storage resources. For example, ... Figure 6 As shown, the virtual storage resources used by the first tenant include those virtual storage resources bound to physical storage resources within the first tenant's virtual storage resources. The virtual storage resources bound to physical storage resources include a portion of the first tenant's virtual storage resources. The virtual storage resources used by the second tenant include those virtual storage resources bound to physical storage resources within the second tenant's virtual storage resources. The virtual storage resources bound to physical storage resources include a portion of the second tenant's virtual storage resources. Used physical storage resources include those bound to the tenant's virtual storage resources. Idle physical storage resources include those not bound to a tenant's virtual storage resources.

[0123] For example, the virtual storage resources of the first tenant include first virtual storage resources and second virtual storage resources. The task data of the first tenant in the storage resource management module includes first task data and second task data.

[0124] When the working module instructs the storage of the first tenant's first task data, the storage resource management module allocates the first physical storage resource to the first virtual storage resource of the first tenant and stores the first task data of the first tenant in the first physical storage resource mapped by the first virtual storage resource of the first tenant.

[0125] When the working module instructs the storage of the second task data of the first tenant, the storage resource management module allocates the second physical storage resource to the second virtual storage resource of the first tenant, and stores the second task data of the first tenant in the second physical storage resource mapped by the second virtual storage resource of the first tenant.

[0126] Optionally, the first task and the second task of the first tenant can be processed by the same working module, and the same working module can instruct the storage of the first task data and the second task data of the first tenant. For example, the first working module processes the first task and the second task of the first tenant and instructs the storage of the first task data and the second task data of the first tenant.

[0127] Optionally, the first task and the second task of the first tenant can be handled by different working modules, and the first task data and the second task data of the first tenant can be stored by different working modules. For example, the first working module handles the first task of the first tenant and instructs the storage of the first task data of the first tenant, while the second working module handles the second task of the first tenant and instructs the storage of the second task data of the first tenant.

[0128] Step 620: The storage resource management module stores the second tenant's task data into the memory of the first computing node based on the second tenant's virtual storage resources.

[0129] When the storage resource management module stores the second tenant's task data using the second tenant's virtual storage resources, it binds the second tenant's virtual storage resources with the physical storage resources in the memory of the first computing node, that is, it establishes a mapping relationship between the second tenant's virtual storage resources and the physical storage resources in the memory of the first computing node.

[0130] In some embodiments, the storage resource management module allocates physical storage resources in memory to the second tenant, that is, establishes a mapping relationship between the second tenant's virtual storage resources and the allocated physical storage resources. The storage resource management module stores the second tenant's task data in the physical storage resources mapped from the second tenant's virtual storage resources, that is, stores the second tenant's task data in the memory of the first computing node.

[0131] For details on the process of storing the second tenant's task data in the memory of the first computing node, please refer to the relevant explanation of the first tenant.

[0132] Since the virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant, that is, the virtual storage resources of the first tenant are different from or do not overlap with the virtual storage resources of the second tenant, and the addresses of the virtual storage resources of the first tenant are not overlapping with the addresses of the virtual storage resources of the second tenant, the physical storage resources mapped by the virtual storage resources of the first tenant are also different from or do not overlap with the physical storage resources of the virtual storage resources of the second tenant. Thus, data isolation of multiple tenants is achieved, improving data security and system performance.

[0133] It should be noted that this application does not limit the working modules for processing the tasks of the first tenant and the tasks of the second tenant. For example, the same working module can process the tasks of the first tenant and the tasks of the second tenant. Alternatively, different working modules can process the tasks of the first tenant and the tasks of the second tenant. For instance, the first computing node includes a first working module and a second working module, where the first working module processes the tasks of the first tenant and the second working module processes the tasks of the second tenant.

[0134] The shared memory mechanism described in this application refers to multiple working modules on a single compute node sharing the node's memory. These modules handle tasks from multiple tenants. Understandably, by sharing the same compute node's memory, multiple tenants can store their task data within the node's memory, enabling efficient data transfer between functions and achieving high performance without copying. However, since each tenant possesses independent virtual storage resources, these resources are isolated, and their physical storage resources within the compute node's memory also differ, thus achieving data isolation and improving data security.

[0135] Each computing node in a data processing system or cloud system can store data based on the virtual storage resource isolation, shared memory and other mechanisms provided in this application, thereby achieving data isolation for multiple tenants.

[0136] After storing the tenant's task data in the memory of the first compute node, the tenant's task data can also be read from memory. For example... Figure 6 As shown in (b) of the diagram, the method includes the following steps.

[0137] Step 630: The storage resource management module reads the task data of the first tenant from the memory of the first computing node according to the virtual storage resources of the first tenant.

[0138] Based on the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources in the memory of the first computing node, the task data of the first tenant is read from the physical storage resources mapped from the virtual storage resources of the first tenant. The storage resource management module then feeds back the task data of the first tenant to the working module.

[0139] Optionally, the same working module can instruct the storage and retrieval of the first tenant's task data. Alternatively, different working modules can instruct the storage and retrieval of the first tenant's task data. For example, the first compute node includes a first working module and a second working module, where the first working module instructs the storage of the first tenant's task data, and the second working module instructs the retrieval of the first tenant's task data. The storage resource management module receives a first request from the first working module and stores the first tenant's task data in the memory of the first compute node according to the first tenant's virtual storage resources. The storage resource management module receives a second request from the second working module, which includes the first tenant's identifier. The storage resource management module determines whether to retrieve the first tenant's task data based on the first tenant's identifier. It then retrieves the first tenant's task data according to the first tenant's virtual storage resources and feeds the first tenant's task data back to the second working module.

[0140] In some embodiments, the virtual storage resources of the first tenant include first virtual storage resources and second virtual storage resources, and the task data of the first tenant includes first task data and second task data.

[0141] When the working module instructs to read the first task data of the first tenant, the storage resource management module reads the first task data of the first tenant from the first physical storage resource mapped from the first virtual storage resource of the first tenant.

[0142] When the working module instructs the reading of the second task data of the first tenant, the storage resource management module reads the second task data of the first tenant from the second physical storage resource mapped from the second virtual storage resource of the first tenant.

[0143] Optionally, the first and second tasks of the first tenant can be processed by the same working module, and the same working module can instruct the reading of the first task data and the second task data of the first tenant. For example, the first working module processes the first and second tasks of the first tenant and instructs the reading of the first task data and the second task data of the first tenant. Alternatively, the first and second tasks of the first tenant can be processed by different working modules, and different working modules can instruct the reading of the first task data and the second task data of the first tenant. For example, the first working module processes the first task of the first tenant and instructs the reading of the first task data of the first tenant, while the second working module processes the second task of the first tenant and instructs the reading of the second task data of the first tenant.

[0144] Optionally, different working modules may store and retrieve the first task data and the second task data of the first tenant. For example, the first working module processes the first task of the first tenant and instructs the storage of the first task data. The second working module processes the second task of the first tenant and instructs the storage of the second task data. The first working module instructs the retrieval of the second task data of the first tenant, and the second working module instructs the retrieval of the first task data.

[0145] Step 640: The storage resource management module reads the task data of the second tenant from the memory of the first computing node according to the virtual storage resources of the second tenant.

[0146] Based on the mapping relationship between the virtual storage resources of the second tenant and the physical storage resources in the memory of the first computing node, the task data of the second tenant is read from the physical storage resources mapped from the virtual storage resources of the second tenant. The storage resource management module then feeds back the acquired task data of the second tenant to the working module.

[0147] For details on the process of reading the second tenant's task data from the memory of the first compute node, please refer to the relevant explanation of the first tenant.

[0148] Understandably, the storage resource management module stores and retrieves tenant task data based on the tenant's virtual storage resources. The virtual storage resources of multiple tenants are isolated, and the physical storage resources of each tenant in the compute node's memory are also different, achieving data isolation among multiple tenants and improving data security.

[0149] In some embodiments, the storage resource management module unbinds the mapping relationship between virtual storage resources and physical storage resources.

[0150] Figure 7 A flowchart illustrating another data processing method provided in this application. Here, it is shown... Figure 2 The computing node shown or Figure 4 The cloud environment illustrated below uses the mapping relationship between virtual storage resources and physical storage resources of unbound tenants as an example for explanation. Figure 7 As shown, the method includes the following steps.

[0151] Step 650: The storage resource management module unbinds the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant.

[0152] In some embodiments, the storage resource management module unbinds the mapping relationship between the virtual storage resources and physical storage resources of the first tenant based on the unbinding timing determined by the available physical storage resources in memory. The unbinding timing includes the end of the data lifecycle in memory or the end of the tenant's task processing, etc.

[0153] For example, if there are sufficient physical storage resources available in memory, such as when the used physical storage resources in memory are below a threshold, it indicates that there are sufficient physical storage resources available in memory. After the first duration, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound.

[0154] For example, when there is insufficient physical storage resources available in memory, such as when the used physical storage resources in memory exceed a threshold, it indicates that the available physical storage resources in memory are insufficient. The mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is then unbound.

[0155] Optionally, the madvise command can be invoked asynchronously to unbind physical storage resources from virtual storage resources, returning the virtual storage resources bound to them to memory. This asynchronous or delayed unbinding improves performance and efficiency.

[0156] Optionally, the `madvise` command can be invoked synchronously to unbind physical storage resources from virtual storage resources, returning the virtual storage resources bound to them to memory. This ensures timely unbinding of the mapping between virtual and physical storage resources, guaranteeing resource availability.

[0157] In addition, this application provides a method for dynamically binding virtual storage resources and physical storage resources. For example, after unbinding the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant, the unbound physical storage resources can still be bound to the virtual storage resources of other tenants, thereby decoupling virtual storage resources and physical storage resources. This allows for flexible allocation of physical storage resources to tenants, binding virtual storage resources and physical storage resources as needed, avoiding waste of physical storage resources, improving storage resource utilization, and reducing fragmentation of physical storage resources.

[0158] The various embodiments of this application can be implemented independently or in combination, without limitation. Unless otherwise specified or in conflict of logic, the terminology and / or descriptions between the different embodiments provided in this application are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0159] It is understood that in the embodiments of this application, the executing entity may perform some or all of the steps in the embodiments of this application. These steps or operations are merely examples, and the embodiments of this application may also perform other operations or variations thereof. Furthermore, the various steps may be executed in different orders as presented in the embodiments of this application, and it is not necessarily necessary to execute all the operations in the embodiments of this application.

[0160] The foregoing primarily describes the solutions provided in this application from the perspective of device-to-device interaction. It is understood that each device, in order to achieve the aforementioned functions, includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] This application embodiment can divide each device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0162] In the case of dividing each function into different modules, this application also provides a data processing apparatus, such as... Figure 8 As shown, the data processing device 800 includes a working module 810 and a storage resource management module 820.

[0163] Work module 810 is used to handle tasks for multiple tenants.

[0164] The storage resource management module 820 is used to store the first tenant's task data into the memory of the first computing node according to the first tenant's virtual storage resources. For example, the storage resource management module 820 is used to execute... Figure 6 Step 610.

[0165] The storage resource management module 820 is also used to store the second tenant's task data into the memory of the first computing node according to the second tenant's virtual storage resources. The virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant.

[0166] For example, the storage resource management module 820 is used to perform... Figure 6 Step 620.

[0167] Both the working module 810 and the storage resource management module 820 can be implemented in software or hardware. For example, the implementation of the working module will be described below. Similarly, the implementation of the storage resource management module can refer to the implementation of the working module.

[0168] Optionally, the data processing device 800 also includes a storage module 830 for storing tenant task data.

[0169] As an example of a software functional unit, a working module may include code running on a computing instance. A computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance may be one or more. For example, a working module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0170] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0171] As an example of a hardware functional unit, a working module includes at least one computing device, such as a server. Alternatively, a working module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0172] The multiple computing devices included in the working module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the working module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the working module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0173] It should be noted that, in other embodiments, the working module is used to execute any step in the data processing method, and the storage resource management module is used to execute any step in the data processing method. The steps implemented by the working module and the storage resource management module can be specified as needed. The data processing device can achieve all its functions by implementing different steps in the data processing method through the working module and the storage resource management module respectively.

[0174] This application also provides a computing device 100. For example... Figure 9 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0175] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0176] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0177] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0178] The memory 106 stores executable program code, and the processor 104 executes this executable program code to implement the functions of the aforementioned working module and storage resource management module, thereby realizing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0179] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0180] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0181] like Figure 10 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing data processing methods.

[0182] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing data processing methods.

[0183] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules in the working module and the storage resource management module.

[0184] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 One possible implementation is shown. For example... Figure 11 As shown, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the working module and the storage resource management module. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of the working module and the storage resource management module.

[0185] It should be understood that Figure 11 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.

[0186] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 10 and Figure 11 The connection method of the computing device cluster shown is different in that the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing data processing methods.

[0187] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0188] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data processing method.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0190] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0191] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0192] It should be understood that in this application, "at least one (item)" means one or more. "More than one" means two or more. "At least two (items)" means two or three or more. "And / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. Both "...when" and "if" indicate that a corresponding action will be taken under certain objective circumstances. They are not time limits, nor do they require a judgment action to be taken when the action is taken, nor do they imply any other limitations.

[0193] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0196] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0198] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of this application embodiment, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, The cloud system includes multiple computing nodes, including a first computing node. The first computing node includes multiple working modules and a storage resource management module. The multiple working modules are used to process tasks from multiple tenants. The method includes: The storage resource management module stores the first tenant's task data into the memory of the first computing node according to the first tenant's virtual storage resources; The storage resource management module stores the second tenant's task data into the memory of the first computing node according to the second tenant's virtual storage resources; The virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant.

2. The method according to claim 1, characterized in that, The storage resource management module stores the first tenant's task data into the memory of the first computing node according to the first tenant's virtual storage resources, including: Allocate the physical storage resources in memory that are mapped to the virtual storage resources of the first tenant; The task data of the first tenant is stored in the physical storage resource mapped by the virtual storage resource of the first tenant.

3. The method according to claim 2, characterized in that, Allocating the physical storage resources in memory mapped from the virtual storage resources of the first tenant includes: The physical storage resources mapped to the virtual storage resources of the first tenant are allocated based on the amount of data in the first tenant's task data.

4. The method according to claim 2, characterized in that, Allocating the physical storage resources in memory mapped from the virtual storage resources of the first tenant includes: The physical storage resources mapped to the virtual storage resources of the first tenant are allocated according to the storage blocks, and the size of the physical storage resources mapped to the virtual storage resources of the first tenant is equal to the size of at least one storage block.

5. The method according to any one of claims 2-4, characterized in that, The virtual storage resources of the first tenant include first virtual storage resources and second virtual storage resources, and the task data of the first tenant includes first task data and second task data; Storing the first tenant's task data in the physical storage resource mapped from the first tenant's virtual storage resource includes: The first task data of the first tenant is stored in the first physical storage resource mapped from the first virtual storage resource of the first tenant. The second task data of the first tenant is stored in the second physical storage resource mapped from the second virtual storage resource of the first tenant.

6. The method according to any one of claims 1-5, characterized in that, The size of the virtual storage resource of the first tenant is greater than or less than the size of the memory.

7. The method according to any one of claims 1-6, characterized in that, Before the storage resource management module stores the first tenant's task data into the memory of the first computing node based on the first tenant's virtual storage resources, the method further includes: The storage resource management module allocates virtual storage resources to the first tenant.

8. The method according to claim 7, characterized in that, The storage resource management module allocates virtual storage resources to the first tenant, including: If the first tenant passes authentication, the storage resource management module allocates virtual storage resources to the first tenant.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The storage resource management module reads the first tenant's task data from the memory of the first computing node according to the first tenant's virtual storage resources; The storage resource management module reads the second tenant's task data from the memory of the first computing node based on the second tenant's virtual storage resources.

10. The method according to claim 9, characterized in that, The virtual storage resources of the first tenant include first virtual storage resources and second virtual storage resources, and the task data of the first tenant includes first task data and second task data; The storage resource management module reads the first tenant's task data from the memory of the first computing node based on the first tenant's virtual storage resources, including: Read the first task data of the first tenant from the first physical storage resource mapped from the first virtual storage resource of the first tenant; Read the second task data of the first tenant from the second physical storage resource mapped from the second virtual storage resource of the first tenant.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: The storage resource management module unbinds the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant.

12. The method according to claim 11, characterized in that, The storage resource management module unbinds the mapping relationship between the first tenant's virtual storage resources and the physical storage resources mapped to the first tenant's virtual storage resources, including: Based on the unbinding timing determined by the available physical storage resources in the memory, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound.

13. The method according to claim 12, characterized in that, Based on the unbinding timing determined by the available physical storage resources in memory, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound, including: If there are sufficient physical storage resources available in the memory, after a first period of time, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound. If the available physical storage resources in the memory are insufficient, the mapping relationship between the virtual storage resources of the first tenant and the physical storage resources mapped to the virtual storage resources of the first tenant is unbound.

14. A data processing apparatus, characterized in that, include: The working module is used to handle tasks from multiple tenants; The storage resource management module is used to store the task data of the first tenant into the memory of the first computing node according to the virtual storage resources of the first tenant; The storage resource management module is also used to store the task data of the second tenant into the memory of the first computing node according to the virtual storage resources of the second tenant; The virtual storage resources of the first tenant are isolated from the virtual storage resources of the second tenant.

15. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the method as described in any one of claims 1-13.

16. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the computing device is used to execute instructions stored in the memory of the computing device to cause the computing device cluster to perform the method as described in any one of claims 1-13.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-13.

18. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-13.

Images