A storage method and device of a resource object, an electronic device and a storage medium

Abstract

The application relates to the computer technical field, and in particular to a resource object storage method and device, electronic equipment and a storage medium, to improve the scheduling performance of a heterogeneous resource cluster. The method comprises the following steps: determining various types of resource objects corresponding to each heterogeneous node in the heterogeneous resource cluster; the resource object is a cloud native object for supporting task execution on the heterogeneous node; constructing a plurality of cloud native storage services according to the total number of various types of resource objects; each cloud native storage service corresponds to a service identifier; dividing various types of resource objects into a plurality of object sets according to the object categories and object identifiers of various types of resource objects, and determining the service identifiers corresponding to each object set; storing each object set into the cloud native storage service represented by the corresponding service identifier for each object set; and each cloud native storage service adopts a master-slave architecture, and the master node in the master-slave architecture is used for processing write requests related to various resource objects.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for storing resource objects. Background Technology

[0002] A heterogeneous resource cluster refers to a cluster composed of different types of computing nodes, used to provide diverse computing capabilities. The computing nodes in this cluster are also called heterogeneous nodes (or heterogeneous computing power nodes), used to execute various computing tasks. These tasks generate a large amount of data during execution, and this data needs to be effectively managed and stored.

[0003] To support efficient task execution and rational resource scheduling on heterogeneous nodes, cloud-native storage provides a reliable storage platform for the resulting data from these nodes, ensuring data security and accessibility. Specifically, by assigning tasks to appropriate cloud-native objects (e.g., specific Pods) for execution and utilizing cloud-native storage to manage the data generated by these tasks, efficient resource scheduling and data management can be achieved, thereby supporting efficient task execution on heterogeneous nodes.

[0004] However, with the rapid development of large-scale artificial intelligence (AI) models, the demand for heterogeneous computing power has increased dramatically, posing a significant challenge to cluster scheduling performance. Current cloud-native storage employs a master-slave architecture, where the master node handles all write requests and replicates data to the slave node via asynchronous synchronization. Because this architecture requires all write operations to be performed through a single master node, allowing multiple master nodes to write simultaneously necessitates complex consistency protocols, a problem inherent in traditional master-slave architectures. Furthermore, regardless of the number of slave nodes added, write requests will still accumulate on the master node. Therefore, during periods of explosive growth in heterogeneous computing demand, current storage methods are ineffective at distributing the load and failing to significantly improve the scheduling performance of heterogeneous resource clusters.

[0005] In summary, how to adjust the storage method of cloud-native objects to improve the scheduling performance of heterogeneous resource clusters is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, and storage medium for storing resource objects, which are used to adjust the storage method of cloud-native objects to improve the scheduling performance of heterogeneous resource clusters.

[0007] This application provides a method for storing resource objects, including:

[0008] For each heterogeneous node in the heterogeneous resource cluster, determine the various types of resource objects corresponding to the heterogeneous node; the resource objects refer to cloud-native objects generated during the production scheduling process of the heterogeneous resource cluster to support task execution on the heterogeneous nodes.

[0009] Based on the total number of the various types of resource objects, multiple cloud-native storage services are constructed; each cloud-native storage service corresponds to a service identifier.

[0010] Based on the object category and object identifier of each type of resource object, the various types of resource objects are divided into multiple object sets, and the service identifier corresponding to each object set is determined.

[0011] For each set of objects, the set of objects is stored in a cloud-native storage service represented by the corresponding service identifier; wherein each cloud-native storage service adopts a primary-backup architecture, and the primary node in the primary-backup architecture is used to handle write requests related to each resource object in the corresponding set of objects.

[0012] This application provides a storage device for resource objects, comprising:

[0013] The resource object determination unit is used to determine the various types of resource objects corresponding to each heterogeneous node in the heterogeneous resource cluster; the resource object refers to the cloud-native object generated during the production scheduling process of the heterogeneous resource cluster to support task execution on the heterogeneous node.

[0014] The construction unit is used to construct multiple cloud-native storage services based on the total number of the various types of resource objects; wherein each cloud-native storage service corresponds to a service identifier;

[0015] The partitioning unit is used to divide the various types of resource objects into multiple object sets according to their respective object categories and object identifiers, and to determine the service identifier corresponding to each object set.

[0016] A storage unit is used to store each set of objects into a cloud-native storage service represented by a corresponding service identifier; wherein each cloud-native storage service adopts a primary-secondary architecture, and the primary node in the primary-secondary architecture is used to handle write requests related to each resource object in the corresponding set of objects.

[0017] Optionally, the device further includes:

[0018] The expansion unit is used to collect the service load information of each cloud-native storage service; the service load information is determined based on the resource usage of each resource object stored in the corresponding cloud-native storage service.

[0019] Based on the service load information of each cloud-native storage service, the cluster load information of the heterogeneous resource cluster is obtained;

[0020] Based on the cluster load information, a scaling decision is made for the heterogeneous resource cluster.

[0021] Optionally, the object categories include: static categories and dynamic categories; the different cloud-native storage services include: a first cloud-native storage service and multiple second cloud-native storage services; then the partitioning unit is used for:

[0022] The resource objects of the static category among the various types of resource objects are divided into the same object set, and the service identifier of the first cloud-native storage service is used as the service identifier corresponding to the same object set; wherein, the resource objects of the static category refer to resource objects of fixed size after being deployed to the heterogeneous resource cluster;

[0023] The resource objects of the dynamic category among the various types of resource objects are divided into at least one object set according to their respective object identifiers, and the service identifiers corresponding to the at least one object set are determined in the service identifiers of the multiple second cloud-native storage services; wherein, the resource objects of the dynamic category refer to resource objects that can be elastically adjusted according to demand after being deployed to the heterogeneous resource cluster.

[0024] Optionally, the resource objects of the dynamic categories include: namespace objects and container group objects; then the partitioning unit is used for:

[0025] For each namespace object, perform the following operations:

[0026] To group a namespace object and the various container group objects belonging to the namespace object into the same object set;

[0027] The hash value is obtained by performing a hash operation on the object identifier of the namespace object.

[0028] The service identifier that the hash value matches in the service identifiers of the multiple second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

[0029] Optionally, the partitioning unit is used for:

[0030] The result is obtained by performing a modulo operation on the hash value and the first quantity; the first quantity is the total number of the plurality of second cloud-native storage services.

[0031] The service identifier that matches the calculation result among the service identifiers corresponding to each of the second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

[0032] Optionally, for each set of objects, after storing the set of objects to a cloud-native storage service represented by the corresponding service identifier, the apparatus further includes:

[0033] The synchronization unit is used to synchronize data between various secondary cloud-native storage services to ensure the consistency of stored data between each secondary cloud-native storage service.

[0034] Optionally, the building unit is used for:

[0035] The third quantity is determined based on the ratio between the second quantity and the unit quantity; wherein, the unit quantity is: the total number of various resource objects supported by each cloud-native storage service; the second quantity is: the total number of various resource objects in the heterogeneous resource cluster;

[0036] Build a third cloud-native storage service and assign a service identifier to each cloud-native storage service.

[0037] Optionally, the expansion unit is used for:

[0038] Based on the cluster load information, analyze whether the operation status of the heterogeneous resource cluster is normal;

[0039] If it is determined that the operation status of the heterogeneous resource cluster is abnormal, then the number of new heterogeneous nodes supported by the heterogeneous resource cluster is determined based on the resource usage trend information of the heterogeneous resource cluster; the resource usage trend information is determined based on the service traffic information of each cloud-native storage service.

[0040] The heterogeneous resource cluster is expanded based on the number of newly added heterogeneous nodes.

[0041] Optionally, a data collection agent is deployed on each cloud-native storage service;

[0042] Each of the service load information is obtained by collecting the resource usage of the master node and each backup node corresponding to the master node through the collection agent on the corresponding cloud-native storage service.

[0043] Each of the aforementioned service traffic information is obtained by collecting the write request processing status of the master node on the corresponding cloud-native storage service through the collection agent on the corresponding cloud-native storage service.

[0044] Optionally, the expansion unit is used for:

[0045] If the cluster load information does not reach the preset threshold, then the heterogeneous resource cluster is determined to be operating normally.

[0046] If the cluster load information reaches the preset threshold, it is determined that the heterogeneous resource cluster is in an abnormal operating state.

[0047] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described resource object storage methods.

[0048] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described resource object storage methods.

[0049] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described resource object storage methods.

[0050] The beneficial effects of this application are as follows:

[0051] This application provides a method, apparatus, electronic device, and storage medium for storing resource objects. First, in a heterogeneous resource cluster, the various resource objects involved in the production scheduling process of each heterogeneous node are identified. These resource objects serve as the basic information units for the heterogeneous nodes to execute tasks; their explicit definition helps establish a mapping relationship between resource usage status and task execution logic, thereby providing object-level processing basis for subsequent storage structure design.

[0052] Next, based on the total number of identified resource objects, a certain number of cloud-native storage services are deployed, and each cloud-native storage service is assigned a unique service identifier. This step dynamically plans the scale of the storage services, ensuring that the system's write capacity keeps pace with the growth of resource objects, avoiding performance bottlenecks caused by a single storage service handling too many write requests, and improving the system's scalability and write stability.

[0053] Then, combining the type of resource objects and their object information, various resource objects are divided into several object sets, and the service identifier of the cloud-native storage service corresponding to each object set is determined based on mapping rules. This division method, by establishing a classification structure on object type and object identifier, can achieve a reasonable mapping from resource objects to storage services, improve the accuracy of object write location, and facilitate balanced writes and orderly object management in multi-storage environments.

[0054] Finally, each object collection is written to its corresponding cloud-native storage service, with each storage service employing a master-slave architecture, where the master node handles write requests. By distributing write operations related to object collections across multiple master nodes, the pressure from centralized write request processing is significantly reduced, effectively alleviating the master node's write bottleneck. This ensures data availability while improving the system's write concurrency capabilities, providing performance assurance for the efficient execution of heterogeneous resource scheduling.

[0055] In summary, the resource object storage method provided in this application can optimize the configuration of resource object write paths through resource object identification, storage service construction, object set partitioning, and parallel write strategies, thereby effectively improving the write performance and scalability of cloud-native storage systems in heterogeneous resource cluster environments and supporting the scheduling needs of high-density heterogeneous nodes.

[0056] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0058] Figure 1A A schematic diagram illustrating the relationship between heterogeneous resource scheduling and resource object generation for related technologies;

[0059] Figure 1B This is a schematic diagram illustrating a method for partitioning heterogeneous resource sub-clusters in related technologies;

[0060] Figure 2A A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0061] Figure 2B A schematic diagram of a resource object management architecture in a heterogeneous resource environment provided in this application embodiment;

[0062] Figure 3 A flowchart illustrating the implementation of a resource object storage method provided in this application embodiment;

[0063] Figure 4 This is a schematic diagram illustrating the generation and association of resource objects in an embodiment of this application;

[0064] Figure 5 This application provides a schematic diagram of the classification of resource objects in an embodiment.

[0065] Figure 6 A schematic diagram of a cloud-native storage service identifier allocation structure provided in this application embodiment;

[0066] Figure 7 A schematic diagram illustrating the classification and cloud-native storage affiliation of resource objects, provided as an embodiment of this application;

[0067] Figure 8 This application provides a schematic diagram of a resource object classification and attribution mapping structure.

[0068] Figure 9 A schematic diagram of a cloud-native storage primary / standby architecture provided in an embodiment of this application;

[0069] Figure 10 A flowchart illustrating an implementation of scaling up a heterogeneous resource cluster, provided in this application embodiment;

[0070] Figure 11A A schematic diagram illustrating a service load information collection and reporting mechanism provided in an embodiment of this application;

[0071] Figure 11B This is a schematic diagram illustrating the generation and visualization of cluster load information as provided in an embodiment of this application.

[0072] Figure 11C This is a schematic diagram illustrating the generation of resource usage trend information and capacity expansion assessment based on service traffic information, provided as an embodiment of this application.

[0073] Figure 12 This is a schematic diagram illustrating the overall process of resource object partitioning and storage management based on a heterogeneous resource cluster, as provided in an embodiment of this application.

[0074] Figure 13 This application provides an overall flowchart for resource object partitioning and heterogeneous resource cluster expansion processing in an embodiment of the present application.

[0075] Figure 14 This is a schematic diagram of the structure of a storage device for resource objects provided in an embodiment of this application;

[0076] Figure 15This is a schematic diagram of the composition structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0078] The following describes some of the concepts involved in the embodiments of this application.

[0079] 1. Heterogeneous Resource Cluster: A heterogeneous resource cluster refers to a cluster composed of computing nodes of different types, used to provide diverse computing capabilities. The computing nodes in this cluster are also called heterogeneous nodes, which are used to execute different types of computing tasks. In this embodiment, the heterogeneous resource cluster is the basic environment for resource scheduling. Resource objects are generated during task production and scheduling, and these resource objects need to be efficiently managed and stored to support the execution of tasks on the heterogeneous nodes.

[0080] 2. Heterogeneous Nodes: Heterogeneous nodes refer to computing nodes in a heterogeneous resource cluster. Different heterogeneous nodes may have different computing resource characteristics and typically execute specific tasks during cluster operation. In the embodiments of this application, heterogeneous nodes are the running units for task execution, and corresponding resource objects are generated during cluster deployment and task scheduling.

[0081] 3. Resource Objects: Resource objects refer to cloud-native objects generated during the production scheduling process of heterogeneous resource clusters to support task execution on heterogeneous nodes. In the embodiments of this application, resource objects are the core data units for system scheduling, management, and storage, including static and dynamic categories of resource objects.

[0082] 4. Cloud-native objects: Cloud-native objects refer to system objects used in cloud-native architectures to represent computing tasks, node information, running status, etc. In the embodiments of this application, cloud-native objects are the specific manifestations of resource objects, including but not limited to node objects, daemonset objects, namespace objects, and pod objects, which participate in the scheduling process and serve as the basic unit of storage operations.

[0083] 5. Cloud-native storage services: Cloud-native storage services refer to storage components used to store resource objects. Each cloud-native storage service has a service identifier, supports object write operations, and has a primary / backup architecture to improve write reliability. In this embodiment, multiple cloud-native storage services can be constructed according to the total number of resource objects to share the object write load.

[0084] The first cloud-native storage service refers to a cloud-native storage service used to store static category resource objects; the second cloud-native storage service refers to multiple cloud-native storage services used to store dynamic category resource objects. In this embodiment, static category resource objects are assigned to the first cloud-native storage service, while dynamic category resource objects are allocated to multiple second cloud-native storage services according to their object identifiers.

[0085] 6. Service Identifier: A service identifier is an identification information used to uniquely identify each cloud-native storage service. In this embodiment, the service identifier is used to map resource object sets to cloud-native storage services, and the object sets are written into the corresponding cloud-native storage services based on the service identifier.

[0086] 7. Object Category and Object Identifier: The object category refers to the classification to which the resource object belongs, which includes static categories and dynamic categories in this embodiment; the object identifier refers to information that identifies the resource object itself. In this embodiment, the resource object is classified according to its object category and object identifier, and its corresponding cloud-native storage service is determined.

[0087] 8. Primary / Secondary Architecture: A primary / secondary architecture refers to a storage structure in which the primary node handles write requests, and the secondary node synchronizes data with the primary node. In this embodiment, each cloud-native storage service adopts a primary / secondary architecture, with the primary node handling write operations to ensure the stability and consistency of data writes.

[0088] 9. Static Resource Objects: Static resource objects refer to resource objects whose size is fixed and does not change dynamically according to tasks after being deployed to a heterogeneous resource cluster. In this embodiment, static resource objects include, but are not limited to, node objects and daemonset objects. In this embodiment, they are divided into the same object set and stored uniformly in the first cloud-native storage service.

[0089] 10. Dynamically Classified Resource Objects: Dynamically classified resource objects refer to resource objects that can be dynamically added or removed according to task requirements after being deployed to a heterogeneous resource cluster. In this embodiment, dynamically classified resource objects include, but are not limited to, namespace objects and pod objects. This embodiment classifies them according to object identifiers and determines the corresponding second cloud-native storage service based on hash values.

[0090] 11. Node Object: A node object is a resource object used to represent a heterogeneous node, describing the node's resource capabilities, operating status, and basic information about its participation in scheduling. In this embodiment, the node object is automatically generated when a compute node (heterogeneous node) deployed in a heterogeneous resource cluster is registered, and belongs to the static category of resource objects. Node objects are typically created during the initialization or expansion of the heterogeneous resource cluster, and their lifecycle is relatively stable. In this embodiment, they can be grouped together with daemon process objects into the same object set and uniformly written into the first cloud-native storage service.

[0091] 12. Daemon Set Object: A daemon set object refers to a resource object used to control the deployment of system daemon tasks on each heterogeneous node. In this embodiment, the daemon set object is used to perform system services such as resource collection and monitoring. The daemon set object does not change with the computing task and belongs to the static category of resource objects. During the resource object partitioning process, the daemon set object and node objects can be grouped into the same object set and uniformly written into the first cloud-native storage service.

[0092] 13. Namespace Objects: Namespace objects are resource objects used to represent logically isolated spaces, typically divided as the namespace to which a task belongs. In this embodiment, a hash calculation can be performed based on the object identifier of the namespace object to determine its cloud-native storage service, and pod objects under that namespace can be grouped into the same object set.

[0093] 14. Container Group Object: A container group object refers to a pod object created under a namespace and used to execute specific tasks. In this embodiment, the container group object, as a dynamic category resource object, can be included in an object collection along with the object of its namespace and written to the corresponding second cloud-native storage service.

[0094] 15. Service Load Information: Service load information refers to information reflecting the resource usage of each cloud-native storage service. In this embodiment, service load information is determined based on the resource usage of resource objects and is used to assess the operating status of each cloud-native storage service, serving as the basis for subsequent load analysis and capacity expansion decisions.

[0095] 16. Cluster Load Information: Cluster load information refers to information obtained by aggregating the service load information of various cloud-native storage services, reflecting the overall load status of the heterogeneous resource cluster. In this embodiment, cluster load information is used to determine whether the cluster is under high load and can serve as the basis for deciding whether to perform scaling up.

[0096] 17. Resource Usage Trend Information: Resource usage trend information refers to trend data formed by changes in service load information or service traffic information over time. In this embodiment, the system can predict changes in resource usage based on resource usage trend information and determine the number of new heterogeneous nodes that can be supported accordingly.

[0097] 18. Service Traffic Information: Service traffic information refers to information reflecting the write request processing status of the master node in cloud-native storage services. In this embodiment, service traffic information is used to assess the write capacity usage status of the master node and participate in resource usage trend analysis and capacity expansion judgment.

[0098] 19. Data Collection Agent: A data collection agent is a component deployed on a cloud-native storage service used to collect service load information and service traffic information. In this embodiment, the data collection agent is responsible for collecting data on the operating status of the master node and its backup nodes, providing supporting data for the system to analyze cluster load and perform scaling.

[0099] The design concept of the embodiments of this application is briefly introduced below:

[0100] A heterogeneous resource cluster refers to a cluster composed of different types of computing nodes, used to provide diverse computing capabilities. The computing nodes in this cluster are also called heterogeneous nodes (or heterogeneous computing power nodes), used to execute various computing tasks. These tasks generate a large amount of data during execution, and this data needs to be effectively managed and stored.

[0101] To support efficient task execution and rational resource scheduling on heterogeneous nodes, cloud-native storage provides a reliable storage platform for the resulting data from these nodes, ensuring data security and accessibility. Specifically, by assigning tasks to appropriate cloud-native objects (such as specific container groups Pods) for execution and utilizing cloud-native storage to manage the data generated by these tasks, efficient resource scheduling and data management can be achieved, thereby supporting efficient task execution on heterogeneous nodes.

[0102] See Figure 1A As shown, it is a schematic diagram of the relationship between heterogeneous resource scheduling and resource object generation provided by related technologies.

[0103] exist Figure 1AIn this process, multiple computing tasks (such as computing task A, computing task B, and computing task C) are assigned to multiple heterogeneous nodes in a heterogeneous resource cluster for execution. During task deployment, relevant cloud-native objects are created. These cloud-native objects represent instances of objects such as node information, task runtime spaces, and task execution units. These cloud-native objects typically include node objects, daemon objects, namespace objects, and container group objects. These cloud-native objects are written to the backend cloud-native storage to support subsequent resource access and scheduling operations.

[0104] However, with the rapid development of large-scale artificial intelligence (AI) models, the demand for heterogeneous computing power has increased dramatically, posing a significant challenge to cluster scheduling performance. Current cloud-native storage employs a master-slave architecture, where the master node handles all write requests and replicates data to the slave node via asynchronous synchronization. Because this architecture requires all write operations to be performed through a single master node, allowing multiple master nodes to write simultaneously necessitates complex consistency protocols, a problem inherent in traditional master-slave architectures. Furthermore, regardless of the number of slave nodes added, write requests will still accumulate on the master node. Therefore, during periods of explosive growth in heterogeneous computing demand, current storage methods are ineffective at distributing the load and failing to significantly improve the scheduling performance of heterogeneous resource clusters.

[0105] In related technologies, to improve resource scheduling performance, a large-scale heterogeneous resource cluster can be split into multiple heterogeneous resource sub-clusters for independent management. The number of nodes in each sub-cluster is controlled within a relatively fixed range (e.g., within a thousand) to avoid write bottlenecks and scheduling performance issues caused by an excessively large single cluster.

[0106] See Figure 1B As shown, this is a schematic diagram of a method for dividing heterogeneous resource sub-clusters provided in related technologies. Figure 1B In this context, multiple computing tasks, after being connected, need to be adapted to different heterogeneous resource sub-clusters and deployed and executed in the corresponding sub-clusters. However, when the computing load of the business increases, or when the computing resource requirements (such as heterogeneous computing power nodes) of a single task change during execution, it is often necessary to split the computing tasks again and reallocate task resources according to the sub-cluster boundaries, which increases the complexity of overall management and scheduling.

[0107] Furthermore, this method of partitioning heterogeneous resource sub-clusters requires computing tasks to specify their corresponding heterogeneous resource sub-clusters before execution. The resource creation and scheduling processes of the task are bound to the sub-cluster structure, failing to decouple the resource layer from the task execution logic. This directly leads to two problems: firstly, business computing needs to frequently perceive and intervene in the underlying resource structure during access and adjustment, lacking scheduling flexibility; secondly, the number of nodes supported by a single sub-cluster is limited, thus restricting the maximum available resource scale for a task within a single cluster. Especially when facing the ever-increasing demand for large model training, this resource service model struggles to meet the scheduling and expansion needs of high-density heterogeneous computing resources.

[0108] In summary, how to adjust the storage method of cloud-native objects to improve the scheduling performance of heterogeneous resource clusters is an urgent problem to be solved.

[0109] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for storing resource objects. Specifically, the method first identifies, in a heterogeneous resource cluster, various resource objects involved in the production scheduling process of each heterogeneous node. These resource objects serve as the basic information units for heterogeneous nodes to execute tasks; their explicit definition helps establish a mapping relationship between resource usage status and task execution logic, thereby providing object-level processing basis for subsequent storage structure design.

[0110] Next, based on the total number of identified resource objects, a certain number of cloud-native storage services are deployed, and each cloud-native storage service is assigned a unique service identifier. This step dynamically plans the scale of the storage services, ensuring that the system's write capacity keeps pace with the growth of resource objects, avoiding performance bottlenecks caused by a single storage service handling too many write requests, and improving the system's scalability and write stability.

[0111] Then, combining the type of resource objects and their object information, various resource objects are divided into several object sets, and the service identifier of the cloud-native storage service corresponding to each object set is determined based on mapping rules. This division method, by establishing a classification structure on object type and object identifier, can achieve a reasonable mapping from resource objects to storage services, improve the accuracy of object write location, and facilitate balanced writes and orderly object management in multi-storage environments.

[0112] Finally, each object collection is written to its corresponding cloud-native storage service, with each storage service employing a master-slave architecture, where the master node handles write requests. By distributing write operations related to object collections across multiple master nodes, the pressure from centralized write request processing is significantly reduced, effectively alleviating the master node's write bottleneck. This ensures data availability while improving the system's write concurrency capabilities, providing performance assurance for the efficient execution of heterogeneous resource scheduling.

[0113] In summary, the resource object storage method provided in this application can optimize the configuration of resource object write paths through resource object identification, storage service construction, object set partitioning, and parallel write strategies, thereby effectively improving the write performance and scalability of cloud-native storage systems in heterogeneous resource cluster environments and supporting the scheduling needs of high-density heterogeneous nodes.

[0114] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0115] See Figure 2A The diagram shown illustrates an application scenario provided by an embodiment of this application. The application scenario diagram includes a terminal device 210 and a server 220.

[0116] In this embodiment, the terminal device 210 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client installed related to heterogeneous resource management or model training. This client can be software (e.g., a browser, AI training management platform), or a webpage, mini-program, etc. The server 220 is the backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for resource storage; this application does not impose specific limitations. The server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0117] It should be noted that the resource object storage method in each embodiment of this application can be executed by an electronic device, which can be a terminal device 210 or a server 220. That is, the method can be executed by the terminal device 210 or the server 220 alone, or by both the terminal device 210 and the server 220. For example, when executed by the server 220 alone, in a heterogeneous task scheduling scenario, when a task request is received, the server 220 executes the operation process for managing and writing resource objects related to the task, including: identifying the cloud-native objects associated with each heterogeneous node in the heterogeneous resource cluster that support task execution; constructing multiple cloud-native storage services based on the total number of all cloud-native objects; dividing them into several object sets according to object category and object identifier, and determining a corresponding storage service identifier for each set; finally, writing each object set to the corresponding cloud-native storage service, and having the master node in the master-slave architecture handle the related write requests, thereby achieving efficient storage and write load balancing of resources.

[0118] In one alternative implementation, the terminal device 210 and the server 220 can communicate via a communication network.

[0119] In one alternative implementation, the communication network is a wired network or a wireless network.

[0120] It should be noted that, Figure 2A The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0121] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the resource object storage method of this embodiment, the resource object data involved can be stored on the blockchain, such as the object set partitioning information, service identifier, etc.

[0122] Furthermore, the embodiments of this application can be applied to various scenarios, including not only resource scheduling and storage management scenarios of heterogeneous resource clusters, but also scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. Several scenarios will be briefly introduced below:

[0123] (I) Heterogeneous computing resource scheduling scenario in large model training platform.

[0124] The training tasks of current large-scale artificial intelligence models place high-concurrency and high-throughput scheduling requirements on heterogeneous computing resources. In such platforms, training tasks are typically broken down into multiple concurrently executed computing units, which are then scheduled to run on heterogeneous computing nodes. During the scheduling process, a large number of resource objects, such as container group objects (pods) and namespace objects, need to be dynamically created. The resource object storage method described in this application allows for the classification of all generated resource objects during the task scheduling phase. Based on object category and identifier, these objects are divided into multiple object sets and mapped to multiple cloud-native storage services. This avoids performance bottlenecks caused by concentrated resource object writing on a single master node, thereby improving the cluster's scheduling response capability and node scalability, and supporting the high-density computing resource usage requirements of large-scale model training tasks.

[0125] (ii) Dynamic resource expansion scenario in edge cloud collaborative environment.

[0126] In edge computing and central cloud collaborative deployment architectures, when edge nodes experience resource constraints or receive a large number of sudden tasks, the central server needs to quickly complete resource scheduling and expansion operations. In this scenario, the resource object storage method of this application embodiment can centrally aggregate and classify resource objects generated in the edge and central nodes. After hash calculation using object identifiers, the objects are written to a distributed cloud-native storage system, achieving efficient synchronization and disk persistence of resource objects. Based on the storage distribution status, the load can be dynamically assessed and expansion decisions can be made, smoothly integrating new heterogeneous nodes into the cluster. This method can break through the edge resource writing bottleneck, improve the efficiency of edge-cloud resource collaboration, and is suitable for scenarios such as scheduling of smart devices at the city edge and expanding intranets in industrial parks.

[0127] (III) Multi-task resource isolation and scheduling scenarios in intelligent vehicle platforms.

[0128] In-vehicle computing platforms typically need to run multiple heterogeneous tasks simultaneously, such as environmental perception, path prediction, vehicle-to-cloud synchronization, and in-vehicle entertainment. These tasks require high levels of resource isolation and real-time performance. In such multi-task environments, the system needs to create corresponding container group objects based on task type and deploy them across heterogeneous in-vehicle nodes. The resource object storage method described in this application can classify resource objects corresponding to various tasks according to their category (e.g., static deployment objects and dynamic task objects) and write them to different cloud-native storage services, reducing write conflicts and access interference. Simultaneously, the distributed write approach using a master-slave architecture ensures write performance stability, effectively supporting the scheduling performance requirements of the in-vehicle system during high task concurrency and rapid resource switching.

[0129] It should be noted that the application scenarios listed above are just simple examples, and other application scenarios are also applicable to the embodiments of this application, which will not be elaborated here.

[0130] See Figure 2B As shown, it is a schematic diagram of a resource object management architecture in a heterogeneous resource environment provided in an embodiment of this application.

[0131] like Figure 2B As shown, the resource object management architecture includes three layers: the resource object layer, the heterogeneous resource cluster layer, and the product application layer, which are used to illustrate the applicability of the resource management mechanism involved in this application in various heterogeneous computing scenarios.

[0132] exist Figure 2B At the bottom resource object layer, different types of resource objects are aggregated and managed, including resource information generated from various heterogeneous nodes. These resource objects can originate from statically deployed system components or dynamically generated task containers during task execution, exhibiting high dynamism and complexity.

[0133] exist Figure 2B The middle heterogeneous resource cluster layer displays a resource pool composed of multiple heterogeneous nodes, used to provide computing power support for upper-layer application tasks. Whether it is the rapid resource scheduling in the face of sudden tasks in the edge cloud environment, or the resource isolation and rapid switching of multiple types of tasks that need to run simultaneously in the vehicle platform, the heterogeneous resource cluster must have the ability to uniformly bear and elastically allocate multi-source resource objects.

[0134] exist Figure 2B The topmost product application layer covers typical AI task types such as speech models, vision models, game models, medical models, and NLP models. These application tasks place differentiated demands on the management and scheduling of computing resources, especially in resource-intensive and real-time scenarios, posing a greater challenge to the dynamic management capabilities of underlying resource objects.

[0135] In the above application scenarios, the method of this application embodiment can manage and store resource objects in heterogeneous resource clusters, and can support the generation, classification, scheduling and expansion of large-scale resource objects to ensure the smooth operation of various business tasks and the continuous stability of the system.

[0136] The following describes the storage method for resource objects provided by exemplary embodiments of this application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0137] See Figure 3The diagram shown is a flowchart of an implementation method for storing resource objects according to an embodiment of this application. The specific implementation process of this method is as follows: steps S31 to S34:

[0138] S31: For each heterogeneous node in the heterogeneous resource cluster, determine the various types of resource objects corresponding to the heterogeneous node; resource objects refer to cloud-native objects generated during the production scheduling process of the heterogeneous resource cluster to support task execution on the heterogeneous nodes.

[0139] Among them, heterogeneous resource clusters refer to clusters composed of heterogeneous nodes of different types, which are used to provide diverse computing capabilities.

[0140] Heterogeneous nodes refer to computing nodes with specific computing capabilities. They can be computing nodes that integrate graphics processing units (GPUs), neural processing units (NPUs), field-programmable gate arrays (FPGAs), or other dedicated accelerators. This article does not make specific limitations on this.

[0141] Heterogeneous nodes are used to execute specific computing tasks allocated by the resource scheduling system, supporting the running of various types of containerized tasks. Different types of heterogeneous nodes can support tasks of different scales or characteristics according to their hardware capabilities, serving as physical units in the task execution process and providing schedulable computing resources to the scheduling system.

[0142] In this embodiment of the application, the heterogeneous resource cluster is the basic environment for resource scheduling. Resource objects are generated during the task production scheduling process. Resource objects need to be efficiently managed and stored to support the execution of tasks on heterogeneous nodes.

[0143] Resource objects refer to cloud-native objects generated during the production scheduling process of heterogeneous resource clusters to support task execution on heterogeneous nodes.

[0144] Cloud-native objects are objects used in cloud-native architectures to represent various runtime entities, scheduled resources, service capabilities, or configuration relationships. They are a crucial foundation for building and managing distributed systems. In practical deployments, cloud-native objects can cover multiple dimensions such as task execution, system configuration, service orchestration, and security control. They possess unified lifecycle management characteristics and can be automatically scheduled, dynamically created, and deleted.

[0145] It should be noted here that resource objects are application abstractions of cloud-native objects under heterogeneous resource scheduling, and belong to a specific subset of cloud-native objects. In the following embodiments, "resource object" is used as a unified term to refer to cloud-native objects that are actually generated by heterogeneous nodes during task scheduling and require storage management.

[0146] In this application embodiment, resource objects may include, but are not limited to, the following: namespace objects for representing task isolation domains, pod objects for deploying task containers, node objects for describing node information, and daemonset objects for running system components resident in nodes, etc.

[0147] Furthermore, for heterogeneous resource clusters with complex business orchestration or advanced scheduling capabilities, the types of resource objects can be further expanded, such as: configuration map (ConfigMap) objects for storing application configuration information, service (Service) objects for representing network communication relationships and service discovery paths between resource objects, job (Job) objects and Cron Job (CronJob) objects for defining one-time computing tasks or periodic execution logic, persistent volume claim (PVC) objects for declaring and mounting persistent storage resources, and role (Role) objects and role binding (RoleBinding) objects for defining access permission policies, etc.

[0148] It should be noted that the resource objects listed above are just simple examples, and other resource objects are also applicable to the embodiments of this application, which will not be described in detail here.

[0149] Furthermore, it should be noted that the following implementation will use namespace objects, container group objects, node objects, and daemon set objects as examples to more clearly demonstrate the resource object partitioning and storage method proposed in this application. Other types of resource objects, if they possess similar structural attributes and lifecycle characteristics, are also applicable to the implementation mechanism provided in this application, and are not specifically limited here.

[0150] See Figure 4 As shown, it is a schematic diagram of the generation and association of resource objects provided in an embodiment of this application.

[0151] like Figure 4As shown, after a heterogeneous node is added to a heterogeneous resource cluster, a corresponding node object is generated for each heterogeneous node to represent the node's basic resource attributes and running status. For example, for nodes node1 to node6 shown in the figure, the system will generate node objects for each node to record information such as its computing power, memory configuration, and running status label.

[0152] Simultaneously, a daemonset object is created on each node to represent a system-level resource entity that resides long-term on that node, such as a monitoring agent, log collector, or state synchronization module. In this embodiment, the daemonset object and its corresponding node object are generated together during the node initialization phase and typically do not change dynamically during task scheduling, exhibiting strong stability and node binding characteristics. The daemonset object helps support the operational assurance and status reporting of basic capabilities in heterogeneous resource clusters and is one of the fundamental supporting objects for resource management and cluster scheduling.

[0153] For example, when a computing task A arrives at the cluster, in order to achieve logical isolation of resources, a namespace object namespace-A is first created for the task. Under this namespace, according to the concurrency requirements and resource planning of the task, multiple container group objects (such as container group objects-A1 to A4) are further created as the actual execution units of the task, and deployed to multiple heterogeneous nodes, such as heterogeneous node 1 (node ​​object is node1) and node 3 (node ​​object is node3).

[0154] It should be noted here that, Figure 4 The example only uses node1 and node3 to demonstrate the deployment path of container group objects. Other heterogeneous nodes (such as node2, node4, node5, and node6) also have node objects and daemon containers, but these are only shown in the example. Figure 4 Container group objects are not drawn in the diagram to simplify the illustration.

[0155] In this embodiment, there is a hierarchical relationship between container group objects and namespace objects, that is, container group objects can belong to their corresponding namespaces; there is a deployment relationship between container group objects and node objects, that is, container group objects are bound to and scheduled to specific heterogeneous nodes during actual runtime.

[0156] In one alternative implementation, in order to improve the management efficiency of resource objects, resource objects can be divided into different object categories according to the life cycle characteristics and resource change frequency of resource objects, so that subsequent targeted partitioning, storage and scheduling control operations can be performed.

[0157] Optionally, the object categories for resource objects include: static categories and dynamic categories.

[0158] Among them, static resource objects refer to resource objects whose size is basically fixed after being deployed to a heterogeneous resource cluster and do not dynamically increase or decrease during task execution. They are usually determined during the cluster planning stage. For example, node objects are used to describe the running status and resource capabilities of each heterogeneous node in the cluster. Their number depends on the size of the nodes and usually does not change frequently. Daemon set objects are used to deploy system-level service components. Their structure is also fixed during the node initialization stage, so they can be classified as static resource objects.

[0159] Dynamic resource objects refer to resource objects that, after being deployed to a heterogeneous resource cluster, can be elastically adjusted according to the concurrency level and scheduling strategy of business tasks. For example, namespace objects are typically created and released as the number of tasks increases; container group objects, as the smallest execution unit of task execution, are also dynamically created and destroyed during task execution, with frequent changes in their number, and therefore can be classified as dynamic resource objects.

[0160] See Figure 5 As shown, it is a schematic diagram of the classification of resource objects provided in an embodiment of this application.

[0161] Figure 5 The document illustrates how resource objects are categorized into two main types based on their lifecycle and change characteristics: static and dynamic. Taking resource objects, including node objects, daemon set objects, namespace objects, and container group objects, as an example, static resource objects include node objects and daemon set objects, while dynamic resource objects include namespace objects and container group objects. Different types of resource objects have different creation times and change frequencies, making them suitable for different management strategies and subsequent processing paths.

[0162] It should be noted that this application uses resource objects, including node objects, daemon set objects, namespace objects, and container group objects, as examples for classification and explanation. If other resource objects with similar lifecycle characteristics exist in actual applications, they can also be classified in the same way as those provided in this application, and no specific limitations are made here.

[0163] By implementing the above methods, classifying resource objects into static and dynamic categories helps to adopt differentiated management strategies based on the changing characteristics of objects when performing subsequent resource object classification, organization, and processing operations, thereby laying a structured foundation for the efficient distributed processing of resource objects.

[0164] In this embodiment of the application, to improve the storage scalability of resource objects and further avoid performance bottlenecks caused by centralized writes, multiple cloud-native storage services can be built to achieve decentralized management of resource objects, which will be described in detail below:

[0165] S32: Based on the total number of various resource objects, construct multiple cloud-native storage services; each cloud-native storage service corresponds to a service identifier.

[0166] Cloud-native storage services refer to storage service units used to store resource object data. They can be flexibly deployed and expanded based on a cloud-native architecture, supporting the writing, querying, and management of resource objects. Each cloud-native storage service is independent of the others, possesses independent data processing capabilities, and can handle the storage needs of different resource objects in parallel.

[0167] It should be noted that, in this embodiment, writing a resource object actually refers to writing key data such as metadata, runtime status information, and scheduling configuration related to the resource object, according to the object's partitioning structure, to its respective cloud-native storage service. This supports the accessibility and management integrity of the resource object in subsequent scheduling, querying, synchronization, and fault tolerance operations. This write operation includes not only the storage of the object's basic attributes but also structured records of object lifecycle management, state synchronization, and relationships with other resource objects.

[0168] In one alternative implementation, in order to achieve fine-grained management of resource objects on a per-volume basis, avoid storage pressure caused by excessive concentration of resource objects, and improve the scalability and load balancing capabilities of storage services, the number of cloud-native storage services to be built can be dynamically calculated based on the total number of resource objects.

[0169] Optionally, a third quantity is determined based on the ratio between the second quantity and the unit quantity; a third quantity of cloud-native storage services is constructed, and a service identifier is assigned to each cloud-native storage service.

[0170] The first quantity refers to the total number of various resource objects that each cloud-native storage service supports storing; the second quantity refers to the total number of various resource objects in the heterogeneous resource cluster.

[0171] In this application embodiment, the unit quantity refers to the upper limit of the resource object capacity that a single cloud-native storage service can support under stable operation. In an optional implementation, the unit quantity can be a preset configuration parameter; the second quantity is the total number of various resource objects in the heterogeneous resource cluster; and the third quantity is the number of cloud-native storage services that need to be built.

[0172] Furthermore, it should be noted that the unit quantity can be not only a preset configuration parameter, but also dynamically set based on the hardware resources (such as the number of CPU cores, memory capacity, and disk I / O capabilities) or software configurations (such as database instance specifications, container runtime configuration parameters, and concurrent connection limits) of the cloud-native storage service's operating environment. By combining the actual performance indicators of heterogeneous node clusters to calculate the unit quantity, more accurate capacity assessment and service partitioning strategies can be achieved, thereby improving the storage stability of resource objects and the operational efficiency of cloud-native storage services.

[0173] In one alternative implementation, a third quantity can be obtained by dividing the second quantity by the unit quantity and rounding up, which can then be used as the number of cloud-native storage services to be deployed. The formula for calculating the third quantity Y is shown in Formula 1 below:

[0174]

[0175] Where Y represents the third quantity, i.e., the number of cloud-native storage services that need to be built; K represents the second quantity, i.e., the total number of all resource objects in the heterogeneous resource cluster; M represents the unit quantity, i.e., the upper limit of the number of resource objects that a single cloud-native storage service can support storing; ∑k1, ∑k2, ∑k3, ∑k4, ..., ∑k n These represent the sum of the number of resource objects of type 1, type 2, ..., type n in the heterogeneous resource cluster, respectively. The symbol indicates rounding up.

[0176] Taking resource objects including node objects, daemon set objects, namespace objects, and container group objects as an example, the formula for calculating the third quantity Y can be as follows: Formula 2:

[0177]

[0178] ∑k node ∑k can represent the total number of node objects. daemonset ∑k represents the total number of daemon set objects. namespace ∑k represents the total number of namespace objects. pod This indicates the total number of container group objects.

[0179] By adding up the quantities of the above-mentioned resource objects, we obtain the second quantity K. Then, by dividing K by the unit quantity M and rounding up, we can determine the number Y of cloud-native storage services that need to be built.

[0180] For example, at a certain moment, the resource objects generated by the heterogeneous resource cluster during task scheduling and execution include: 200 node objects, 200 daemon set objects, 4000 namespace objects, and 25000 container group objects, totaling 29400 resource objects. If the unit quantity M is configured to support 10000 resource objects per cloud-native storage service, then the calculation is as follows:

[0181]

[0182] That is, three cloud-native storage services should be built for the subsequent partitioning and writing of resource objects.

[0183] After building the third cloud-native storage service, assign a service identifier to each cloud-native storage service. The service identifier is used to identify the logical number of each cloud-native storage service and is the basis for resource object mapping and storage operations.

[0184] In this embodiment, service identifiers can be assigned using consecutive numbering. Furthermore, prefix numbers or encoding rules can be set based on service deployment region, service role, or priority to support more complex service identification needs and enhance the semantic expressiveness and management flexibility of service identifiers. The following will use consecutively numbered service identifiers as an example to illustrate the mapping and ownership relationships of each resource object in the storage process.

[0185] For example, when the third quantity is 3, service identifiers 0, 1 and 2 can be assigned respectively, corresponding to the first, second and third cloud-native storage services.

[0186] For example, if a resource object is calculated to have a service identifier of 1 during a subsequent mapping process, then that resource object will be written to the cloud-native storage service with a service identifier of 1. The introduction of service identifiers establishes a clear one-to-one correspondence between resource objects and target storage, which helps to achieve precise control over object partitioning, writing, and reading.

[0187] Through the above implementation methods, the required number of cloud-native storage services can be dynamically determined and constructed based on the total number of various resource objects in the heterogeneous resource cluster, achieving decentralized management of resource objects and avoiding write bottlenecks caused by centralized writing to a single storage service. Furthermore, by setting a unit limit on the capacity of resource objects supported by each cloud-native storage service, and rationally allocating the number of storage services based on this capacity, it is beneficial to maintain the scalability and stability of the storage architecture as the number of resource objects continues to grow.

[0188] Furthermore, by assigning a unique service identifier to each cloud-native storage service, a basis for identification can be provided for the subsequent partitioning, mapping, and writing of resource objects, thereby improving the granularity of resource management and the accuracy of write control, and laying the foundation for subsequent efficient scheduling and read / write optimization.

[0189] After completing the above service identifier allocation, in order to further improve the flexibility and structural clarity of resource object partitioning and management, in an optional implementation, different cloud-native storage services include: a first cloud-native storage service and multiple second cloud-native storage services.

[0190] The first and second cloud-native storage services are logically categorized based on functional roles and management strategies, used to differentiate the mapping paths and processing methods for different resource objects. The first cloud-native storage service is a single instance, while multiple second cloud-native storage services constitute a scalable set of cloud-native storage services.

[0191] In one optional implementation, if the number of cloud-native storage services constructed is N+1, then each cloud-native storage service corresponds to a service identifier, and the value of the service identifier ranges from 0 to N. Specifically, the cloud-native storage service with service identifier N is the first cloud-native storage service, and the cloud-native storage services with service identifiers from 0 to N-1 are multiple second cloud-native storage services.

[0192] It should be noted that the above classification method and service identifier allocation logic are only used to establish the structural correspondence between different cloud-native storage services. They do not limit the mapping method between specific resource objects and services, but rather provide a structural basis for subsequent resource object division and service identifier determination.

[0193] Furthermore, it should be noted that the number of the first cloud-native storage service and the second cloud-native storage service can be adjusted according to actual needs. For example, in some implementation scenarios, a structure of multiple first cloud-native storage services and multiple second cloud-native storage services can be set to meet the needs of more complex resource allocation strategies. The following embodiments of this application use the structure of "one first cloud-native storage service and multiple second cloud-native storage services" as an example for illustration. Other structural forms can also be applied to this application, and no specific limitations are imposed.

[0194] In one optional implementation, assuming the total number of cloud-native storage services to be built is N+1, a unique service identifier can be assigned to each cloud-native storage service, with the identifier ranging from 0 to N. The cloud-native storage service with service identifier N is the first cloud-native storage service, and the remaining cloud-native storage services with service identifiers from 0 to N-1 are the multiple second cloud-native storage services.

[0195] See Figure 6As shown, it is a schematic diagram of a cloud-native storage service identifier allocation structure provided in an embodiment of this application.

[0196] like Figure 6 As shown in this embodiment, assuming the total number of cloud-native storage services to be built is N+1, a unique service identifier can be assigned to each cloud-native storage service to identify its logical number in the heterogeneous resource cluster. The service identifiers are assigned using a sequential numbering method starting from 0, with a range of 0 to N.

[0197] Specifically, the cloud-native storage service numbered N is the first cloud-native storage service, and the cloud-native storage services numbered 0 to N-1 are multiple second cloud-native storage services. This numbering method can achieve a clear division of the first cloud-native storage service and the second cloud-native storage service in terms of logical structure.

[0198] Figure 6 The structural layout of the identifier allocation relationship is further illustrated: at the top is the first cloud-native storage service, corresponding to service identifier N; below are multiple second cloud-native storage services numbered N-1, N-2, up to 0, each cloud-native storage service is bound to the corresponding service identifier.

[0199] By logically dividing cloud-native storage services as described above, the role of cloud-native storage services can be clearly defined in the overall architecture, laying the foundation for the group management and distributed control of different types of resource objects, which is conducive to improving the clarity and controllability of subsequent resource object storage.

[0200] The above process describes how to build multiple cloud-native storage services based on the total number of resource objects and assign service identifiers, establish a one-to-one mapping relationship between resource objects and cloud-native storage services, and introduce the structural division logic of "first cloud-native storage service" and "multiple second cloud-native storage services", providing a role basis and structural support for the subsequent classification, storage and management of different types of resource objects.

[0201] The following section will introduce how to classify various resource objects according to their object categories and object identifiers, and determine their corresponding cloud-native storage services, thereby achieving the grouping and accurate mapping of resource objects.

[0202] S33: Based on the object category and object identifier of each type of resource object, divide each type of resource object into multiple object sets, and determine the service identifier corresponding to each object set.

[0203] In this context, an object set refers to a collection of resource objects that share the same category attributes and object identifiers, aggregated together during the resource object partitioning process. As the basic granularity for partitioning and writing, the object set identifies the cloud-native storage service to which this batch of resource objects should belong.

[0204] The above process has described how to build multiple cloud-native storage services and assign service identifiers. It also explains that resource objects can be distinguished by object category. In an optional implementation, to better adapt to the differences in storage structure caused by different lifecycle characteristics of resource objects, this application also proposes a classification and attribution method based on object category. This method achieves the classified storage and distributed writing of cloud-native objects by centrally mapping statically categorized resource objects and distributively mapping dynamically categorized resource objects. Optionally, an optional implementation of S33 is as follows: steps S331 to S332 (… Figure 3 (Not shown):

[0205] S331: Allocate static resource objects from various resource objects to the same object set, and use the service identifier of the first cloud-native storage service as the service identifier corresponding to the same object set; wherein, static resource objects refer to resource objects with a fixed size after being deployed to a heterogeneous resource cluster.

[0206] In this embodiment, statically categorized resource objects may include node objects and daemon set objects, the number of which is determined during cluster initialization or planning and is typically not adjusted during business scheduling. By grouping these objects into the same object set and uniformly mapping them to the service identifier of the first cloud-native storage service, the efficiency of centralized management of subsequent resource objects can be improved, providing a basic mapping path for centralized object writing.

[0207] S332: Divide dynamic resource objects among various resource objects into at least one object set according to their respective object identifiers, and determine the service identifier corresponding to at least one object set in the service identifiers of multiple second cloud-native storage services; wherein, dynamic resource objects refer to resource objects that can be elastically adjusted according to demand after being deployed to a heterogeneous resource cluster.

[0208] In this embodiment, dynamically categorized resource objects may include namespace objects and container group objects. These objects are typically dynamically created and released by the scheduling process during task issuance or execution. By performing mapping partitioning based on object identifiers and determining the service identifier to which an object belongs from among the service identifiers of multiple second cloud-native storage services, distributed partitioning and fine-grained management of objects can be achieved, providing efficient support for the distributed writing and scheduling of subsequent resource objects.

[0209] See Figure 7 As shown, it is a schematic diagram of the resource object classification and cloud-native storage affiliation provided in an embodiment of this application.

[0210] like Figure 7 As shown, resource objects are first categorized into static and dynamic categories based on their object type. Static resource objects are uniformly grouped into a single object set, which is then mapped to a service identifier of the first cloud-native storage service, enabling centralized management and unified writing of static objects. Dynamic resource objects are further divided into multiple object sets based on their respective object identifiers. Each object set is mapped to multiple service identifiers of the second cloud-native storage service, enabling distributed management and distributed writing of dynamic objects.

[0211] In this embodiment, static resource objects include node objects and daemon set objects. Their size is set during the initialization phase of the heterogeneous resource cluster. They have long lifecycles and low change frequency, making them suitable for a centralized ownership strategy. Dynamic resource objects include namespace objects and container group objects. Their number changes dynamically with the start and termination of computing tasks. They are suitable for hash mapping using object identifiers and allocation to different second cloud-native storage services, thereby distributing storage load and improving concurrent writes.

[0212] Through such Figure 7 The partitioning mechanism shown can determine the service identifier of a resource object at the generation stage, thereby realizing a resource ownership strategy of "static centralization and dynamic decentralization". While ensuring the clarity of the storage structure, it significantly improves the resource write performance, scheduling flexibility and system scalability, and builds a unified, clear and efficient underlying support architecture for resource object management in heterogeneous resource clusters.

[0213] Through the above implementation methods, a differentiated partitioning mechanism can be established for resource objects with different lifecycle characteristics. Static resource objects are centrally classified into the same object set and uniformly assigned to the first cloud-native storage service, which helps to improve the centralized management efficiency and access consistency of stable objects. At the same time, dynamic resource objects are divided into multiple object sets based on object identifiers and distributed to multiple second cloud-native storage services. This enables the decentralized carrying and fine-grained control of large-scale dynamic objects, reduces single-point pressure, and improves the throughput and scalability of resource writes. This optimizes the overall storage management structure of resource objects and provides stable support for subsequent scheduling execution, read and write performance optimization, and system elastic expansion.

[0214] To further improve the consistency of ownership and access aggregation of dynamic category resource objects during the partitioning process, this application also proposes an object partitioning method based on namespace ownership. Optionally, dynamic category resource objects include: namespace objects and container group objects. Namespace objects are typically used to represent the logical isolation environment of a task, while container group objects serve as the specific task units running under that namespace. The two have a natural subordinate relationship and high access relevance in the actual resource scheduling and task execution process.

[0215] Given the binding relationship between namespaces and container groups, and their frequent use as a unified context for read and write operations in access scenarios, it is necessary to bind them during the partitioning phase to ensure the aggregation of subsequent ownership and the coordination of operations in order to maintain the consistency of object partitioning and access paths.

[0216] Optionally, for each namespace object, perform the following operations respectively: S3321 to S3323 ( Figure 3 (Not shown in the image):

[0217] S3321: Group a namespace object and its container groups into the same object set.

[0218] In this embodiment, namespace objects and container group objects constitute a hierarchical structure of resource organization. Container groups are typically derived from namespace objects and are bound and persist throughout their lifecycle.

[0219] Including namespace objects and container group objects in the same object set enables synchronization of resource object ownership identifiers, avoiding issues such as cross-service read / write, information inconsistency, or operation delays caused by object ownership separation during subsequent storage access.

[0220] For example, in a scenario where two computing tasks are received, to achieve logical isolation of task resources, corresponding namespace objects are created for each task. Task A corresponds to namespace-A, and task B corresponds to namespace-B. Under namespace-A, there are multiple container group objects pod-A1 and pod-A2; under namespace-B, there are container group objects pod-B1, pod-B2, and pod-B3. During resource object partitioning, namespace-A and its subordinate pods pod-A1 and pod-A2 are grouped into one set of objects, while namespace-B and its subordinate pods pod-B1, pod-B2, and pod-B3 are grouped into another set of objects. This ensures that each namespace object and its associated container group objects can subsequently belong to the same cloud-native storage service, achieving consistency in object ownership and aggregation of access paths. This avoids problems such as access redirection, information redundancy, or degraded write performance caused by resource objects belonging to the same task being distributed to different services.

[0221] The above describes the method for handling the consistency of ownership between namespace objects and their subordinate container group objects during the partitioning process. In order to further realize the automatic mapping ownership between object collections and specific cloud-native storage services, the following will introduce a hash operation mechanism based on the object identifier of namespace objects, which is used to determine the service identifier of the cloud-native storage service to which it belongs.

[0222] S3322: Obtain a hash value by performing a hash operation on the object identifier of a namespace object.

[0223] In this embodiment, since both namespace objects and container group objects are dynamic resource objects with scalability and high frequency of change, static mapping can easily lead to uneven distribution and difficulty in expansion. Therefore, a hash operation based on the object identifier of the namespace object can be used to achieve automatic mapping and ownership.

[0224] In this embodiment of the application, the formula for hash operation is shown in Formula 3 below:

[0225] H = hash(value(namespace)) (Formula 3)

[0226] Where H is the hash result of the namespace object, value(namespace) is the object identifier of the namespace object, and hash is the hash algorithm.

[0227] In this embodiment of the application, the object identifier of the namespace object can be a unique code of the namespace, such as a globally unique name generated when it is registered in the cluster (e.g., namespace-a-training-job-001), or its task number, business ID, etc. during the scheduling process, to ensure its uniqueness and stability in the system.

[0228] In one alternative implementation, the hash operation algorithm can be the MD5 hash algorithm, which has the advantages of fast calculation speed and uniform distribution, and is suitable for resource object mapping scenarios.

[0229] For example, if the object identifier of a namespace object is namespace-a-training-job-001, it can be used as input and processed using Message Digest Algorithm 5 (MD5). The resulting 128-bit hash digest value is 3e4f2a10b7c88f52c1d0eae8391c2a9f. Taking the first 8 bits, 3e4f2a10, and converting them to a decimal integer, we get the hash value 1047781648. This hash value will be used as the basis for calculating the service identifier of the object set to which the namespace object belongs, and will participate in the subsequent service identifier determination operation.

[0230] In another alternative implementation, the hash operation algorithm can also be Secure Hash Algorithm 1 (SHA-1) or Cyclic Redundancy Check 32 (CRC32). SHA-1 is suitable for scenarios with low tolerance for hash collisions and high security requirements, while CRC32 is suitable for lightweight and fast mapping scenarios with high requirements for computation speed and resource consumption.

[0231] It should be noted that the hash algorithms listed above are just simple examples, and other hash algorithms are also applicable to the embodiments of this application, which will not be described in detail here.

[0232] By performing a hash operation on the object identifier of a namespace object using a hash algorithm, the corresponding hash value can be obtained. This hash value is used as the basis for subsequently determining the service identifier of the cloud-native storage service, thereby enabling the automatic distribution and ownership determination of resource objects.

[0233] S3323: Take the service identifier that matches the hash value in the service identifiers of multiple second cloud-native storage services and use it as the service identifier corresponding to the object set to which a namespace object belongs.

[0234] In this embodiment, after obtaining the hash value of a namespace object, a service identifier needs to be determined from multiple service identifiers of the second cloud-native storage service as the corresponding service identifier of the object set to which the namespace object belongs. This "hit" relationship means that by processing the hash value according to specific rules (e.g., performing a modulo operation on the hash value and the total number of the second cloud-native storage services, dividing the hash value into intervals and mapping, or combining load weights for weighted selection, etc.), a unique identifier is selected from multiple preset service identifiers for subsequent resource object writing.

[0235] This processing method is typically based on a mapping rule between hash values ​​and service identifiers to ensure that the same namespace object can be identified with the same service identifier in each processing step, thus achieving consistency in the ownership of resource objects.

[0236] This approach not only ensures that namespace objects and their subordinate container group objects are managed uniformly within the same service, but also enables the reasonable distribution of dynamic category resource objects across multiple cloud-native storage services. This improves write performance, reduces cross-service read / write overhead, and provides structured support for subsequent data access and scheduling.

[0237] Through the above implementation method, namespace objects and their container group objects can establish a unified object set affiliation during the partitioning phase, and use a hash mapping mechanism to match the corresponding second cloud-native storage service identifier, thereby realizing the logical aggregation, load balancing, and access coordination of dynamic resource objects. This approach not only improves the structural consistency of resource object affiliation relationships but also enhances the efficiency and stability of subsequent distributed write and scheduling operations, making it particularly suitable for heterogeneous computing scenarios with a large number of dynamic objects that change frequently.

[0238] To further improve the computational efficiency and operability of the service identifier allocation process, this application also proposes a service identifier matching method based on a modulo algorithm, which is used to quickly determine the service identifier to which an object set belongs in multiple second cloud-native storage services. In one optional implementation, the hash value corresponding to the namespace object is taken as input, and the service identifier corresponding to the namespace object is directly matched by performing a modulo operation with the total number of second cloud-native storage services, thereby achieving efficient distribution and balanced mapping of resource objects.

[0239] Optionally, one possible implementation of S3323 is as follows: S33231~S33232 ( Figure 3 (Not shown):

[0240] S33231: Obtain the result by performing a modulo operation on the hash value and the first quantity; the first quantity is the total number of multiple second cloud-native storage services.

[0241] In this embodiment, step S33231 is used to uniformly map the hash value of the namespace object to the range of service identifiers of each second cloud-native storage service, thereby achieving distributed ownership of resource objects and avoiding performance bottlenecks caused by centralized writes. The formula for the modulo operation is shown in Formula 3 below:

[0242] R = H % G (Formula 4)

[0243] Where R represents the result of the modulo operation; H represents the result of the hash operation on the namespace object; and G represents the total number of the second cloud-native storage services, i.e., the first number.

[0244] For example, taking the object identifier of the namespace object as namespace-a-training-job-001, the hash value obtained by MD5 hashing is: H = 1047781648. Assuming that three second cloud-native storage services have been built in the current cluster, i.e., the first quantity G = 3, the modulo calculation result is:

[0245] R = 1047781648 mod 3 = 2

[0246] After obtaining the calculation results, the corresponding service identifier can be determined from the service identifiers of multiple second cloud-native storage services, which will be described in detail below.

[0247] S33232: Take the service identifier that matches the calculation result from the service identifiers corresponding to each of the second cloud-native storage services and use it as the service identifier of the object set to which the namespace object belongs.

[0248] Taking the above example again, assuming that the service identifiers of the three second cloud-native storage services are 0, 1, and 2 respectively, the calculation result R=2 will match the second cloud-native storage service with service identifier 2.

[0249] At this point, the namespace object namespace-a-training-job-001 and its associated container group objects (such as pod-A1 and pod-A2) can be included in the object set corresponding to the service identifier and written to the second cloud-native storage service with service identifier 2.

[0250] By employing the above implementation methods, combining hash operations and modulo operations, service identifier allocation can be completed quickly and stably during the resource object partitioning phase. This not only ensures the uniformity and predictability of resource object distribution but also improves the efficiency of resource write path location and reduces the computational complexity of service identifier allocation. Furthermore, this approach exhibits good scalability; when the number of second cloud-native storage services changes, it can still dynamically adapt to the new service structure based on the same computational logic, making it suitable for the rapid writing and ownership management needs of dynamic objects in large-scale heterogeneous resource clusters.

[0251] See Figure 8 As shown, it is a schematic diagram of the structure of resource object classification and attribution mapping provided in an embodiment of this application.

[0252] like Figure 8 As shown, during the partitioning phase, resource objects are first categorized into static and dynamic categories based on their object type. Static resource objects (such as node objects and daemon set objects) are uniformly grouped into the same object set and directly assigned to the First Cloud Native Storage Service, enabling centralized writing and management.

[0253] Dynamically categorized resource objects (such as namespace objects and container group objects) undergo a hash operation based on their respective object identifiers to obtain a hash value. Subsequently, by performing a modulo operation between the hash value and the number of services in the second cloud-native storage service, the target service identifier to which the dynamic object belongs is determined. Finally, the object is assigned to the corresponding object set and written to the corresponding second cloud-native storage service.

[0254] The aforementioned structure, through a "static unification and dynamic fragmentation" partitioning approach, achieves hierarchical organization and precise data storage of resource objects. This improves write efficiency while enhancing the system's responsiveness to dynamic loads. A first cloud-native storage service provides stable support, while multiple second cloud-native storage services enable parallel scaling, effectively improving the overall storage architecture's elasticity and scalability. This makes it suitable for resource scheduling scenarios involving high-frequency writes and high-concurrency access in large-scale heterogeneous resource clusters.

[0255] In addition to achieving the target service identifier by performing a modulo operation on the hash value and the total number of second cloud-native storage services, the following methods can also be used in this embodiment to determine the service identifier corresponding to the namespace object:

[0256] Method 1: Use consistent hashing algorithm. This involves constructing a virtual hash ring and mapping the service identifiers of all second cloud-native storage services to this hash space. The hash value of the namespace object will be mapped onto the hash ring, and the first service identifier matched in the clockwise direction belongs to it. This method has the characteristics of high scalability and minimal impact of node changes on data migration, and is suitable for dynamic environments with frequent additions and subtractions of nodes.

[0257] Method 2: Use a hash segmentation mapping mechanism, which divides the hash value space into several fixed segments. Each segment corresponds to a service identifier. After the namespace object is hashed, it directly matches the corresponding service identifier based on the segment it falls into. This method is suitable for scenarios where the number of service identifiers is fixed.

[0258] Method 3: Use the service weight ratio mapping method, which combines the load capacity or resource allocation of each second cloud-native storage service, sets different weight ranges, and then calculates the belonging service identifier through hash value and range mapping rules to achieve capacity-based load balancing writes.

[0259] By introducing the aforementioned multiple service identification methods, we can not only adapt to different types of resource distribution strategies and business complexity requirements, but also further enhance the flexibility of resource object partitioning logic and the scalability of system architecture. Especially in heterogeneous resource clusters, where nodes frequently expand and shrink and the number of resource objects changes drastically, these methods can effectively reduce data migration costs, reduce hotspot concentration, optimize write paths, and improve the system's stable operation and resource scheduling efficiency in high-concurrency, multi-tasking scenarios.

[0260] It should be noted that the service identifier determination methods listed above are just simple examples, and the service identifier determination methods are also applicable to the embodiments of this application, and will not be described in detail here.

[0261] The above process describes how to automatically partition and map resource objects across multiple cloud-native storage services based on their category and object identifier, using hash calculation and service identifier matching.

[0262] In addition to the service identifier allocation method based on hash calculation in this application embodiment, a method such as equal allocation based on the number of resource objects can also be adopted. That is, during the writing process, resource objects are sequentially allocated to the service identifiers of each cloud-native storage service according to a round-robin or equal-division strategy to achieve initial load balancing among services. Furthermore, static binding or policy-driven allocation can also be performed based on rules such as business domains and scheduling tags. The above methods can be flexibly selected or combined according to actual needs and are also applicable to the embodiments of this application, and are not specifically limited here.

[0263] To complete the storage of resource objects and support subsequent access and scheduling operations, the actual storage process of the object collection will be introduced below.

[0264] S34: For each set of objects, store the set of objects in the cloud-native storage service represented by the corresponding service identifier; wherein, each cloud-native storage service adopts a primary-secondary architecture, and the primary node in the primary-secondary architecture is used to handle write requests related to each resource object in the corresponding set of objects.

[0265] In this embodiment, once resource objects are partitioned into object sets and their respective cloud-native storage service identifiers are determined, the system can write the resource object data in the object set to the corresponding cloud-native storage service. Each cloud-native storage service is built on a master-slave architecture, containing one master node (M) and at least one slave node (S). The master node is responsible for processing write requests for all resource objects in the object set, ensuring the order and consistency of data writing; the slave node receives data changes from the master node through asynchronous or semi-synchronous mechanisms to ensure high availability and fault tolerance.

[0266] See Figure 9 As shown, it is a schematic diagram of a cloud-native storage primary-backup architecture provided in an embodiment of this application.

[0267] Figure 9 The diagram illustrates the primary and secondary node configuration structure of the first cloud-native storage service and multiple secondary cloud-native storage services. Each cloud-native storage service contains one primary node (M) and several secondary nodes (S). When writing to an object collection, all write operations are first received and processed by the primary node. After successful writing, the data is then synchronized to the corresponding secondary nodes, thereby achieving a unified guarantee of write reliability and storage security.

[0268] The write mechanism of the master-slave architecture described above can not only effectively support high-concurrency writes to object collections, but also ensure uninterrupted business operation through master-slave failover in the event of node failure, thereby improving the robustness of resource object storage and the overall stability of the system. This mechanism is particularly suitable for scenarios requiring parallel storage and high-frequency updates of multiple types of resource objects in heterogeneous resource clusters.

[0269] To further enhance the data consistency guarantee capability of resource objects in a large-scale distributed environment, in one optional implementation, this application also proposes a data synchronization mechanism for multiple second cloud-native storage services to eliminate the data silo problem caused by scattered resource writing.

[0270] Optionally, for each set of objects, after storing the set of objects in the cloud-native storage service represented by the corresponding service identifier, data synchronization is performed between the various secondary cloud-native storage services to ensure the consistency of stored data between each secondary cloud-native storage service.

[0271] In this embodiment, multiple second cloud-native storage services are deployed independently, each carrying storage and write requests for different object sets. To support system operation requirements such as cross-set queries, task scheduling switching, and read / write route adjustments, a cross-service synchronization mechanism can be introduced: when a second cloud-native storage service completes data writing or updating, in one optional implementation, the newly added or modified data can be synchronized to other designated second cloud-native storage services through timed synchronization, message-driven methods, or log subscriptions, thereby achieving asynchronous or eventually consistent data sharing.

[0272] For example, suppose the namespace object namespace-A and its container group objects A1 and A2 are stored in the second cloud-native storage service A. After the write is completed, the system can trigger a cross-service synchronization operation to copy the information of these resource objects to the second cloud-native storage services B and C, so that they have the same object view, so as to achieve smooth switching and data inheritance during service scheduling switching or disaster recovery fault tolerance.

[0273] After completing the above data synchronization operations, multiple secondary cloud-native storage services will maintain copies of the same object set, meaning that each secondary cloud-native storage service contains all resource object data from the corresponding object set. Therefore, when performing subsequent resource object read operations, the system can retrieve the complete data content of the required resource object from any secondary cloud-native storage service based on access policies, service load, or operating status.

[0274] This approach not only improves the flexibility of resource object read paths but also enhances the system's availability and fault tolerance in the event of node anomalies, task migration, or service rescheduling. Simultaneously, enabling read request load balancing across multiple secondary cloud-native storage services helps reduce single-node access pressure, optimizes overall read / write performance, and ensures stable readability and service continuity of resource objects in heterogeneous resource clusters under high-concurrency and strong consistency requirements.

[0275] The above implementation methods effectively enhance the system's collaborative capabilities in a multi-cloud native storage service parallel deployment environment, ensuring consistent data visibility across multiple services and reducing data access inconsistencies caused by node isolation, request drift, or abnormal recovery. Simultaneously, it provides underlying consistency guarantees for subsequent implementations of distributed resource object indexing, multi-service read / write redundancy, and global task scheduling.

[0276] To further achieve stable access of resource objects in high-concurrency environments and dynamic expansion capabilities of heterogeneous resource clusters, this application also proposes a collection and analysis mechanism based on the operating status of storage services. By acquiring the service load information of each cloud-native storage service, the overall operating pressure of the cluster is assessed, and automated expansion decisions are made accordingly.

[0277] See Figure 10 The diagram shown is a flowchart of an implementation method for expanding a heterogeneous resource cluster according to an embodiment of this application. The specific implementation process is as follows: steps S101 to S103:

[0278] S101: Collect service load information for each cloud-native storage service; whereby each service load information is determined based on the resource usage of each resource object stored in the corresponding cloud-native storage service.

[0279] Among them, service load information refers to the real-time resource usage of the current cloud-native storage service during the processing of resource objects, which is used to characterize the operating load status of the storage service.

[0280] In one optional implementation, the service load information may include, but is not limited to, the following: CPU utilization, memory utilization, and disk I / O utilization of the primary and backup nodes within a specified monitoring period.

[0281] The service load information mentioned above allows for a comprehensive understanding of the current operational pressure and resource consumption trends of each cloud-native storage service, providing a foundation for subsequent cluster load analysis and capacity expansion assessment.

[0282] To achieve continuous monitoring and automated data collection of service load, this application also proposes a resource monitoring mechanism based on a data collection agent. Optionally, a data collection agent is deployed on each cloud-native storage service. In one optional implementation, the service load information is obtained by collecting the resource usage of the primary node and its corresponding backup nodes on the corresponding cloud-native storage service through the data collection agent on that cloud-native storage service.

[0283] Resource usage may include, but is not limited to, usage metrics such as CPU utilization, memory utilization, and disk usage.

[0284] In one optional implementation, the data collection agent can sample and record resource usage at fixed time intervals (e.g., every 10 seconds, 30 seconds, or 1 minute). During collection, it can obtain host metrics such as CPU utilization and memory usage by calling resource monitoring interfaces provided by the operating system (e.g., process information file systems), or obtain container-level or node-level resource usage by accessing metric interfaces exposed by container management components or monitoring modules. After obtaining resource usage data, the data collection agent can parse it into structured data and write it uniformly into a database to support subsequent resource load assessment and scheduling strategies.

[0285] For example, if the collection agent detects that the CPU utilization of the master node of the second cloud-native storage service reaches 91%, the memory utilization reaches 87%, and the disk utilization reaches 93% during a certain monitoring period, the collection agent corresponding to the second cloud-native storage service can report the service load information during the monitoring period to the database.

[0286] In this embodiment of the application, in order to realize the automatic generation and visualization of cluster load information, the service load information reported by each cloud-native storage service can be written into a centralized database.

[0287] See Figure 11A As shown, it is a schematic diagram of a service load information collection and reporting mechanism provided in an embodiment of this application.

[0288] Figure 11A The document demonstrates the primary and backup node configuration structure of two cloud-native storage services. Each cloud-native storage service is equipped with a data collection agent, which periodically collects load data from the primary node (M) and backup node (S) and reports the data to the database.

[0289] Through the above implementation methods, automatic collection, real-time updating, and structured management of service load information are achieved, avoiding the information lag or missing issues caused by manual monitoring. The collection agent can cover the resource usage of both primary and backup nodes, forming multi-dimensional, multi-node operational data. This provides a highly reliable decision-making basis for cluster expansion strategies, effectively enhancing the self-awareness and response speed of the entire cloud-native storage system, and improving the system's operational stability and elastic scalability.

[0290] The above process describes how to collect and obtain service load information for each cloud-native storage service. The following section will describe in detail how to obtain cluster load information for heterogeneous resource clusters based on the above service load information.

[0291] S102: Obtain the cluster load information of the heterogeneous resource cluster based on the service load information of each cloud-native storage service.

[0292] In this embodiment, cluster load information refers to the result obtained by uniformly analyzing the service load information of all cloud-native storage services in a heterogeneous resource cluster, reflecting the overall operating pressure level and resource usage trend of the cluster. Its purpose is to assess resource bottlenecks and determine expansion needs from a global perspective, and to provide data support for subsequent expansion decisions.

[0293] In this embodiment of the application, the latest service load information can be read from the database periodically, and a cluster-level service load information chart, i.e., cluster load information, can be dynamically drawn based on the service load information. The cluster load information includes, but is not limited to: CPU utilization curve of the master node, memory usage curve of the master node, disk usage histogram of the master node, etc.

[0294] See Figure 11B As shown, it is a schematic diagram of cluster load information generation and visualization provided in an embodiment of this application.

[0295] Figure 11B The diagram illustrates how resource usage data recorded in the database is extracted to create a view of cluster load information at the cluster level, including CPU curves, memory change trend charts, and other multi-dimensional perspectives.

[0296] For example, read the CPU, memory utilization curves, and disk utilization of all cloud-native storage service master nodes from the database within the past hour, and generate the above data. Figure 11B If the cluster load information shows that the current average CPU utilization is 78%, some nodes have a peak CPU utilization of 91%, memory utilization exceeds 85%, and the disk utilization distribution chart shows that multiple nodes have disk occupancy approaching 90%, then it can be judged that the cluster as a whole is on the verge of high-pressure operation.

[0297] This method allows for rapid perception of the overall operational status of the cluster within a specific time period and identification of potential bottleneck nodes, providing timely and effective data support for dynamic adjustments to resource scheduling strategies and capacity expansion decisions.

[0298] Through the above implementation methods, unified cluster load information can be aggregated and generated at the cluster level based on the service load information of various cloud-native storage services, enabling a global perception of the overall operating status of heterogeneous resource clusters. This approach helps improve the real-time performance and accuracy of cluster-level resource monitoring, providing unified and quantitative data support for subsequent operational status assessment and scaling decisions, and enhancing the adaptive and resource regulation capabilities of heterogeneous resource clusters in high-concurrency and dynamic scheduling scenarios.

[0299] The above process describes how to obtain cluster load information. The following section will introduce the scaling decision-making method based on cluster load information.

[0300] S103: Make scaling decisions for heterogeneous resource clusters based on cluster load information.

[0301] In this embodiment of the application, S103 is used to dynamically determine the overall operating status of the current heterogeneous resource cluster based on the cluster load information obtained in the aforementioned step S102, and determine whether there is a need for expansion.

[0302] When determining whether there is a need for capacity expansion, a comprehensive evaluation can be conducted using multiple performance metrics, such as CPU utilization, memory utilization, disk usage, and write latency of the master and slave nodes. If one or more of these metrics reach or approach a preset warning threshold for several consecutive periods, it can be considered that the cluster's resource capacity is becoming strained, indicating a potential need for expansion. Based on this judgment, the expansion assessment phase can be further initiated, combining resource usage trend information to perform quantitative calculations of expansion capacity and formulate an expansion decision.

[0303] For example, if, within a certain monitoring period, cluster load information shows that the CPU utilization of multiple cloud-native storage service master nodes exceeds 85%, memory utilization is close to the upper limit, and overall write latency shows a continuous upward trend, it can be preliminarily determined that the current heterogeneous resource cluster is experiencing resource shortages and high operational pressure, necessitating expansion. At this point, an expansion assessment process can be triggered, and further, combined with resource usage trend information, the number of new heterogeneous nodes that the heterogeneous resource cluster can currently support can be calculated to guide subsequent node resource application and deployment operations.

[0304] Through the above implementation methods, real-time monitoring and data collection of the operational status of each cloud-native storage service can be achieved. Based on service load information, cluster load information reflecting the overall operational status of the cluster can be constructed, thereby dynamically assessing resource usage from a global perspective. When the cluster load continues to rise or approaches a preset threshold, an automatic scaling decision process can be triggered, enabling on-demand expansion of the heterogeneous resource cluster. This approach not only enhances the system's self-awareness and adaptive capabilities in scenarios with high-concurrency writes or rapid growth of resource objects, but also effectively avoids scheduling blockages or performance degradation caused by resource bottlenecks, thereby enhancing the overall stability, scalability, and scheduling efficiency of the heterogeneous resource cluster.

[0305] In this application embodiment, to ensure that the heterogeneous resource cluster still has good task scheduling capabilities and storage processing performance when under high load or resource bottlenecks, and to improve the overall availability and elasticity of the system, this application proposes a scaling decision mechanism based on cluster load information. This mechanism acquires and analyzes the operating status of each cloud-native storage service in the heterogeneous resource cluster in real time. After identifying potential resource bottlenecks or high-pressure nodes, it automatically infers whether the current resource structure can still support task operation, and determines whether scaling is needed, as well as the number of new heterogeneous nodes required for scaling. Therefore, an optional implementation of S103 includes the following steps S1031 to S1033:

[0306] S1031: Analyze whether the heterogeneous resource cluster is operating normally based on the cluster load information.

[0307] In this embodiment of the application, cluster load information has been obtained through the above step S102, and the operating status of the heterogeneous resource cluster can be analyzed based on the cluster load information to determine whether it is normal.

[0308] The above lists several types of cluster load information, including but not limited to: CPU utilization, memory utilization, and disk utilization of the primary and backup nodes of each cloud-native storage service.

[0309] Optionally, the normal operating status of the heterogeneous resource cluster can be analyzed based on various cluster load information.

[0310] In this embodiment, the cluster load information is obtained by aggregating the service load information of each cloud-native storage service in the heterogeneous resource cluster, reflecting the overall resource object write pressure and the operating trend of the storage service. To achieve dynamic judgment of the operating status of the heterogeneous resource cluster, optionally, one implementation of step S1031 is as follows: steps S10311~S10312:

[0311] S10311: If the cluster load information does not reach the preset threshold, then the heterogeneous resource cluster is determined to be operating normally.

[0312] S10312: If the cluster load information reaches the preset threshold, it is determined that the operation status of the heterogeneous resource cluster is abnormal.

[0313] The preset threshold refers to the performance indicator threshold used to determine whether the current operating status of the heterogeneous resource cluster has reached the load bottleneck, and is the judgment threshold for triggering expansion judgment conditions. The preset threshold may include, but is not limited to, any one or more of the following setting standards: whether the CPU utilization of the master node exceeds 80%, whether the memory utilization of the master node exceeds 85%, whether the disk utilization of the master node exceeds 90%, etc.

[0314] It should be noted that the preset threshold can be flexibly configured based on specific business scenarios and can be dynamically adjusted according to time windows to adapt to the monitoring and evaluation needs of system load fluctuations at different stages.

[0315] For example, if the CPU utilization of a primary node of a second cloud-native storage service reaches 91%, the memory utilization reaches 87%, and the disk utilization reaches 93% within a certain monitoring period, it indicates that the current node is already under high pressure. The system can mark the heterogeneous resource cluster as "abnormal operation status" based on this and enter the subsequent expansion assessment stage.

[0316] For example, during a certain monitoring period, the master nodes of multiple cloud-native storage services exhibit the following operational data: CPU utilization remained below 65%, memory utilization stabilized at around 70%, disk utilization was approximately 60%, and write request response latency was within 150 milliseconds with no abnormal queuing. At this point, all key indicators did not reach the preset threshold range, indicating that the heterogeneous resource cluster was currently in a state of moderate resource utilization and stable operation. Based on the cluster load information, the heterogeneous resource cluster was determined to be operating normally, and no expansion decision process was required.

[0317] Through the above implementation methods, the health status of heterogeneous resource clusters can be automatically determined, which helps to identify potential bottlenecks in a timely manner when resource write pressure rises rapidly, improves the agility and initiative of resource scheduling, and supports the stable storage and distributed processing of resource objects in high-concurrency write scenarios, thereby further improving the overall scalability and scheduling efficiency of heterogeneous resource clusters.

[0318] S1032: If it is determined that the operation status of the heterogeneous resource cluster is abnormal, the number of new heterogeneous nodes supported by the heterogeneous resource cluster shall be determined based on the resource usage trend information of the heterogeneous resource cluster; the resource usage trend information is determined based on the service traffic information of each cloud-native storage service.

[0319] In this embodiment, resource usage trend information refers to a set of indicators reflecting the changing trend of resource carrying capacity in a heterogeneous resource cluster. This information is used to dynamically assess whether resource usage is becoming strained and to predict whether the heterogeneous resource cluster needs expansion. This information can be represented as the growth trajectory of resource write pressure, and the key is to measure the changing trend between the consumption rate of the resource buffer capacity and the available resource balance.

[0320] The resource usage trend information is generated based on service traffic information, namely the resource write request rate processed by each cloud-native storage service master node per unit time. This metric can be expressed as queries per second (QPS) and is an important parameter for measuring resource write intensity. In this application, service traffic information can be collected in real time and reported periodically by a collection agent deployed in each cloud-native storage service.

[0321] In one optional implementation, a write limit can be preset for each cloud-native storage service master node (e.g., a maximum of 1000 write requests per second). By collecting the real-time actual write rate reported by the acquisition agent and calculating the difference between this and the limit value, the remaining capacity (i.e., buffer capacity) of each master node can be obtained. Then, by summing the buffer capacities of all master nodes in the cluster and combining them with the current write request growth rate (which can be estimated based on historical multi-period QPS change trends), the current overall system write growth trend can be derived.

[0322] Furthermore, by combining the average write capacity of a single heterogeneous node in the heterogeneous resource cluster (e.g., a node supports an average of 800 QPS), the current buffer capacity can be divided by the single-node capacity to calculate the number of new heterogeneous nodes the system can still support. For example, if the total cluster buffer capacity is 1600 QPS and the single-node capacity is 800 QPS, then it can support 2 new heterogeneous nodes.

[0323] The above methods enable trend analysis based on underlying real-time service traffic, and dynamic derivation of the current reasonable expansion scale by combining the node write capability model. This allows for on-demand expansion of heterogeneous resource clusters, ensuring their stability and continuous carrying capacity in high-concurrency scenarios.

[0324] In this embodiment of the application, in order to determine the number of new heterogeneous nodes supported by a heterogeneous resource cluster based on its resource usage trend information, an optional implementation method is as follows:

[0325] Based on the currently collected service traffic information and the corresponding master node's preset service traffic threshold, the current buffer capacity can be calculated, the expansion capacity of the heterogeneous resource cluster can be determined, and the number of new heterogeneous nodes can be determined based on the determined capacity.

[0326] Optionally, based on the service traffic information reported in real time by the collection agents deployed in various cloud-native storage services, i.e., the write request rate (queries per second, QPS) processed by the master node per unit time, the actual level of the current resource object write load can be determined. In this embodiment, a preset upper limit for the QPS processing capacity of each master node (e.g., 1000 QPS) can be used. The difference between the collected write QPS and this upper limit value is calculated to obtain the redundancy space or buffer capacity of the current resource write load.

[0327] For example, if a master node currently has a QPS of 850 and a QPS limit of 1000, then the current Buffer = 1000 - 850 = 150, indicating that the node still has 150 QPS of capacity. Trend extrapolation and resource quantity calculation can be performed by combining the total Buffer of multiple nodes, the current growth trend of write requests, and the average amount of resource object writes handled by each heterogeneous node.

[0328] Based on the above analysis, we can further estimate how many additional heterogeneous nodes the current heterogeneous resource cluster can support for write load. For example, if each new heterogeneous node is expected to add 100 write QPS, and the current cluster's total remaining buffer is 500 QPS, then the number of new heterogeneous nodes that can be supported is 500 / 100 = 5.

[0329] Through the above implementation methods, resource usage trend information generated based on service traffic information can be combined with preset performance limits and current actual load to achieve quantitative prediction of cluster expansion needs, providing on-demand and accurate expansion decision support for heterogeneous resource clusters, and avoiding resource redundancy or performance bottleneck problems.

[0330] See Figure 11C As shown, it is a schematic diagram of resource usage trend information generation and capacity expansion assessment based on service traffic information provided in an embodiment of this application.

[0331] like Figure 11C As shown, Figure 11C The image, from left to right, shows the collection agent modules deployed in multiple cloud-native storage services (including the primary node M and the backup node S). Each collection agent is responsible for collecting two types of operational information for its corresponding cloud-native storage service: service load information and service traffic information. Service load information indicates the utilization of resources such as CPU, memory, and disk on the node, while service traffic information primarily collects the write request rate (QPS) of the primary node per unit of time.

[0332] All collected information is reported in real time to an intermediate database for unified storage and centralized analysis. Service load information from the database is read to generate "cluster load information," such as... Figure 11C The aggregated charts shown in the upper right corner, including CPU utilization curves and memory usage curves, are used to assess whether there are resource bottlenecks in the overall operation of the cluster.

[0333] Service traffic information in the database is used to generate "resource usage trend information," such as... Figure 11C As shown in the lower right corner, the difference between the current QPS of the master node and the preset QPS limit can be calculated to analyze the current buffer balance. Combined with the write trend, the write pressure that can be supported can be extrapolated to deduce the number of new heterogeneous nodes that the current cluster can support, thus providing predictive support for scaling decisions.

[0334] Through the above implementation methods, a resource usage trend view can be constructed based on service traffic information, enabling dynamic perception and trend prediction of resource carrying capacity. Expansion operations can be completed in advance before the cluster load reaches its critical point, avoiding task scheduling congestion or write failure caused by resource shortages, and improving the overall system's stability, adaptability, and scalability.

[0335] It should be noted that, besides generating resource usage trend information based on service traffic data to determine the required expansion, other methods can also be used to determine the number of new heterogeneous nodes supported by the heterogeneous resource cluster. For example, based on historical write trends of resource objects, periodic write peak statistics, changes in task scheduling queue length, and the failure rate of resource object creation in the cluster, combined with machine learning prediction models or rule engines, trend extrapolation and pressure prediction can be performed to dynamically calculate the required number of new heterogeneous nodes. The above methods can also be considered as an optional expansion decision-making approach in this application, applicable to more complex or dynamically changing resource scheduling scenarios, and are not specifically limited here.

[0336] To further improve the automation level and response efficiency of the expansion operation, the following section will describe in detail how to perform the expansion operation on the heterogeneous resource cluster based on the number of new heterogeneous nodes determined above.

[0337] S1033: Expand the heterogeneous resource cluster based on the number of newly added heterogeneous nodes.

[0338] In this embodiment, an expansion request can be sent to the resource scheduling platform based on the number of new heterogeneous nodes calculated in step S1032. The scheduling platform then requests a corresponding number of heterogeneous computing resources (e.g., GPU nodes or NPU nodes) and connects the requested heterogeneous nodes to the heterogeneous resource cluster to complete the node registration and initialization operations.

[0339] In one optional implementation, the above expansion process can also be combined with the elastic container service provided by the cloud platform to achieve automatic resource orchestration and deployment configuration. After the new node is connected, it will participate in the subsequent receiving, processing and writing tasks of resource objects according to the preset resource scheduling strategy and object distribution mechanism, ensuring that the cluster service capacity is horizontally expanded on the original basis.

[0340] Furthermore, it should be noted that after adding heterogeneous nodes to the heterogeneous resource cluster, a batch of new resource objects, including node objects, daemon set objects, and container group objects, will be introduced, resulting in a change in the total number (second quantity) of resource objects in the heterogeneous resource cluster. To ensure that the newly added resource objects can be reasonably partitioned and stored, this application proposes that after the expansion is completed, the existing cloud-native storage service structure be dynamically adapted in conjunction with the resource object partitioning mechanism.

[0341] Specifically, based on the updated second quantity after expansion, the number of cloud-native storage services to be built (i.e., the third quantity) can be recalculated in conjunction with the unit quantity, and the allocation range of service identifiers can be dynamically adjusted accordingly. After recalculating the new service identifier structure, the system can identify the object category and extract the object identifier for the newly added resource objects according to the aforementioned division method: if it is a static resource object (such as a node object or a daemon set object), it will be uniformly assigned to the first cloud-native storage service; if it is a dynamic resource object (such as a namespace object or a container group object), a hash operation will be performed on its object identifier, and the service identifier assignment will be determined based on the modulo result of the hash value and the current number of second cloud-native storage services.

[0342] Ultimately, the resource objects of the newly added heterogeneous nodes will be written as a collection of objects to the target cloud-native storage service represented by their service identifiers. This ensures the continuity of resource object partitioning and consistency of ownership, thereby supporting a smooth transition and continuous stable operation of the system after expansion. This approach not only guarantees a clear structure and load balancing of the resource object write path but also further enhances the sustainable evolution capability of heterogeneous resource clusters in dynamic expansion scenarios and the elastic scalability of the cloud-native storage system.

[0343] Through the above implementation methods, cluster load information can be generated based on the service load information of each cloud-native storage service in the heterogeneous resource cluster. Based on this, it can be determined whether the current operating status of the cluster is normal. If the determination result is abnormal, resource usage trend information can be calculated by combining service traffic information, thereby assessing the number of new heterogeneous nodes that the current cluster can support. Based on this number, precise expansion operations can be performed to achieve dynamic elastic expansion of the heterogeneous resource cluster. This avoids service interruptions and scheduling congestion caused by resource bottlenecks, and also prevents the waste of computing power caused by resource redundancy. It helps to improve the overall stability, adaptability and resource utilization efficiency of the system, enhance the system's ability to bear high-concurrency write scenarios and service continuity, and ensure that the heterogeneous resource cluster always maintains an efficient, stable and scalable operating status under variable task loads.

[0344] Additionally, it should be noted that in this embodiment, after performing database sharding on the cloud-native objects of heterogeneous nodes (i.e., in this embodiment, resource objects of different object categories are classified and stored, and different cloud-native storage services are built for each category), the write operation in the above implementation achieves single-point splitting, thereby supporting the horizontal scaling of the heterogeneous node scale. This is referred to as heterogeneous cloud-native object database sharding and table partitioning.

[0345] See Figure 12 The diagram shown illustrates the overall process of resource object partitioning and storage management based on a heterogeneous resource cluster, as provided in an embodiment of this application. Figure 12 This application demonstrates a full-process management mechanism for resource objects, covering key aspects such as resource generation, object classification, storage partitioning, and expansion feedback.

[0346] like Figure 12 As shown, after heterogeneous nodes are connected to the heterogeneous resource cluster, the resource cloud-native clustering is completed under the control of the resource scheduling platform. The scheduling and production module instantiates heterogeneous resources into cloud-native resource objects (such as node objects, container objects, namespace objects, daemon set objects, etc.). Subsequently, the system performs classification based on the attribute characteristics of the resource objects: static category objects (such as nodes and daemon sets) are uniformly assigned to the static object set and mapped to the first cloud-native storage service; dynamic category objects (such as namespaces and container groups) are input into the hash algorithm module, which performs hash calculation based on the namespace identifier, outputs the corresponding second cloud-native storage service number, and completes the attribution and distributed storage of the object set.

[0347] After the resource objects are written, each cloud-native storage service continuously collects load data and service traffic data from the primary and backup nodes through an internal collection agent, and periodically reports it to the service load database. The system further aggregates this data to form cluster load information and resource usage trend information, and dynamically predicts the number of new heterogeneous nodes that can be supported based on the changing trend of write pressure, thereby realizing horizontal scaling and adaptive elastic scheduling of the cluster.

[0348] The overall process enables automatic identification, fine-grained partitioning, and on-demand storage of resource objects, supports efficient resource scheduling, performance awareness, and dynamic expansion, and has the advantages of clear structure, high execution efficiency, and strong scalability. It is suitable for distributed management and high-concurrency processing scenarios of resource objects in large-scale heterogeneous resource environments.

[0349] See Figure 13 The diagram shown is an overall flowchart of resource object partitioning and heterogeneous resource cluster expansion processing provided in this application embodiment. The flowchart clearly illustrates how to complete the entire process from building a heterogeneous resource cluster, sequentially including resource object classification, mapping and storage, runtime status determination, and expansion decision based on cluster load information. Specific implementation may include the following steps S1301 to S1316:

[0350] S1301: Start.

[0351] Initiating the resource management process prepares for subsequent resource object processing and cluster expansion. This step serves as the starting point of the entire process, marking the system's entry into the resource scheduling and storage initialization phase.

[0352] S1302: Construct a heterogeneous resource cluster with heterogeneous nodes.

[0353] At the cluster level, a heterogeneous resource cluster consisting of multiple heterogeneous nodes is constructed to provide a basic platform for subsequent scheduling of resource objects. This step ensures that the cluster has the ability to access heterogeneous computing resources.

[0354] S1303: Summarize the total number of resource objects in the heterogeneous resource cluster.

[0355] The total number of registered and scheduled resource objects in the cluster is counted, including node objects, namespace objects, container group objects, etc., for subsequent storage service capacity planning.

[0356] S1304: Calculate the number of cloud-native storage services that need to be deployed based on the total number of resource objects, and divide the cloud-native storage services into multiple secondary cloud-native storage services.

[0357] Based on the total number of resource objects and the capacity supported by a single service, the required number of cloud-native storage services is calculated, and the system is divided into multiple second cloud-native storage services that can run in parallel.

[0358] S1305: Divide the resource objects of the heterogeneous resource cluster into static categories and dynamic categories.

[0359] Resource objects are categorized based on their lifecycle and change frequency, with static categories such as node objects and daemon set objects, and dynamic categories such as namespaces and container group objects, providing a basis for subsequent storage strategy selection.

[0360] S1306: Store static category resource objects in the First Cloud Native Storage Service.

[0361] Static category resource objects with stable lifecycles are uniformly assigned to the First Cloud Native Storage Service, which facilitates centralized management and ensures consistent access.

[0362] S1307: Perform a hash operation on a resource object of a dynamic category.

[0363] Perform a hash operation on the namespace object in the dynamic resource object to generate the corresponding hash value, which serves as the basic input for subsequent service mapping.

[0364] S1308: The value output by the hit logic matches the service identifier of the corresponding second cloud-native storage service. Based on the hash value, modulo operation or consistent hashing is performed to determine the second cloud-native storage service to which the resource object belongs, thus achieving distributed storage for dynamic objects.

[0365] S1309: Determine whether the storage of a dynamic category resource object has been completed.

[0366] Check whether each dynamic resource object has been correctly written to the target storage service as a status judgment condition for subsequent processes. If yes, continue to step S1310; if not, return to step S1307 to re-execute the object hashing and storage processing flow.

[0367] S1310: Assign collection agents to various cloud-native storage services.

[0368] Deploy a data collection agent on each cloud-native storage service to collect service load information of the primary and backup nodes, enabling real-time monitoring of the system's operating status.

[0369] S1311: After the collection agent executes the collection service load information, it generates cluster load information and builds it into the database.

[0370] The data acquisition agent periodically obtains information on the CPU, memory, and other resource usage of each node, summarizes and generates cluster load information, and stores it in the database to provide a data foundation for subsequent expansion decisions.

[0371] S1312: Read service load information and service traffic information from the database.

[0372] Extract current service load and write traffic information from the database to assess cluster operating pressure and resource usage trends.

[0373] S1313: Draw a graph showing cluster load information and resource usage trends.

[0374] Based on the extracted data, curves of CPU, memory, and disk usage, as well as resource write trend charts, are plotted to assess whether the conditions for expansion have been met.

[0375] S1314: Determine whether the current heterogeneous resource cluster is operating normally.

[0376] The current operating status is determined based on cluster load information and preset thresholds. If the status is normal, the process ends. (Specifically, this means that no expansion is needed at this point. This process can be repeated when heterogeneous nodes or corresponding resource objects in the heterogeneous resource cluster change.) Figure 13 (The process shown); if not, that is, the running status is abnormal, then jump to step S1315.

[0377] S1315: Determine whether the resource usage trend information of the heterogeneous resource cluster meets the expansion conditions.

[0378] If yes, it indicates that the current resource write pressure is continuously increasing and the buffer space is becoming tight, meaning the expansion conditions are met, and step S1316 continues; if no, meaning the expansion conditions are not met, the process ends, and expansion is not performed (same as above, specifically meaning that expansion is not required at this time; when heterogeneous nodes or corresponding resource objects in the heterogeneous resource cluster change, this step can be repeated). Figure 13 (The process shown).

[0379] S1316: Estimate the number of new heterogeneous nodes based on the resource usage trend information of the heterogeneous resource cluster, and add the new heterogeneous nodes to the heterogeneous resource cluster.

[0380] By calculating the difference between the current write QPS and the preset upper limit, the number of new nodes that can handle the write load is assessed, and they are added to the cluster to complete elastic scaling.

[0381] It should be noted here that, Figure 13 The flowchart shown is only used to illustrate the overall processing logic of the embodiments of this application, and to help understand the overall architecture and step relationships of resource object partitioning and dynamic expansion of heterogeneous resource clusters in this application. The specific processing logic and implementation method of each step can be referred to the foregoing specific embodiments section, and will not be repeated here.

[0382] Through such Figure 13The processing flow shown enables fine-grained partitioning and mapping of resource objects on the storage side. Combined with load awareness and trend prediction, it enables on-demand expansion of heterogeneous resource clusters, improving the write performance, scheduling efficiency and scalability of the entire system. It is suitable for resource management needs in large-scale heterogeneous computing environments.

[0383] Through the method of this application embodiment, this application can combine the actual service scenarios of heterogeneous resource clusters to classify and independently map various types of resource objects, and assign resource objects to different cloud-native storage services according to object categories, realizing the classified management and distributed writing of resource objects in the storage structure; at the same time, for resource objects with dynamic categories, a hash calculation method based on the object identifier of the namespace object is introduced, combined with the service identifier allocation mechanism, to complete the automatic scattering and balanced distribution of resource objects, thereby effectively improving resource write performance and avoiding the bottleneck problem caused by concentrated writes; the solution proposed in this application supports the horizontal scaling of the scale of resource objects in heterogeneous resource clusters, and can meet the requirements of large-scale resource scheduling and high-frequency task execution for the throughput and response speed of the storage system; furthermore, by implementing read-write path separation, load sharing and object affiliation aggregation on the resource write side, it helps to improve the flexibility and stability of resource scheduling, reduce task startup latency, improve the overall utilization of heterogeneous resources, and ultimately effectively reduce system operating costs and improve resource scheduling efficiency and business development efficiency.

[0384] Based on the same inventive concept, embodiments of this application also provide a storage device 1400 for resource objects. For example... Figure 14 As shown, this is a structural diagram of the storage device 1400 for resource objects, which may include:

[0385] The resource object determination unit 1401 is used to determine various types of resource objects corresponding to each heterogeneous node in the heterogeneous resource cluster; the resource object refers to a cloud-native object generated during the production scheduling process of the heterogeneous resource cluster to support task execution on the heterogeneous node.

[0386] The construction unit 1402 is used to construct multiple cloud-native storage services based on the total number of the various types of resource objects; wherein each cloud-native storage service corresponds to a service identifier;

[0387] The partitioning unit 1403 is used to divide the various types of resource objects into multiple object sets according to their respective object categories and object identifiers, and to determine the service identifier corresponding to each object set.

[0388] Storage unit 1404 is used to store each object set in a cloud-native storage service represented by a corresponding service identifier; wherein each cloud-native storage service adopts a primary-backup architecture, and the primary node in the primary-backup architecture is used to handle write requests related to each resource object in the corresponding object set.

[0389] Optionally, the device 1400 further includes:

[0390] The expansion unit 1405 is used to collect the service load information of each cloud-native storage service; wherein, each service load information is determined based on the resource usage of each resource object stored in the corresponding cloud-native storage service.

[0391] Based on the service load information of each cloud-native storage service, the cluster load information of the heterogeneous resource cluster is obtained;

[0392] Based on the cluster load information, a scaling decision is made for the heterogeneous resource cluster.

[0393] Optionally, the object categories include: static categories and dynamic categories; the different cloud-native storage services include: a first cloud-native storage service and multiple second cloud-native storage services; then the partitioning unit 1403 is used for:

[0394] The resource objects of the static category among the various types of resource objects are divided into the same object set, and the service identifier of the first cloud-native storage service is used as the service identifier corresponding to the same object set; wherein, the resource objects of the static category refer to resource objects of fixed size after being deployed to the heterogeneous resource cluster;

[0395] The resource objects of the dynamic category among the various types of resource objects are divided into at least one object set according to their respective object identifiers, and the service identifiers corresponding to the at least one object set are determined in the service identifiers of the multiple second cloud-native storage services; wherein, the resource objects of the dynamic category refer to resource objects that can be elastically adjusted according to demand after being deployed to the heterogeneous resource cluster.

[0396] Optionally, the resource objects of the dynamic categories include: namespace objects and container group objects; then the partitioning unit 1403 is used for:

[0397] For each namespace object, perform the following operations:

[0398] To group a namespace object and the various container group objects belonging to the namespace object into the same object set;

[0399] The hash value is obtained by performing a hash operation on the object identifier of the namespace object.

[0400] The service identifier that the hash value matches in the service identifiers of the multiple second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

[0401] Optionally, the partitioning unit 1403 is used for:

[0402] The result is obtained by performing a modulo operation on the hash value and the first quantity; the first quantity is the total number of the plurality of second cloud-native storage services.

[0403] The service identifier that matches the calculation result among the service identifiers corresponding to each of the second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

[0404] Optionally, for each set of objects, after storing the set of objects in a cloud-native storage service represented by the corresponding service identifier, the apparatus 1400 further includes:

[0405] Synchronization unit 1406 is used to synchronize data between various second cloud-native storage services to ensure the consistency of stored data between each second cloud-native storage service.

[0406] Optionally, the building unit 1402 is used for:

[0407] The third quantity is determined based on the ratio between the second quantity and the unit quantity; wherein, the unit quantity is: the total number of various resource objects supported by each cloud-native storage service; the second quantity is: the total number of various resource objects in the heterogeneous resource cluster;

[0408] Build a third cloud-native storage service and assign a service identifier to each cloud-native storage service.

[0409] Optionally, the expansion unit 1405 is used for:

[0410] Based on the cluster load information, analyze whether the operation status of the heterogeneous resource cluster is normal;

[0411] If it is determined that the operation status of the heterogeneous resource cluster is abnormal, then the number of new heterogeneous nodes supported by the heterogeneous resource cluster is determined based on the resource usage trend information of the heterogeneous resource cluster; the resource usage trend information is determined based on the service traffic information of each cloud-native storage service.

[0412] The heterogeneous resource cluster is expanded based on the number of newly added heterogeneous nodes.

[0413] Optionally, a data collection agent is deployed on each cloud-native storage service;

[0414] Each of the service load information is obtained by collecting the resource usage of the master node and each backup node corresponding to the master node through the collection agent on the corresponding cloud-native storage service.

[0415] Each of the aforementioned service traffic information is obtained by collecting the write request processing status of the master node on the corresponding cloud-native storage service through the collection agent on the corresponding cloud-native storage service.

[0416] Optionally, the expansion unit 1405 is used for:

[0417] If the cluster load information does not reach the preset threshold, then the heterogeneous resource cluster is determined to be operating normally.

[0418] If the cluster load information reaches the preset threshold, it is determined that the heterogeneous resource cluster is in an abnormal operating state.

[0419] In summary, the resource object storage device provided in this application can optimize the configuration of resource object write paths through resource object identification, storage service construction, object set partitioning, and parallel write strategies, thereby effectively improving the write performance and scalability of cloud-native storage systems in heterogeneous resource cluster environments and supporting the scheduling needs of high-density heterogeneous nodes.

[0420] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0421] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0422] Having introduced the resource object storage method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0423] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0424] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 2A The server 220 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 15 As shown, it includes a memory 1501, a communication module 1503, and one or more processors 1502.

[0425] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0426] Memory 1501 may be volatile memory, such as random-access memory (RAM); memory 1501 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1501 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1501 may be a combination of the above-described memories.

[0427] Processor 1502 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1502 is used to implement the storage method of the above-mentioned resource objects when calling computer programs stored in memory 1501.

[0428] The communication module 1503 is used to communicate with terminal devices and other servers.

[0429] This application embodiment does not limit the specific connection medium between the memory 1501, communication module 1503, and processor 1502. This application embodiment... Figure 15 The memory 1501 and the processor 1502 are connected via a bus 1504, and the bus 1504 is in Figure 15 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1504 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 15 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0430] The memory 1501 stores a computer storage medium, which stores computer-executable instructions for implementing the resource object storage method of this application embodiment. The processor 1502 is used to execute the above-described resource object storage method, such as... Figure 3 As shown.

[0431] In some possible implementations, various aspects of the resource object storage method provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the resource object storage method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 3 The steps are shown in the figure.

[0432] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0433] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0434] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0435] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0436] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0437] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0438] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0439] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0440] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0441] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0442] These computer program commands may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the commands executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0443] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0444] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for storing resource objects, characterized in that, The method includes: For each heterogeneous node in the heterogeneous resource cluster, determine the various types of resource objects corresponding to the heterogeneous node; the resource objects refer to cloud-native objects generated during the production scheduling process of the heterogeneous resource cluster to support task execution on the heterogeneous nodes. Based on the total number of the various types of resource objects, multiple cloud-native storage services are constructed; each cloud-native storage service corresponds to a service identifier. Based on the object category and object identifier of each type of resource object, the various types of resource objects are divided into multiple object sets, and the service identifier corresponding to each object set is determined. For each set of objects, the set of objects is stored in a cloud-native storage service represented by the corresponding service identifier; wherein each cloud-native storage service adopts a primary-backup architecture, and the primary node in the primary-backup architecture is used to handle write requests related to each resource object in the corresponding set of objects.

2. The method as described in claim 1, characterized in that, The method further includes: The service load information of each cloud-native storage service is collected separately; the service load information of each service is determined based on the resource usage of each resource object stored in the corresponding cloud-native storage service. Based on the service load information of each cloud-native storage service, the cluster load information of the heterogeneous resource cluster is obtained; Based on the cluster load information, a scaling decision is made for the heterogeneous resource cluster.

3. The method as described in claim 1, characterized in that, The object categories include: static categories and dynamic categories; the different cloud-native storage services include: a first cloud-native storage service and multiple second cloud-native storage services; The step of dividing the various resource objects into multiple object sets according to their respective object categories and object identifiers, and determining the service identifier corresponding to each object set, includes: The resource objects of the static category among the various types of resource objects are divided into the same object set, and the service identifier of the first cloud-native storage service is used as the service identifier corresponding to the same object set; wherein, the resource objects of the static category refer to resource objects of fixed size after being deployed to the heterogeneous resource cluster; The resource objects of the dynamic category among the various types of resource objects are divided into at least one object set according to their respective object identifiers, and the service identifiers corresponding to the at least one object set are determined in the service identifiers of the multiple second cloud-native storage services; wherein, the resource objects of the dynamic category refer to resource objects that can be elastically adjusted according to demand after being deployed to the heterogeneous resource cluster.

4. The method as described in claim 3, characterized in that, The resource objects of the dynamic categories include: namespace objects and container group objects; The step of classifying dynamic category resource objects among the various resource objects into at least one object set according to their respective object identifiers, and determining the service identifier corresponding to each of the at least one object set from the service identifiers of the multiple second cloud-native storage services, includes: For each namespace object, perform the following operations: To group a namespace object and the various container group objects belonging to the namespace object into the same object set; The hash value is obtained by performing a hash operation on the object identifier of the namespace object. The service identifier that the hash value matches in the service identifiers of the multiple second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

5. The method as described in claim 4, characterized in that, The step of using the service identifier whose hash value matches one of the service identifiers of the plurality of second cloud-native storage services as the service identifier corresponding to the object set to which the namespace object belongs includes: The result is obtained by performing a modulo operation on the hash value and the first quantity; the first quantity is the total number of the plurality of second cloud-native storage services. The service identifier that matches the calculation result among the service identifiers corresponding to each of the second cloud-native storage services is used as the service identifier corresponding to the object set to which the namespace object belongs.

6. The method according to any one of claims 1 to 5, characterized in that, For each set of objects, after storing the set of objects in the cloud-native storage service represented by the corresponding service identifier, the method further includes: Data is synchronized between various secondary cloud-native storage services to ensure data consistency across each service.

7. The method according to any one of claims 1 to 5, characterized in that, The construction of multiple cloud-native storage services based on the total number of the various resource objects includes: The third quantity is determined based on the ratio between the second quantity and the unit quantity; wherein, the unit quantity is: the total number of various resource objects supported by each cloud-native storage service; the second quantity is: the total number of various resource objects in the heterogeneous resource cluster; Build a third cloud-native storage service and assign a service identifier to each cloud-native storage service.

8. The method as described in claim 2, characterized in that, The step of making scaling decisions for the heterogeneous resource cluster based on the cluster load information includes: Based on the cluster load information, analyze whether the operation status of the heterogeneous resource cluster is normal; If it is determined that the operation status of the heterogeneous resource cluster is abnormal, then the number of new heterogeneous nodes supported by the heterogeneous resource cluster is determined based on the resource usage trend information of the heterogeneous resource cluster; the resource usage trend information is determined based on the service traffic information of each cloud-native storage service. The heterogeneous resource cluster is expanded based on the number of newly added heterogeneous nodes.

9. The method as described in claim 8, characterized in that, Each cloud-native storage service has a data collection agent deployed on it; Each of the service load information is obtained by collecting the resource usage of the master node and each backup node corresponding to the master node through the collection agent on the corresponding cloud-native storage service. Each of the aforementioned service traffic information is obtained by collecting the write request processing status of the master node on the corresponding cloud-native storage service through the collection agent on the corresponding cloud-native storage service.

10. The method as described in claim 8, characterized in that, The step of analyzing whether the heterogeneous resource cluster is operating normally based on the cluster load information includes: If the cluster load information does not reach the preset threshold, then the heterogeneous resource cluster is determined to be operating normally. If the cluster load information reaches the preset threshold, it is determined that the heterogeneous resource cluster is in an abnormal operating state.

11. A storage device for resource objects, characterized in that, include: The resource object determination unit is used to determine the various types of resource objects corresponding to each heterogeneous node in the heterogeneous resource cluster. The resource object refers to a cloud-native object generated during the production scheduling process of the heterogeneous resource cluster, used to support task execution on heterogeneous nodes; The construction unit is used to construct multiple cloud-native storage services based on the total number of the various types of resource objects; wherein each cloud-native storage service corresponds to a service identifier; The partitioning unit is used to divide the various types of resource objects into multiple object sets according to their respective object categories and object identifiers, and to determine the service identifier corresponding to each object set. A storage unit is used to store each set of objects into a cloud-native storage service represented by a corresponding service identifier; wherein each cloud-native storage service adopts a primary-secondary architecture, and the primary node in the primary-secondary architecture is used to handle write requests related to each resource object in the corresponding set of objects.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 10.

14. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 10.

Images