A data loading method and device of a distributed platform and a medium

Abstract

The application discloses a data loading method and device of a distributed platform and a medium, and is suitable for the technical field of the distributed platform. The method comprises the following steps: acquiring mirror image data and cache information of a data set of a current job; determining a corresponding event queue according to the cache information and a corresponding node cache instance; updating node cache task state information of a target node and performing a corresponding cache task queue operation according to each event queue and the cache information, so as to load the cache information to the target node. In the preloading mode, the cache information is directly loaded to the target node without the need of making a request, and the setting of a transfer station is not needed, thereby reducing the interaction process with the target node. Through the design of the double queues (a mirror image cache task queue and a data set cache task queue), concurrent operation processing is realized. In combination with the massive data of the distributed platform, the node cache task state information is updated in real time, data errors are avoided, and the operation efficiency of the platform is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed platform technology, and in particular to a data loading method, apparatus and medium for a distributed platform. Background Technology

[0002] Existing data caching technologies are used to cache data between client and server-side business software and to read mirrored data via target paths. However, these technologies are clearly unsuitable for the large number of cluster nodes, network conditions, and reliability of communication between edge nodes and the cluster center in distributed platform systems.

[0003] Existing data caching technologies only consider the data needs of the business layer, using mirrors and dataset central repositories to store all data as transit points. When a job starts, the client requests the repository center to load the image and data, resulting in frequent server-client interactions and a lengthy loading process. Alternatively, multiple regions are divided, each with its own local image repository. These local repositories only distribute jobs to nodes within their respective regions, leading to delays in retrieving images and datasets, impacting platform resource utilization. Furthermore, in a distributed platform system, the state of images and datasets changes during training tasks. Current methods only process the flow without labeling the state of individual data points. Due to the massive amounts of data in a distributed platform system, this can lead to data errors during job execution, affecting platform job performance.

[0004] Therefore, finding a data loading method for a distributed platform is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a data loading method, apparatus, and medium for a distributed platform, avoiding the latency issues caused by existing distribution methods using a central repository or local image repository. Concurrent operation processing is achieved through a dual-queue design (image cache task queue and dataset cache task queue). Combined with the massive amounts of data on the distributed platform, data errors are avoided by real-time updating of node cache task status information, thereby improving the platform's operational efficiency.

[0006] To address the aforementioned technical problems, this invention provides a data loading method for a distributed platform, comprising:

[0007] Obtain the image data and cache information of the dataset for the current job, wherein the cache information is obtained through resource configuration preloading or by the target node based on the current job's execution request.

[0008] The corresponding event queue is determined based on the cache information and the corresponding node cache instance;

[0009] The node cache task status information of the target node is updated according to each event queue and the cache information, and the corresponding cache task queue operation is executed to load the cache information into the target node. The cache task queue includes a mirror cache task queue and a dataset cache task queue.

[0010] Preferably, the cache information is obtained through the resource configuration preloading method, and the step of obtaining the cache information of the current job's image data and dataset includes:

[0011] Based on the resource configuration information and the current job, obtain the corresponding image data and dataset as cache information;

[0012] Correspondingly, the acquisition of each of the resource configuration information is achieved through the resource configuration preloading method, specifically including:

[0013] Establish the image list and the first preload cache node;

[0014] Obtain the first pull encryption / decryption configuration information for the image data;

[0015] The image list, the first preloaded cache node, and the first pull encryption / decryption configuration information are used as the resource configuration information for the image data.

[0016] Create a dataset list and a second preload cache node;

[0017] Obtain the second pull encryption / decryption configuration information for the dataset;

[0018] The image list, the second preloaded cache node, and the second pull encryption / decryption configuration information are used as the resource configuration information of the dataset.

[0019] Preferably, establishing the mirror list and the first preloaded cache node includes:

[0020] If no node selection label or node list is set in the image list, the image data is preloaded to all nodes in the current cluster as the first preload cache node;

[0021] When the node selection label and the node list are set in the image list, the image data is cached to the node to be cached that matches the node selection label and is recorded in the node list as the first preloaded cache node;

[0022] Correspondingly, the method of establishing the dataset list and the second preloaded cache node is the same as that of establishing the mirror list and the first preloaded cache node.

[0023] Preferably, the cache information is obtained by the target node according to the current job execution request method, and obtaining the cache information of the current job's image data and dataset includes:

[0024] Determine whether the node cache instance corresponding to the target node has changed;

[0025] If it exists, the changed data of the node cache instance will be used as the cached data;

[0026] Receive the node request from the target node to obtain the cached data.

[0027] Preferably, determining the corresponding event queue based on the cache information and the corresponding node cache instance includes:

[0028] The corresponding event handling type is determined based on the node cache instance, wherein the event handling type includes node cache event handling type, node status event handling type and node garbage cleanup event type;

[0029] Each event processing type is entered into the corresponding event queue;

[0030] Correspondingly, the node cache event handling type is determined based on the node cache instance, including:

[0031] Node cache events are generated based on the node cache instance, and the event type of the node cache event is determined. The event type includes creation type, update type, and retry type. When the cache information is obtained through the resource configuration preloading method, the node cache instance is obtained through the cache information.

[0032] Preferably, updating the node cache task status information of the target node and performing the corresponding cache task queue operation according to each event queue and the cache information includes:

[0033] The step of updating the node cache task status information of the target node according to each of the event queues and the cache information includes:

[0034] Create a node cache task queue;

[0035] Update the node cache task status information in the node cache task queue according to the event processing type of the event queue;

[0036] Correspondingly, the corresponding cache task queue operation is performed based on the event queue and the cache information, including:

[0037] The corresponding task type is determined based on the event processing type and the cache information in the event queue, wherein the task type includes node dataset download type, node dataset deletion type, node dataset re-download type, node image download type, node image deletion type, or node image re-download type;

[0038] Determine the corresponding program components based on each task type;

[0039] Each program component performs the corresponding operation of the cached task queue.

[0040] Preferably, the node cache instance includes a node snapshot mechanism, and further includes:

[0041] When the network of the distributed platform is abnormal, the cached information is retained through the node snapshot mechanism so that the cached information can be loaded when the network of the distributed platform is normal.

[0042] To address the aforementioned technical problems, the present invention also provides a data loading device for a distributed platform, comprising:

[0043] The acquisition module is used to acquire the image data and cache information of the dataset of the current job, wherein the cache information is obtained by resource configuration preloading or by the target node according to the current job execution request.

[0044] The determination module is used to determine the corresponding event queue based on the cache information and the corresponding node cache instance;

[0045] The update execution module is used to update the node cache task status information of the target node according to each event queue and the cache information, and to execute the corresponding cache task queue operation so as to load the cache information into the target node, wherein the cache task queue includes a mirror cache task queue and a dataset cache task queue.

[0046] To address the aforementioned technical problems, the present invention also provides a data loading device for a distributed platform, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor, used to implement the steps of the data loading method for a distributed platform as described above when executing the computer program.

[0049] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data loading method for a distributed platform as described above.

[0050] This invention provides a data loading method for a distributed platform, comprising: acquiring the image data and dataset cache information of the current job, wherein the cache information is obtained through resource configuration preloading or by the target node based on the current job's execution request; determining the corresponding event queue based on the cache information and the corresponding node cache instance; updating the node cache task status information of the target node and executing the corresponding cache task queue operation according to each event queue and the cache information to load the cache information to the target node, wherein the cache task queue includes an image cache task queue and a dataset cache task queue. This method, in the preloading mode, does not require a request and directly loads the data to the target node, reducing the interaction process with the target node. Simultaneously, loading the cache information directly to the target node eliminates the need for a relay station, avoiding the latency issues caused by distribution through a central repository or local image repository. The dual-queue design (image cache task queue and dataset cache task queue) enables concurrent operation processing. Combined with the massive data of the distributed platform, real-time updates of node cache task status information prevent data errors, thereby improving the platform's job execution efficiency.

[0051] In addition, the present invention also provides a data loading device and medium for a distributed platform, which has the same beneficial effects as the data loading method for a distributed platform described above. Attached Figure Description

[0052] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a data loading method for a distributed platform provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating image and dataset loading based on a distributed cloud platform, provided as an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram illustrating the application of node caching in an embodiment of the present invention.

[0056] Figure 4 A structural diagram of a data loading device for a distributed platform provided in an embodiment of the present invention;

[0057] Figure 5 This is a structural diagram of another data loading device for a distributed platform provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0059] The core of this invention is to provide a data loading method, apparatus, and medium for a distributed platform, avoiding the latency issues caused by existing distribution methods using a central repository or local image repository. Concurrent operation processing is achieved through a dual-queue design (image cache task queue and dataset cache task queue). Combined with the massive data of the distributed platform, data errors are avoided by real-time updating of node cache task status information, thereby improving the platform's operational efficiency.

[0060] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] It's important to note that existing data loading methods rely on the standard cached data processing flow of ordinary business software. This involves first constructing a cache request based on the user's client's region and sending it to the corresponding server. The server then preloads the data, and the user subsequently requests the server to load the matching data. This method involves multiple interactions between the client and server, and while the process is relatively simple, it's unsuitable for distributed cloud platform systems (such as the image and dataset types commonly used in AI platforms). Furthermore, distributed cloud platform clusters are typically large, ranging from hundreds to thousands of nodes. If data is loaded through multi-node applications using the above process, the data loading process becomes lengthy, and data errors during loading can significantly reduce platform efficiency.

[0062] Another existing data loading method selects a path cache node to read image data, requiring path determination each time image data is retrieved. Existing data loading methods only consider the data requirements of the business layer, designing an optimal selection algorithm and using the optimal cache node for data distribution and download. Forwarding through a central repository introduces significant latency, impacting platform resource utilization. The distributed platform data loading method provided in this invention solves the latency problem of loading image data and datasets in cloud platform cluster systems, ensuring the stability and continuity of platform business operations.

[0063] Figure 1 A flowchart of a data loading method for a distributed platform provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0064] S11: Obtain the image data and cache information of the current job, where the cache information is obtained through resource configuration preloading or by the target node based on the current job's execution request.

[0065] S12: Determine the corresponding event queue based on the cache information and the corresponding node cache instance;

[0066] S13: Update the node cache task status information of the target node according to each event queue and cache information, and execute the corresponding cache task queue operation to load the cache information into the target node. The cache task queue includes the image cache task queue and the dataset cache task queue.

[0067] Specifically, obtaining the cached information of the current job's image data and dataset can be understood in two ways: one is through resource configuration preloading, and the other is through the target node's business request based on the current job's execution. When obtaining it through resource configuration preloading, the target node has no image data or dataset and is in a state where the controller component has just started. Obtaining it through the business request based on the current job's execution occurs when the corresponding node's cache instance changes during the controller component's runtime; for example, the preloaded cache information has been downloaded, or the image data and dataset required by the node job have been added to the cache instance's dynamic cache. Regardless of the method, the cached image data and dataset of the current job can be obtained.

[0068] As one embodiment, cache information is obtained through resource configuration preloading, acquiring the cache information of the current job's image data and dataset, including:

[0069] Based on the resource configuration information and the current job, obtain the corresponding image data and dataset as cache information;

[0070] Correspondingly, the resource configuration information is obtained through resource configuration preloading, specifically including:

[0071] Establish the image list and the first preload cache node;

[0072] Obtain the first pull encryption / decryption configuration information for the image data;

[0073] The image list, the first preloaded cache node, and the first pull encryption / decryption configuration information are used as the resource configuration information for the image data.

[0074] Create a dataset list and a second preload cache node;

[0075] Retrieve the second pull encryption / decryption configuration information for the dataset;

[0076] The image list, the second preloaded cache node, and the second pull encryption / decryption configuration information are used as the resource configuration information for the dataset.

[0077] Specifically, Figure 2 This invention provides a schematic diagram of image and dataset loading based on a distributed cloud platform, as shown in the embodiment of the invention. Figure 2 As shown, the cluster node cache resource configuration device primarily handles the configuration of pre-loadable image data and datasets for cluster nodes. The node cache controller can then batch-download the corresponding node caches based on this configuration. If the image data and datasets required by the platform job are all in this configuration, the platform job can run immediately after scheduling via pre-loading. The cache controller pre-loads cached data upon startup, eliminating delays caused by jobs waiting for image data and dataset downloads. It's important to note that the cluster node cache resource configuration is not a simple configuration file, but rather persisted to the distributed system's resource storage, similar to a text file stored on the host, and can be linked to generate node cache instances on specific nodes. For distributed platforms, for jobs with high timeliness requirements, the required image data and datasets can be configured here.

[0078] The node cache configuration information that needs to be preloaded is provided to the node cache controller. If the platform job needs to quickly start and run services, it can add the required image data and dataset as cache information to the resource configuration. The resource configuration provided in this embodiment can flexibly configure the preloaded cache information required by each node to facilitate the generation of cache instances.

[0079] Specifically, based on the resource configuration information and the current job's operational needs, corresponding image data and datasets are obtained and used as cached data. Resource configuration information is obtained through preloading, first establishing an image list and a first preloaded cache node. It's important to note that the first preloaded cache node does not represent the number of nodes, but rather a node category, all of which appear as the first preloaded cache node. The first pull encryption / decryption configuration information for the image data is obtained, and the image list, the first preloaded cache node, and the first pull encryption / decryption configuration information are used as the resource configuration information for the image data. The method for obtaining the resource configuration information for the dataset is the same as that for the image data; please refer to the example of the resource configuration information for the image data for details, which will not be repeated here.

[0080] Correspondingly, as one embodiment, establishing the image list and the first preloaded cache node includes:

[0081] If no node selection label or node list is set in the image list, the image data will be preloaded to all nodes in the current cluster as the first preload cache node.

[0082] When a node selection label and a node list are set in the image list, the image data is cached to the node to be cached that matches the node selection label and is recorded in the node list as the first preloaded cache node.

[0083] Correspondingly, the process of creating the dataset list and the second preloaded cache node is the same as that of creating the mirror list and the first preloaded cache node.

[0084] Specifically, when neither the node selection label `nodeSelector` nor the node list `nodenames` is set in the image list, the image data is preloaded to all nodes in the current cluster as the first preload cache nodes. If the node selection label `nodeSelector` and the node list `nodenames` are set, the image data is cached to the nodes that match the node selection label and are recorded in the node list as the first preload cache nodes. The method for establishing the dataset list and the second preload cache node is the same as the method for establishing the image list and the first preload cache node described above.

[0085] For example, the cluster node cache resource configuration NodeCacheResource is designed as follows:

[0086]

[0087]

[0088]

[0089] In another preferred embodiment of the acquisition method, the cache information is obtained by the target node according to the current job execution request method, including obtaining the cache information of the current job's image data and dataset, including:

[0090] Determine if the node cache instance corresponding to the target node has changed;

[0091] If it exists, the changed data of the node cache instance will be used as the cache data;

[0092] Receive node requests from the target node to retrieve cached data.

[0093] Specifically, the system checks for changes in the node cache instance corresponding to the target node. If changes occur, it indicates that node cache information and status are maintained through the node cache instance. Each node corresponds to one node cache instance. The controller first generates preloaded cache content based on the cache configuration and then updates its dynamic content based on the cache status during runtime. The changed data of the node cache instance is used as cache data, and node requests from the target node are received to retrieve this cache data. The cache information within the node cache instance is obtained in the same way as the configuration information of the resource configuration in the above embodiment; please refer to the above method for reference.

[0094] For example, the node cache instance is designed as follows:

[0095]

[0096]

[0097]

[0098]

[0099] After obtaining the cache information in step S11, the corresponding event queue is determined based on the cache information and the corresponding node cache instance. Figure 3 This is an application diagram of a node cache provided in an embodiment of the present invention, such as... Figure 3 The two acquisition methods described in the above embodiments are: preloading the resource configuration of the node cache resources and determining the target node into the corresponding event queue through the corresponding node cache instance event. In the first acquisition method, a node cache running instance is generated through the event queue; in the second acquisition method, the node cache running instance in the target node is added to the corresponding event queue based on changes in its configuration information.

[0100] The system updates node cache task status information and executes corresponding cache task queue operations based on event queues and cache information. The cache task queue is a dual queue (image cache task queue and dataset cache task queue), which improves the response speed of the node cache controller in handling cache events and tasks. For task queue processing, image and dataset caching tasks can be performed simultaneously; for example, image data and dataset downloads can be performed concurrently on a single node to improve cache execution efficiency. For `nodeCacheJob` running on a node, different types of specific Job processes can run simultaneously. For example, downloading image A and deleting image B can be performed simultaneously. For specific `nodeCacheJob` type tasks running on a node, concurrent operations on multiple images and datasets can be handled according to the performance configuration of the `nodeCacheJob`. For example, the `imagePullJob` can perform multiple image download operations simultaneously to improve download efficiency. When a platform job starts, it first queries the cache instance on the node based on the required images and datasets. If the data is preloaded, it runs immediately; if the data is not in the node cache instance, the data to be downloaded is added to the instance to update the cache instance. This approach departs from the traditional client-server application programming interface (API) method, enabling self-maintenance and updating of cached information. It is particularly well-suited for handling edge nodes with unstable or unreliable network conditions.

[0101] This invention provides a data loading method for a distributed platform, comprising: acquiring the image data and dataset cache information of the current job, wherein the cache information is obtained through resource configuration preloading or by the target node based on the current job's execution request; determining the corresponding event queue based on the cache information and the corresponding node cache instance; updating the node cache task status information of the target node and executing the corresponding cache task queue operation based on each event queue and the cache information to load the cache information to the target node, wherein the cache task queue includes an image cache task queue and a dataset cache task queue. This method, in the preloading mode, does not require a request and directly loads the data to the target node, reducing the interaction process with the target node. Simultaneously, loading the cache information directly to the target node eliminates the need for a relay station, avoiding the latency issues caused by distribution through a central repository or local image repository. The dual-queue design (image cache task queue and dataset cache task queue) enables concurrent operation processing. Combined with the massive data of the distributed platform, real-time updates of the node cache task status information prevent data errors, thereby improving the platform's job execution efficiency.

[0102] Based on the above embodiments, step S12, which involves determining the corresponding event queue according to the cache information and the corresponding node cache instance, includes:

[0103] The corresponding event handling type is determined based on the node cache instance, where the event handling type includes node cache event handling type, node status event handling type, and node garbage cleanup event type;

[0104] Each event is entered into the corresponding event queue according to its processing type;

[0105] Correspondingly, the node cache event handling type is determined based on the node cache instance, including:

[0106] Node cache events are generated based on the node cache instance, and the event type of the node cache event is determined. The event type includes creation type, update type, and retry type. When the cache information is obtained through resource configuration preloading, the node cache instance obtains the cache information.

[0107] Specifically, the event handling type is determined based on the node cache implementation, and events generated based on changes to the node cache instance are entered into the event queue. Event handling types include node cache event handling types, node state event handling types, and node garbage collection event types. Each event handling type is then entered into its corresponding event queue.

[0108] like Figure 3 As shown, node cache event handling types include creation, update, and retry types. Node cache event handling is performed by the CacheWorker in the controller. It is primarily used in the following scenarios:

[0109] 1. When the controller component starts, it first parses the cluster node preloading information based on the cache resource configuration and generates the corresponding cache instance for each node. At this point, the cache instance contains the preloaded cache information for that node. A cache instance for node1 is created, and a node cache event is generated (created), which is then added to the event queue. The CacheWorker will create node cache tasks based on the cache instance associated with the event, such as the node1 node cache task, the node2 node cache task, etc., and add these cache tasks to the cache task queue. The cluster node cache preloading process is then ready to begin.

[0110] 2. During the execution of the controller component, if the node cache instance changes, such as when a preloaded cache has been downloaded, or when the image and dataset required by the node job are added to the dynamic cache of the cache instance, a node cache event is generated. The event type is update, and the event is added to the event queue. CacheWorker will update the node cache task status or create a cache download task based on the cache instance associated with the event.

[0111] 3. During the operation of the controller component, if there are abnormal changes in the node cache instance, such as the failure of preloading the cache of a certain image of a certain node, it is necessary to delete and retry, generate a node cache event, the event type is retry (repull), and re-download the failed node cache item.

[0112] The corresponding node status event handling types specifically include monitoring node and cache health status. Monitoring node and cache health status is implemented by MonitorWroker, which performs business health checks on node links, heartbeat checks, monitors node image space and data space metrics, and tracks job usage of node images and datasets. It also triggers cache update or cache deletion tasks.

[0113] The specific event type for node garbage cleanup is node cache garbage cleanup, which is implemented by RefreshWorker. Based on MonitorWorker's monitoring metrics, such as insufficient remaining space in the node cache, node cache garbage cleanup is performed, triggering cache update or cache deletion tasks. Before issuing the cleanup operation, it is necessary to ensure that these images and datasets to be cleaned have not been used by node jobs. If it is found that some pre-loaded images and datasets are missing (these caches have been deleted or new nodes have been added, etc.), a cache update task will be triggered, notifying the re-download of these node caches.

[0114] The event queue determination process provided in this embodiment of the invention enables the control of cluster node cache download, deletion, and update processes; maintains and updates the cluster node cache status; monitors the health status of nodes and caches while simultaneously cleaning up node cache garbage to save node resources.

[0115] Based on the above embodiments, step S13, which involves updating the node cache task status information of the target node according to each event queue and cache information, and executing the corresponding cache task queue operation, includes:

[0116] Update the node cache task status information of the target node based on each event queue and cache information, including:

[0117] Create a node cache task queue;

[0118] Update the node cache task status information in the node cache task queue according to the event processing type of the event queue;

[0119] Correspondingly, based on the event queue and cache information, the corresponding cache task queue operations are executed, including:

[0120] The corresponding task type is determined based on the event processing type and cache information of the event queue. The task types include node dataset download type, node dataset deletion type, node dataset re-download type, node image download type, node image deletion type, or node image re-download type.

[0121] Determine the corresponding program components based on each task type;

[0122] Each program component executes the corresponding cache task queue operation.

[0123] Specifically, the node cache task status information is updated based on each event queue and cache information. First, a node cache task queue is created, and then the node cache task status information is updated according to the event processing type in the above embodiment.

[0124] In executing corresponding cache task queue operations based on event queue and cache information, the task type is first determined, the corresponding program component is determined based on each task type, and the cache task queue operation is executed based on each program component. For example... Figure 3 As shown, the datasetManager cache manager is used to perform node dataset caching operations. Depending on the task type, the specific operations include:

[0125] Node dataset download: datasetPullJob

[0126] Node dataset deletion: datasetDeleteJob

[0127] Node dataset re-download: datasetRePullJob

[0128] The imageManager cache manager is used to perform node dataset caching operations. Depending on the task type, the specific operations include:

[0129] Node image download: imagePullJob

[0130] Node mirror deletion: imageDeleteJob

[0131] Node image re-download: imageRePullJob

[0132] `nodeCacheJob` is a program component that performs node caching. Specific caching task programs include: `datasetPullJob`, `datasetDeleteJob`, `datasetRePullJob`, `imagePullJob`, `imageDeleteJob`, and `imageRePullJob`. These `nodeCacheJobs` are not traditional program components that run permanently on nodes. Instead, they are created by `datasetManager` and `imageManager` based on the actual caching task requirements and run on the node specified by the caching task. Once the caching task is completed, the `nodeCacheJob` running on the node will exit and no longer occupy cloud platform node system resources. In addition, during the caching process, `nodeCacheJob` also synchronously updates node cache instance information, such as the cache information list, cache size, cache results, and status.

[0133] The node caching task component is a program component that runs on a node and performs specific operations on caching node images and datasets. It can download or delete node caches based on the node's cache resource instances and download command configuration. Unlike traditional methods where a resident program component runs continuously on the node, this caching task component is remotely invoked and terminated by the node cache controller, avoiding the drawback of traditional components constantly consuming cloud platform resources.

[0134] In addition, Figure 3 The node cache controller device and its submodules are as follows:

[0135] The node cache controller controls the image and dataset caching process of running cluster nodes, and collects and processes dynamic data related to the cache status and lifecycle of job services.

[0136] The image manager loads image pull configuration information, preloads cached information for nodes, dynamically caches information for nodes, and controls the image cache download process.

[0137] The dataset manager loads dataset download configuration information, preloads cache information for nodes, dynamically caches cache information for nodes, and controls the dataset cache download process.

[0138] The cleanup manager marks cached images and datasets on nodes as junk to be cleaned up, such as non-preloaded images and datasets that have not been used for a long time, and cleans them up to free up node storage space.

[0139] The monitoring manager performs cluster node health monitoring and cache progress status detection, enabling the image manager and dataset manager to handle node cache download and update in abnormal node scenarios.

[0140] As one embodiment, the node cache instance includes a node snapshot mechanism, and also includes:

[0141] When the network of the distributed platform is abnormal, the cached information is retained through the node snapshot mechanism so that the cached information can be loaded when the network of the distributed platform is normal.

[0142] Specifically, node cache resource configurations and node cache instances are persisted to the distributed system, providing reliable services for node access to resources. The node cache control can also update and synchronize these resource configurations online. For example, if the platform's pre-configured images and datasets are updated, simply changing the node cache resource configuration will immediately notify the node cache instances to synchronize and update, triggering the corresponding sub-modules in the controller to handle the changes. Furthermore, a node snapshot mechanism is added to the node cache instances. When the platform network experiences an anomaly, and edge nodes cannot access the cluster masternode or restart abnormally, the system can still perceive the current cached information of the nodes, meeting the data requirements for node job execution.

[0143] The node cache task status information update and cache task queue operation execution provided by the embodiments of the present invention can ensure that platform components and user jobs can reliably access node cache information and instantly and quickly perceive changes in cache information.

[0144] The foregoing has described in detail various embodiments of the data loading method for a distributed platform. Based on this, the present invention also discloses a data loading apparatus for a distributed platform corresponding to the above-described method. Figure 4 This is a structural diagram of a data loading device for a distributed platform provided in an embodiment of the present invention. Figure 4 As shown, the data loading device of the distributed platform includes:

[0145] The acquisition module 11 is used to acquire the image data and cache information of the dataset of the current job, wherein the cache information is obtained by preloading the resource configuration or by the target node according to the current job's execution request.

[0146] Module 12 is used to determine the corresponding event queue based on the cache information and the corresponding node cache instance;

[0147] The update execution module 13 is used to update the node cache task status information of the target node according to each event queue and cache information, and to execute the corresponding cache task queue operation so as to load the cache information to the target node. The cache task queue includes the image cache task queue and the dataset cache task queue.

[0148] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0149] For a description of the data loading device for a distributed platform provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above-described data loading method for a distributed platform.

[0150] Figure 5 A structural diagram of another data loading device for a distributed platform provided in an embodiment of the present invention is shown below. Figure 5 As shown, the device includes:

[0151] Memory 21 is used to store computer programs;

[0152] Processor 22 is used to implement the steps of a data loading method for a distributed platform when executing a computer program.

[0153] The data loading device for the distributed platform provided in this embodiment may include, but is not limited to, tablet computers, laptop computers, or desktop computers.

[0154] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0155] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the data loading method for the distributed platform disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the data loading method of the distributed platform.

[0156] In some embodiments, the data loading device of the distributed platform may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0157] Those skilled in the field can understand, Figure 5 The structure shown does not constitute a limitation on the data loading device of a distributed platform and may include more or fewer components than illustrated.

[0158] The processor 22 implements the data loading method of the distributed platform provided in any of the above embodiments by calling instructions stored in the memory 21.

[0159] For a description of the data loading device for a distributed platform provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above-described data loading method for a distributed platform.

[0160] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by processor 22, implements the steps of the data loading method for the distributed platform described above.

[0161] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the data loading method of the above distributed platform.

[0163] The foregoing has provided a detailed description of a data loading method, a data loading device, and a medium for a distributed platform provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0164] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data loading method for a distributed platform, characterized in that, include: Obtain the image data and cache information of the dataset for the current job, wherein the cache information is obtained through resource configuration preloading or by the target node based on the current job's execution request. The corresponding event queue is determined based on the cache information and the corresponding node cache instance; The node cache task status information of the target node is updated according to each event queue and the cache information, and the corresponding cache task queue operation is executed to load the cache information into the target node. The cache task queue includes a mirror cache task queue and a dataset cache task queue. Correspondingly, the cache information is obtained through the resource configuration preloading method, and obtaining the cache information of the current job's image data and dataset includes: Based on the resource configuration information and the current job, obtain the corresponding image data and dataset as cache information; Correspondingly, the acquisition of each of the resource configuration information is achieved through the resource configuration preloading method, specifically including: Establish the image list and the first preload cache node; Obtain the first pull encryption / decryption configuration information for the image data; The image list, the first preloaded cache node, and the first pull encryption / decryption configuration information are used as the resource configuration information for the image data. Create a dataset list and a second preload cache node; Obtain the second pull encryption / decryption configuration information for the dataset; The image list, the second preloaded cache node, and the second pull encryption / decryption configuration information are used as the resource configuration information of the dataset.

2. The data loading method for a distributed platform according to claim 1, characterized in that, The establishment of the image list and the first preloaded cache node includes: If no node selection label or node list is set in the image list, the image data is preloaded to all nodes in the current cluster as the first preload cache node; When the node selection label and the node list are set in the image list, the image data is cached to the node to be cached that matches the node selection label and is recorded in the node list as the first preloaded cache node; Correspondingly, the method of establishing the dataset list and the second preloaded cache node is the same as that of establishing the mirror list and the first preloaded cache node.

3. The data loading method for a distributed platform according to claim 1, characterized in that, The cache information is obtained by the target node according to the current job execution request method. Obtaining the cache information of the current job's image data and dataset includes: Determine whether the node cache instance corresponding to the target node has changed; If it exists, the changed data of the node cache instance will be used as the cache information; Receive the node request from the target node to obtain the cache information.

4. The data loading method for a distributed platform according to claim 1, characterized in that, The step of determining the corresponding event queue based on the cache information and the corresponding node cache instance includes: The corresponding event handling type is determined based on the node cache instance, wherein the event handling type includes node cache event handling type, node status event handling type and node garbage cleanup event type; Each event processing type is entered into the corresponding event queue; Correspondingly, the node cache event handling type is determined based on the node cache instance, including: Node cache events are generated based on the node cache instance, and the event type of the node cache event is determined. The event type includes creation type, update type, and retry type. When the cache information is obtained through the resource configuration preloading method, the node cache instance is obtained through the cache information.

5. The data loading method for a distributed platform according to claim 4, characterized in that, Update the node cache task status information of the target node and execute the corresponding cache task queue operation according to each event queue and the cache information, including: The step of updating the node cache task status information of the target node according to each of the event queues and the cache information includes: Create a node cache task queue; Update the node cache task status information in the node cache task queue according to the event processing type of the event queue; Correspondingly, the corresponding cache task queue operation is performed based on the event queue and the cache information, including: The corresponding task type is determined based on the event processing type and the cache information in the event queue, wherein the task type includes node dataset download type, node dataset deletion type, node dataset re-download type, node image download type, node image deletion type, or node image re-download type; Determine the corresponding program components based on each task type; Each program component performs the corresponding operation of the cached task queue.

6. The data loading method for a distributed platform according to any one of claims 1 to 5, characterized in that, The node cache instance includes a node snapshot mechanism, and also includes: When the network of the distributed platform is abnormal, the cached information is retained through the node snapshot mechanism so that the cached information can be loaded when the network of the distributed platform is normal.

7. A data loading device for a distributed platform, characterized in that, include: The acquisition module is used to acquire the image data and cache information of the dataset of the current job, wherein the cache information is obtained by resource configuration preloading or by the target node according to the current job execution request. The determination module is used to determine the corresponding event queue based on the cache information and the corresponding node cache instance; The update execution module is used to update the node cache task status information of the target node according to each event queue and the cache information, and to execute the corresponding cache task queue operation so as to load the cache information into the target node, wherein the cache task queue includes a mirror cache task queue and a dataset cache task queue; Correspondingly, the cache information is obtained through the resource configuration preloading method, and obtaining the cache information of the current job's image data and dataset includes: Based on the resource configuration information and the current job, obtain the corresponding image data and dataset as cache information; Correspondingly, the acquisition of each of the resource configuration information is achieved through the resource configuration preloading method, specifically including: Establish the image list and the first preload cache node; Obtain the first pull encryption / decryption configuration information for the image data; The image list, the first preloaded cache node, and the first pull encryption / decryption configuration information are used as the resource configuration information for the image data. Create a dataset list and a second preload cache node; Obtain the second pull encryption / decryption configuration information for the dataset; The image list, the second preloaded cache node, and the second pull encryption / decryption configuration information are used as the resource configuration information of the dataset.

8. A data loading device for a distributed platform, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data loading method for a distributed platform as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data loading method for a distributed platform as described in any one of claims 1 to 6.

Images