Data processing method and system, and electronic device, storage medium and product

Abstract

The present disclosure relates to the technical field of artificial intelligence. Provided are a data processing method and apparatus, and an electronic device, a computer-readable storage medium and a computer program product. The data processing method comprises: determining a target data set required for a target container to execute a model training task; determining in a data cache layer a target storage volume for storing the target data set; and mounting the target storage volume to the target container, such that the target container accesses the target storage volume and uses the target data set to execute the model training task. Unlike deploying a data set in a cloud or storing the data set locally, the data processing method in the present disclosure directly deploys a data set as cache resources of a cluster, so as to directly mount in a cluster container a storage volume corresponding to the data set, thereby breaking through a transmission bottleneck caused by the network speed and the hard disk speed, and greatly improving the data access speed and the model training efficiency.

Description

[0001] Data Processing Method, System, Electronic Device, Storage Medium, and Product This disclosure claims priority to Chinese Patent Application No. 202410382424.2, filed with the China Patent Office on March 29, 2024, entitled "Data Processing Method, System, Electronic Device, Storage Medium, and Product," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of artificial intelligence technology, and more particularly to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background: Cloud native is a method for building and running applications in a cloud environment. Currently, more and more enterprises are leveraging cloud native architecture principles to build model training environments for training and inference tasks. While using cloud native architecture to run model training tasks offers the advantages of elastic computing, it also presents challenges such as complex adaptation to diverse storage, data access latency, and high bandwidth overhead for remote data pulls. For example, in deep learning training scenarios, the need to remotely read large amounts of training data can significantly slow down computing efficiency. When deploying large-model inference, due to the generally large size of the models and the large number of available models, the need to compare the differences between multiple large models can also lead to the need to download and store large amounts of data multiple times. SUMMARY OF THE INVENTION The present disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product to alleviate or resolve one or more technical problems existing in the prior art. In a first aspect, the present disclosure provides a data processing method, comprising: determining a target dataset required for a target container to perform a model training task; determining a target storage volume in a data cache layer to store the target dataset; and mounting the target storage volume to a target container, so that the target container accesses the target storage volume and performs the model training task using the target dataset. In a second aspect, the present disclosure provides a data processing system, comprising a cluster and a data cache layer. The cluster includes multiple worker nodes, and the worker nodes are configured to execute any of the methods of the present disclosure to enable containers on the worker nodes to perform the model training task using the dataset cached in the data cache layer. In a third aspect, embodiments of the present disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, the method described in any one of the embodiments of the present disclosure is implemented. In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium, the computer-readable storage medium storing the computer program. When the processor executes the computer program, the method described in any one of the embodiments of the present disclosure is implemented.In a fifth aspect, embodiments of the present disclosure provide a computer program product, including a computer program. When executed by a processor, the computer program implements any method of the embodiments of the present disclosure. The data processing method based on the embodiments of the present disclosure, unlike deploying datasets in the cloud or local storage, directly deploys the datasets as cache resources in a cluster, directly mounting the storage volumes corresponding to the datasets to cluster containers. This overcomes transmission bottlenecks caused by network and hard disk speeds, significantly improving data access speed and model training efficiency. The above description is only an overview of the technical solutions of the present disclosure. To better understand the technical solutions of the present disclosure, implementation should be carried out in accordance with the contents of this specification. To further enhance the understanding of the above and other objectives, features, and advantages of the present disclosure, specific embodiments of the present disclosure are described below. BRIEF DESCRIPTION OF THE DRAWINGS In the accompanying drawings, unless otherwise specified, identical reference numerals throughout the multiple drawings indicate identical or similar components or elements. The drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments of the present disclosure and should not be construed as limiting the scope of the present disclosure. Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure; Figure 2 is a flow chart of a data processing method according to an embodiment of the present disclosure; Figure 3 is an architectural diagram of a data processing system according to an embodiment of the present disclosure; Figure 4 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure; and Figure 5 is a block diagram of an electronic device used to implement an embodiment of the present disclosure. Specific Implementations: Below, only certain exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description are to be regarded as illustrative in nature, rather than restrictive. To facilitate understanding of the technical solutions of the embodiments of the present disclosure, the following describes related technologies. The following related technologies, as optional solutions, can be combined with the technical solutions of the embodiments of the present disclosure in any manner and fall within the scope of protection of the embodiments of the present disclosure. The following terminology will be used below. It should be noted that the following is a brief description of the basic concepts involved in the embodiments of the present disclosure. It should be understood that the basic concepts introduced below do not limit the embodiments of the present disclosure. Cloud native: Cloud native is a distributed cloud based on distributed deployment and unified operation and management, and a set of cloud technology product systems built on technologies such as containers, Kubernetes, and microservices.

[0002] Kubernetes (also known as K8S): Kubernetes is an open-source, portable container orchestration and management tool for container services, used to automate the deployment, scaling, and management of application containers. Kubernetes groups multiple hosts into a cluster for centralized management of containerized applications, and clusters can be deployed across physical or virtual machines. Pod: A Pod is the fundamental deployment unit in Kubernetes and the minimal resource object for running containerized applications. It consists of one or more containers (such as Docker containers), which share storage and networking resources and run according to a common set of configurations. Persistent Volume (PV): A PV is a resource in Kubernetes that represents a storage area in the cluster. It is typically pre-created by the cluster administrator or automatically created through dynamic provisioning. It exists in the storage layer of the Kubernetes cluster and its lifecycle is independent of any specific worker node. Persistent Volume Claim (PVC): PVC is a declaration of user storage, representing a user's request for storage. The user can specify the storage size and access mode (such as read-write once or read-write multiple times) in the PVC. Container Storage Interface (CSI): CSI is a standard interface for providing persistent storage for Kubernetes clusters, which allows third parties to create and manage persistent storage plug-ins. The relationship between PVC, PV and Pod is usually as follows: (1) The user creates a PVC, specifying the required storage size and access mode; (2) The Kubernetes cluster will find a storage volume that meets the conditions in the available PVs and bind it to the PVC. If there is no existing PV that can match the PVC requirements, the cluster will dynamically create a new PV to meet the PVC; (3) Once the PVC is bound to the PV, the PVC will remain bound to the PV until the PVC is deleted. Containers (Pods) can use the PV by referencing the PVC in their configuration, so that the Pod can access the data on the PV. In one implementation, deep learning training platforms are all built based on Kubernetes clusters. However, current deep learning training platforms based on Kubernetes clusters have I / when performing distributed training.Problems such as high communication pressure (read and write) and complex data management lead to slow containerized distributed training, and I / O bottlenecks affect resource utilization. When datasets are stored in the cloud, the speed of distributed training is limited by the communication capabilities between the network and cluster hosts; when datasets are stored locally, the speed of distributed training is limited by the network latency and disk-level I / O reads caused by mounting the dataset to the Network File System (NFS). Therefore, in this implementation, whether the dataset is stored in the cloud or locally, a significant I / O communication bottleneck exists, resulting in low distributed training speed. Based on this, the present embodiment provides a data processing method that can improve access speed to data storage in a cloud-native architecture. This method improves resource utilization and mitigates data access latency and high bandwidth overhead associated with remote data pull, while leveraging the elasticity and cost advantages of public cloud services. Figure 1 is a schematic diagram of an application scenario of the present embodiment, illustrating the storage format of training data and how containers in a Kubernetes cluster read training data when performing model training tasks. As shown in Figure 1, in an embodiment of the present disclosure, training data can be divided into different sets to form different datasets. Multiple different PVCs are defined based on the different datasets and submitted to the Kubernetes cluster. The Kubernetes cluster then determines the PVs that satisfy each PVC from the cluster's cache resources and binds the PVs to the PVCs they satisfy. The datasets corresponding to the PVCs are then stored in the PVs bound to the PVCs, so that different PVs store different datasets. The training data is persistently stored as datasets in the Kubernetes cluster's cache resources. Furthermore, a dataset directory can be constructed based on the datasets already stored in the Kubernetes cluster's cache resources. The dataset directory can record the mapping between different datasets and their defined PVCs. Specifically, different datasets can be represented in the dataset directory using labels, with different labels corresponding to different PVCs. Therefore, when utilizing the Kubernetes cluster to perform model inference or distributed training tasks, multiple pods can be created on the Kubernetes cluster, allowing each pod to complete the corresponding task using the training data stored in the Kubernetes cluster's cache resources. The following describes the process of obtaining training data (including the target dataset) for a single pod:

[0003] 1) Based on the training tasks that the Pod needs to perform, determine the target dataset required to complete the training tasks;

[0004] 2) Search the target dataset's label in the dataset directory to determine the target PVC corresponding to the target dataset;

[0005] 3) Configure the Pod to reference the target PVC, thereby mounting the target PV bound to the target PVC to the Pod. After the mount is complete, the Pod can access the target PV and read the target dataset stored on the target PV. Unlike traditional methods of deploying datasets in the cloud or locally, the technical solution of the disclosed embodiment is to persistently cache the dataset as a Kubernetes cluster resource, allowing containers on the cluster and applications or processes on the containers to directly call cluster resources to perform related tasks, greatly improving data access speed. Furthermore, when the training task executed by the Kubernetes cluster requires new training data, the corresponding dataset may not exist in the Kubernetes cluster's cache resources. In this case, the process for the Pod to obtain training data is as follows:

[0006] 1) Based on the training tasks that the Pod needs to perform, determine the target dataset required to complete the training tasks;

[0007] 2) If the label corresponding to the target dataset is not found in the dataset directory, download the target dataset from the cloud or local storage;

[0008] 3) Define a new PVC based on the target dataset as the target PVC. Then, determine a PV that satisfies the target PVC in the cluster's cache resources and bind it to the target PVC. The target dataset is stored in the target PV.

[0009] 4) Configure the Pod to reference the target PVC, thereby mounting the target PV bound to the target PVC to the Pod. After the mount is completed, the Pod can access the target PV and read the target dataset stored in the target PV.

[0010] 5) Update the dataset directory, recording the mapping relationship between the target PVC and the target dataset in the dataset directory. Based on this, when new training data needs arise, the new training data can be downloaded and cached as cluster resources in the Kubernetes cluster, facilitating the cluster's subsequent use of training data, effectively reducing data duplication and lowering network costs. Furthermore, cluster resource expansion is more convenient. In the event of resource shortages, allocation can be requested immediately. Compared to building local distributed storage that requires reserving excess storage space, this disclosed embodiment effectively utilizes resource pools and avoids resource waste. It should be noted that the aforementioned application scenarios or application examples of the data processing methods provided in this disclosed embodiment are for ease of understanding and do not specifically limit the application of the data processing methods. Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) referred to in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation portals are provided for users to select, edit, authorize, or reject. The following describes in detail the technical solution of the present disclosure and how the technical solution of the present disclosure solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. FIG2 shows a flow chart of a data processing method according to an embodiment of the present disclosure. The data processing method can be applied to a data processing device. As shown in FIG2 , the data processing method may include steps S201, S202, and S203. Step S201: Determine the target dataset required for the target container to perform the model training task; Step S202: Determine the target storage volume for storing the target dataset in the data cache layer; Step S203: Mount the target storage volume to the target container so that the target container can access the target storage volume and use the target dataset to perform the model training task.For example, the target container can be a Pod container in a Kubernetes cluster, and the data cache layer can be a cluster resource of the cluster where the target container resides. Training data can be stored in the form of datasets in different storage volumes in the data cache layer. When a target container is created in the cluster to perform a model training task, the required target dataset can be determined based on the model training task, and the target storage volume for storing the target dataset can be determined in the data cache layer. The target storage volume is then mounted to the target container, enabling the application or process performing the model training task in the target container to read the target dataset in the target storage volume at high speed to complete the model training task. It can be understood that the data cache layer is middleware located between the underlying distributed file system and the upper-layer distributed computing framework in the Kubernetes cluster. Its primary responsibility is to provide data access services in memory or other storage facilities in the form of files. The disclosed embodiments generate training data in the form of datasets as a cluster cache resource, significantly improving data access speed. In one embodiment, the target storage volume is a persistent volume (PV). Mounting the target storage volume to the target container in step S203 may include configuring the target container to reference a persistent volume claim (PVC) bound to the target storage volume, thereby mounting the target storage volume to the target container. For example, when a dataset is first loaded into the data cache layer, a PVC is typically created based on the dataset's storage requirements. The Kubernetes cluster then determines a PV from available PVs that meets the storage requirements, binds it to the PVC, and stores the dataset in the PV bound to the corresponding PVC. Therefore, each PV storing a dataset is typically bound to a PVC. By referencing this PVC, the target container can bind the PVC to the target PV using the Filesystem in Userspace (FUSE) and mount it as a plugin, thereby directly reading data from the target PV. Using the methods of the embodiments of the present disclosure, datasets are bound to PVCs, enabling containers in the cluster to quickly access datasets stored on the PV bound to the PVC by referencing the PVC, further improving data access speed and reducing data access latency.In one embodiment, in step S202, determining a target storage volume for storing a target dataset in the data cache layer may include: in response to a dataset directory recording the target dataset, determining, from the multiple storage volumes in the data cache layer according to the dataset directory, a storage volume corresponding to the target dataset as the target storage volume, wherein the dataset directory records a mapping relationship between the dataset and the storage volumes in the data cache layer. Exemplarily, the dataset directory may collect statistics on each dataset stored in the data cache layer and record the storage location (i.e., storage volume) of each dataset. For example, the dataset directory may divide each dataset into labels and record the mapping relationship between each dataset label and storage location in the dataset directory. Thus, for a target dataset required to execute a model training task on a target container, the target dataset label may be retrieved from the dataset directory to quickly determine the storage location of the target dataset. Optionally, since the datasets in the embodiments of the present disclosure are stored in different PVs, the dataset directory can also record the PVC bound to the PV storing the dataset instead of the PV. Specifically, this records the mapping relationship between the dataset tag and the PVC. Therefore, after retrieving the target dataset's tag from the dataset directory, the corresponding PVC can be directly determined using the mapping relationship. By configuring the target container to reference the PVC, the target dataset in the target PV can be directly read. Using the method of the embodiments of the present disclosure, by statistically organizing the datasets stored in the data cache layer to form a dataset directory, the storage address of the required data can be quickly retrieved based on the data requirements of different containers, thereby improving data access speed. In one embodiment, determining the target storage volume for storing the target dataset in the data cache layer in step S202 may further include: responsive to the target dataset not being recorded in the dataset directory, retrieving the target dataset from local storage and / or cloud storage; allocating a target storage volume for the target dataset in the data cache layer; and downloading the target dataset to the target storage volume. It is understandable that if the target dataset is not retrieved in the dataset directory, it means that the model training task to be performed by the target container has new training data requirements, and the target dataset can be obtained from local storage and / or cloud storage to perform the model training task.Furthermore, when the target dataset is first loaded into the data cache layer, a PVC can be created based on the required storage size. A matching PV from the available PV resources in the data cache layer is then bound to the created PVC as the target PV. The acquired target dataset is then downloaded to the target PV. The target PV is then mounted to the target container by referencing the PVC, allowing the container to read the target dataset and execute model training tasks. Furthermore, if the data cache layer lacks matching PV resources, cluster resources can be expanded in real time to create a new PV that meets the PVC requirements. Based on this, the datasets stored in the data cache layer are dynamically updated based on data demand. New training data is downloaded and cached as cluster resources for the Kubernetes cluster. Compared to deploying datasets in the cloud or locally, this effectively reduces data duplication and lowers network costs. Compared to building local distributed storage that requires reserving excess storage space, cluster resource expansion is more convenient. In the event of resource shortages, allocation can be requested immediately, effectively utilizing the resource pool and avoiding resource waste. Furthermore, new training data is obtained from cloud storage when data demand arises, leveraging the elasticity and cost advantages of the public cloud. In one embodiment, after downloading the target dataset to the target storage volume, the process also includes updating the dataset directory to record the mapping relationship between the target dataset and the target storage volume. It is understood that for the target dataset newly cached in the data cache layer, the dataset directory must record the mapping relationship between the new target dataset and the target PV storing the new dataset. Therefore, when subsequent model training tasks involve the target dataset, the updated dataset directory can be used to quickly determine the target dataset's storage location or the corresponding PVC, allowing for rapid access to new data. Based on the method of the embodiments of the present disclosure, the dataset directory is synchronously updated based on updates to the dataset stored in the data cache layer, thereby facilitating subsequent use of training data by the cluster, establishing a comprehensive dataset update mechanism, and improving data access speed. In one embodiment, downloading the target dataset to the target storage volume may include: dividing the target dataset into N data subsets, where N is an integer greater than 1; and downloading the N data subsets to the target storage volume using M worker nodes deployed in the same cluster, where M is an integer greater than 1, and the M worker nodes include the worker node to which the target container belongs.For example, the target container runs on a worker node in a Kubernetes cluster. To download a target dataset, the target dataset can be divided into different data subsets, and the download tasks are assigned to different worker nodes in the Kubernetes cluster. This allows the different worker nodes to collaboratively complete the download task for the target dataset. Each downloaded data subset is then transferred to the target storage volume via a peer-to-peer (P2P) mechanism to form the target dataset. Using the method of the present embodiment, when new training data needs to be downloaded from the cloud or local storage, multiple nodes in the cluster can be leveraged to collaboratively complete the download task, maximizing the cluster's scheduling capabilities to achieve efficient file transfer and, in turn, improve access speed for uncached data. Corresponding to the data processing method provided in the above embodiment, as shown in FIG3 , the present embodiment further provides a data processing system 30 comprising a cluster 301 and a data caching layer 302. Cluster 301 includes multiple worker nodes, which are configured to execute the data processing method of the present embodiment so that containers on the worker nodes perform model training tasks using the datasets cached in the data caching layer. For example, cluster 301 may be a Kubernetes cluster. Pod containers and container groups are deployed on each worker node of the Kubernetes cluster. Data cache layer 302 is configured as a cluster resource of the Kubernetes cluster. Different datasets are cached in different PVs in data cache layer 302. Each PV storing a dataset is bound to a corresponding PVC. When a Pod container in the Kubernetes cluster is configured to reference a PVC, the Pod container can read the dataset stored in the PV bound to the PVC. In one embodiment, the datasets stored in data cache layer 302 can be statistically organized to form a dataset directory. Because a dataset is first loaded into data cache layer 302, a PVC must be created, thereby specifying a PV in the PV resource for storing the dataset. Therefore, the PVC can directly point to the dataset stored in the bound PV. Therefore, the dataset directory can directly record the mapping between each PVC and each dataset. When a Pod needs to read a dataset, it can directly search the dataset directory for the PVC corresponding to the dataset and read the corresponding dataset by referencing the PVC.In one embodiment, when a Kubernetes cluster initially builds a data cache layer for storing datasets, and when datasets need to be expanded based on data demands, the data processing system 30 can distribute data loading tasks to multiple worker nodes for collaborative execution based on the capacity of each worker node. After downloading data from local storage or the cloud, the multiple worker nodes transfer the corresponding data via a peer-to-peer (P2P) mechanism to the PVs allocated for different datasets in the data cache layer 302, thereby integrating the datasets into cluster resources within the Kubernetes cluster. The technical solution based on the embodiments of this disclosure, unlike deploying datasets in the cloud or local storage, directly deploys datasets as cluster cache resources and establishes a dataset directory. By searching the dataset directory, the PVCs for configured containers can be quickly retrieved, significantly improving the speed at which each container in the cluster can read training data, thereby enhancing model training efficiency. Another beneficial effect of the technical solution of the disclosed embodiments is that, compared to building local distributed storage that requires reserving excess storage space, cluster resource expansion is more convenient. In the event of resource shortages, allocation can be requested immediately, effectively utilizing the resource pool and avoiding resource waste. Furthermore, when new data requirements arise, the cluster's scheduling capabilities can be leveraged to coordinate multiple nodes to complete data acquisition tasks, improving access speed to uncached data. Corresponding to the application scenarios and data processing methods provided in the disclosed embodiments, the disclosed embodiments also provide a data processing device 400. This device may include: a first determination module 401 for determining a target dataset required for a target container to perform a model training task; a second determination module 402 for determining a target storage volume in a data cache layer to store the target dataset; and a processing module 403 for mounting the target storage volume to a target container, enabling the target container to access the target storage volume and perform the model training task using the target dataset. Exemplarily, the second determination module 402 is configured to: in response to a target dataset being recorded in the dataset directory, determine, from the multiple storage volumes in the data cache layer, a storage volume corresponding to the target dataset as the target storage volume based on the dataset directory, wherein the dataset directory records a mapping relationship between datasets and storage volumes in the data cache layer. Exemplarily, the second determination module 402 is further configured to: in response to the target dataset not being recorded in the dataset directory, obtain the target dataset from local storage and / or cloud storage; allocate a target storage volume for the target dataset in the data cache layer; and download the target dataset to the target storage volume.Exemplarily, the apparatus further includes an update module for updating the dataset directory to record the mapping relationship between the target dataset and the target storage volume in the dataset directory. Exemplarily, the update module is specifically configured to divide the target dataset into N data subsets, where N is an integer greater than 1; and download the N data subsets to the target storage volume using M worker nodes deployed in the same cluster, where M is an integer greater than 1 and the M worker nodes include the worker node to which the target container belongs. Exemplarily, the target storage volume is a persistent volume (PV), and the processing module 403 is specifically configured to configure the target container to reference the persistent volume claim (PVC) bound to the target storage volume, so that the target storage volume is mounted to the target container. The functions of each module in each apparatus of the embodiments of the present disclosure can be found in the corresponding descriptions of the above-mentioned methods, and they have corresponding beneficial effects, and are not further described here. FIG5 is a block diagram of an electronic device for implementing the embodiments of the present disclosure. As shown in FIG5 , the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program executable on the processor 502. When the processor 502 executes the computer program, the method of the above-mentioned embodiment is implemented. The number of memory 501 and processor 502 can be one or more. The electronic device also includes a communication interface 503 for communicating with external devices and exchanging data. If the memory 501, processor 502, and communication interface 503 are implemented independently, the memory 501, processor 502, and communication interface 503 can be interconnected via a bus to facilitate communication. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG5 shows only one thick line, but this does not mean that there is only one bus or only one type of bus. Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, the memory 501, processor 502, and communication interface 503 can communicate with each other via an internal interface.The present disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the methods provided in the present disclosure. Furthermore, the present disclosure also provides a computer program product including the computer program. When executed by a processor, the computer program implements the methods provided in the present disclosure. The present disclosure also provides a chip including a processor configured to retrieve and execute instructions stored in a memory, thereby enabling a communication device equipped with the chip to perform the methods provided in the present disclosure. The present disclosure also provides a chip including an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected via an internal connection path. The processor executes code in the memory. When the code is executed, the processor performs the methods provided in the present disclosure. It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture. Furthermore, optionally, the memory described above may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory.Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM RAM (DR RAM). In the above embodiments, all or part of them may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When computer program instructions are loaded and executed on a computer, the processes or functions according to the present disclosure are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicates that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present disclosure. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples described in this specification, as well as features from different embodiments or examples, without conflict. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed to indicate or imply relative importance or to implicitly specify the number of technical features indicated. Therefore, features identified as "first" or "second" may explicitly or implicitly include at least one of those features.In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined. Any process or method depicted in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code comprising one or more executable instructions for implementing a specific logical function or process step. Furthermore, the scope of the preferred embodiments of this disclosure includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions substantially simultaneously or in reverse order depending on the functions involved. The logic and / or steps depicted in a flowchart or otherwise described herein can, for example, be considered a sequenced list of executable instructions for implementing the logical functions and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). It should be understood that various aspects of this disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above method embodiments can be performed by a program instructing the relevant hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of the present disclosure can be integrated into a single processing module, each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated modules can be implemented in hardware or as software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a magnetic disk, or an optical disk, etc. The above are merely exemplary embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Anyone skilled in the art can easily conceive of various variations or substitutions within the technical scope of the present disclosure, and such variations or substitutions are intended to be encompassed by the scope of protection of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

Claims 1. A data processing method, comprising: Determine the target dataset required by the target container to perform the model training task; Determining, in the data cache layer, a target storage volume for storing the target data set; The target storage volume is mounted to the target container, so that the target container accesses the target storage volume and uses the target dataset to perform the model training task.

2. The method according to claim 1, wherein: Determining a target storage volume for storing the target data set in a data cache layer includes: in response to a data set directory recording the target data set, determining a storage volume corresponding to the target data set from multiple storage volumes in the data cache layer according to the data set directory as the target storage volume, wherein the data set directory records a mapping relationship between the data set and the storage volume in the data cache layer.

3. The method according to claim 2, wherein: Determining a target storage volume for storing the target data set in the data cache layer also includes: in response to the target data set not being recorded in the data set directory, obtaining the target data set from local storage and / or cloud storage; allocating the target storage volume for the target data set in the data cache layer; and downloading the target data set to the target storage volume.

4. The method according to claim 3, wherein: After downloading the target dataset to the target storage volume, the method further includes: updating the dataset directory to record a mapping relationship between the target dataset and the target storage volume in the dataset directory.

5. The method according to claim 3 or 4, wherein: Downloading the target data set to the target storage volume includes: dividing the target data set into N data subsets, where N is an integer greater than 1; and downloading the N data subsets to the target storage volume using M worker nodes deployed in the same cluster, where M is an integer greater than 1, and the M worker nodes include the worker node to which the target container belongs.

6. The method according to any one of claims 1 to 5, wherein: The target storage volume is a persistent volume PV, and mounting the target storage volume to the target container includes: configuring the target container to reference a persistent volume claim PVC bound to the target storage volume, so that the target storage volume is mounted to the target container.

7. A data processing system comprising a cluster and a data cache layer, wherein: The cluster includes multiple working nodes, and the working nodes are used to execute the data processing method according to any one of claims 1 to 6 so that the containers on the working nodes use the data sets cached in the data cache layer to perform model training tasks.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program which, when executed by a processor, Now according to the method according to any one of claims 1-6.

Images